From solipsis at pitrou.net  Mon Jan  1 04:14:17 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 01 Jan 2018 09:14:17 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=2
Message-ID: <20180101091417.1.E733A849134DEC68@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, 0, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogR6n_56', '--timeout', '7200']

From webhook-mailer at python.org  Mon Jan  1 12:39:41 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 01 Jan 2018 17:39:41 -0000
Subject: [Python-checkins] bpo-32416: Add two new tests in
 test_sys_settrace. (#5072)
Message-ID: <mailman.107.1514828383.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/e8ed96550c6aa9a1e39c36e67e892994e25e2c41
commit: e8ed96550c6aa9a1e39c36e67e892994e25e2c41
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-01T19:39:38+02:00
summary:

bpo-32416: Add two new tests in test_sys_settrace. (#5072)

Move other test to more proper place.

files:
M Lib/test/test_sys_settrace.py

diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py
index b4b578cc9dd..e6eb80ad86a 100644
--- a/Lib/test/test_sys_settrace.py
+++ b/Lib/test/test_sys_settrace.py
@@ -789,6 +789,14 @@ def test_jump_within_except_block(output):
             output.append(6)
         output.append(7)
 
+    @jump_test(2, 4, [1, 4, 5, -4])
+    def test_jump_across_with(output):
+        output.append(1)
+        with tracecontext(output, 2):
+            output.append(3)
+        with tracecontext(output, 4):
+            output.append(5)
+
     @jump_test(8, 11, [1, 3, 5, 11, 12])
     def test_jump_out_of_complex_nested_blocks(output):
         output.append(1)
@@ -840,6 +848,17 @@ def test_jump_over_break_in_try_finally_block(output):
             break
         output.append(13)
 
+    @jump_test(1, 7, [7, 8])
+    def test_jump_over_for_block_before_else(output):
+        output.append(1)
+        if not output:  # always false
+            for i in [3]:
+                output.append(4)
+        else:
+            output.append(6)
+            output.append(7)
+        output.append(8)
+
     # The second set of 'jump' tests are for things that are not allowed:
 
     @jump_test(2, 3, [1], (ValueError, 'after'))
@@ -997,21 +1016,24 @@ def test_no_jump_out_of_finally_block(output):
         finally:
             output.append(5)
 
-    @jump_test(2, 4, [1, 4, 5, -4])
-    def test_jump_across_with(output):
+    @jump_test(3, 5, [1, 2, -2], (ValueError, 'into'))
+    def test_no_jump_between_with_blocks(output):
         output.append(1)
         with tracecontext(output, 2):
             output.append(3)
         with tracecontext(output, 4):
             output.append(5)
 
-    @jump_test(3, 5, [1, 2, -2], (ValueError, 'into'))
-    def test_jump_across_with_2(output):
+    @jump_test(7, 4, [1, 6], (ValueError, 'into'))
+    def test_no_jump_into_for_block_before_else(output):
         output.append(1)
-        with tracecontext(output, 2):
-            output.append(3)
-        with tracecontext(output, 4):
-            output.append(5)
+        if not output:  # always false
+            for i in [3]:
+                output.append(4)
+        else:
+            output.append(6)
+            output.append(7)
+        output.append(8)
 
     def test_no_jump_to_non_integers(self):
         self.run_test(no_jump_to_non_integers, 2, "Spam", [True])


From webhook-mailer at python.org  Mon Jan  1 13:51:15 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 01 Jan 2018 18:51:15 -0000
Subject: [Python-checkins] bpo-32416: Add two new tests in
 test_sys_settrace. (GH-5072) (#5074)
Message-ID: <mailman.108.1514832676.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/2de47ca109d0418c7c01e451b5614f748ad76ee9
commit: 2de47ca109d0418c7c01e451b5614f748ad76ee9
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-01T20:51:12+02:00
summary:

bpo-32416: Add two new tests in test_sys_settrace. (GH-5072) (#5074)

Move other test to more proper place.
(cherry picked from commit e8ed96550c6aa9a1e39c36e67e892994e25e2c41)

files:
M Lib/test/test_sys_settrace.py

diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py
index e5b380fbe0b..02524588252 100644
--- a/Lib/test/test_sys_settrace.py
+++ b/Lib/test/test_sys_settrace.py
@@ -715,6 +715,14 @@ def test_jump_within_except_block(output):
             output.append(6)
         output.append(7)
 
+    @jump_test(2, 4, [1, 4, 5, -4])
+    def test_jump_across_with(output):
+        output.append(1)
+        with tracecontext(output, 2):
+            output.append(3)
+        with tracecontext(output, 4):
+            output.append(5)
+
     @jump_test(8, 11, [1, 3, 5, 11, 12])
     def test_jump_out_of_complex_nested_blocks(output):
         output.append(1)
@@ -766,6 +774,17 @@ def test_jump_over_break_in_try_finally_block(output):
             break
         output.append(13)
 
+    @jump_test(1, 7, [7, 8])
+    def test_jump_over_for_block_before_else(output):
+        output.append(1)
+        if not output:  # always false
+            for i in [3]:
+                output.append(4)
+        else:
+            output.append(6)
+            output.append(7)
+        output.append(8)
+
     # The second set of 'jump' tests are for things that are not allowed:
 
     @jump_test(2, 3, [1], (ValueError, 'after'))
@@ -911,21 +930,24 @@ def test_no_jump_out_of_finally_block(output):
         finally:
             output.append(5)
 
-    @jump_test(2, 4, [1, 4, 5, -4])
-    def test_jump_across_with(output):
+    @jump_test(3, 5, [1, 2, -2], (ValueError, 'into'))
+    def test_no_jump_between_with_blocks(output):
         output.append(1)
         with tracecontext(output, 2):
             output.append(3)
         with tracecontext(output, 4):
             output.append(5)
 
-    @jump_test(3, 5, [1, 2, -2], (ValueError, 'into'))
-    def test_jump_across_with_2(output):
+    @jump_test(7, 4, [1, 6], (ValueError, 'into'))
+    def test_no_jump_into_for_block_before_else(output):
         output.append(1)
-        with tracecontext(output, 2):
-            output.append(3)
-        with tracecontext(output, 4):
-            output.append(5)
+        if not output:  # always false
+            for i in [3]:
+                output.append(4)
+        else:
+            output.append(6)
+            output.append(7)
+        output.append(8)
 
     def test_no_jump_to_non_integers(self):
         self.run_test(no_jump_to_non_integers, 2, "Spam", [True])


From webhook-mailer at python.org  Mon Jan  1 13:51:33 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 01 Jan 2018 18:51:33 -0000
Subject: [Python-checkins] bpo-32416: Add two new tests in
 test_sys_settrace. (GH-5072) (#5073)
Message-ID: <mailman.109.1514832694.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/439ce8a93936d92e4f10765c482195a28e93ec76
commit: 439ce8a93936d92e4f10765c482195a28e93ec76
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-01T20:51:30+02:00
summary:

bpo-32416: Add two new tests in test_sys_settrace. (GH-5072) (#5073)

Move other test to more proper place.
(cherry picked from commit e8ed96550c6aa9a1e39c36e67e892994e25e2c41)

files:
M Lib/test/test_sys_settrace.py

diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py
index 49fbf4d750d..5b3b0660b89 100644
--- a/Lib/test/test_sys_settrace.py
+++ b/Lib/test/test_sys_settrace.py
@@ -743,6 +743,14 @@ def test_jump_within_except_block(output):
             output.append(6)
         output.append(7)
 
+    @jump_test(2, 4, [1, 4, 5, -4])
+    def test_jump_across_with(output):
+        output.append(1)
+        with tracecontext(output, 2):
+            output.append(3)
+        with tracecontext(output, 4):
+            output.append(5)
+
     @jump_test(8, 11, [1, 3, 5, 11, 12])
     def test_jump_out_of_complex_nested_blocks(output):
         output.append(1)
@@ -794,6 +802,17 @@ def test_jump_over_break_in_try_finally_block(output):
             break
         output.append(13)
 
+    @jump_test(1, 7, [7, 8])
+    def test_jump_over_for_block_before_else(output):
+        output.append(1)
+        if not output:  # always false
+            for i in [3]:
+                output.append(4)
+        else:
+            output.append(6)
+            output.append(7)
+        output.append(8)
+
     # The second set of 'jump' tests are for things that are not allowed:
 
     @jump_test(2, 3, [1], (ValueError, 'after'))
@@ -951,21 +970,24 @@ def test_no_jump_out_of_finally_block(output):
         finally:
             output.append(5)
 
-    @jump_test(2, 4, [1, 4, 5, -4])
-    def test_jump_across_with(output):
+    @jump_test(3, 5, [1, 2, -2], (ValueError, 'into'))
+    def test_no_jump_between_with_blocks(output):
         output.append(1)
         with tracecontext(output, 2):
             output.append(3)
         with tracecontext(output, 4):
             output.append(5)
 
-    @jump_test(3, 5, [1, 2, -2], (ValueError, 'into'))
-    def test_jump_across_with_2(output):
+    @jump_test(7, 4, [1, 6], (ValueError, 'into'))
+    def test_no_jump_into_for_block_before_else(output):
         output.append(1)
-        with tracecontext(output, 2):
-            output.append(3)
-        with tracecontext(output, 4):
-            output.append(5)
+        if not output:  # always false
+            for i in [3]:
+                output.append(4)
+        else:
+            output.append(6)
+            output.append(7)
+        output.append(8)
 
     def test_no_jump_to_non_integers(self):
         self.run_test(no_jump_to_non_integers, 2, "Spam", [True])


From webhook-mailer at python.org  Mon Jan  1 19:38:38 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 02 Jan 2018 00:38:38 -0000
Subject: [Python-checkins] bpo-32478: Add tests for 'break' and 'return'
 inside 'finally' clause. (#5078)
Message-ID: <mailman.110.1514853519.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/7cc42c356b0dc5ad9eaa9392789e84bd4aa1c7de
commit: 7cc42c356b0dc5ad9eaa9392789e84bd4aa1c7de
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-02T02:38:35+02:00
summary:

bpo-32478: Add tests for 'break' and 'return' inside 'finally' clause. (#5078)

files:
M Lib/test/test_grammar.py

diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index 823315f8cd0..9f26e9c4fb0 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -805,6 +805,80 @@ def g2(): return 1
         x = g2()
         check_syntax_error(self, "class foo:return 1")
 
+    def test_break_in_finally(self):
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                pass
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                continue
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                1/0
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                pass
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                continue
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                1/0
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+    def test_return_in_finally(self):
+        def g1():
+            try:
+                pass
+            finally:
+                return 1
+        self.assertEqual(g1(), 1)
+
+        def g2():
+            try:
+                return 2
+            finally:
+                return 3
+        self.assertEqual(g2(), 3)
+
+        def g3():
+            try:
+                1/0
+            finally:
+                return 4
+        self.assertEqual(g3(), 4)
+
     def test_yield(self):
         # Allowed as standalone statement
         def g(): yield 1


From webhook-mailer at python.org  Mon Jan  1 20:34:56 2018
From: webhook-mailer at python.org (R. David Murray)
Date: Tue, 02 Jan 2018 01:34:56 -0000
Subject: [Python-checkins] bpo-32452: clarify term 'brackets' in generator
 tutorial (#5079)
Message-ID: <mailman.111.1514856898.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/f190eb59e60e2ae7a7cbd396458389a7a076e0d3
commit: f190eb59e60e2ae7a7cbd396458389a7a076e0d3
branch: master
author: Emily Morehouse <emily at cuttlesoft.com>
committer: R. David Murray <rdmurray at bitdance.com>
date: 2018-01-01T20:34:53-05:00
summary:

bpo-32452: clarify term 'brackets' in generator tutorial (#5079)

Updates documentation for generator expressions in classes tutorial: Clarify usage of ambiguous term "brackets" by replacing with "square brackets". Updated subsequent lines to respect line breaks. (#5079)

files:
M Doc/tutorial/classes.rst

diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst
index 95141f94d6a..4676ef4b8a6 100644
--- a/Doc/tutorial/classes.rst
+++ b/Doc/tutorial/classes.rst
@@ -876,9 +876,9 @@ Generator Expressions
 =====================
 
 Some simple generators can be coded succinctly as expressions using a syntax
-similar to list comprehensions but with parentheses instead of brackets.  These
-expressions are designed for situations where the generator is used right away
-by an enclosing function.  Generator expressions are more compact but less
+similar to list comprehensions but with parentheses instead of square brackets.
+These expressions are designed for situations where the generator is used right
+away by an enclosing function.  Generator expressions are more compact but less
 versatile than full generator definitions and tend to be more memory friendly
 than equivalent list comprehensions.
 


From webhook-mailer at python.org  Tue Jan  2 03:20:15 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 02 Jan 2018 08:20:15 -0000
Subject: [Python-checkins] bpo-32478: Add tests for 'break' and 'return'
 inside 'finally' clause. (GH-5078) (#5083)
Message-ID: <mailman.112.1514881216.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/aea95c2fbdec05dddd8b833a74eeb7ec8dc69706
commit: aea95c2fbdec05dddd8b833a74eeb7ec8dc69706
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-02T10:20:12+02:00
summary:

bpo-32478: Add tests for 'break' and 'return' inside 'finally' clause. (GH-5078) (#5083)

(cherry picked from commit 7cc42c356b0dc5ad9eaa9392789e84bd4aa1c7de)

files:
M Lib/test/test_grammar.py

diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index 65e26bfd383..62f3251f051 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -805,6 +805,80 @@ def g2(): return 1
         x = g2()
         check_syntax_error(self, "class foo:return 1")
 
+    def test_break_in_finally(self):
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                pass
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                continue
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                1/0
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                pass
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                continue
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                1/0
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+    def test_return_in_finally(self):
+        def g1():
+            try:
+                pass
+            finally:
+                return 1
+        self.assertEqual(g1(), 1)
+
+        def g2():
+            try:
+                return 2
+            finally:
+                return 3
+        self.assertEqual(g2(), 3)
+
+        def g3():
+            try:
+                1/0
+            finally:
+                return 4
+        self.assertEqual(g3(), 4)
+
     def test_yield(self):
         # Allowed as standalone statement
         def g(): yield 1


From webhook-mailer at python.org  Tue Jan  2 03:56:48 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 02 Jan 2018 08:56:48 -0000
Subject: [Python-checkins] [2.7] bpo-32478: Add tests for 'break' and
 'return' inside 'finally' clause. (GH-5078). (#5084)
Message-ID: <mailman.113.1514883409.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/b495377a8bdd711e9aab0885c60cd148284156e8
commit: b495377a8bdd711e9aab0885c60cd148284156e8
branch: 2.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-02T10:56:40+02:00
summary:

[2.7] bpo-32478: Add tests for 'break' and 'return' inside 'finally' clause. (GH-5078). (#5084)

(cherry picked from commit 7cc42c356b0dc5ad9eaa9392789e84bd4aa1c7de)

files:
M Lib/test/test_grammar.py

diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index fc675c35a30..47b6b3e0d15 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -490,6 +490,80 @@ def g2(): return 1
         x = g2()
         check_syntax_error(self, "class foo:return 1")
 
+    def test_break_in_finally(self):
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                pass
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                continue
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        count = 0
+        while count < 2:
+            count += 1
+            try:
+                1/0
+            finally:
+                break
+        self.assertEqual(count, 1)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                pass
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                continue
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+        for count in [0, 1]:
+            self.assertEqual(count, 0)
+            try:
+                1/0
+            finally:
+                break
+        self.assertEqual(count, 0)
+
+    def test_return_in_finally(self):
+        def g1():
+            try:
+                pass
+            finally:
+                return 1
+        self.assertEqual(g1(), 1)
+
+        def g2():
+            try:
+                return 2
+            finally:
+                return 3
+        self.assertEqual(g2(), 3)
+
+        def g3():
+            try:
+                1/0
+            finally:
+                return 4
+        self.assertEqual(g3(), 4)
+
     def testYield(self):
         check_syntax_error(self, "class foo:yield 1")
 


From solipsis at pitrou.net  Tue Jan  2 04:13:13 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 02 Jan 2018 09:13:13 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=9
Message-ID: <20180102091313.1.7D2263C6DA5098E8@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_asyncio leaked [0, 0, 3] memory blocks, sum=3
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [0, -2, 2] memory blocks, sum=0
test_multiprocessing_spawn leaked [2, 0, 0] memory blocks, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogjkuXfD', '--timeout', '7200']

From webhook-mailer at python.org  Tue Jan  2 08:36:13 2018
From: webhook-mailer at python.org (R. David Murray)
Date: Tue, 02 Jan 2018 13:36:13 -0000
Subject: [Python-checkins] bpo-32452: clarify term 'brackets' in generator
 tutorial (GH-5079) (#5081)
Message-ID: <mailman.114.1514900174.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/f24c1857a8a1ba3efb3f957d43371bc9499e3c86
commit: f24c1857a8a1ba3efb3f957d43371bc9499e3c86
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: R. David Murray <rdmurray at bitdance.com>
date: 2018-01-02T08:36:06-05:00
summary:

bpo-32452: clarify term 'brackets' in generator tutorial (GH-5079) (#5081)

Updates documentation for generator expressions in classes tutorial: Clarify usage of ambiguous term "brackets" by replacing with "square brackets". Updated subsequent lines to respect line breaks. (GH-5079)
(cherry picked from commit f190eb59e60e2ae7a7cbd396458389a7a076e0d3)

files:
M Doc/tutorial/classes.rst

diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst
index 95141f94d6a..4676ef4b8a6 100644
--- a/Doc/tutorial/classes.rst
+++ b/Doc/tutorial/classes.rst
@@ -876,9 +876,9 @@ Generator Expressions
 =====================
 
 Some simple generators can be coded succinctly as expressions using a syntax
-similar to list comprehensions but with parentheses instead of brackets.  These
-expressions are designed for situations where the generator is used right away
-by an enclosing function.  Generator expressions are more compact but less
+similar to list comprehensions but with parentheses instead of square brackets.
+These expressions are designed for situations where the generator is used right
+away by an enclosing function.  Generator expressions are more compact but less
 versatile than full generator definitions and tend to be more memory friendly
 than equivalent list comprehensions.
 


From webhook-mailer at python.org  Tue Jan  2 10:42:18 2018
From: webhook-mailer at python.org (R. David Murray)
Date: Tue, 02 Jan 2018 15:42:18 -0000
Subject: [Python-checkins] bpo-32452: clarify term 'brackets' in generator
 tutorial (GH-5079) (#5082)
Message-ID: <mailman.115.1514907740.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/0e0d1017a4c8ad6f77ee42d7b640463058037f62
commit: 0e0d1017a4c8ad6f77ee42d7b640463058037f62
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: R. David Murray <rdmurray at bitdance.com>
date: 2018-01-02T10:42:12-05:00
summary:

bpo-32452: clarify term 'brackets' in generator tutorial (GH-5079) (#5082)

Updates documentation for generator expressions in classes tutorial: Clarify usage of ambiguous term "brackets" by replacing with "square brackets". Updated subsequent lines to respect line breaks. (GH-5079)
(cherry picked from commit f190eb59e60e2ae7a7cbd396458389a7a076e0d3)

files:
M Doc/tutorial/classes.rst

diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst
index 216dcae6b60..7f45c7632d5 100644
--- a/Doc/tutorial/classes.rst
+++ b/Doc/tutorial/classes.rst
@@ -873,9 +873,9 @@ Generator Expressions
 =====================
 
 Some simple generators can be coded succinctly as expressions using a syntax
-similar to list comprehensions but with parentheses instead of brackets.  These
-expressions are designed for situations where the generator is used right away
-by an enclosing function.  Generator expressions are more compact but less
+similar to list comprehensions but with parentheses instead of square brackets.
+These expressions are designed for situations where the generator is used right
+away by an enclosing function.  Generator expressions are more compact but less
 versatile than full generator definitions and tend to be more memory friendly
 than equivalent list comprehensions.
 


From webhook-mailer at python.org  Tue Jan  2 12:25:50 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 02 Jan 2018 17:25:50 -0000
Subject: [Python-checkins] bpo-31530: fix crash when multiple threads
 iterate over a file, round 2 (#5060)
Message-ID: <mailman.116.1514913951.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/dbf52e02f18dac6f5f0a64f78932f3dc6efc056b
commit: dbf52e02f18dac6f5f0a64f78932f3dc6efc056b
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-02T09:25:41-08:00
summary:

bpo-31530: fix crash when multiple threads iterate over a file, round 2 (#5060)

Multiple threads iterating over a file can corrupt the file's internal readahead
buffer resulting in crashes. To fix this, cache buffer state thread-locally for
the duration of a file_iternext call and only update the file's internal state
after reading completes.

No attempt is made to define or provide "reasonable" semantics for iterating
over a file on multiple threads. (Non-crashing) races are still
present. Duplicated, corrupt, and missing data will happen.

This was originally fixed by 6401e5671781eb217ee1afb4603cc0d1b0367ae6, which
raised an exception from seek() and next() when concurrent operations were
detected. Alas, this simpler solution breaks legitimate use cases such as
capturing the standard streams when multiple threads are logging.

files:
M Lib/test/test_file2k.py
M Misc/NEWS.d/next/Core and Builtins/2017-09-20-18-28-09.bpo-31530.CdLOM7.rst
M Objects/fileobject.c

diff --git a/Lib/test/test_file2k.py b/Lib/test/test_file2k.py
index d8966e034e0..c73e8d8dc45 100644
--- a/Lib/test/test_file2k.py
+++ b/Lib/test/test_file2k.py
@@ -653,18 +653,15 @@ def io_func():
         self._test_close_open_io(io_func)
 
     def test_iteration_torture(self):
-        # bpo-31530: Crash when concurrently iterate over a file.
+        # bpo-31530
         with open(self.filename, "wb") as fp:
             for i in xrange(2**20):
                 fp.write(b"0"*50 + b"\n")
         with open(self.filename, "rb") as f:
-            def iterate():
-                try:
-                    for l in f:
-                        pass
-                except IOError:
+            def it():
+                for l in f:
                     pass
-            self._run_workers(iterate, 10)
+            self._run_workers(it, 10)
 
     def test_iteration_seek(self):
         # bpo-31530: Crash when concurrently seek and iterate over a file.
@@ -674,17 +671,15 @@ def test_iteration_seek(self):
         with open(self.filename, "rb") as f:
             it = iter([1] + [0]*10)  # one thread reads, others seek
             def iterate():
-                try:
-                    if next(it):
-                        for l in f:
-                            pass
-                    else:
-                        for i in range(100):
-                            f.seek(i*100, 0)
-                except IOError:
-                    pass
+                if next(it):
+                    for l in f:
+                        pass
+                else:
+                    for i in xrange(100):
+                        f.seek(i*100, 0)
             self._run_workers(iterate, 10)
 
+
 @unittest.skipUnless(os.name == 'posix', 'test requires a posix system.')
 class TestFileSignalEINTR(unittest.TestCase):
     def _test_reading(self, data_to_write, read_and_verify_code, method_name,
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-09-20-18-28-09.bpo-31530.CdLOM7.rst b/Misc/NEWS.d/next/Core and Builtins/2017-09-20-18-28-09.bpo-31530.CdLOM7.rst
index a6cb6c9e9ba..beb09b5ae65 100644
--- a/Misc/NEWS.d/next/Core and Builtins/2017-09-20-18-28-09.bpo-31530.CdLOM7.rst	
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-09-20-18-28-09.bpo-31530.CdLOM7.rst	
@@ -1,4 +1 @@
 Fixed crashes when iterating over a file on multiple threads.
-seek() and next() methods of file objects now raise an exception during
-concurrent operation on the same file object.
-A lock can be used to prevent the error.
diff --git a/Objects/fileobject.c b/Objects/fileobject.c
index 8d1c5812f0d..270b28264a8 100644
--- a/Objects/fileobject.c
+++ b/Objects/fileobject.c
@@ -609,7 +609,12 @@ err_iterbuffered(void)
     return NULL;
 }
 
-static void drop_readahead(PyFileObject *);
+static void
+drop_file_readahead(PyFileObject *f)
+{
+    PyMem_FREE(f->f_buf);
+    f->f_buf = NULL;
+}
 
 /* Methods */
 
@@ -632,7 +637,7 @@ file_dealloc(PyFileObject *f)
     Py_XDECREF(f->f_mode);
     Py_XDECREF(f->f_encoding);
     Py_XDECREF(f->f_errors);
-    drop_readahead(f);
+    drop_file_readahead(f);
     Py_TYPE(f)->tp_free((PyObject *)f);
 }
 
@@ -767,13 +772,7 @@ file_seek(PyFileObject *f, PyObject *args)
 
     if (f->f_fp == NULL)
         return err_closed();
-    if (f->unlocked_count > 0) {
-        PyErr_SetString(PyExc_IOError,
-            "seek() called during concurrent "
-            "operation on the same file object");
-        return NULL;
-    }
-    drop_readahead(f);
+    drop_file_readahead(f);
     whence = 0;
     if (!PyArg_ParseTuple(args, "O|i:seek", &offobj, &whence))
         return NULL;
@@ -2242,12 +2241,16 @@ static PyGetSetDef file_getsetlist[] = {
     {0},
 };
 
+typedef struct {
+    char *buf, *bufptr, *bufend;
+} readaheadbuffer;
+
 static void
-drop_readahead(PyFileObject *f)
+drop_readaheadbuffer(readaheadbuffer *rab)
 {
-    if (f->f_buf != NULL) {
-        PyMem_Free(f->f_buf);
-        f->f_buf = NULL;
+    if (rab->buf != NULL) {
+        PyMem_FREE(rab->buf);
+        rab->buf = NULL;
     }
 }
 
@@ -2255,36 +2258,34 @@ drop_readahead(PyFileObject *f)
    (unless at EOF) and no more than bufsize.  Returns negative value on
    error, will set MemoryError if bufsize bytes cannot be allocated. */
 static int
-readahead(PyFileObject *f, Py_ssize_t bufsize)
+readahead(PyFileObject *f, readaheadbuffer *rab, Py_ssize_t bufsize)
 {
     Py_ssize_t chunksize;
 
-    assert(f->unlocked_count == 0);
-    if (f->f_buf != NULL) {
-        if( (f->f_bufend - f->f_bufptr) >= 1)
+    if (rab->buf != NULL) {
+        if ((rab->bufend - rab->bufptr) >= 1)
             return 0;
         else
-            drop_readahead(f);
+            drop_readaheadbuffer(rab);
     }
-    if ((f->f_buf = (char *)PyMem_Malloc(bufsize)) == NULL) {
+    if ((rab->buf = PyMem_MALLOC(bufsize)) == NULL) {
         PyErr_NoMemory();
         return -1;
     }
     FILE_BEGIN_ALLOW_THREADS(f)
     errno = 0;
-    chunksize = Py_UniversalNewlineFread(
-        f->f_buf, bufsize, f->f_fp, (PyObject *)f);
+    chunksize = Py_UniversalNewlineFread(rab->buf, bufsize, f->f_fp, (PyObject *)f);
     FILE_END_ALLOW_THREADS(f)
     if (chunksize == 0) {
         if (ferror(f->f_fp)) {
             PyErr_SetFromErrno(PyExc_IOError);
             clearerr(f->f_fp);
-            drop_readahead(f);
+            drop_readaheadbuffer(rab);
             return -1;
         }
     }
-    f->f_bufptr = f->f_buf;
-    f->f_bufend = f->f_buf + chunksize;
+    rab->bufptr = rab->buf;
+    rab->bufend = rab->buf + chunksize;
     return 0;
 }
 
@@ -2294,51 +2295,43 @@ readahead(PyFileObject *f, Py_ssize_t bufsize)
    logarithmic buffer growth to about 50 even when reading a 1gb line. */
 
 static PyStringObject *
-readahead_get_line_skip(PyFileObject *f, Py_ssize_t skip, Py_ssize_t bufsize)
+readahead_get_line_skip(PyFileObject *f, readaheadbuffer *rab, Py_ssize_t skip, Py_ssize_t bufsize)
 {
     PyStringObject* s;
     char *bufptr;
     char *buf;
     Py_ssize_t len;
 
-    if (f->unlocked_count > 0) {
-        PyErr_SetString(PyExc_IOError,
-            "next() called during concurrent "
-            "operation on the same file object");
-        return NULL;
-    }
-    if (f->f_buf == NULL)
-        if (readahead(f, bufsize) < 0)
+    if (rab->buf == NULL)
+        if (readahead(f, rab, bufsize) < 0)
             return NULL;
 
-    len = f->f_bufend - f->f_bufptr;
+    len = rab->bufend - rab->bufptr;
     if (len == 0)
-        return (PyStringObject *)
-            PyString_FromStringAndSize(NULL, skip);
-    bufptr = (char *)memchr(f->f_bufptr, '\n', len);
+        return (PyStringObject *)PyString_FromStringAndSize(NULL, skip);
+    bufptr = (char *)memchr(rab->bufptr, '\n', len);
     if (bufptr != NULL) {
         bufptr++;                               /* Count the '\n' */
-        len = bufptr - f->f_bufptr;
-        s = (PyStringObject *)
-            PyString_FromStringAndSize(NULL, skip + len);
+        len = bufptr - rab->bufptr;
+        s = (PyStringObject *)PyString_FromStringAndSize(NULL, skip + len);
         if (s == NULL)
             return NULL;
-        memcpy(PyString_AS_STRING(s) + skip, f->f_bufptr, len);
-        f->f_bufptr = bufptr;
-        if (bufptr == f->f_bufend)
-            drop_readahead(f);
+        memcpy(PyString_AS_STRING(s) + skip, rab->bufptr, len);
+        rab->bufptr = bufptr;
+        if (bufptr == rab->bufend)
+            drop_readaheadbuffer(rab);
     } else {
-        bufptr = f->f_bufptr;
-        buf = f->f_buf;
-        f->f_buf = NULL;                /* Force new readahead buffer */
+        bufptr = rab->bufptr;
+        buf = rab->buf;
+        rab->buf = NULL;                /* Force new readahead buffer */
         assert(len <= PY_SSIZE_T_MAX - skip);
-        s = readahead_get_line_skip(f, skip + len, bufsize + (bufsize>>2));
+        s = readahead_get_line_skip(f, rab, skip + len, bufsize + (bufsize>>2));
         if (s == NULL) {
-            PyMem_Free(buf);
+            PyMem_FREE(buf);
             return NULL;
         }
         memcpy(PyString_AS_STRING(s) + skip, bufptr, len);
-        PyMem_Free(buf);
+        PyMem_FREE(buf);
     }
     return s;
 }
@@ -2356,7 +2349,30 @@ file_iternext(PyFileObject *f)
     if (!f->readable)
         return err_mode("reading");
 
-    l = readahead_get_line_skip(f, 0, READAHEAD_BUFSIZE);
+    {
+        /*
+          Multiple threads can enter this method while the GIL is released
+          during file read and wreak havoc on the file object's readahead
+          buffer. To avoid dealing with cross-thread coordination issues, we
+          cache the file buffer state locally and only set it back on the file
+          object when we're done.
+        */
+        readaheadbuffer rab = {f->f_buf, f->f_bufptr, f->f_bufend};
+        f->f_buf = NULL;
+        l = readahead_get_line_skip(f, &rab, 0, READAHEAD_BUFSIZE);
+        /*
+          Make sure the file's internal read buffer is cleared out. This will
+          only do anything if some other thread interleaved with us during
+          readahead. We want to drop any changeling buffer, so we don't leak
+          memory. We may lose data, but that's what you get for reading the same
+          file object in multiple threads.
+        */
+        drop_file_readahead(f);
+        f->f_buf = rab.buf;
+        f->f_bufptr = rab.bufptr;
+        f->f_bufend = rab.bufend;
+    }
+
     if (l == NULL || PyString_GET_SIZE(l) == 0) {
         Py_XDECREF(l);
         return NULL;


From webhook-mailer at python.org  Tue Jan  2 18:52:49 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 02 Jan 2018 23:52:49 -0000
Subject: [Python-checkins] coalesce GILless sections in new_buffersize
 (#5059)
Message-ID: <mailman.117.1514937170.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/eb08a9290f28e602ba035acfe8c2d3cd2b9965b0
commit: eb08a9290f28e602ba035acfe8c2d3cd2b9965b0
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-02T15:52:42-08:00
summary:

coalesce GILless sections in new_buffersize (#5059)

830daae1c82ed33deef0086b7b6323e5be0b0cc8 added some new GIL-releasing to new_buffersize. This is fine, but it's better to avoid reacquiring the GIL for as long as possible. Also, it should use FILE_(BEGIN|END)_ALLOW_THREADS to avoid having the file closed from under it.

files:
M Objects/fileobject.c

diff --git a/Objects/fileobject.c b/Objects/fileobject.c
index 270b28264a8..b524f09e0a3 100644
--- a/Objects/fileobject.c
+++ b/Objects/fileobject.c
@@ -1015,10 +1015,10 @@ new_buffersize(PyFileObject *f, size_t currentsize)
     off_t pos, end;
     struct stat st;
     int res;
+    size_t bufsize = 0;
 
-    Py_BEGIN_ALLOW_THREADS
+    FILE_BEGIN_ALLOW_THREADS(f)
     res = fstat(fileno(f->f_fp), &st);
-    Py_END_ALLOW_THREADS
 
     if (res == 0) {
         end = st.st_size;
@@ -1032,9 +1032,7 @@ new_buffersize(PyFileObject *f, size_t currentsize)
            need to take the amount of buffered data into account.
            (Yet another reason why stdio stinks. :-) */
 
-        Py_BEGIN_ALLOW_THREADS
         pos = lseek(fileno(f->f_fp), 0L, SEEK_CUR);
-        Py_END_ALLOW_THREADS
 
         if (pos >= 0) {
             pos = ftell(f->f_fp);
@@ -1042,9 +1040,12 @@ new_buffersize(PyFileObject *f, size_t currentsize)
         if (pos < 0)
             clearerr(f->f_fp);
         if (end > pos && pos >= 0)
-            return currentsize + end - pos + 1;
+            bufsize = currentsize + end - pos + 1;
         /* Add 1 so if the file were to grow we'd notice. */
     }
+    FILE_END_ALLOW_THREADS(f)
+    if (bufsize != 0)
+        return bufsize;
 #endif
     /* Expand the buffer by an amount proportional to the current size,
        giving us amortized linear-time behavior. Use a less-than-double


From lp_benchmark_robot at intel.com  Tue Jan  2 19:19:34 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Tue, 2 Jan 2018 16:19:34 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-02
Message-ID: <20f10ab4-0fd4-4b9c-b4b8-e7eb8d9a6a96@orsmsx104.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-02 12:54:41-08:00.
  - commit:		f190eb5
  - previous commit:	e325608
  - revision date:	2018-01-01 20:34:53-05:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.628% |  +0.750%   |  +7.909%   |  +6.905%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.249% |  -1.681%   |  +22.446%  |  +10.724%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.021% |  -0.797%   |  +24.941%  |  +8.176%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 1.275% |  -0.965%   |  +22.191%  |  +11.543%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 3.673% |  -0.546%   |  +8.132%   |  +12.181%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.870% |  -0.329%   |  +9.353%   |  +11.350%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 0.685% |  -0.122%   |  +5.590%   |  +12.351%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.905% |  +0.650%   |  -0.977%   |  +8.520%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 1.463% |  +0.961%   |  +11.570%  |  +16.173%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.590% |  +0.852%   |  +12.859%  |  +13.514%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.231% |  -0.141%   |  +3.629%   |  +7.408%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.904% |  +0.006%   |  +5.276%   |  +5.428%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 1.764% |  -0.361%   |  +1.598%   |  +8.848%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.154% |  -0.846%   |  +12.084%  |  +9.207%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.662% |  -0.262%   |  +8.569%   |  +8.333%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.996% |  -0.205%   |  +6.115%   |  +12.340%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.536% |  +0.318%   |  +10.529%  |  +10.506%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.629% |  +0.919%   |  +7.630%   |  +10.362%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.121% |  -0.568%   |  +3.155%   |  +8.066%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 4.018% |  -0.971%   |  -0.795%   |  +13.669%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.690% |  +0.936%   |  +7.066%   |  +13.093%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.152% |  +1.681%   |  +47.676%  |  +11.273%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.685% |  +1.534%   |  +7.929%   |  +14.336%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.757% |  +0.581%   |  +17.153%  |  +12.344%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.726% |  +0.488%   |  +11.204%  |  +7.556%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.708% |  -0.827%   |  +4.328%   |  +4.984%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 1.009% |  -1.435%   |  -2.100%   |  +1.161%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.869% |  +0.178%   |  +4.526%   |  +5.738%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.315% |  -1.902%   |  +0.940%   |  +11.691%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.253% |  +0.803%   |  +1.149%   |  +19.569%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 1.104% |  +2.406%   |  +1.800%   |  +17.925%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.689% |  +1.205%   |  +3.772%   |  +18.242%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 2.900% |  +1.436%   |  +11.530%  |  +10.437%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.472% |  -0.204%   |  -0.171%   |  +9.644%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.142% |  +0.224%   |  +10.670%  |  +4.839%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.107% |  -0.013%   |  +1.960%   |  +5.148%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.810% |  +0.696%   |  +10.455%  |  +14.272%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.326% |  -1.118%   |  +4.950%   |  +9.056%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.629% |  -0.270%   |  -2.544%   |  +11.464%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.847% |  +1.020%   |  -5.220%   |  +5.141%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 1.235% |  -0.642%   |  +3.830%   |  +6.260%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.137% |  +1.298%   |  +9.088%   |  +14.883%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.632% |  +0.958%   |  -1.503%   |  +2.844%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.294% |  +0.471%   |  +27.328%  |  +7.329%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.257% |  +0.323%   |  +4.810%   |  +5.277%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.573% |  +0.276%   |  +14.522%  |  +8.771%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.740% |  +0.866%   |  -2.746%   |  -1.194%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.439% |  +0.436%   |  +5.850%   |  +2.336%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.552% |  +0.437%   |  +6.948%   |  +6.692%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.486% |  +0.688%   |  +6.508%   |  +5.829%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.659% |  -1.415%   |  +19.000%  |  +6.349%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 1.984% |  +0.285%   |  +12.287%  |  +5.841%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.127% |  -0.010%   |  +9.068%   |  +6.141%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 4.490% |  -0.131%   |  +11.699%  |  +7.881%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 5.808% |  +0.275%   |  +10.954%  |  +7.149%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 4.827% |  +1.036%   |  +20.006%  |  +11.780%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.958% |  -0.017%   |  +6.104%   |  +6.524%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.388% |  +0.319%   |  +2.019%   |  -0.897%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 6.934% |  -1.154%   |  +8.350%   |  +21.270%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 2.498% |  +2.675%   |  -3.986%   |  +14.267%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 1.352% |  +0.804%   |  +8.694%   |  +5.699%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.091% |  +0.116%   |  +1.971%   |  +10.304%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.406% |  +1.191%   |  +3.710%   |  +6.418%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 3.522% |  +1.823%   |  -8.935%   |  +13.501%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.350% |  -0.096%   |  +4.185%   |  +9.883%   |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-02

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Wed Jan  3 04:16:04 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 03 Jan 2018 09:16:04 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=2
Message-ID: <20180103091604.1.6EF98A5719944971@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [-1, 1, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloguFaMBU', '--timeout', '7200']

From webhook-mailer at python.org  Wed Jan  3 14:32:31 2018
From: webhook-mailer at python.org (Brett Cannon)
Date: Wed, 03 Jan 2018 19:32:31 -0000
Subject: [Python-checkins] Ignore importlib.h for automatic review requests
 from the import team. (GH-5087)
Message-ID: <mailman.118.1515007952.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/811b2878dfc68dc3ecd81fb4b370c6e1f9ec69c2
commit: 811b2878dfc68dc3ecd81fb4b370c6e1f9ec69c2
branch: master
author: Brett Cannon <brettcannon at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-03T11:32:22-08:00
summary:

Ignore importlib.h for automatic review requests from the import team. (GH-5087)

Otherwise the import team gets flagged for reviews any time the bytecode for
importlib.h changes (e.g new bytecode, optimizations, etc.).

files:
M .github/CODEOWNERS

diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index d751a918d0d..b42e1c93d8f 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -14,8 +14,13 @@
 **/*hashlib*                  @python/crypto-team
 **/*pyhash*                   @python/crypto-team
 
-# Import (including importlib)
-**/*import*                   @python/import-team
+# Import (including importlib).
+# Ignoring importlib.h so as to not get flagged on
+# all pull requests that change the the emitted
+# bytecode.
+**/*import*.c                 @python/import-team
+**/*import*.py                @python/import-team
+
 
 # SSL
 **/*ssl*                      @python/crypto-team


From lp_benchmark_robot at intel.com  Wed Jan  3 18:38:31 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Wed, 3 Jan 2018 15:38:31 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-02
Message-ID: <e4dda3b9-6438-445b-8f34-a59d33dc0c6a@orsmsx103.amr.corp.intel.com>


No new revisions. Here are the previous results:

Results for project python/master, build date: 2018-01-02 20:54:41+00:00.
  - commit:		f190eb5
  - previous commit:	e325608
  - revision date:	2018-01-01 20:34:53-05:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.628% |  +0.750%   |  +7.909%   |  +6.905%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.249% |  -1.681%   |  +22.446%  |  +10.724%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.021% |  -0.797%   |  +24.941%  |  +8.176%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 1.275% |  -0.965%   |  +22.191%  |  +11.543%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 3.673% |  -0.546%   |  +8.132%   |  +12.181%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.870% |  -0.329%   |  +9.353%   |  +11.350%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 0.685% |  -0.122%   |  +5.590%   |  +12.351%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.905% |  +0.650%   |  -0.977%   |  +8.520%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 1.463% |  +0.961%   |  +11.570%  |  +16.173%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.590% |  +0.852%   |  +12.859%  |  +13.514%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.231% |  -0.141%   |  +3.629%   |  +7.408%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.904% |  +0.006%   |  +5.276%   |  +5.428%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 1.764% |  -0.361%   |  +1.598%   |  +8.848%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.154% |  -0.846%   |  +12.084%  |  +9.207%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.662% |  -0.262%   |  +8.569%   |  +8.333%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.996% |  -0.205%   |  +6.115%   |  +12.340%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.536% |  +0.318%   |  +10.529%  |  +10.506%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.629% |  +0.919%   |  +7.630%   |  +10.362%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.121% |  -0.568%   |  +3.155%   |  +8.066%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 4.018% |  -0.971%   |  -0.795%   |  +13.669%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.690% |  +0.936%   |  +7.066%   |  +13.093%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.152% |  +1.681%   |  +47.676%  |  +11.273%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.685% |  +1.534%   |  +7.929%   |  +14.336%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.757% |  +0.581%   |  +17.153%  |  +12.344%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.726% |  +0.488%   |  +11.204%  |  +7.556%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.708% |  -0.827%   |  +4.328%   |  +4.984%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 1.009% |  -1.435%   |  -2.100%   |  +1.161%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.869% |  +0.178%   |  +4.526%   |  +5.738%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.315% |  -1.902%   |  +0.940%   |  +11.691%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.253% |  +0.803%   |  +1.149%   |  +19.569%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 1.104% |  +2.406%   |  +1.800%   |  +17.925%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.689% |  +1.205%   |  +3.772%   |  +18.242%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 2.900% |  +1.436%   |  +11.530%  |  +10.437%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.472% |  -0.204%   |  -0.171%   |  +9.644%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.142% |  +0.224%   |  +10.670%  |  +4.839%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.107% |  -0.013%   |  +1.960%   |  +5.148%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.810% |  +0.696%   |  +10.455%  |  +14.272%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.326% |  -1.118%   |  +4.950%   |  +9.056%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.629% |  -0.270%   |  -2.544%   |  +11.464%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.847% |  +1.020%   |  -5.220%   |  +5.141%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 1.235% |  -0.642%   |  +3.830%   |  +6.260%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.137% |  +1.298%   |  +9.088%   |  +14.883%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.632% |  +0.958%   |  -1.503%   |  +2.844%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.294% |  +0.471%   |  +27.328%  |  +7.329%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.257% |  +0.323%   |  +4.810%   |  +5.277%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.573% |  +0.276%   |  +14.522%  |  +8.771%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.740% |  +0.866%   |  -2.746%   |  -1.194%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.439% |  +0.436%   |  +5.850%   |  +2.336%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.552% |  +0.437%   |  +6.948%   |  +6.692%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.486% |  +0.688%   |  +6.508%   |  +5.829%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.659% |  -1.415%   |  +19.000%  |  +6.349%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 1.984% |  +0.285%   |  +12.287%  |  +5.841%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.127% |  -0.010%   |  +9.068%   |  +6.141%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 4.490% |  -0.131%   |  +11.699%  |  +7.881%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 5.808% |  +0.275%   |  +10.954%  |  +7.149%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 4.827% |  +1.036%   |  +20.006%  |  +11.780%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.958% |  -0.017%   |  +6.104%   |  +6.524%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.388% |  +0.319%   |  +2.019%   |  -0.897%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 6.934% |  -1.154%   |  +8.350%   |  +21.270%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 2.498% |  +2.675%   |  -3.986%   |  +14.267%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 1.352% |  +0.804%   |  +8.694%   |  +5.699%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.091% |  +0.116%   |  +1.971%   |  +10.304%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.406% |  +1.191%   |  +3.710%   |  +6.418%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 3.522% |  +1.823%   |  -8.935%   |  +13.501%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.350% |  -0.096%   |  +4.185%   |  +9.883%   |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-02

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Thu Jan  4 03:36:18 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 08:36:18 -0000
Subject: [Python-checkins] bpo-32482: Improve syntax and grammar tests.
 (#5085)
Message-ID: <mailman.119.1515054978.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/0bcba372bcd564555c98a65be4fb1309ff1ae5d5
commit: 0bcba372bcd564555c98a65be4fb1309ff1ae5d5
branch: 2.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T10:36:14+02:00
summary:

bpo-32482: Improve syntax and grammar tests. (#5085)

files:
M Lib/test/test_grammar.py
M Lib/test/test_syntax.py
M Lib/test/test_sys_settrace.py

diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index 47b6b3e0d15..23b6ce8d3c5 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -11,7 +11,7 @@
 
 class TokenTests(unittest.TestCase):
 
-    def testBackslash(self):
+    def test_backslash(self):
         # Backslash means line continuation:
         x = 1 \
         + 1
@@ -21,29 +21,42 @@ def testBackslash(self):
         x = 0
         self.assertEqual(x, 0, 'backslash ending comment')
 
-    def testPlainIntegers(self):
+    def test_plain_integers(self):
+        self.assertEqual(type(000), type(0))
         self.assertEqual(0xff, 255)
         self.assertEqual(0377, 255)
+        self.assertEqual(0o377, 255)
         self.assertEqual(2147483647, 017777777777)
+        self.assertEqual(2147483647, 0o17777777777)
+        self.assertEqual(0b1001, 9)
         # "0x" is not a valid literal
         self.assertRaises(SyntaxError, eval, "0x")
         from sys import maxint
         if maxint == 2147483647:
-            self.assertEqual(-2147483647-1, -020000000000)
+            self.assertEqual(-2147483647-1, -0o20000000000)
             # XXX -2147483648
             self.assertTrue(037777777777 > 0)
+            self.assertTrue(0o37777777777 > 0)
             self.assertTrue(0xffffffff > 0)
-            for s in '2147483648', '040000000000', '0x100000000':
+            self.assertTrue(0b1111111111111111111111111111111 > 0)
+            for s in ('2147483648', '040000000000', '0o40000000000',
+                      '0x100000000',
+                      '0b10000000000000000000000000000000'):
                 try:
                     x = eval(s)
                 except OverflowError:
                     self.fail("OverflowError on huge integer literal %r" % s)
         elif maxint == 9223372036854775807:
             self.assertEqual(-9223372036854775807-1, -01000000000000000000000)
+            self.assertEqual(-9223372036854775807-1, -0o1000000000000000000000)
             self.assertTrue(01777777777777777777777 > 0)
+            self.assertTrue(0o1777777777777777777777 > 0)
             self.assertTrue(0xffffffffffffffff > 0)
+            self.assertTrue(0b11111111111111111111111111111111111111111111111111111111111111 > 0)
             for s in '9223372036854775808', '02000000000000000000000', \
-                     '0x10000000000000000':
+                     '0o2000000000000000000000', \
+                     '0x10000000000000000', \
+                     '0b100000000000000000000000000000000000000000000000000000000000000':
                 try:
                     x = eval(s)
                 except OverflowError:
@@ -51,7 +64,7 @@ def testPlainIntegers(self):
         else:
             self.fail('Weird maxint value %r' % maxint)
 
-    def testLongIntegers(self):
+    def test_long_integers(self):
         x = 0L
         x = 0l
         x = 0xffffffffffffffffL
@@ -61,7 +74,7 @@ def testLongIntegers(self):
         x = 123456789012345678901234567890L
         x = 123456789012345678901234567890l
 
-    def testFloats(self):
+    def test_floats(self):
         x = 3.14
         x = 314.
         x = 0.314
@@ -81,7 +94,7 @@ def test_float_exponent_tokenization(self):
         self.assertEqual(1 if 0else 0, 0)
         self.assertRaises(SyntaxError, eval, "0 if 1Else 0")
 
-    def testStringLiterals(self):
+    def test_string_literals(self):
         x = ''; y = ""; self.assertTrue(len(x) == 0 and x == y)
         x = '\''; y = "'"; self.assertTrue(len(x) == 1 and x == y and ord(x) == 39)
         x = '"'; y = "\""; self.assertTrue(len(x) == 1 and x == y and ord(x) == 34)
@@ -133,11 +146,11 @@ class GrammarTests(unittest.TestCase):
     # expr_input: testlist NEWLINE
     # XXX Hard to test -- used only in calls to input()
 
-    def testEvalInput(self):
+    def test_eval_input(self):
         # testlist ENDMARKER
         x = eval('1, 0 or 1')
 
-    def testFuncdef(self):
+    def test_funcdef(self):
         ### 'def' NAME parameters ':' suite
         ### parameters: '(' [varargslist] ')'
         ### varargslist: (fpdef ['=' test] ',')* ('*' NAME [',' ('**'|'*' '*') NAME]
@@ -153,9 +166,10 @@ def f1(): pass
         f1(*(), **{})
         def f2(one_argument): pass
         def f3(two, arguments): pass
-        # Silence Py3k warning
-        exec('def f4(two, (compound, (argument, list))): pass')
-        exec('def f5((compound, first), two): pass')
+        with check_py3k_warnings(('tuple parameter unpacking has been removed',
+                                  SyntaxWarning)):
+            exec('def f4(two, (compound, (argument, list))): pass')
+            exec('def f5((compound, first), two): pass')
         self.assertEqual(f2.func_code.co_varnames, ('one_argument',))
         self.assertEqual(f3.func_code.co_varnames, ('two', 'arguments'))
         if sys.platform.startswith('java'):
@@ -174,8 +188,9 @@ def a2(two, args,): pass
         def v0(*rest): pass
         def v1(a, *rest): pass
         def v2(a, b, *rest): pass
-        # Silence Py3k warning
-        exec('def v3(a, (b, c), *rest): return a, b, c, rest')
+        with check_py3k_warnings(('tuple parameter unpacking has been removed',
+                                  SyntaxWarning)):
+            exec('def v3(a, (b, c), *rest): return a, b, c, rest')
 
         f1()
         f2(1)
@@ -212,7 +227,9 @@ def d01(a=1): pass
         d01()
         d01(1)
         d01(*(1,))
+        d01(*[] or [2])
         d01(**{'a':2})
+        d01(**{'a':2} or {})
         def d11(a, b=1): pass
         d11(1)
         d11(1, 2)
@@ -280,9 +297,12 @@ def d22v(a, b, c=1, d=2, *rest): pass
         d22v(*(1, 2, 3, 4))
         d22v(1, 2, *(3, 4, 5))
         d22v(1, *(2, 3), **{'d': 4})
-        # Silence Py3k warning
-        exec('def d31v((x)): pass')
-        exec('def d32v((x,)): pass')
+        with check_py3k_warnings(('parenthesized argument names are invalid',
+                                  SyntaxWarning)):
+            exec('def d31v((x)): pass')
+        with check_py3k_warnings(('tuple parameter unpacking has been removed',
+                                  SyntaxWarning)):
+            exec('def d32v((x,)): pass')
         d31v(1)
         d32v((1,))
 
@@ -293,12 +313,19 @@ def f(*args, **kwargs):
                                                     {'x':2, 'y':5}))
         self.assertRaises(SyntaxError, eval, "f(1, *(2,3), 4)")
         self.assertRaises(SyntaxError, eval, "f(1, x=2, *(3,4), x=5)")
+        self.assertEqual(f(**{'eggs':'scrambled', 'spam':'fried'}),
+                         ((), {'eggs':'scrambled', 'spam':'fried'}))
+        self.assertEqual(f(spam='fried', **{'eggs':'scrambled'}),
+                         ((), {'eggs':'scrambled', 'spam':'fried'}))
 
         # Check ast errors in *args and *kwargs
         check_syntax_error(self, "f(*g(1=2))")
         check_syntax_error(self, "f(**g(1=2))")
 
-    def testLambdef(self):
+        # Check trailing commas are permitted in funcdef argument list
+        def f(a,): pass
+
+    def test_lambdef(self):
         ### lambdef: 'lambda' [varargslist] ':' test
         l1 = lambda : 0
         self.assertEqual(l1(), 0)
@@ -311,12 +338,18 @@ def testLambdef(self):
         self.assertEqual(l5(1, 2), 5)
         self.assertEqual(l5(1, 2, 3), 6)
         check_syntax_error(self, "lambda x: x = 2")
-        check_syntax_error(self, "lambda (None,): None")
+        with check_py3k_warnings(('tuple parameter unpacking has been removed',
+                                  SyntaxWarning)):
+            check_syntax_error(self, "lambda (None,): None")
+
+        # check that trailing commas are permitted
+        l10 = lambda a,: 0
+
 
     ### stmt: simple_stmt | compound_stmt
     # Tested below
 
-    def testSimpleStmt(self):
+    def test_simple_stmt(self):
         ### simple_stmt: small_stmt (';' small_stmt)* [';']
         x = 1; pass; del x
         def foo():
@@ -327,7 +360,7 @@ def foo():
     ### small_stmt: expr_stmt | print_stmt  | pass_stmt | del_stmt | flow_stmt | import_stmt | global_stmt | access_stmt | exec_stmt
     # Tested below
 
-    def testExprStmt(self):
+    def test_expr_stmt(self):
         # (exprlist '=')* exprlist
         1
         1, 2, 3
@@ -340,7 +373,7 @@ def testExprStmt(self):
         check_syntax_error(self, "x + 1 = 1")
         check_syntax_error(self, "a + 1 = b + 2")
 
-    def testPrintStmt(self):
+    def test_print_stmt(self):
         # 'print' (test ',')* [test]
         import StringIO
 
@@ -410,7 +443,7 @@ def tellme(file=None):
         check_syntax_error(self, 'print ,')
         check_syntax_error(self, 'print >> x,')
 
-    def testDelStmt(self):
+    def test_del_stmt(self):
         # 'del' exprlist
         abc = [1,2,3]
         x, y, z = abc
@@ -419,18 +452,18 @@ def testDelStmt(self):
         del abc
         del x, y, (z, xyz)
 
-    def testPassStmt(self):
+    def test_pass_stmt(self):
         # 'pass'
         pass
 
     # flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt
     # Tested below
 
-    def testBreakStmt(self):
+    def test_break_stmt(self):
         # 'break'
         while 1: break
 
-    def testContinueStmt(self):
+    def test_continue_stmt(self):
         # 'continue'
         i = 1
         while i: i = 0; continue
@@ -482,7 +515,7 @@ def test_inner(extra_burning_oil = 1, count=0):
                 self.fail("continue then break in try/except in loop broken!")
         test_inner()
 
-    def testReturn(self):
+    def test_return(self):
         # 'return' [testlist]
         def g1(): return
         def g2(): return 1
@@ -513,7 +546,7 @@ def test_break_in_finally(self):
         while count < 2:
             count += 1
             try:
-                1/0
+                1.0/0.0
             finally:
                 break
         self.assertEqual(count, 1)
@@ -537,7 +570,7 @@ def test_break_in_finally(self):
         for count in [0, 1]:
             self.assertEqual(count, 0)
             try:
-                1/0
+                1.0/0.0
             finally:
                 break
         self.assertEqual(count, 0)
@@ -559,13 +592,33 @@ def g2():
 
         def g3():
             try:
-                1/0
+                1.0/0.0
             finally:
                 return 4
         self.assertEqual(g3(), 4)
 
-    def testYield(self):
+    def test_yield(self):
+        # Allowed as standalone statement
+        def g(): yield 1
+        # Allowed as RHS of assignment
+        def g(): x = yield 1
+        # Ordinary yield accepts implicit tuples
+        def g(): yield 1, 1
+        def g(): x = yield 1, 1
+        # Requires parentheses as subexpression
+        def g(): 1, (yield 1)
+        check_syntax_error(self, "def g(): 1, yield 1")
+        # Requires parentheses as call argument
+        def g(): f((yield 1))
+        def g(): f((yield 1), 1)
+        check_syntax_error(self, "def g(): f(yield 1)")
+        check_syntax_error(self, "def g(): f(yield 1, 1)")
+        # Not allowed at top level
+        check_syntax_error(self, "yield")
+        # Not allowed at class scope
         check_syntax_error(self, "class foo:yield 1")
+        # Check annotation refleak on SyntaxError
+        check_syntax_error(self, "def g(a:(yield)): pass")
 
     def test_yield_in_comprehensions(self):
         # Check yield in comprehensions
@@ -607,14 +660,14 @@ def check(code, warntext):
         check("((yield x) for x in ())",
               "'yield' inside generator expression")
 
-    def testRaise(self):
+    def test_raise(self):
         # 'raise' test [',' test]
         try: raise RuntimeError, 'just testing'
         except RuntimeError: pass
         try: raise KeyboardInterrupt
         except KeyboardInterrupt: pass
 
-    def testImport(self):
+    def test_import(self):
         # 'import' dotted_as_names
         import sys
         import time, sys
@@ -627,13 +680,13 @@ def testImport(self):
         from sys import (path, argv)
         from sys import (path, argv,)
 
-    def testGlobal(self):
+    def test_global(self):
         # 'global' NAME (',' NAME)*
         global a
         global a, b
         global one, two, three, four, five, six, seven, eight, nine, ten
 
-    def testExec(self):
+    def test_exec(self):
         # 'exec' expr ['in' expr [',' expr]]
         z = None
         del z
@@ -665,7 +718,7 @@ def testExec(self):
         if (g, l) != ({'a':1}, {'b':2}):
             self.fail('exec ... in g (%s), l (%s)' %(g,l))
 
-    def testAssert(self):
+    def test_assert(self):
         # assertTruestmt: 'assert' test [',' test]
         assert 1
         assert 1, 1
@@ -704,7 +757,7 @@ def testAssert2(self):
     ### compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | funcdef | classdef
     # Tested below
 
-    def testIf(self):
+    def test_if(self):
         # 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
         if 1: pass
         if 1: pass
@@ -717,7 +770,7 @@ def testIf(self):
         elif 0: pass
         else: pass
 
-    def testWhile(self):
+    def test_while(self):
         # 'while' test ':' suite ['else' ':' suite]
         while 0: pass
         while 0: pass
@@ -732,7 +785,7 @@ def testWhile(self):
             x = 2
         self.assertEqual(x, 2)
 
-    def testFor(self):
+    def test_for(self):
         # 'for' exprlist 'in' exprlist ':' suite ['else' ':' suite]
         for i in 1, 2, 3: pass
         for i, j, k in (): pass
@@ -759,30 +812,30 @@ def __getitem__(self, i):
             result.append(x)
         self.assertEqual(result, [1, 2, 3])
 
-    def testTry(self):
+    def test_try(self):
         ### try_stmt: 'try' ':' suite (except_clause ':' suite)+ ['else' ':' suite]
         ###         | 'try' ':' suite 'finally' ':' suite
         ### except_clause: 'except' [expr [('as' | ',') expr]]
         try:
-            1/0
+            1/0.0
         except ZeroDivisionError:
             pass
         else:
             pass
-        try: 1/0
+        try: 1/0.0
         except EOFError: pass
         except TypeError as msg: pass
         except RuntimeError, msg: pass
         except: pass
         else: pass
-        try: 1/0
+        try: 1/0.0
         except (EOFError, TypeError, ZeroDivisionError): pass
-        try: 1/0
+        try: 1/0.0
         except (EOFError, TypeError, ZeroDivisionError), msg: pass
         try: pass
         finally: pass
 
-    def testSuite(self):
+    def test_suite(self):
         # simple_stmt | NEWLINE INDENT NEWLINE* (stmt NEWLINE*)+ DEDENT
         if 1: pass
         if 1:
@@ -797,7 +850,7 @@ def testSuite(self):
             pass
             #
 
-    def testTest(self):
+    def test_test(self):
         ### and_test ('or' and_test)*
         ### and_test: not_test ('and' not_test)*
         ### not_test: 'not' not_test | comparison
@@ -808,7 +861,7 @@ def testTest(self):
         if not 1 and 1 and 1: pass
         if 1 and 1 or 1 and 1 and 1 or not 1 and 1: pass
 
-    def testComparison(self):
+    def test_comparison(self):
         ### comparison: expr (comp_op expr)*
         ### comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'
         if 1: pass
@@ -824,40 +877,49 @@ def testComparison(self):
         if 1 in (): pass
         if 1 not in (): pass
         if 1 < 1 > 1 == 1 >= 1 <= 1 != 1 in 1 not in 1 is 1 is not 1: pass
-        # Silence Py3k warning
-        if eval('1 <> 1'): pass
-        if eval('1 < 1 > 1 == 1 >= 1 <= 1 <> 1 != 1 in 1 not in 1 is 1 is not 1'): pass
+        with check_py3k_warnings(('<> not supported in 3.x; use !=',
+                                  DeprecationWarning)):
+            if eval('1 <> 1'): pass
+        with check_py3k_warnings(('<> not supported in 3.x; use !=',
+                                  DeprecationWarning)):
+            if eval('1 < 1 > 1 == 1 >= 1 <= 1 <> 1 != 1 in 1 not in 1 is 1 is not 1'): pass
 
-    def testBinaryMaskOps(self):
+    def test_binary_mask_ops(self):
         x = 1 & 1
         x = 1 ^ 1
         x = 1 | 1
 
-    def testShiftOps(self):
+    def test_shift_ops(self):
         x = 1 << 1
         x = 1 >> 1
         x = 1 << 1 >> 1
 
-    def testAdditiveOps(self):
+    def test_additive_ops(self):
         x = 1
         x = 1 + 1
         x = 1 - 1 - 1
         x = 1 - 1 + 1 - 1 + 1
 
-    def testMultiplicativeOps(self):
+    def test_multiplicative_ops(self):
         x = 1 * 1
-        x = 1 / 1
+        with check_py3k_warnings(('classic int division', DeprecationWarning)):
+            x = 1 / 1
+        x = 1 / 1.0
         x = 1 % 1
-        x = 1 / 1 * 1 % 1
+        with check_py3k_warnings(('classic int division', DeprecationWarning)):
+            x = 1 / 1 * 1 % 1
+        x = 1 / 1.0 * 1 % 1
 
-    def testUnaryOps(self):
+    def test_unary_ops(self):
         x = +1
         x = -1
         x = ~1
         x = ~1 ^ 1 & 1 | 1 & 1 ^ -1
-        x = -1*1/1 + 1*1 - ---1*1
+        with check_py3k_warnings(('classic int division', DeprecationWarning)):
+            x = -1*1/1 + 1*1 - ---1*1
+        x = -1*1/1.0 + 1*1 - ---1*1
 
-    def testSelectors(self):
+    def test_selectors(self):
         ### trailer: '(' [testlist] ')' | '[' subscript ']' | '.' NAME
         ### subscript: expr | [expr] ':' [expr]
 
@@ -884,10 +946,10 @@ def testSelectors(self):
         d[1,2] = 3
         d[1,2,3] = 4
         L = list(d)
-        L.sort()
+        L.sort(key=lambda x: (type(x).__name__, x))
         self.assertEqual(str(L), '[1, (1,), (1, 2), (1, 2, 3)]')
 
-    def testAtoms(self):
+    def test_atoms(self):
         ### atom: '(' [testlist] ')' | '[' [testlist] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING
         ### dictorsetmaker: (test ':' test (',' test ':' test)* [',']) | (test (',' test)* [','])
 
@@ -914,10 +976,10 @@ def testAtoms(self):
         x = {'one', 'two', 'three'}
         x = {2, 3, 4,}
 
-        # Silence Py3k warning
-        x = eval('`x`')
-        x = eval('`1 or 2 or 3`')
-        self.assertEqual(eval('`1,2`'), '(1, 2)')
+        with check_py3k_warnings(('backquote not supported', SyntaxWarning)):
+            x = eval('`x`')
+            x = eval('`1 or 2 or 3`')
+            self.assertEqual(eval('`1,2`'), '(1, 2)')
 
         x = x
         x = 'x'
@@ -927,7 +989,7 @@ def testAtoms(self):
     ### testlist: test (',' test)* [',']
     # These have been exercised enough above
 
-    def testClassdef(self):
+    def test_classdef(self):
         # 'class' NAME ['(' [testlist] ')'] ':' suite
         class B: pass
         class B2(): pass
@@ -938,6 +1000,7 @@ class C:
             def meth1(self): pass
             def meth2(self, arg): pass
             def meth3(self, a1, a2): pass
+
         # decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
         # decorators: decorator+
         # decorated: decorators (classdef | funcdef)
@@ -949,14 +1012,14 @@ class G:
             pass
         self.assertEqual(G.decorated, True)
 
-    def testDictcomps(self):
+    def test_dictcomps(self):
         # dictorsetmaker: ( (test ':' test (comp_for |
         #                                   (',' test ':' test)* [','])) |
         #                   (test (comp_for | (',' test)* [','])) )
         nums = [1, 2, 3]
         self.assertEqual({i:i+1 for i in nums}, {1: 2, 2: 3, 3: 4})
 
-    def testListcomps(self):
+    def test_listcomps(self):
         # list comprehension tests
         nums = [1, 2, 3, 4, 5]
         strs = ["Apple", "Banana", "Coconut"]
@@ -979,7 +1042,7 @@ def testListcomps(self):
                          [[1], [1, 1], [1, 2, 4], [1, 3, 9, 27], [1, 4, 16, 64, 256]])
 
         def test_in_func(l):
-            return [None < x < 3 for x in l if x > 2]
+            return [0 < x < 3 for x in l if x > 2]
 
         self.assertEqual(test_in_func(nums), [False, False, False])
 
@@ -1019,7 +1082,7 @@ def test_nested_front():
         self.assertEqual(x, [('Boeing', 'Airliner'), ('Boeing', 'Engine'), ('Ford', 'Engine'),
                              ('Macdonalds', 'Cheeseburger')])
 
-    def testGenexps(self):
+    def test_genexps(self):
         # generator expression tests
         g = ([x for x in range(10)] for x in range(1))
         self.assertEqual(g.next(), [x for x in range(10)])
@@ -1054,7 +1117,7 @@ def testGenexps(self):
         check_syntax_error(self, "foo(x for x in range(10), 100)")
         check_syntax_error(self, "foo(100, x for x in range(10))")
 
-    def testComprehensionSpecials(self):
+    def test_comprehension_specials(self):
         # test for outmost iterable precomputation
         x = 10; g = (i for i in range(x)); x = 5
         self.assertEqual(len(list(g)), 10)
@@ -1093,11 +1156,11 @@ def __exit__(self, *args):
         with manager() as x, manager():
             pass
 
-    def testIfElseExpr(self):
+    def test_if_else_expr(self):
         # Test ifelse expressions in various cases
         def _checkeval(msg, ret):
             "helper to check that evaluation of expressions is done correctly"
-            print x
+            print(msg)
             return ret
 
         self.assertEqual([ x() for x in lambda: True, lambda: False if x() ], [True])
@@ -1116,7 +1179,8 @@ def _checkeval(msg, ret):
         self.assertEqual((6 + 1 if 1 else 2), 7)
         self.assertEqual((6 - 1 if 1 else 2), 5)
         self.assertEqual((6 * 2 if 1 else 4), 12)
-        self.assertEqual((6 / 2 if 1 else 3), 3)
+        with check_py3k_warnings(('classic int division', DeprecationWarning)):
+            self.assertEqual((6 / 2 if 1 else 3), 3)
         self.assertEqual((6 < 4 if 0 else 2), 2)
 
     def test_paren_evaluation(self):
@@ -1129,13 +1193,7 @@ def test_paren_evaluation(self):
 
 
 def test_main():
-    with check_py3k_warnings(
-            ("backquote not supported", SyntaxWarning),
-            ("tuple parameter unpacking has been removed", SyntaxWarning),
-            ("parenthesized argument names are invalid", SyntaxWarning),
-            ("classic int division", DeprecationWarning),
-            (".+ not supported in 3.x", DeprecationWarning)):
-        run_unittest(TokenTests, GrammarTests)
+    run_unittest(TokenTests, GrammarTests)
 
 if __name__ == '__main__':
     test_main()
diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py
index 0a61462a608..103b4c11093 100644
--- a/Lib/test/test_syntax.py
+++ b/Lib/test/test_syntax.py
@@ -73,13 +73,12 @@
 
 >>> "abc" = 1
 Traceback (most recent call last):
-  File "<doctest test.test_syntax[8]>", line 1
+  File "<doctest test.test_syntax[9]>", line 1
 SyntaxError: can't assign to literal
 
->>> `1` = 1
+>>> b"" = 1
 Traceback (most recent call last):
-  File "<doctest test.test_syntax[10]>", line 1
-SyntaxError: can't assign to repr
+SyntaxError: can't assign to literal
 
 If the left-hand side of an assignment is a list or tuple, an illegal
 expression inside that contain should still cause a syntax error.
@@ -148,7 +147,7 @@
 
 From ast_for_call():
 
->>> def f(it, *varargs):
+>>> def f(it, *varargs, **kwargs):
 ...     return list(it)
 >>> L = range(10)
 >>> f(x for x in L)
@@ -157,73 +156,179 @@
 Traceback (most recent call last):
   File "<doctest test.test_syntax[23]>", line 1
 SyntaxError: Generator expression must be parenthesized if not sole argument
+>>> f(x for x in L, y=1)
+Traceback (most recent call last):
+SyntaxError: Generator expression must be parenthesized if not sole argument
+>>> f(L, x for x in L)
+Traceback (most recent call last):
+SyntaxError: Generator expression must be parenthesized if not sole argument
+>>> f(x for x in L, y for y in L)
+Traceback (most recent call last):
+SyntaxError: Generator expression must be parenthesized if not sole argument
+>>> f(x for x in L,)
+[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
 >>> f((x for x in L), 1)
 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
 
->>> f(i0,  i1,  i2,  i3,  i4,  i5,  i6,  i7,  i8,  i9,  i10,  i11,
-...   i12,  i13,  i14,  i15,  i16,  i17,  i18,  i19,  i20,  i21,  i22,
-...   i23,  i24,  i25,  i26,  i27,  i28,  i29,  i30,  i31,  i32,  i33,
-...   i34,  i35,  i36,  i37,  i38,  i39,  i40,  i41,  i42,  i43,  i44,
-...   i45,  i46,  i47,  i48,  i49,  i50,  i51,  i52,  i53,  i54,  i55,
-...   i56,  i57,  i58,  i59,  i60,  i61,  i62,  i63,  i64,  i65,  i66,
-...   i67,  i68,  i69,  i70,  i71,  i72,  i73,  i74,  i75,  i76,  i77,
-...   i78,  i79,  i80,  i81,  i82,  i83,  i84,  i85,  i86,  i87,  i88,
-...   i89,  i90,  i91,  i92,  i93,  i94,  i95,  i96,  i97,  i98,  i99,
-...   i100,  i101,  i102,  i103,  i104,  i105,  i106,  i107,  i108,
-...   i109,  i110,  i111,  i112,  i113,  i114,  i115,  i116,  i117,
-...   i118,  i119,  i120,  i121,  i122,  i123,  i124,  i125,  i126,
-...   i127,  i128,  i129,  i130,  i131,  i132,  i133,  i134,  i135,
-...   i136,  i137,  i138,  i139,  i140,  i141,  i142,  i143,  i144,
-...   i145,  i146,  i147,  i148,  i149,  i150,  i151,  i152,  i153,
-...   i154,  i155,  i156,  i157,  i158,  i159,  i160,  i161,  i162,
-...   i163,  i164,  i165,  i166,  i167,  i168,  i169,  i170,  i171,
-...   i172,  i173,  i174,  i175,  i176,  i177,  i178,  i179,  i180,
-...   i181,  i182,  i183,  i184,  i185,  i186,  i187,  i188,  i189,
-...   i190,  i191,  i192,  i193,  i194,  i195,  i196,  i197,  i198,
-...   i199,  i200,  i201,  i202,  i203,  i204,  i205,  i206,  i207,
-...   i208,  i209,  i210,  i211,  i212,  i213,  i214,  i215,  i216,
-...   i217,  i218,  i219,  i220,  i221,  i222,  i223,  i224,  i225,
-...   i226,  i227,  i228,  i229,  i230,  i231,  i232,  i233,  i234,
-...   i235,  i236,  i237,  i238,  i239,  i240,  i241,  i242,  i243,
-...   i244,  i245,  i246,  i247,  i248,  i249,  i250,  i251,  i252,
-...   i253,  i254,  i255)
+>>> def g(*args, **kwargs):
+...     print args, sorted(kwargs.items())
+>>> g(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
+...   20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
+...   38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
+...   56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
+...   74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
+...   92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
+...   108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121,
+...   122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135,
+...   136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149,
+...   150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163,
+...   164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
+...   178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
+...   192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205,
+...   206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
+...   220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
+...   234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
+...   248, 249, 250, 251, 252, 253, 254)  # doctest: +ELLIPSIS
+(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ..., 252, 253, 254) []
+>>> g(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
+...   20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
+...   38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
+...   56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
+...   74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
+...   92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
+...   108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121,
+...   122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135,
+...   136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149,
+...   150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163,
+...   164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
+...   178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
+...   192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205,
+...   206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
+...   220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
+...   234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
+...   248, 249, 250, 251, 252, 253, 254, 255)
 Traceback (most recent call last):
   File "<doctest test.test_syntax[25]>", line 1
 SyntaxError: more than 255 arguments
 
-The actual error cases counts positional arguments, keyword arguments,
-and generator expression arguments separately.  This test combines the
-three.
-
->>> f(i0,  i1,  i2,  i3,  i4,  i5,  i6,  i7,  i8,  i9,  i10,  i11,
-...   i12,  i13,  i14,  i15,  i16,  i17,  i18,  i19,  i20,  i21,  i22,
-...   i23,  i24,  i25,  i26,  i27,  i28,  i29,  i30,  i31,  i32,  i33,
-...   i34,  i35,  i36,  i37,  i38,  i39,  i40,  i41,  i42,  i43,  i44,
-...   i45,  i46,  i47,  i48,  i49,  i50,  i51,  i52,  i53,  i54,  i55,
-...   i56,  i57,  i58,  i59,  i60,  i61,  i62,  i63,  i64,  i65,  i66,
-...   i67,  i68,  i69,  i70,  i71,  i72,  i73,  i74,  i75,  i76,  i77,
-...   i78,  i79,  i80,  i81,  i82,  i83,  i84,  i85,  i86,  i87,  i88,
-...   i89,  i90,  i91,  i92,  i93,  i94,  i95,  i96,  i97,  i98,  i99,
-...   i100,  i101,  i102,  i103,  i104,  i105,  i106,  i107,  i108,
-...   i109,  i110,  i111,  i112,  i113,  i114,  i115,  i116,  i117,
-...   i118,  i119,  i120,  i121,  i122,  i123,  i124,  i125,  i126,
-...   i127,  i128,  i129,  i130,  i131,  i132,  i133,  i134,  i135,
-...   i136,  i137,  i138,  i139,  i140,  i141,  i142,  i143,  i144,
-...   i145,  i146,  i147,  i148,  i149,  i150,  i151,  i152,  i153,
-...   i154,  i155,  i156,  i157,  i158,  i159,  i160,  i161,  i162,
-...   i163,  i164,  i165,  i166,  i167,  i168,  i169,  i170,  i171,
-...   i172,  i173,  i174,  i175,  i176,  i177,  i178,  i179,  i180,
-...   i181,  i182,  i183,  i184,  i185,  i186,  i187,  i188,  i189,
-...   i190,  i191,  i192,  i193,  i194,  i195,  i196,  i197,  i198,
-...   i199,  i200,  i201,  i202,  i203,  i204,  i205,  i206,  i207,
-...   i208,  i209,  i210,  i211,  i212,  i213,  i214,  i215,  i216,
-...   i217,  i218,  i219,  i220,  i221,  i222,  i223,  i224,  i225,
-...   i226,  i227,  i228,  i229,  i230,  i231,  i232,  i233,  i234,
-...   i235, i236,  i237,  i238,  i239,  i240,  i241,  i242,  i243,
-...   (x for x in i244),  i245,  i246,  i247,  i248,  i249,  i250,  i251,
-...    i252=1, i253=1,  i254=1,  i255=1)
-Traceback (most recent call last):
-  File "<doctest test.test_syntax[26]>", line 1
+>>> g(a000=0, a001=1, a002=2, a003=3, a004=4, a005=5, a006=6, a007=7, a008=8,
+...   a009=9, a010=10, a011=11, a012=12, a013=13, a014=14, a015=15, a016=16,
+...   a017=17, a018=18, a019=19, a020=20, a021=21, a022=22, a023=23, a024=24,
+...   a025=25, a026=26, a027=27, a028=28, a029=29, a030=30, a031=31, a032=32,
+...   a033=33, a034=34, a035=35, a036=36, a037=37, a038=38, a039=39, a040=40,
+...   a041=41, a042=42, a043=43, a044=44, a045=45, a046=46, a047=47, a048=48,
+...   a049=49, a050=50, a051=51, a052=52, a053=53, a054=54, a055=55, a056=56,
+...   a057=57, a058=58, a059=59, a060=60, a061=61, a062=62, a063=63, a064=64,
+...   a065=65, a066=66, a067=67, a068=68, a069=69, a070=70, a071=71, a072=72,
+...   a073=73, a074=74, a075=75, a076=76, a077=77, a078=78, a079=79, a080=80,
+...   a081=81, a082=82, a083=83, a084=84, a085=85, a086=86, a087=87, a088=88,
+...   a089=89, a090=90, a091=91, a092=92, a093=93, a094=94, a095=95, a096=96,
+...   a097=97, a098=98, a099=99, a100=100, a101=101, a102=102, a103=103,
+...   a104=104, a105=105, a106=106, a107=107, a108=108, a109=109, a110=110,
+...   a111=111, a112=112, a113=113, a114=114, a115=115, a116=116, a117=117,
+...   a118=118, a119=119, a120=120, a121=121, a122=122, a123=123, a124=124,
+...   a125=125, a126=126, a127=127, a128=128, a129=129, a130=130, a131=131,
+...   a132=132, a133=133, a134=134, a135=135, a136=136, a137=137, a138=138,
+...   a139=139, a140=140, a141=141, a142=142, a143=143, a144=144, a145=145,
+...   a146=146, a147=147, a148=148, a149=149, a150=150, a151=151, a152=152,
+...   a153=153, a154=154, a155=155, a156=156, a157=157, a158=158, a159=159,
+...   a160=160, a161=161, a162=162, a163=163, a164=164, a165=165, a166=166,
+...   a167=167, a168=168, a169=169, a170=170, a171=171, a172=172, a173=173,
+...   a174=174, a175=175, a176=176, a177=177, a178=178, a179=179, a180=180,
+...   a181=181, a182=182, a183=183, a184=184, a185=185, a186=186, a187=187,
+...   a188=188, a189=189, a190=190, a191=191, a192=192, a193=193, a194=194,
+...   a195=195, a196=196, a197=197, a198=198, a199=199, a200=200, a201=201,
+...   a202=202, a203=203, a204=204, a205=205, a206=206, a207=207, a208=208,
+...   a209=209, a210=210, a211=211, a212=212, a213=213, a214=214, a215=215,
+...   a216=216, a217=217, a218=218, a219=219, a220=220, a221=221, a222=222,
+...   a223=223, a224=224, a225=225, a226=226, a227=227, a228=228, a229=229,
+...   a230=230, a231=231, a232=232, a233=233, a234=234, a235=235, a236=236,
+...   a237=237, a238=238, a239=239, a240=240, a241=241, a242=242, a243=243,
+...   a244=244, a245=245, a246=246, a247=247, a248=248, a249=249, a250=250,
+...   a251=251, a252=252, a253=253, a254=254)  # doctest: +ELLIPSIS
+() [('a000', 0), ('a001', 1), ('a002', 2), ..., ('a253', 253), ('a254', 254)]
+>>> g(a000=0, a001=1, a002=2, a003=3, a004=4, a005=5, a006=6, a007=7, a008=8,
+...   a009=9, a010=10, a011=11, a012=12, a013=13, a014=14, a015=15, a016=16,
+...   a017=17, a018=18, a019=19, a020=20, a021=21, a022=22, a023=23, a024=24,
+...   a025=25, a026=26, a027=27, a028=28, a029=29, a030=30, a031=31, a032=32,
+...   a033=33, a034=34, a035=35, a036=36, a037=37, a038=38, a039=39, a040=40,
+...   a041=41, a042=42, a043=43, a044=44, a045=45, a046=46, a047=47, a048=48,
+...   a049=49, a050=50, a051=51, a052=52, a053=53, a054=54, a055=55, a056=56,
+...   a057=57, a058=58, a059=59, a060=60, a061=61, a062=62, a063=63, a064=64,
+...   a065=65, a066=66, a067=67, a068=68, a069=69, a070=70, a071=71, a072=72,
+...   a073=73, a074=74, a075=75, a076=76, a077=77, a078=78, a079=79, a080=80,
+...   a081=81, a082=82, a083=83, a084=84, a085=85, a086=86, a087=87, a088=88,
+...   a089=89, a090=90, a091=91, a092=92, a093=93, a094=94, a095=95, a096=96,
+...   a097=97, a098=98, a099=99, a100=100, a101=101, a102=102, a103=103,
+...   a104=104, a105=105, a106=106, a107=107, a108=108, a109=109, a110=110,
+...   a111=111, a112=112, a113=113, a114=114, a115=115, a116=116, a117=117,
+...   a118=118, a119=119, a120=120, a121=121, a122=122, a123=123, a124=124,
+...   a125=125, a126=126, a127=127, a128=128, a129=129, a130=130, a131=131,
+...   a132=132, a133=133, a134=134, a135=135, a136=136, a137=137, a138=138,
+...   a139=139, a140=140, a141=141, a142=142, a143=143, a144=144, a145=145,
+...   a146=146, a147=147, a148=148, a149=149, a150=150, a151=151, a152=152,
+...   a153=153, a154=154, a155=155, a156=156, a157=157, a158=158, a159=159,
+...   a160=160, a161=161, a162=162, a163=163, a164=164, a165=165, a166=166,
+...   a167=167, a168=168, a169=169, a170=170, a171=171, a172=172, a173=173,
+...   a174=174, a175=175, a176=176, a177=177, a178=178, a179=179, a180=180,
+...   a181=181, a182=182, a183=183, a184=184, a185=185, a186=186, a187=187,
+...   a188=188, a189=189, a190=190, a191=191, a192=192, a193=193, a194=194,
+...   a195=195, a196=196, a197=197, a198=198, a199=199, a200=200, a201=201,
+...   a202=202, a203=203, a204=204, a205=205, a206=206, a207=207, a208=208,
+...   a209=209, a210=210, a211=211, a212=212, a213=213, a214=214, a215=215,
+...   a216=216, a217=217, a218=218, a219=219, a220=220, a221=221, a222=222,
+...   a223=223, a224=224, a225=225, a226=226, a227=227, a228=228, a229=229,
+...   a230=230, a231=231, a232=232, a233=233, a234=234, a235=235, a236=236,
+...   a237=237, a238=238, a239=239, a240=240, a241=241, a242=242, a243=243,
+...   a244=244, a245=245, a246=246, a247=247, a248=248, a249=249, a250=250,
+...   a251=251, a252=252, a253=253, a254=254, a255=255)
+Traceback (most recent call last):
+  File "<doctest test.test_syntax[35]>", line 1
+SyntaxError: more than 255 arguments
+
+>>> class C:
+...     def meth(self, *args):
+...         return args
+>>> obj = C()
+>>> obj.meth(
+...   0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
+...   20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
+...   38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
+...   56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
+...   74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
+...   92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
+...   108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121,
+...   122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135,
+...   136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149,
+...   150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163,
+...   164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
+...   178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
+...   192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205,
+...   206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
+...   220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
+...   234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
+...   248, 249, 250, 251, 252, 253, 254)  # doctest: +ELLIPSIS
+(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ..., 252, 253, 254)
+>>> obj.meth(
+...   0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
+...   20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
+...   38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
+...   56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
+...   74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
+...   92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
+...   108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121,
+...   122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135,
+...   136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149,
+...   150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163,
+...   164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
+...   178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
+...   192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205,
+...   206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
+...   220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
+...   234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
+...   248, 249, 250, 251, 252, 253, 254, 255)
+Traceback (most recent call last):
+  File "<doctest test.test_syntax[38]>", line 1
 SyntaxError: more than 255 arguments
 
 >>> f(lambda x: x[0] = 3)
@@ -403,6 +508,16 @@
      ...
    SyntaxError: too many statically nested blocks
 
+Misuse of the global statement can lead to a few unique syntax errors.
+
+   >>> def f(x):
+   ...     global x
+   ... # doctest: +ELLIPSIS
+   Traceback (most recent call last):
+     ...
+   SyntaxError: name 'x' is local and global (<doctest ...>, line 1)
+
+
 This tests assignment-context; there was a bug in Python 2.5 where compiling
 a complex 'if' (one with 'elif') would fail to notice an invalid suite,
 leading to spurious errors.
@@ -458,9 +573,21 @@
      File "<doctest test.test_syntax[48]>", line 6
    SyntaxError: can't assign to function call
 
+Test the "raise X, Y[, Z]" form:
+
+   >>> raise ArithmeticError, 'bad number'
+   Traceback (most recent call last):
+     ...
+   ArithmeticError: bad number
+   >>> raise ArithmeticError, 'bad number', None
+   Traceback (most recent call last):
+     ...
+   ArithmeticError: bad number
+
+
 >>> f(a=23, a=234)
 Traceback (most recent call last):
-   ...
+  ...
   File "<doctest test.test_syntax[49]>", line 1
 SyntaxError: keyword argument repeated
 
@@ -476,6 +603,12 @@
    File "<doctest test.test_syntax[50]>", line 1
 SyntaxError: can't assign to literal
 
+Corner-case that used to fail to raise the correct error:
+
+    >>> def f(x=lambda __debug__:0): pass
+    Traceback (most recent call last):
+    SyntaxError: cannot assign to __debug__
+
 Corner-case that used to crash:
 
     >>> def f(*xx, **__debug__): pass
@@ -488,12 +621,12 @@
 import unittest
 import warnings
 
-from test import test_support
+from test import support
 
 class SyntaxTestCase(unittest.TestCase):
 
     def _check_error(self, code, errtext,
-                     filename="<testcase>", mode="exec", subclass=None):
+                     filename="<testcase>", mode="exec", subclass=None, lineno=None, offset=None):
         """Check that compiling code raises SyntaxError with errtext.
 
         errtest is a regular expression that must be present in the
@@ -501,13 +634,19 @@ def _check_error(self, code, errtext,
         is the expected subclass of SyntaxError (e.g. IndentationError).
         """
         try:
-            compile(code, filename, mode)
-        except SyntaxError, err:
+            compile(code, filename or "<testcase>", mode)
+        except SyntaxError as err:
             if subclass and not isinstance(err, subclass):
                 self.fail("SyntaxError is not a %s" % subclass.__name__)
             mo = re.search(errtext, str(err))
             if mo is None:
                 self.fail("%s did not contain '%r'" % (err, errtext,))
+            if filename is not None:
+                self.assertEqual(err.filename, filename)
+            if lineno is not None:
+                self.assertEqual(err.lineno, lineno)
+            if offset is not None:
+                self.assertEqual(err.offset, offset)
         else:
             self.fail("compile() did not raise SyntaxError")
 
@@ -515,6 +654,11 @@ def test_paren_arg_with_default(self):
         self._check_error("def f((x)=23): pass",
                           "parenthesized arg with default")
 
+    def test_assign_repr(self):
+        with support.check_py3k_warnings(('backquote not supported',
+                                          SyntaxWarning)):
+            self._check_error("`1` = 1", "assign to repr")
+
     def test_assign_call(self):
         self._check_error("f() = 1", "assign")
 
@@ -524,28 +668,48 @@ def test_assign_del(self):
     def test_global_err_then_warn(self):
         # Bug tickler:  The SyntaxError raised for one global statement
         # shouldn't be clobbered by a SyntaxWarning issued for a later one.
-        source = re.sub('(?m)^ *:', '', """\
-            :def error(a):
-            :    global a  # SyntaxError
-            :def warning():
-            :    b = 1
-            :    global b  # SyntaxWarning
-            :""")
-        warnings.filterwarnings(action='ignore', category=SyntaxWarning)
-        self._check_error(source, "global")
-        warnings.filters.pop(0)
+        source = """if 1:
+            def error(a):
+                global a  # SyntaxError
+            def warning():
+                b = 1
+                global b  # SyntaxWarning
+            """
+        with support.check_warnings((".*assigned to before global declaration",
+                                     SyntaxWarning)):
+            self._check_error(source, "local and global", lineno=2)
+
+    def test_misuse_global(self):
+        source = """if 1:
+            def f():
+                print(x)
+                global x
+            """
+        with support.check_warnings(('.*used prior to global declaration',
+                                     SyntaxWarning)):
+            compile(source, '<testcase>', 'exec')
+
+    def test_misuse_global_2(self):
+        source = """if 1:
+            def f():
+                x = 1
+                global x
+            """
+        with support.check_warnings(('.*assigned to before global declaration',
+                                     SyntaxWarning)):
+            compile(source, '<testcase>', 'exec')
 
     def test_break_outside_loop(self):
         self._check_error("break", "outside loop")
 
     def test_delete_deref(self):
-        source = re.sub('(?m)^ *:', '', """\
-            :def foo(x):
-            :  def bar():
-            :    print x
-            :  del x
-            :""")
-        self._check_error(source, "nested scope")
+        source = """if 1:
+            def foo(x):
+                def bar():
+                    print(x)
+                del x
+            """
+        self._check_error(source, "nested scope", filename=None)
 
     def test_unexpected_indent(self):
         self._check_error("foo()\n bar()\n", "unexpected indent",
@@ -564,11 +728,9 @@ def test_kwargs_last(self):
         self._check_error("int(base=10, '2')", "non-keyword arg")
 
 def test_main():
-    test_support.run_unittest(SyntaxTestCase)
+    support.run_unittest(SyntaxTestCase)
     from test import test_syntax
-    with test_support.check_py3k_warnings(("backquote not supported",
-                                             SyntaxWarning)):
-        test_support.run_doctest(test_syntax, verbosity=True)
+    support.run_doctest(test_syntax, verbosity=True)
 
 if __name__ == "__main__":
     test_main()
diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py
index 02524588252..7e1d7a131f5 100644
--- a/Lib/test/test_sys_settrace.py
+++ b/Lib/test/test_sys_settrace.py
@@ -619,7 +619,7 @@ def test_jump_in_nested_finally(output):
     def test_jump_in_nested_finally_2(output):
         try:
             output.append(2)
-            1/0
+            1.0/0.0
             return
         finally:
             output.append(6)
@@ -630,7 +630,7 @@ def test_jump_in_nested_finally_2(output):
     def test_jump_in_nested_finally_3(output):
         try:
             output.append(2)
-            1/0
+            1.0/0.0
             return
         finally:
             output.append(6)
@@ -697,7 +697,7 @@ def test_jump_backwards_out_of_try_except_block(output):
     @jump_test(5, 7, [4, 7, 8])
     def test_jump_between_except_blocks(output):
         try:
-            1/0
+            1.0/0.0
         except ZeroDivisionError:
             output.append(4)
             output.append(5)
@@ -708,7 +708,7 @@ def test_jump_between_except_blocks(output):
     @jump_test(5, 6, [4, 6, 7])
     def test_jump_within_except_block(output):
         try:
-            1/0
+            1.0/0.0
         except:
             output.append(4)
             output.append(5)


From webhook-mailer at python.org  Thu Jan  4 03:36:38 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 08:36:38 -0000
Subject: [Python-checkins] bpo-32482: Fix suspicious code in tests for
 syntax and grammar. (#5086)
Message-ID: <mailman.120.1515055000.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/0cc99c8cd70d422e4b345837a907db30e9180ab9
commit: 0cc99c8cd70d422e4b345837a907db30e9180ab9
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T10:36:35+02:00
summary:

bpo-32482: Fix suspicious code in tests for syntax and grammar. (#5086)

files:
M Lib/test/test_grammar.py
M Lib/test/test_syntax.py

diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index 9f26e9c4fb0..88c22b89d44 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -575,6 +575,10 @@ def f(*args, **kwargs):
         self.assertEqual(f(spam='fried', **{'eggs':'scrambled'}),
                          ((), {'eggs':'scrambled', 'spam':'fried'}))
 
+        # Check ast errors in *args and *kwargs
+        check_syntax_error(self, "f(*g(1=2))")
+        check_syntax_error(self, "f(**g(1=2))")
+
         # argument annotation tests
         def f(x) -> list: pass
         self.assertEqual(f.__annotations__, {'return': list})
@@ -616,10 +620,6 @@ def f(x=1): return closure
         def f(*, k=1): return closure
         def f() -> int: return closure
 
-        # Check ast errors in *args and *kwargs
-        check_syntax_error(self, "f(*g(1=2))")
-        check_syntax_error(self, "f(**g(1=2))")
-
         # Check trailing commas are permitted in funcdef argument list
         def f(a,): pass
         def f(*args,): pass
@@ -1091,7 +1091,6 @@ def test_try(self):
         try: 1/0
         except EOFError: pass
         except TypeError as msg: pass
-        except RuntimeError as msg: pass
         except: pass
         else: pass
         try: 1/0
@@ -1200,7 +1199,7 @@ def test_selectors(self):
         d[1,2] = 3
         d[1,2,3] = 4
         L = list(d)
-        L.sort(key=lambda x: x if isinstance(x, tuple) else ())
+        L.sort(key=lambda x: (type(x).__name__, x))
         self.assertEqual(str(L), '[1, (1,), (1, 2), (1, 2, 3)]')
 
     def test_atoms(self):
diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py
index e161f56d30f..2b96a94401a 100644
--- a/Lib/test/test_syntax.py
+++ b/Lib/test/test_syntax.py
@@ -668,12 +668,12 @@ def test_kwargs_last(self):
                           "positional argument follows keyword argument")
 
     def test_kwargs_last2(self):
-        self._check_error("int(**{base: 10}, '2')",
+        self._check_error("int(**{'base': 10}, '2')",
                           "positional argument follows "
                           "keyword argument unpacking")
 
     def test_kwargs_last3(self):
-        self._check_error("int(**{base: 10}, *['2'])",
+        self._check_error("int(**{'base': 10}, *['2'])",
                           "iterable argument unpacking follows "
                           "keyword argument unpacking")
 


From webhook-mailer at python.org  Thu Jan  4 04:06:16 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 09:06:16 -0000
Subject: [Python-checkins] bpo-32308: Replace empty matches adjacent to a
 previous non-empty match in re.sub(). (#4846)
Message-ID: <mailman.121.1515056777.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/fbb490fd2f38bd817d99c20c05121ad0168a38ee
commit: fbb490fd2f38bd817d99c20c05121ad0168a38ee
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T11:06:13+02:00
summary:

bpo-32308: Replace empty matches adjacent to a previous non-empty match in re.sub(). (#4846)

files:
A Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst
M Doc/howto/regex.rst
M Doc/library/re.rst
M Doc/whatsnew/3.7.rst
M Lib/test/test_re.py
M Modules/_sre.c

diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst
index fa8c6939408..87a6b1aba59 100644
--- a/Doc/howto/regex.rst
+++ b/Doc/howto/regex.rst
@@ -1140,12 +1140,12 @@ new string value and the number of replacements  that were performed::
    >>> p.subn('colour', 'no colours at all')
    ('no colours at all', 0)
 
-Empty matches are replaced only when they're not adjacent to a previous match.
+Empty matches are replaced only when they're not adjacent to a previous empty match.
 ::
 
    >>> p = re.compile('x*')
    >>> p.sub('-', 'abxd')
-   '-a-b-d-'
+   '-a-b--d-'
 
 If *replacement* is a string, any backslash escapes in it are processed.  That
 is, ``\n`` is converted to a single newline character, ``\r`` is converted to a
diff --git a/Doc/library/re.rst b/Doc/library/re.rst
index dae1d7ea10a..9b175f4e967 100644
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -708,12 +708,15 @@ form.
    That way, separator components are always found at the same relative
    indices within the result list.
 
-   The pattern can match empty strings. ::
+   Empty matches for the pattern split the string only when not adjacent
+   to a previous empty match.
 
       >>> re.split(r'\b', 'Words, words, words.')
       ['', 'Words', ', ', 'words', ', ', 'words', '.']
+      >>> re.split(r'\W*', '...words...')
+      ['', '', 'w', 'o', 'r', 'd', 's', '', '']
       >>> re.split(r'(\W*)', '...words...')
-      ['', '...', 'w', '', 'o', '', 'r', '', 'd', '', 's', '...', '']
+      ['', '...', '', '', 'w', '', 'o', '', 'r', '', 'd', '', 's', '...', '', '', '']
 
    .. versionchanged:: 3.1
       Added the optional flags argument.
@@ -778,8 +781,8 @@ form.
    The optional argument *count* is the maximum number of pattern occurrences to be
    replaced; *count* must be a non-negative integer.  If omitted or zero, all
    occurrences will be replaced. Empty matches for the pattern are replaced only
-   when not adjacent to a previous match, so ``sub('x*', '-', 'abc')`` returns
-   ``'-a-b-c-'``.
+   when not adjacent to a previous empty match, so ``sub('x*', '-', 'abxd')`` returns
+   ``'-a-b--d-'``.
 
    In string-type *repl* arguments, in addition to the character escapes and
    backreferences described above,
@@ -805,6 +808,9 @@ form.
       Unknown escapes in *repl* consisting of ``'\'`` and an ASCII letter
       now are errors.
 
+      Empty matches for the pattern are replaced when adjacent to a previous
+      non-empty match.
+
 
 .. function:: subn(pattern, repl, string, count=0, flags=0)
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 1924881219a..1311e9e2016 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -881,8 +881,9 @@ Changes in the Python API
 * The result of splitting a string on a :mod:`regular expression <re>`
   that could match an empty string has been changed.  For example
   splitting on ``r'\s*'`` will now split not only on whitespaces as it
-  did previously, but also between any pair of non-whitespace
-  characters.  The previous behavior can be restored by changing the pattern
+  did previously, but also on empty strings before all non-whitespace
+  characters and just before the end of the string.
+  The previous behavior can be restored by changing the pattern
   to ``r'\s+'``.  A :exc:`FutureWarning` was emitted for such patterns since
   Python 3.5.
 
@@ -893,7 +894,13 @@ Changes in the Python API
   positions 2--3.  To match only blank lines, the pattern should be rewritten
   as ``r'(?m)^[^\S\n]*$'``.
 
-  (Contributed by Serhiy Storchaka in :issue:`25054`.)
+  :func:`re.sub()` now replaces empty matches adjacent to a previous
+  non-empty match.  For example ``re.sub('x*', '-', 'abxd')`` returns now
+  ``'-a-b--d-'`` instead of ``'-a-b--d-'`` (the first minus between 'b' and
+  'd' replaces 'x', and the second minus replaces an empty string between
+  'x' and 'd').
+
+  (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.)
 
 * :class:`tracemalloc.Traceback` frames are now sorted from oldest to most
   recent to be more consistent with :mod:`traceback`.
diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
index aaed3d893aa..9fed4bef880 100644
--- a/Lib/test/test_re.py
+++ b/Lib/test/test_re.py
@@ -213,11 +213,6 @@ def test_bug_114660(self):
         self.assertEqual(re.sub(r'(\S)\s+(\S)', r'\1 \2', 'hello  there'),
                          'hello there')
 
-    def test_bug_462270(self):
-        # Test for empty sub() behaviour, see SF bug #462270
-        self.assertEqual(re.sub('x*', '-', 'abxd'), '-a-b-d-')
-        self.assertEqual(re.sub('x+', '-', 'abxd'), 'ab-d')
-
     def test_symbolic_groups(self):
         re.compile(r'(?P<a>x)(?P=a)(?(a)y)')
         re.compile(r'(?P<a1>x)(?P=a1)(?(a1)y)')
@@ -331,10 +326,10 @@ def test_re_split(self):
                          ['', 'a', '', '', 'c'])
 
         for sep, expected in [
-            (':*', ['', 'a', 'b', 'c', '']),
-            ('(?::*)', ['', 'a', 'b', 'c', '']),
-            ('(:*)', ['', ':', 'a', ':', 'b', '::', 'c', '', '']),
-            ('(:)*', ['', ':', 'a', ':', 'b', ':', 'c', None, '']),
+            (':*', ['', '', 'a', '', 'b', '', 'c', '']),
+            ('(?::*)', ['', '', 'a', '', 'b', '', 'c', '']),
+            ('(:*)', ['', ':', '', '', 'a', ':', '', '', 'b', '::', '', '', 'c', '', '']),
+            ('(:)*', ['', ':', '', None, 'a', ':', '', None, 'b', ':', '', None, 'c', None, '']),
         ]:
             with self.subTest(sep=sep):
                 self.assertTypedEqual(re.split(sep, ':a:b::c'), expected)
@@ -357,7 +352,7 @@ def test_qualified_re_split(self):
         self.assertEqual(re.split("(:+)", ":a:b::c", maxsplit=2),
                          ['', ':', 'a', ':', 'b::c'])
         self.assertEqual(re.split("(:*)", ":a:b::c", maxsplit=2),
-                         ['', ':', 'a', ':', 'b::c'])
+                         ['', ':', '', '', 'a:b::c'])
 
     def test_re_findall(self):
         self.assertEqual(re.findall(":+", "abc"), [])
@@ -1753,13 +1748,13 @@ def test_match_repr(self):
     def test_zerowidth(self):
         # Issues 852532, 1647489, 3262, 25054.
         self.assertEqual(re.split(r"\b", "a::bc"), ['', 'a', '::', 'bc', ''])
-        self.assertEqual(re.split(r"\b|:+", "a::bc"), ['', 'a', '', 'bc', ''])
-        self.assertEqual(re.split(r"(?<!\w)(?=\w)|:+", "a::bc"), ['', 'a', 'bc'])
+        self.assertEqual(re.split(r"\b|:+", "a::bc"), ['', 'a', '', '', 'bc', ''])
+        self.assertEqual(re.split(r"(?<!\w)(?=\w)|:+", "a::bc"), ['', 'a', '', 'bc'])
         self.assertEqual(re.split(r"(?<=\w)(?!\w)|:+", "a::bc"), ['a', '', 'bc', ''])
 
         self.assertEqual(re.sub(r"\b", "-", "a::bc"), '-a-::-bc-')
-        self.assertEqual(re.sub(r"\b|:+", "-", "a::bc"), '-a--bc-')
-        self.assertEqual(re.sub(r"(\b|:+)", r"[\1]", "a::bc"), '[]a[][::]bc[]')
+        self.assertEqual(re.sub(r"\b|:+", "-", "a::bc"), '-a---bc-')
+        self.assertEqual(re.sub(r"(\b|:+)", r"[\1]", "a::bc"), '[]a[][::][]bc[]')
 
         self.assertEqual(re.findall(r"\b|:+", "a::bc"), ['', '', '::', '', ''])
         self.assertEqual(re.findall(r"\b|\w+", "a::bc"),
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst b/Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst
new file mode 100644
index 00000000000..d760d6089ba
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst
@@ -0,0 +1,2 @@
+:func:`re.sub()` now replaces empty matches adjacent to a previous non-empty
+match.
diff --git a/Modules/_sre.c b/Modules/_sre.c
index 68fc523c251..b6be6f6ffa6 100644
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -955,7 +955,7 @@ _sre_SRE_Pattern_split_impl(PatternObject *self, PyObject *string,
         }
 
         n = n + 1;
-        state.must_advance = 1;
+        state.must_advance = (state.ptr == state.start);
         last = state.start = state.ptr;
 
     }
@@ -1109,7 +1109,7 @@ pattern_subx(PatternObject* self, PyObject* ptemplate, PyObject* string,
 
         i = e;
         n = n + 1;
-        state.must_advance = 1;
+        state.must_advance = (state.ptr == state.start);
         state.start = state.ptr;
     }
 


From webhook-mailer at python.org  Thu Jan  4 04:08:27 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 09:08:27 -0000
Subject: [Python-checkins] bpo-32211: Document the existing bug in
 re.findall() and re.finditer(). (#4695)
Message-ID: <mailman.122.1515056909.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/1e6d8525f9dd3dcdc83adb93b164082c8b95d17a
commit: 1e6d8525f9dd3dcdc83adb93b164082c8b95d17a
branch: 3.6
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T11:08:24+02:00
summary:

bpo-32211: Document the existing bug in re.findall() and re.finditer(). (#4695)

files:
M Doc/library/re.rst

diff --git a/Doc/library/re.rst b/Doc/library/re.rst
index fae8945f8b8..874c8ddce67 100644
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -719,14 +719,21 @@ form.
       Splitting on a pattern that could match an empty string now raises
       a warning.  Patterns that can only match empty strings are now rejected.
 
+
 .. function:: findall(pattern, string, flags=0)
 
    Return all non-overlapping matches of *pattern* in *string*, as a list of
    strings.  The *string* is scanned left-to-right, and matches are returned in
    the order found.  If one or more groups are present in the pattern, return a
    list of groups; this will be a list of tuples if the pattern has more than
-   one group.  Empty matches are included in the result unless they touch the
-   beginning of another match.
+   one group.  Empty matches are included in the result.
+
+   .. note::
+
+      Due to the limitation of the current implementation the character
+      following an empty match is not included in a next match, so
+      ``findall(r'^|\w+', 'two words')`` returns ``['', 'wo', 'words']``
+      (note missed "t").  This is changed in Python 3.7.
 
 
 .. function:: finditer(pattern, string, flags=0)
@@ -734,8 +741,7 @@ form.
    Return an :term:`iterator` yielding :ref:`match objects <match-objects>` over
    all non-overlapping matches for the RE *pattern* in *string*.  The *string*
    is scanned left-to-right, and matches are returned in the order found.  Empty
-   matches are included in the result unless they touch the beginning of another
-   match.
+   matches are included in the result.  See also the note about :func:`findall`.
 
 
 .. function:: sub(pattern, repl, string, count=0, flags=0)


From solipsis at pitrou.net  Thu Jan  4 04:08:56 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 04 Jan 2018 09:08:56 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=10
Message-ID: <20180104090856.1.A18E564691E79EAD@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, 7] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [-2, 2, -1] memory blocks, sum=-1


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogg7TW73', '--timeout', '7200']

From webhook-mailer at python.org  Thu Jan  4 04:15:47 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 09:15:47 -0000
Subject: [Python-checkins] bpo-31778: Make ast.literal_eval() more strict.
 (#4035)
Message-ID: <mailman.123.1515057348.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/d8ac4d1d5ac256ebf3d8d38c226049abec82a2a0
commit: d8ac4d1d5ac256ebf3d8d38c226049abec82a2a0
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T11:15:39+02:00
summary:

bpo-31778: Make ast.literal_eval() more strict. (#4035)

Addition and subtraction of arbitrary numbers no longer allowed.

files:
A Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst
M Lib/ast.py
M Lib/test/test_ast.py
M Lib/test/test_inspect.py

diff --git a/Lib/ast.py b/Lib/ast.py
index 070c2bee7f9..2ecb03f38bc 100644
--- a/Lib/ast.py
+++ b/Lib/ast.py
@@ -35,8 +35,6 @@ def parse(source, filename='<unknown>', mode='exec'):
     return compile(source, filename, mode, PyCF_ONLY_AST)
 
 
-_NUM_TYPES = (int, float, complex)
-
 def literal_eval(node_or_string):
     """
     Safely evaluate an expression node or a string containing a Python
@@ -48,6 +46,21 @@ def literal_eval(node_or_string):
         node_or_string = parse(node_or_string, mode='eval')
     if isinstance(node_or_string, Expression):
         node_or_string = node_or_string.body
+    def _convert_num(node):
+        if isinstance(node, Constant):
+            if isinstance(node.value, (int, float, complex)):
+                return node.value
+        elif isinstance(node, Num):
+            return node.n
+        raise ValueError('malformed node or string: ' + repr(node))
+    def _convert_signed_num(node):
+        if isinstance(node, UnaryOp) and isinstance(node.op, (UAdd, USub)):
+            operand = _convert_num(node.operand)
+            if isinstance(node.op, UAdd):
+                return + operand
+            else:
+                return - operand
+        return _convert_num(node)
     def _convert(node):
         if isinstance(node, Constant):
             return node.value
@@ -62,26 +75,19 @@ def _convert(node):
         elif isinstance(node, Set):
             return set(map(_convert, node.elts))
         elif isinstance(node, Dict):
-            return dict((_convert(k), _convert(v)) for k, v
-                        in zip(node.keys, node.values))
+            return dict(zip(map(_convert, node.keys),
+                            map(_convert, node.values)))
         elif isinstance(node, NameConstant):
             return node.value
-        elif isinstance(node, UnaryOp) and isinstance(node.op, (UAdd, USub)):
-            operand = _convert(node.operand)
-            if isinstance(operand, _NUM_TYPES):
-                if isinstance(node.op, UAdd):
-                    return + operand
-                else:
-                    return - operand
         elif isinstance(node, BinOp) and isinstance(node.op, (Add, Sub)):
-            left = _convert(node.left)
-            right = _convert(node.right)
-            if isinstance(left, _NUM_TYPES) and isinstance(right, _NUM_TYPES):
+            left = _convert_signed_num(node.left)
+            right = _convert_num(node.right)
+            if isinstance(left, (int, float)) and isinstance(right, complex):
                 if isinstance(node.op, Add):
                     return left + right
                 else:
                     return left - right
-        raise ValueError('malformed node or string: ' + repr(node))
+        return _convert_signed_num(node)
     return _convert(node_or_string)
 
 
diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
index aa53503e3b5..67f363ad31f 100644
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -551,14 +551,37 @@ def test_literal_eval(self):
         self.assertEqual(ast.literal_eval('{1, 2, 3}'), {1, 2, 3})
         self.assertEqual(ast.literal_eval('b"hi"'), b"hi")
         self.assertRaises(ValueError, ast.literal_eval, 'foo()')
+        self.assertEqual(ast.literal_eval('6'), 6)
+        self.assertEqual(ast.literal_eval('+6'), 6)
         self.assertEqual(ast.literal_eval('-6'), -6)
-        self.assertEqual(ast.literal_eval('-6j+3'), 3-6j)
         self.assertEqual(ast.literal_eval('3.25'), 3.25)
-
-    def test_literal_eval_issue4907(self):
-        self.assertEqual(ast.literal_eval('2j'), 2j)
-        self.assertEqual(ast.literal_eval('10 + 2j'), 10 + 2j)
-        self.assertEqual(ast.literal_eval('1.5 - 2j'), 1.5 - 2j)
+        self.assertEqual(ast.literal_eval('+3.25'), 3.25)
+        self.assertEqual(ast.literal_eval('-3.25'), -3.25)
+        self.assertEqual(repr(ast.literal_eval('-0.0')), '-0.0')
+        self.assertRaises(ValueError, ast.literal_eval, '++6')
+        self.assertRaises(ValueError, ast.literal_eval, '+True')
+        self.assertRaises(ValueError, ast.literal_eval, '2+3')
+
+    def test_literal_eval_complex(self):
+        # Issue #4907
+        self.assertEqual(ast.literal_eval('6j'), 6j)
+        self.assertEqual(ast.literal_eval('-6j'), -6j)
+        self.assertEqual(ast.literal_eval('6.75j'), 6.75j)
+        self.assertEqual(ast.literal_eval('-6.75j'), -6.75j)
+        self.assertEqual(ast.literal_eval('3+6j'), 3+6j)
+        self.assertEqual(ast.literal_eval('-3+6j'), -3+6j)
+        self.assertEqual(ast.literal_eval('3-6j'), 3-6j)
+        self.assertEqual(ast.literal_eval('-3-6j'), -3-6j)
+        self.assertEqual(ast.literal_eval('3.25+6.75j'), 3.25+6.75j)
+        self.assertEqual(ast.literal_eval('-3.25+6.75j'), -3.25+6.75j)
+        self.assertEqual(ast.literal_eval('3.25-6.75j'), 3.25-6.75j)
+        self.assertEqual(ast.literal_eval('-3.25-6.75j'), -3.25-6.75j)
+        self.assertEqual(ast.literal_eval('(3+6j)'), 3+6j)
+        self.assertRaises(ValueError, ast.literal_eval, '-6j+3')
+        self.assertRaises(ValueError, ast.literal_eval, '-6j+3j')
+        self.assertRaises(ValueError, ast.literal_eval, '3+-6j')
+        self.assertRaises(ValueError, ast.literal_eval, '3+(0+6j)')
+        self.assertRaises(ValueError, ast.literal_eval, '-(3+6j)')
 
     def test_bad_integer(self):
         # issue13436: Bad error message with invalid numeric values
@@ -1077,11 +1100,11 @@ def test_literal_eval(self):
         ast.copy_location(new_left, binop.left)
         binop.left = new_left
 
-        new_right = ast.Constant(value=20)
+        new_right = ast.Constant(value=20j)
         ast.copy_location(new_right, binop.right)
         binop.right = new_right
 
-        self.assertEqual(ast.literal_eval(binop), 30)
+        self.assertEqual(ast.literal_eval(binop), 10+20j)
 
 
 def main():
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
index e8eddbedf7e..cb51f8aff29 100644
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -2074,7 +2074,7 @@ def p(name): return signature.parameters[name].default
         self.assertEqual(p('f'), False)
         self.assertEqual(p('local'), 3)
         self.assertEqual(p('sys'), sys.maxsize)
-        self.assertEqual(p('exp'), sys.maxsize - 1)
+        self.assertNotIn('exp', signature.parameters)
 
         test_callable(object)
 
diff --git a/Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst b/Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst
new file mode 100644
index 00000000000..452ad6e4bd2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst
@@ -0,0 +1,2 @@
+ast.literal_eval() is now more strict. Addition and subtraction of
+arbitrary numbers no longer allowed.


From webhook-mailer at python.org  Thu Jan  4 04:32:55 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 09:32:55 -0000
Subject: [Python-checkins] bpo-32482: Fix suspicious code in tests for
 syntax and grammar. (GH-5086) (#5095)
Message-ID: <mailman.124.1515058377.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/a70d5ff992e5e123a3fa6f0e7f672026db8ed1f7
commit: a70d5ff992e5e123a3fa6f0e7f672026db8ed1f7
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-04T11:32:53+02:00
summary:

bpo-32482: Fix suspicious code in tests for syntax and grammar. (GH-5086) (#5095)

(cherry picked from commit 0cc99c8cd70d422e4b345837a907db30e9180ab9)

files:
M Lib/test/test_grammar.py
M Lib/test/test_syntax.py

diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index 62f3251f051..ac8d85a3c41 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -575,6 +575,10 @@ def f(*args, **kwargs):
         self.assertEqual(f(spam='fried', **{'eggs':'scrambled'}),
                          ((), {'eggs':'scrambled', 'spam':'fried'}))
 
+        # Check ast errors in *args and *kwargs
+        check_syntax_error(self, "f(*g(1=2))")
+        check_syntax_error(self, "f(**g(1=2))")
+
         # argument annotation tests
         def f(x) -> list: pass
         self.assertEqual(f.__annotations__, {'return': list})
@@ -616,10 +620,6 @@ def f(x=1): return closure
         def f(*, k=1): return closure
         def f() -> int: return closure
 
-        # Check ast errors in *args and *kwargs
-        check_syntax_error(self, "f(*g(1=2))")
-        check_syntax_error(self, "f(**g(1=2))")
-
         # Check trailing commas are permitted in funcdef argument list
         def f(a,): pass
         def f(*args,): pass
@@ -1056,7 +1056,6 @@ def test_try(self):
         try: 1/0
         except EOFError: pass
         except TypeError as msg: pass
-        except RuntimeError as msg: pass
         except: pass
         else: pass
         try: 1/0
@@ -1165,7 +1164,7 @@ def test_selectors(self):
         d[1,2] = 3
         d[1,2,3] = 4
         L = list(d)
-        L.sort(key=lambda x: x if isinstance(x, tuple) else ())
+        L.sort(key=lambda x: (type(x).__name__, x))
         self.assertEqual(str(L), '[1, (1,), (1, 2), (1, 2, 3)]')
 
     def test_atoms(self):
diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py
index b7095a2cdb0..26f508316b6 100644
--- a/Lib/test/test_syntax.py
+++ b/Lib/test/test_syntax.py
@@ -600,12 +600,12 @@ def test_kwargs_last(self):
                           "positional argument follows keyword argument")
 
     def test_kwargs_last2(self):
-        self._check_error("int(**{base: 10}, '2')",
+        self._check_error("int(**{'base': 10}, '2')",
                           "positional argument follows "
                           "keyword argument unpacking")
 
     def test_kwargs_last3(self):
-        self._check_error("int(**{base: 10}, *['2'])",
+        self._check_error("int(**{'base': 10}, *['2'])",
                           "iterable argument unpacking follows "
                           "keyword argument unpacking")
 


From webhook-mailer at python.org  Thu Jan  4 05:24:52 2018
From: webhook-mailer at python.org (Berker Peksag)
Date: Thu, 04 Jan 2018 10:24:52 -0000
Subject: [Python-checkins] Fix outdated comment in typeobject.c (GH-5090)
Message-ID: <mailman.125.1515061493.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/78e24d4415c37e9e9951920bc31e9ee4868d86cb
commit: 78e24d4415c37e9e9951920bc31e9ee4868d86cb
branch: master
author: Berker Peksag <berker.peksag at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T13:24:45+03:00
summary:

Fix outdated comment in typeobject.c (GH-5090)

The comment was added in 3abca127fe1565ec80f6b0a1ae48d65186ad887d.
In d78448e912126410117723c7d240bbdfff06df16, it was changed to
use PyArg_ParseTuple instead.

files:
M Objects/typeobject.c

diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index 40c8fadc52e..e570c4145df 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -2372,7 +2372,7 @@ type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
         }
 
         /* SF bug 475327 -- if that didn't trigger, we need 3
-           arguments. but PyArg_ParseTupleAndKeywords below may give
+           arguments. but PyArg_ParseTuple below may give
            a msg saying type() needs exactly 3. */
         if (nargs != 3) {
             PyErr_SetString(PyExc_TypeError,


From webhook-mailer at python.org  Thu Jan  4 06:06:51 2018
From: webhook-mailer at python.org (Berker Peksag)
Date: Thu, 04 Jan 2018 11:06:51 -0000
Subject: [Python-checkins] Fix outdated comment in typeobject.c (GH-5090)
Message-ID: <mailman.126.1515064011.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/78e43a59acd2c5c2325e52f37ddd59d2079391ee
commit: 78e43a59acd2c5c2325e52f37ddd59d2079391ee
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Berker Peksag <berker.peksag at gmail.com>
date: 2018-01-04T14:06:47+03:00
summary:

Fix outdated comment in typeobject.c (GH-5090)

The comment was added in 3abca127fe1565ec80f6b0a1ae48d65186ad887d.
In d78448e912126410117723c7d240bbdfff06df16, it was changed to
use PyArg_ParseTuple instead.

(cherry picked from commit 78e24d4415c37e9e9951920bc31e9ee4868d86cb)

files:
M Objects/typeobject.c

diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index cb5e235628d..5a231c81d4e 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -2313,7 +2313,7 @@ type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
         }
 
         /* SF bug 475327 -- if that didn't trigger, we need 3
-           arguments. but PyArg_ParseTupleAndKeywords below may give
+           arguments. but PyArg_ParseTuple below may give
            a msg saying type() needs exactly 3. */
         if (nargs != 3) {
             PyErr_SetString(PyExc_TypeError,


From webhook-mailer at python.org  Thu Jan  4 07:08:34 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 12:08:34 -0000
Subject: [Python-checkins] [2.7] bpo-32211: Document the existing bug in
 re.findall() and re.finditer(). (GH-4695). (#5096)
Message-ID: <mailman.127.1515067715.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/ca54740f257086393106d242644d450485180b96
commit: ca54740f257086393106d242644d450485180b96
branch: 2.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T14:08:27+02:00
summary:

[2.7] bpo-32211: Document the existing bug in re.findall() and re.finditer(). (GH-4695). (#5096)

(cherry picked from commit 1e6d8525f9dd3dcdc83adb93b164082c8b95d17a)

files:
M Doc/library/re.rst

diff --git a/Doc/library/re.rst b/Doc/library/re.rst
index 22340818e26..c424230e13a 100644
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -611,14 +611,21 @@ form.
       Added the optional flags argument.
 
 
+
 .. function:: findall(pattern, string, flags=0)
 
    Return all non-overlapping matches of *pattern* in *string*, as a list of
    strings.  The *string* is scanned left-to-right, and matches are returned in
    the order found.  If one or more groups are present in the pattern, return a
    list of groups; this will be a list of tuples if the pattern has more than
-   one group.  Empty matches are included in the result unless they touch the
-   beginning of another match.
+   one group.  Empty matches are included in the result.
+
+   .. note::
+
+      Due to the limitation of the current implementation the character
+      following an empty match is not included in a next match, so
+      ``findall(r'^|\w+', 'two words')`` returns ``['', 'wo', 'words']``
+      (note missed "t").  This is changed in Python 3.7.
 
    .. versionadded:: 1.5.2
 
@@ -631,8 +638,7 @@ form.
    Return an :term:`iterator` yielding :class:`MatchObject` instances over all
    non-overlapping matches for the RE *pattern* in *string*.  The *string* is
    scanned left-to-right, and matches are returned in the order found.  Empty
-   matches are included in the result unless they touch the beginning of another
-   match.
+   matches are included in the result.  See also the note about :func:`findall`.
 
    .. versionadded:: 2.2
 


From webhook-mailer at python.org  Thu Jan  4 12:20:15 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 17:20:15 -0000
Subject: [Python-checkins] bpo-31672: Restore the former behavior when
 override flags in Template. (#5099)
Message-ID: <mailman.128.1515086416.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/87be28f4a1c5b76926c71a3d9f92503f9eb82d51
commit: 87be28f4a1c5b76926c71a3d9f92503f9eb82d51
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-04T19:20:11+02:00
summary:

bpo-31672: Restore the former behavior when override flags in Template. (#5099)

Overriding flags to 0 will make the default pattern matching only
lower case letters.

files:
M Doc/library/string.rst
M Lib/string.py
M Lib/test/test_string.py

diff --git a/Doc/library/string.rst b/Doc/library/string.rst
index e9606783ef7..fc3d94bfbfc 100644
--- a/Doc/library/string.rst
+++ b/Doc/library/string.rst
@@ -755,19 +755,14 @@ attributes:
 
 * *idpattern* -- This is the regular expression describing the pattern for
   non-braced placeholders.  The default value is the regular expression
-  ``(?a:[_a-zA-Z][_a-zA-Z0-9]*)``.  If this is given and *braceidpattern* is
+  ``(?a:[_a-z][_a-z0-9]*)``.  If this is given and *braceidpattern* is
   ``None`` this pattern will also apply to braced placeholders.
 
   .. note::
 
      Since default *flags* is ``re.IGNORECASE``, pattern ``[a-z]`` can match
      with some non-ASCII characters. That's why we use the local ``a`` flag
-     here.  Further, with the default *flags* value, including ``A-Z`` in the
-     ranges is redundant, but required for backward compatibility.
-
-     While *flags* is kept to ``re.IGNORECASE`` for backward compatibility,
-     you can override it to ``0`` or ``re.IGNORECASE | re.ASCII`` when
-     subclassing.
+     here.
 
   .. versionchanged:: 3.7
      *braceidpattern* can be used to define separate patterns used inside and
diff --git a/Lib/string.py b/Lib/string.py
index fd4b1f7a62f..b9d6f5eb567 100644
--- a/Lib/string.py
+++ b/Lib/string.py
@@ -82,11 +82,8 @@ class Template(metaclass=_TemplateMetaclass):
     # r'[a-z]' matches to non-ASCII letters when used with IGNORECASE, but
     # without the ASCII flag.  We can't add re.ASCII to flags because of
     # backward compatibility.  So we use the ?a local flag and [a-z] pattern.
-    # We also can't remove the A-Z ranges, because although they are
-    # technically redundant with the IGNORECASE flag, the value is part of the
-    # publicly documented API.
     # See https://bugs.python.org/issue31672
-    idpattern = r'(?a:[_a-zA-Z][_a-zA-Z0-9]*)'
+    idpattern = r'(?a:[_a-z][_a-z0-9]*)'
     braceidpattern = None
     flags = _re.IGNORECASE
 
diff --git a/Lib/test/test_string.py b/Lib/test/test_string.py
index 3480459c282..0be28fdb609 100644
--- a/Lib/test/test_string.py
+++ b/Lib/test/test_string.py
@@ -219,6 +219,16 @@ def test_regular_templates_with_braces(self):
         self.assertRaises(KeyError, s.substitute,
                           dict(who='tim', what='ham'))
 
+    def test_regular_templates_with_upper_case(self):
+        s = Template('$WHO likes ${WHAT} for ${MEAL}')
+        d = dict(WHO='tim', WHAT='ham', MEAL='dinner')
+        self.assertEqual(s.substitute(d), 'tim likes ham for dinner')
+
+    def test_regular_templates_with_non_letters(self):
+        s = Template('$_wh0_ likes ${_w_h_a_t_} for ${mea1}')
+        d = dict(_wh0_='tim', _w_h_a_t_='ham', mea1='dinner')
+        self.assertEqual(s.substitute(d), 'tim likes ham for dinner')
+
     def test_escapes(self):
         eq = self.assertEqual
         s = Template('$who likes to eat a bag of $$what worth $$100')
@@ -288,6 +298,14 @@ class PathPattern(Template):
         s = PathPattern('$bag.foo.who likes to eat a bag of $bag.what')
         self.assertEqual(s.substitute(m), 'tim likes to eat a bag of ham')
 
+    def test_flags_override(self):
+        class MyPattern(Template):
+            flags = 0
+        s = MyPattern('$wHO likes ${WHAT} for ${meal}')
+        d = dict(wHO='tim', WHAT='ham', meal='dinner', w='fred')
+        self.assertRaises(ValueError, s.substitute, d)
+        self.assertEqual(s.safe_substitute(d), 'fredHO likes ${WHAT} for dinner')
+
     def test_idpattern_override_inside_outside(self):
         # bpo-1198569: Allow the regexp inside and outside braces to be
         # different when deriving from Template.


From webhook-mailer at python.org  Thu Jan  4 17:21:50 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 04 Jan 2018 22:21:50 -0000
Subject: [Python-checkins] bpo-32226: Make __class_getitem__ an automatic
 class method. (#5098)
Message-ID: <mailman.129.1515104511.2882.python-checkins@python.org>

https://github.com/python/cpython/commit/ce5b0e9db1b9698e6ffc43ae41cf3a22ca5a6ba6
commit: ce5b0e9db1b9698e6ffc43ae41cf3a22ca5a6ba6
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-05T00:21:41+02:00
summary:

bpo-32226: Make __class_getitem__ an automatic class method. (#5098)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst
M Lib/test/test_genericclass.py
M Modules/_testcapimodule.c
M Objects/abstract.c
M Objects/typeobject.c

diff --git a/Lib/test/test_genericclass.py b/Lib/test/test_genericclass.py
index 2057fc000c0..37e755b1167 100644
--- a/Lib/test/test_genericclass.py
+++ b/Lib/test/test_genericclass.py
@@ -183,12 +183,21 @@ def __class_getitem__(cls, item):
         self.assertEqual(D[int], 'D[int]')
         self.assertEqual(D[D], 'D[D]')
 
+    def test_class_getitem_classmethod(self):
+        class C:
+            @classmethod
+            def __class_getitem__(cls, item):
+                return f'{cls.__name__}[{item.__name__}]'
+        class D(C): ...
+        self.assertEqual(D[int], 'D[int]')
+        self.assertEqual(D[D], 'D[D]')
+
     def test_class_getitem_patched(self):
         class C:
             def __init_subclass__(cls):
                 def __class_getitem__(cls, item):
                     return f'{cls.__name__}[{item.__name__}]'
-                cls.__class_getitem__ = __class_getitem__
+                cls.__class_getitem__ = classmethod(__class_getitem__)
         class D(C): ...
         self.assertEqual(D[int], 'D[int]')
         self.assertEqual(D[D], 'D[D]')
@@ -254,7 +263,7 @@ class CAPITest(unittest.TestCase):
 
     def test_c_class(self):
         from _testcapi import Generic, GenericAlias
-        self.assertIsInstance(Generic.__class_getitem__(Generic, int), GenericAlias)
+        self.assertIsInstance(Generic.__class_getitem__(int), GenericAlias)
 
         IntGeneric = Generic[int]
         self.assertIs(type(IntGeneric), GenericAlias)
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst
new file mode 100644
index 00000000000..842775f3b93
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst	
@@ -0,0 +1 @@
+``__class_getitem__`` is now an automatic class method.
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 47b50640f5d..7b2e2a3e5a4 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -5104,17 +5104,13 @@ typedef struct {
 } PyGenericObject;
 
 static PyObject *
-generic_class_getitem(PyObject *self, PyObject *args)
+generic_class_getitem(PyObject *type, PyObject *item)
 {
-    PyObject *type, *item;
-    if (!PyArg_UnpackTuple(args, "__class_getitem__", 2, 2, &type, &item)) {
-        return NULL;
-    }
     return generic_alias_new(item);
 }
 
 static PyMethodDef generic_methods[] = {
-    {"__class_getitem__", generic_class_getitem, METH_VARARGS|METH_STATIC, NULL},
+    {"__class_getitem__", generic_class_getitem, METH_O|METH_CLASS, NULL},
     {NULL}  /* sentinel */
 };
 
diff --git a/Objects/abstract.c b/Objects/abstract.c
index 0105c5d1696..a68253bdb43 100644
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -169,11 +169,11 @@ PyObject_GetItem(PyObject *o, PyObject *key)
     }
 
     if (PyType_Check(o)) {
-        PyObject *meth, *result, *stack[2] = {o, key};
+        PyObject *meth, *result, *stack[1] = {key};
         _Py_IDENTIFIER(__class_getitem__);
         meth = _PyObject_GetAttrId(o, &PyId___class_getitem__);
         if (meth) {
-            result = _PyObject_FastCall(meth, stack, 2);
+            result = _PyObject_FastCall(meth, stack, 1);
             Py_DECREF(meth);
             return result;
         }
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index e570c4145df..0c8f0adfb78 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -55,6 +55,7 @@ static size_t method_cache_collisions = 0;
 /* alphabetical order */
 _Py_IDENTIFIER(__abstractmethods__);
 _Py_IDENTIFIER(__class__);
+_Py_IDENTIFIER(__class_getitem__);
 _Py_IDENTIFIER(__delitem__);
 _Py_IDENTIFIER(__dict__);
 _Py_IDENTIFIER(__doc__);
@@ -2694,8 +2695,8 @@ type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
         Py_DECREF(tmp);
     }
 
-    /* Special-case __init_subclass__: if it's a plain function,
-       make it a classmethod */
+    /* Special-case __init_subclass__ and __class_getitem__:
+       if they are plain functions, make them classmethods */
     tmp = _PyDict_GetItemId(dict, &PyId___init_subclass__);
     if (tmp != NULL && PyFunction_Check(tmp)) {
         tmp = PyClassMethod_New(tmp);
@@ -2708,6 +2709,18 @@ type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
         Py_DECREF(tmp);
     }
 
+    tmp = _PyDict_GetItemId(dict, &PyId___class_getitem__);
+    if (tmp != NULL && PyFunction_Check(tmp)) {
+        tmp = PyClassMethod_New(tmp);
+        if (tmp == NULL)
+            goto error;
+        if (_PyDict_SetItemId(dict, &PyId___class_getitem__, tmp) < 0) {
+            Py_DECREF(tmp);
+            goto error;
+        }
+        Py_DECREF(tmp);
+    }
+
     /* Add descriptors for custom slots from __slots__, or for __dict__ */
     mp = PyHeapType_GET_MEMBERS(et);
     slotoffset = base->tp_basicsize;


From lp_benchmark_robot at intel.com  Thu Jan  4 17:52:15 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Thu, 4 Jan 2018 14:52:15 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-04
Message-ID: <e0280af8-8180-4377-9896-83d02dda4cdb@orsmsx152.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-04 03:04:09-08:00.
  - commit:		78e24d4
  - previous commit:	f190eb5
  - revision date:	2018-01-04 13:24:45+03:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.797% |  +0.133%   |  +8.032%   |  +6.092%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.105% |  +0.028%   |  +22.468%  |  +11.210%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.876% |  -0.275%   |  +24.734%  |  +9.827%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.988% |  +0.233%   |  +22.372%  |  +10.117%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.518% |  +1.300%   |  +9.326%   |  +12.009%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.844% |  +0.137%   |  +9.478%   |  +14.814%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 2.200% |  +0.357%   |  +5.928%   |  +11.537%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.616% |  +0.250%   |  -0.725%   |  +9.190%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 0.988% |  -0.169%   |  +11.421%  |  +16.770%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.348% |  +1.230%   |  +13.931%  |  +13.320%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.217% |  -0.220%   |  +3.417%   |  +6.512%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.353% |  -0.635%   |  +4.675%   |  +6.306%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.706% |  +0.801%   |  +2.386%   |  +6.078%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.284% |  +0.169%   |  +12.233%  |  +10.870%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.323% |  -0.603%   |  +8.018%   |  +10.753%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 1.060% |  +0.108%   |  +6.217%   |  +11.330%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.767% |  -0.035%   |  +10.498%  |  +11.191%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.773% |  -0.508%   |  +7.161%   |  +10.447%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 2.057% |  -1.026%   |  +2.161%   |  +10.118%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 1.319% |  +0.606%   |  -0.184%   |  +12.863%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.657% |  +1.078%   |  +8.068%   |  +11.898%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.877% |  -0.956%   |  +47.176%  |  +13.350%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.741% |  +1.404%   |  +9.222%   |  +13.028%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.703% |  -0.039%   |  +17.121%  |  +12.590%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 1.545% |  +0.594%   |  +11.732%  |  +6.808%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.454% |  -0.270%   |  +4.070%   |  +5.999%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.409% |  -1.703%   |  -3.838%   |  +2.823%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 2.173% |  -0.228%   |  +4.309%   |  +8.196%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.174% |  -0.220%   |  +0.722%   |  +13.097%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.295% |  +0.128%   |  +1.276%   |  +20.244%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.420% |  +0.276%   |  +2.071%   |  +22.753%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.817% |  -0.030%   |  +3.743%   |  +21.988%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 3.435% |  -0.327%   |  +11.240%  |  +10.329%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.052% |  +0.091%   |  -0.080%   |  +10.317%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.136% |  +0.028%   |  +10.695%  |  +4.989%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.108% |  +0.067%   |  +2.025%   |  +5.068%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.210% |  -0.022%   |  +10.435%  |  +13.817%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.521% |  -0.279%   |  +4.685%   |  +11.187%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.487% |  +0.054%   |  -2.489%   |  +12.251%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.374% |  -0.070%   |  -5.294%   |  +7.086%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 1.061% |  -0.050%   |  +3.782%   |  +8.280%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.975% |  -0.305%   |  +8.811%   |  +15.339%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 2.590% |  -0.975%   |  -2.493%   |  +3.452%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.813% |  -0.845%   |  +26.714%  |  +8.635%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.560% |  -0.653%   |  +4.189%   |  +7.086%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.929% |  -0.254%   |  +14.304%  |  +7.591%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 1.776% |  -0.630%   |  -3.393%   |  -2.623%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.580% |  -0.528%   |  +5.353%   |  +4.259%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.593% |  +0.282%   |  +7.210%   |  +5.771%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 2.974% |  +0.689%   |  +7.152%   |  +3.229%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 3.729% |  +0.393%   |  +19.319%  |  +8.864%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 1.926% |  +0.099%   |  +12.374%  |  +5.317%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.095% |  -0.203%   |  +8.883%   |  +6.382%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.460% |  +0.604%   |  +12.232%  |  +7.890%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.381% |  -1.832%   |  +9.323%   |  +10.574%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 5.579% |  -0.893%   |  +19.292%  |  +12.342%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.998% |  +0.233%   |  +6.324%   |  +5.886%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.090% |  +0.335%   |  +2.347%   |  +3.000%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 8.031% |  -1.749%   |  +6.747%   |  +23.029%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 2.646% |  +0.220%   |  -3.757%   |  +17.204%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.861% |  -0.189%   |  +8.522%   |  +6.625%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.037% |  +0.195%   |  +2.162%   |  +12.480%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.382% |  -0.192%   |  +3.526%   |  +7.787%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 2.581% |  +0.508%   |  -8.382%   |  +12.486%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.286% |  +0.241%   |  +4.416%   |  +11.897%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-04

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Fri Jan  5 01:34:23 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Fri, 05 Jan 2018 06:34:23 -0000
Subject: [Python-checkins] advance copyright years to 2018 (#5094)
Message-ID: <mailman.0.1515134064.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/65f2a6dcc2bc28a8566b74c8e9273f982331ec48
commit: 65f2a6dcc2bc28a8566b74c8e9273f982331ec48
branch: master
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-04T22:34:19-08:00
summary:

advance copyright years to 2018 (#5094)

files:
M Doc/copyright.rst
M Doc/license.rst
M LICENSE
M Mac/IDLE/IDLE.app/Contents/Info.plist
M Mac/PythonLauncher/Info.plist.in
M Mac/Resources/app/Info.plist.in
M Mac/Resources/framework/Info.plist.in
M Python/getcopyright.c
M README.rst

diff --git a/Doc/copyright.rst b/Doc/copyright.rst
index 2b5400cd9fb..540ff5ef059 100644
--- a/Doc/copyright.rst
+++ b/Doc/copyright.rst
@@ -4,7 +4,7 @@ Copyright
 
 Python and this documentation is:
 
-Copyright ? 2001-2017 Python Software Foundation. All rights reserved.
+Copyright ? 2001-2018 Python Software Foundation. All rights reserved.
 
 Copyright ? 2000 BeOpen.com. All rights reserved.
 
diff --git a/Doc/license.rst b/Doc/license.rst
index cd018689f87..aa4b1450e91 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -87,7 +87,7 @@ PSF LICENSE AGREEMENT FOR PYTHON |release|
       analyze, test, perform and/or display publicly, prepare derivative works,
       distribute, and otherwise use Python |release| alone or in any derivative
       version, provided, however, that PSF's License Agreement and PSF's notice of
-      copyright, i.e., "Copyright ? 2001-2017 Python Software Foundation; All Rights
+      copyright, i.e., "Copyright ? 2001-2018 Python Software Foundation; All Rights
       Reserved" are retained in Python |release| alone or in any derivative version
       prepared by Licensee.
 
diff --git a/LICENSE b/LICENSE
index 529349e4b38..1afbedba92b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -73,9 +73,9 @@ analyze, test, perform and/or display publicly, prepare derivative works,
 distribute, and otherwise use Python alone or in any derivative version,
 provided, however, that PSF's License Agreement and PSF's notice of copyright,
 i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
-2011, 2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation; All Rights
-Reserved" are retained in Python alone or in any derivative version prepared by
-Licensee.
+2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation; All
+Rights Reserved" are retained in Python alone or in any derivative version
+prepared by Licensee.
 
 3. In the event Licensee prepares a derivative work that is based on
 or incorporates Python or any part thereof, and wants to make
diff --git a/Mac/IDLE/IDLE.app/Contents/Info.plist b/Mac/IDLE/IDLE.app/Contents/Info.plist
index 5507687e90e..826d3793f75 100644
--- a/Mac/IDLE/IDLE.app/Contents/Info.plist
+++ b/Mac/IDLE/IDLE.app/Contents/Info.plist
@@ -36,7 +36,7 @@
 	<key>CFBundleExecutable</key>
 	<string>IDLE</string>
 	<key>CFBundleGetInfoString</key>
-	<string>%version%, ? 2001-2017 Python Software Foundation</string>
+	<string>%version%, ? 2001-2018 Python Software Foundation</string>
 	<key>CFBundleIconFile</key>
 	<string>IDLE.icns</string>
 	<key>CFBundleIdentifier</key>
diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in
index f1ab79f3d44..5fa346ed4d3 100644
--- a/Mac/PythonLauncher/Info.plist.in
+++ b/Mac/PythonLauncher/Info.plist.in
@@ -40,7 +40,7 @@
 	<key>CFBundleExecutable</key>
 	<string>Python Launcher</string>
 	<key>CFBundleGetInfoString</key>
-	<string>%VERSION%, ? 2001-2017 Python Software Foundation</string>
+	<string>%VERSION%, ? 2001-2018 Python Software Foundation</string>
 	<key>CFBundleIconFile</key>
 	<string>PythonLauncher.icns</string>
 	<key>CFBundleIdentifier</key>
diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in
index a23166e6d32..abe9ae23e34 100644
--- a/Mac/Resources/app/Info.plist.in
+++ b/Mac/Resources/app/Info.plist.in
@@ -37,7 +37,7 @@
 	<key>CFBundleInfoDictionaryVersion</key>
 	<string>6.0</string>
 	<key>CFBundleLongVersionString</key>
-	<string>%version%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%version%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleName</key>
 	<string>Python</string>
 	<key>CFBundlePackageType</key>
diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in
index 7a64619e295..c1ea9f68892 100644
--- a/Mac/Resources/framework/Info.plist.in
+++ b/Mac/Resources/framework/Info.plist.in
@@ -17,9 +17,9 @@
 	<key>CFBundlePackageType</key>
 	<string>FMWK</string>
 	<key>CFBundleShortVersionString</key>
-	<string>%VERSION%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleLongVersionString</key>
-	<string>%VERSION%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleSignature</key>
 	<string>????</string>
 	<key>CFBundleVersion</key>
diff --git a/Python/getcopyright.c b/Python/getcopyright.c
index cac647c3ce4..51772ecef31 100644
--- a/Python/getcopyright.c
+++ b/Python/getcopyright.c
@@ -4,7 +4,7 @@
 
 static const char cprt[] =
 "\
-Copyright (c) 2001-2017 Python Software Foundation.\n\
+Copyright (c) 2001-2018 Python Software Foundation.\n\
 All Rights Reserved.\n\
 \n\
 Copyright (c) 2000 BeOpen.com.\n\
diff --git a/README.rst b/README.rst
index d60fcac35d5..346eb4a32db 100644
--- a/README.rst
+++ b/README.rst
@@ -14,7 +14,7 @@ This is Python version 3.7.0 alpha 3+
    :target: https://codecov.io/gh/python/cpython
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
-2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation.  All rights
+2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation.  All rights
 reserved.
 
 See the end of this file for further copyright and license information.
@@ -233,7 +233,7 @@ Copyright and License Information
 ---------------------------------
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
-2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation.  All rights
+2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation.  All rights
 reserved.
 
 Copyright (c) 2000 BeOpen.com.  All rights reserved.


From webhook-mailer at python.org  Fri Jan  5 02:02:14 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Fri, 05 Jan 2018 07:02:14 -0000
Subject: [Python-checkins] [2.7] advance copyright years to 2018 (GH-5094).
 (#5105)
Message-ID: <mailman.1.1515135735.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/e13698172f5da96c2639224c216efb05b48ceac9
commit: e13698172f5da96c2639224c216efb05b48ceac9
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-04T23:02:11-08:00
summary:

[2.7] advance copyright years to 2018 (GH-5094). (#5105)

(cherry picked from commit 65f2a6dcc2bc28a8566b74c8e9273f982331ec48)

files:
M Doc/copyright.rst
M Doc/license.rst
M LICENSE
M Mac/PythonLauncher/Info.plist.in
M Mac/Resources/app/Info.plist.in
M Mac/Resources/framework/Info.plist.in
M Python/getcopyright.c
M README

diff --git a/Doc/copyright.rst b/Doc/copyright.rst
index 22d7705846e..540ff5ef059 100644
--- a/Doc/copyright.rst
+++ b/Doc/copyright.rst
@@ -4,7 +4,7 @@ Copyright
 
 Python and this documentation is:
 
-Copyright ? 2001-2016 Python Software Foundation. All rights reserved.
+Copyright ? 2001-2018 Python Software Foundation. All rights reserved.
 
 Copyright ? 2000 BeOpen.com. All rights reserved.
 
diff --git a/Doc/license.rst b/Doc/license.rst
index 942ad20e47b..f33495ab1ef 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -87,7 +87,7 @@ PSF LICENSE AGREEMENT FOR PYTHON |release|
       analyze, test, perform and/or display publicly, prepare derivative works,
       distribute, and otherwise use Python |release| alone or in any derivative
       version, provided, however, that PSF's License Agreement and PSF's notice of
-      copyright, i.e., "Copyright ? 2001-2017 Python Software Foundation; All Rights
+      copyright, i.e., "Copyright ? 2001-2018 Python Software Foundation; All Rights
       Reserved" are retained in Python |release| alone or in any derivative version
       prepared by Licensee.
 
diff --git a/LICENSE b/LICENSE
index 529349e4b38..1afbedba92b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -73,9 +73,9 @@ analyze, test, perform and/or display publicly, prepare derivative works,
 distribute, and otherwise use Python alone or in any derivative version,
 provided, however, that PSF's License Agreement and PSF's notice of copyright,
 i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
-2011, 2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation; All Rights
-Reserved" are retained in Python alone or in any derivative version prepared by
-Licensee.
+2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation; All
+Rights Reserved" are retained in Python alone or in any derivative version
+prepared by Licensee.
 
 3. In the event Licensee prepares a derivative work that is based on
 or incorporates Python or any part thereof, and wants to make
diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in
index 1a8e2b44553..b84fffeec64 100644
--- a/Mac/PythonLauncher/Info.plist.in
+++ b/Mac/PythonLauncher/Info.plist.in
@@ -40,7 +40,7 @@
 	<key>CFBundleExecutable</key>
 	<string>PythonLauncher</string>
 	<key>CFBundleGetInfoString</key>
-	<string>%VERSION%, ? 2001-2017 Python Software Foundation</string>
+	<string>%VERSION%, ? 2001-2018 Python Software Foundation</string>
 	<key>CFBundleIconFile</key>
 	<string>PythonLauncher.icns</string>
 	<key>CFBundleIdentifier</key>
diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in
index a23166e6d32..abe9ae23e34 100644
--- a/Mac/Resources/app/Info.plist.in
+++ b/Mac/Resources/app/Info.plist.in
@@ -37,7 +37,7 @@
 	<key>CFBundleInfoDictionaryVersion</key>
 	<string>6.0</string>
 	<key>CFBundleLongVersionString</key>
-	<string>%version%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%version%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleName</key>
 	<string>Python</string>
 	<key>CFBundlePackageType</key>
diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in
index 7a64619e295..c1ea9f68892 100644
--- a/Mac/Resources/framework/Info.plist.in
+++ b/Mac/Resources/framework/Info.plist.in
@@ -17,9 +17,9 @@
 	<key>CFBundlePackageType</key>
 	<string>FMWK</string>
 	<key>CFBundleShortVersionString</key>
-	<string>%VERSION%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleLongVersionString</key>
-	<string>%VERSION%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleSignature</key>
 	<string>????</string>
 	<key>CFBundleVersion</key>
diff --git a/Python/getcopyright.c b/Python/getcopyright.c
index c37f8fa81cb..1b69012fbc1 100644
--- a/Python/getcopyright.c
+++ b/Python/getcopyright.c
@@ -4,7 +4,7 @@
 
 static char cprt[] = 
 "\
-Copyright (c) 2001-2017 Python Software Foundation.\n\
+Copyright (c) 2001-2018 Python Software Foundation.\n\
 All Rights Reserved.\n\
 \n\
 Copyright (c) 2000 BeOpen.com.\n\
diff --git a/README b/README
index 00a6b394655..387b437cc29 100644
--- a/README
+++ b/README
@@ -2,7 +2,7 @@ This is Python version 2.7.14
 =============================
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
-2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation.  All rights
+2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation.  All rights
 reserved.
 
 Copyright (c) 2000 BeOpen.com.


From solipsis at pitrou.net  Fri Jan  5 04:14:17 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 05 Jan 2018 09:14:17 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=9
Message-ID: <20180105091417.1.18E93BB5250E5A0D@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_asyncio leaked [0, 0, 3] memory blocks, sum=3
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [0, 0, 2] memory blocks, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogzzquDX', '--timeout', '7200']

From webhook-mailer at python.org  Fri Jan  5 05:15:58 2018
From: webhook-mailer at python.org (Antoine Pitrou)
Date: Fri, 05 Jan 2018 10:15:58 -0000
Subject: [Python-checkins] bpo-31699 Deadlocks in
 `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)
Message-ID: <mailman.2.1515147360.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/94459fd7dc25ce19096f2080eb7339497d319eb0
commit: 94459fd7dc25ce19096f2080eb7339497d319eb0
branch: master
author: Thomas Moreau <thomas.moreau.2010 at gmail.com>
committer: Antoine Pitrou <pitrou at free.fr>
date: 2018-01-05T11:15:54+01:00
summary:

bpo-31699 Deadlocks in `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)

Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.

files:
A Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst
M Lib/concurrent/futures/process.py
M Lib/multiprocessing/queues.py
M Lib/test/_test_multiprocessing.py
M Lib/test/test_concurrent_futures.py

diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
index 35af65d0bee..aaa5151e017 100644
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -8,10 +8,10 @@
 |======================= In-process =====================|== Out-of-process ==|
 
 +----------+     +----------+       +--------+     +-----------+    +---------+
-|          |  => | Work Ids |    => |        |  => | Call Q    | => |         |
-|          |     +----------+       |        |     +-----------+    |         |
-|          |     | ...      |       |        |     | ...       |    |         |
-|          |     | 6        |       |        |     | 5, call() |    |         |
+|          |  => | Work Ids |       |        |     | Call Q    |    | Process |
+|          |     +----------+       |        |     +-----------+    |  Pool   |
+|          |     | ...      |       |        |     | ...       |    +---------+
+|          |     | 6        |    => |        |  => | 5, call() | => |         |
 |          |     | 7        |       |        |     | ...       |    |         |
 | Process  |     | ...      |       | Local  |     +-----------+    | Process |
 |  Pool    |     +----------+       | Worker |                      |  #1..n  |
@@ -52,6 +52,7 @@
 from queue import Full
 import multiprocessing as mp
 from multiprocessing.connection import wait
+from multiprocessing.queues import Queue
 import threading
 import weakref
 from functools import partial
@@ -72,16 +73,31 @@
 # workers to exit when their work queues are empty and then waits until the
 # threads/processes finish.
 
-_threads_queues = weakref.WeakKeyDictionary()
+_threads_wakeups = weakref.WeakKeyDictionary()
 _global_shutdown = False
 
+
+class _ThreadWakeup:
+    __slot__ = ["_state"]
+
+    def __init__(self):
+        self._reader, self._writer = mp.Pipe(duplex=False)
+
+    def wakeup(self):
+        self._writer.send_bytes(b"")
+
+    def clear(self):
+        while self._reader.poll():
+            self._reader.recv_bytes()
+
+
 def _python_exit():
     global _global_shutdown
     _global_shutdown = True
-    items = list(_threads_queues.items())
-    for t, q in items:
-        q.put(None)
-    for t, q in items:
+    items = list(_threads_wakeups.items())
+    for _, thread_wakeup in items:
+        thread_wakeup.wakeup()
+    for t, _ in items:
         t.join()
 
 # Controls how many more calls than processes will be queued in the call queue.
@@ -90,6 +106,7 @@ def _python_exit():
 # (Futures in the call queue cannot be cancelled).
 EXTRA_QUEUED_CALLS = 1
 
+
 # Hack to embed stringification of remote traceback in local traceback
 
 class _RemoteTraceback(Exception):
@@ -132,6 +149,25 @@ def __init__(self, work_id, fn, args, kwargs):
         self.kwargs = kwargs
 
 
+class _SafeQueue(Queue):
+    """Safe Queue set exception to the future object linked to a job"""
+    def __init__(self, max_size=0, *, ctx, pending_work_items):
+        self.pending_work_items = pending_work_items
+        super().__init__(max_size, ctx=ctx)
+
+    def _on_queue_feeder_error(self, e, obj):
+        if isinstance(obj, _CallItem):
+            tb = traceback.format_exception(type(e), e, e.__traceback__)
+            e.__cause__ = _RemoteTraceback('\n"""\n{}"""'.format(''.join(tb)))
+            work_item = self.pending_work_items.pop(obj.work_id, None)
+            # work_item can be None if another process terminated. In this case,
+            # the queue_manager_thread fails all work_items with BrokenProcessPool
+            if work_item is not None:
+                work_item.future.set_exception(e)
+        else:
+            super()._on_queue_feeder_error(e, obj)
+
+
 def _get_chunks(*iterables, chunksize):
     """ Iterates over zip()ed iterables in chunks. """
     it = zip(*iterables)
@@ -152,6 +188,17 @@ def _process_chunk(fn, chunk):
     """
     return [fn(*args) for args in chunk]
 
+
+def _sendback_result(result_queue, work_id, result=None, exception=None):
+    """Safely send back the given result or exception"""
+    try:
+        result_queue.put(_ResultItem(work_id, result=result,
+                                     exception=exception))
+    except BaseException as e:
+        exc = _ExceptionWithTraceback(e, e.__traceback__)
+        result_queue.put(_ResultItem(work_id, exception=exc))
+
+
 def _process_worker(call_queue, result_queue, initializer, initargs):
     """Evaluates calls from call_queue and places the results in result_queue.
 
@@ -183,10 +230,9 @@ def _process_worker(call_queue, result_queue, initializer, initargs):
             r = call_item.fn(*call_item.args, **call_item.kwargs)
         except BaseException as e:
             exc = _ExceptionWithTraceback(e, e.__traceback__)
-            result_queue.put(_ResultItem(call_item.work_id, exception=exc))
+            _sendback_result(result_queue, call_item.work_id, exception=exc)
         else:
-            result_queue.put(_ResultItem(call_item.work_id,
-                                         result=r))
+            _sendback_result(result_queue, call_item.work_id, result=r)
 
         # Liberate the resource as soon as possible, to avoid holding onto
         # open files or shared memory that is not needed anymore
@@ -230,12 +276,14 @@ def _add_call_item_to_queue(pending_work_items,
                 del pending_work_items[work_id]
                 continue
 
+
 def _queue_management_worker(executor_reference,
                              processes,
                              pending_work_items,
                              work_ids_queue,
                              call_queue,
-                             result_queue):
+                             result_queue,
+                             thread_wakeup):
     """Manages the communication between this process and the worker processes.
 
     This function is run in a local thread.
@@ -253,6 +301,9 @@ def _queue_management_worker(executor_reference,
             derived from _WorkItems for processing by the process workers.
         result_queue: A ctx.SimpleQueue of _ResultItems generated by the
             process workers.
+        thread_wakeup: A _ThreadWakeup to allow waking up the
+            queue_manager_thread from the main Thread and avoid deadlocks
+            caused by permanently locked queues.
     """
     executor = None
 
@@ -261,10 +312,21 @@ def shutting_down():
                 or executor._shutdown_thread)
 
     def shutdown_worker():
-        # This is an upper bound
-        nb_children_alive = sum(p.is_alive() for p in processes.values())
-        for i in range(0, nb_children_alive):
-            call_queue.put_nowait(None)
+        # This is an upper bound on the number of children alive.
+        n_children_alive = sum(p.is_alive() for p in processes.values())
+        n_children_to_stop = n_children_alive
+        n_sentinels_sent = 0
+        # Send the right number of sentinels, to make sure all children are
+        # properly terminated.
+        while n_sentinels_sent < n_children_to_stop and n_children_alive > 0:
+            for i in range(n_children_to_stop - n_sentinels_sent):
+                try:
+                    call_queue.put_nowait(None)
+                    n_sentinels_sent += 1
+                except Full:
+                    break
+            n_children_alive = sum(p.is_alive() for p in processes.values())
+
         # Release the queue's resources as soon as possible.
         call_queue.close()
         # If .join() is not called on the created processes then
@@ -272,19 +334,37 @@ def shutdown_worker():
         for p in processes.values():
             p.join()
 
-    reader = result_queue._reader
+    result_reader = result_queue._reader
+    wakeup_reader = thread_wakeup._reader
+    readers = [result_reader, wakeup_reader]
 
     while True:
         _add_call_item_to_queue(pending_work_items,
                                 work_ids_queue,
                                 call_queue)
 
-        sentinels = [p.sentinel for p in processes.values()]
-        assert sentinels
-        ready = wait([reader] + sentinels)
-        if reader in ready:
-            result_item = reader.recv()
-        else:
+        # Wait for a result to be ready in the result_queue while checking
+        # that all worker processes are still running, or for a wake up
+        # signal send. The wake up signals come either from new tasks being
+        # submitted, from the executor being shutdown/gc-ed, or from the
+        # shutdown of the python interpreter.
+        worker_sentinels = [p.sentinel for p in processes.values()]
+        ready = wait(readers + worker_sentinels)
+
+        cause = None
+        is_broken = True
+        if result_reader in ready:
+            try:
+                result_item = result_reader.recv()
+                is_broken = False
+            except BaseException as e:
+                cause = traceback.format_exception(type(e), e, e.__traceback__)
+
+        elif wakeup_reader in ready:
+            is_broken = False
+            result_item = None
+        thread_wakeup.clear()
+        if is_broken:
             # Mark the process pool broken so that submits fail right now.
             executor = executor_reference()
             if executor is not None:
@@ -293,14 +373,15 @@ def shutdown_worker():
                                     'usable anymore')
                 executor._shutdown_thread = True
                 executor = None
+            bpe = BrokenProcessPool("A process in the process pool was "
+                                    "terminated abruptly while the future was "
+                                    "running or pending.")
+            if cause is not None:
+                bpe.__cause__ = _RemoteTraceback(
+                    f"\n'''\n{''.join(cause)}'''")
             # All futures in flight must be marked failed
             for work_id, work_item in pending_work_items.items():
-                work_item.future.set_exception(
-                    BrokenProcessPool(
-                        "A process in the process pool was "
-                        "terminated abruptly while the future was "
-                        "running or pending."
-                    ))
+                work_item.future.set_exception(bpe)
                 # Delete references to object. See issue16284
                 del work_item
             pending_work_items.clear()
@@ -329,6 +410,9 @@ def shutdown_worker():
                     work_item.future.set_result(result_item.result)
                 # Delete references to object. See issue16284
                 del work_item
+            # Delete reference to result_item
+            del result_item
+
         # Check whether we should start shutting down.
         executor = executor_reference()
         # No more work items can be added if:
@@ -348,8 +432,11 @@ def shutdown_worker():
                 pass
         executor = None
 
+
 _system_limits_checked = False
 _system_limited = None
+
+
 def _check_system_limits():
     global _system_limits_checked, _system_limited
     if _system_limits_checked:
@@ -369,7 +456,8 @@ def _check_system_limits():
         # minimum number of semaphores available
         # according to POSIX
         return
-    _system_limited = "system provides too few semaphores (%d available, 256 necessary)" % nsems_max
+    _system_limited = ("system provides too few semaphores (%d"
+                       " available, 256 necessary)" % nsems_max)
     raise NotImplementedError(_system_limited)
 
 
@@ -415,6 +503,7 @@ def __init__(self, max_workers=None, mp_context=None,
                 raise ValueError("max_workers must be greater than 0")
 
             self._max_workers = max_workers
+
         if mp_context is None:
             mp_context = mp.get_context()
         self._mp_context = mp_context
@@ -424,34 +513,52 @@ def __init__(self, max_workers=None, mp_context=None,
         self._initializer = initializer
         self._initargs = initargs
 
+        # Management thread
+        self._queue_management_thread = None
+
+        # Map of pids to processes
+        self._processes = {}
+
+        # Shutdown is a two-step process.
+        self._shutdown_thread = False
+        self._shutdown_lock = threading.Lock()
+        self._broken = False
+        self._queue_count = 0
+        self._pending_work_items = {}
+
+        # Create communication channels for the executor
         # Make the call queue slightly larger than the number of processes to
         # prevent the worker processes from idling. But don't make it too big
         # because futures in the call queue cannot be cancelled.
         queue_size = self._max_workers + EXTRA_QUEUED_CALLS
-        self._call_queue = mp_context.Queue(queue_size)
+        self._call_queue = _SafeQueue(
+            max_size=queue_size, ctx=self._mp_context,
+            pending_work_items=self._pending_work_items)
         # Killed worker processes can produce spurious "broken pipe"
         # tracebacks in the queue's own worker thread. But we detect killed
         # processes anyway, so silence the tracebacks.
         self._call_queue._ignore_epipe = True
         self._result_queue = mp_context.SimpleQueue()
         self._work_ids = queue.Queue()
-        self._queue_management_thread = None
-        # Map of pids to processes
-        self._processes = {}
 
-        # Shutdown is a two-step process.
-        self._shutdown_thread = False
-        self._shutdown_lock = threading.Lock()
-        self._broken = False
-        self._queue_count = 0
-        self._pending_work_items = {}
+        # _ThreadWakeup is a communication channel used to interrupt the wait
+        # of the main loop of queue_manager_thread from another thread (e.g.
+        # when calling executor.submit or executor.shutdown). We do not use the
+        # _result_queue to send the wakeup signal to the queue_manager_thread
+        # as it could result in a deadlock if a worker process dies with the
+        # _result_queue write lock still acquired.
+        self._queue_management_thread_wakeup = _ThreadWakeup()
 
     def _start_queue_management_thread(self):
-        # When the executor gets lost, the weakref callback will wake up
-        # the queue management thread.
-        def weakref_cb(_, q=self._result_queue):
-            q.put(None)
         if self._queue_management_thread is None:
+            # When the executor gets garbarge collected, the weakref callback
+            # will wake up the queue management thread so that it can terminate
+            # if there is no pending work item.
+            def weakref_cb(_,
+                           thread_wakeup=self._queue_management_thread_wakeup):
+                mp.util.debug('Executor collected: triggering callback for'
+                              ' QueueManager wakeup')
+                thread_wakeup.wakeup()
             # Start the processes so that their sentinels are known.
             self._adjust_process_count()
             self._queue_management_thread = threading.Thread(
@@ -461,10 +568,13 @@ def weakref_cb(_, q=self._result_queue):
                       self._pending_work_items,
                       self._work_ids,
                       self._call_queue,
-                      self._result_queue))
+                      self._result_queue,
+                      self._queue_management_thread_wakeup),
+                name="QueueManagerThread")
             self._queue_management_thread.daemon = True
             self._queue_management_thread.start()
-            _threads_queues[self._queue_management_thread] = self._result_queue
+            _threads_wakeups[self._queue_management_thread] = \
+                self._queue_management_thread_wakeup
 
     def _adjust_process_count(self):
         for _ in range(len(self._processes), self._max_workers):
@@ -491,7 +601,7 @@ def submit(self, fn, *args, **kwargs):
             self._work_ids.put(self._queue_count)
             self._queue_count += 1
             # Wake up queue management thread
-            self._result_queue.put(None)
+            self._queue_management_thread_wakeup.wakeup()
 
             self._start_queue_management_thread()
             return f
@@ -531,7 +641,7 @@ def shutdown(self, wait=True):
             self._shutdown_thread = True
         if self._queue_management_thread:
             # Wake up queue management thread
-            self._result_queue.put(None)
+            self._queue_management_thread_wakeup.wakeup()
             if wait:
                 self._queue_management_thread.join()
         # To reduce the risk of opening too many files, remove references to
diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index 328efbd95fe..d66d37a5c3e 100644
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -160,9 +160,10 @@ def _start_thread(self):
         self._thread = threading.Thread(
             target=Queue._feed,
             args=(self._buffer, self._notempty, self._send_bytes,
-                  self._wlock, self._writer.close, self._ignore_epipe),
+                  self._wlock, self._writer.close, self._ignore_epipe,
+                  self._on_queue_feeder_error),
             name='QueueFeederThread'
-            )
+        )
         self._thread.daemon = True
 
         debug('doing self._thread.start()')
@@ -201,7 +202,8 @@ def _finalize_close(buffer, notempty):
             notempty.notify()
 
     @staticmethod
-    def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe):
+    def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
+              onerror):
         debug('starting thread to feed data to pipe')
         nacquire = notempty.acquire
         nrelease = notempty.release
@@ -253,8 +255,17 @@ def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe):
                     info('error in queue thread: %s', e)
                     return
                 else:
-                    import traceback
-                    traceback.print_exc()
+                    onerror(e, obj)
+
+    @staticmethod
+    def _on_queue_feeder_error(e, obj):
+        """
+        Private API hook called when feeding data in the background thread
+        raises an exception.  For overriding by concurrent.futures.
+        """
+        import traceback
+        traceback.print_exc()
+
 
 _sentinel = object()
 
diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index 7575c5d3681..05166b91ba8 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -1029,6 +1029,43 @@ def __reduce__(self):
             self.assertTrue(q.get(timeout=1.0))
             close_queue(q)
 
+    def test_queue_feeder_on_queue_feeder_error(self):
+        # bpo-30006: verify feeder handles exceptions using the
+        # _on_queue_feeder_error hook.
+        if self.TYPE != 'processes':
+            self.skipTest('test not appropriate for {}'.format(self.TYPE))
+
+        class NotSerializable(object):
+            """Mock unserializable object"""
+            def __init__(self):
+                self.reduce_was_called = False
+                self.on_queue_feeder_error_was_called = False
+
+            def __reduce__(self):
+                self.reduce_was_called = True
+                raise AttributeError
+
+        class SafeQueue(multiprocessing.queues.Queue):
+            """Queue with overloaded _on_queue_feeder_error hook"""
+            @staticmethod
+            def _on_queue_feeder_error(e, obj):
+                if (isinstance(e, AttributeError) and
+                        isinstance(obj, NotSerializable)):
+                    obj.on_queue_feeder_error_was_called = True
+
+        not_serializable_obj = NotSerializable()
+        # The captured_stderr reduces the noise in the test report
+        with test.support.captured_stderr():
+            q = SafeQueue(ctx=multiprocessing.get_context())
+            q.put(not_serializable_obj)
+
+            # Verify that q is still functionning correctly
+            q.put(True)
+            self.assertTrue(q.get(timeout=1.0))
+
+        # Assert that the serialization and the hook have been called correctly
+        self.assertTrue(not_serializable_obj.reduce_was_called)
+        self.assertTrue(not_serializable_obj.on_queue_feeder_error_was_called)
 #
 #
 #
diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py
index 76878992f9a..675cd7ae05e 100644
--- a/Lib/test/test_concurrent_futures.py
+++ b/Lib/test/test_concurrent_futures.py
@@ -18,6 +18,7 @@
 import time
 import unittest
 import weakref
+from pickle import PicklingError
 
 from concurrent import futures
 from concurrent.futures._base import (
@@ -394,16 +395,17 @@ def test_del_shutdown(self):
         queue_management_thread = executor._queue_management_thread
         processes = executor._processes
         call_queue = executor._call_queue
+        queue_management_thread = executor._queue_management_thread
         del executor
 
+        # Make sure that all the executor ressources were properly cleaned by
+        # the shutdown process
         queue_management_thread.join()
         for p in processes.values():
             p.join()
-        call_queue.close()
         call_queue.join_thread()
 
 
-
 create_executor_tests(ProcessPoolShutdownTest,
                       executor_mixins=(ProcessPoolForkMixin,
                                        ProcessPoolForkserverMixin,
@@ -784,6 +786,172 @@ def test_ressources_gced_in_workers(self):
                                        ProcessPoolForkserverMixin,
                                        ProcessPoolSpawnMixin))
 
+def hide_process_stderr():
+    import io
+    sys.stderr = io.StringIO()
+
+
+def _crash(delay=None):
+    """Induces a segfault."""
+    if delay:
+        time.sleep(delay)
+    import faulthandler
+    faulthandler.disable()
+    faulthandler._sigsegv()
+
+
+def _exit():
+    """Induces a sys exit with exitcode 1."""
+    sys.exit(1)
+
+
+def _raise_error(Err):
+    """Function that raises an Exception in process."""
+    hide_process_stderr()
+    raise Err()
+
+
+def _return_instance(cls):
+    """Function that returns a instance of cls."""
+    hide_process_stderr()
+    return cls()
+
+
+class CrashAtPickle(object):
+    """Bad object that triggers a segfault at pickling time."""
+    def __reduce__(self):
+        _crash()
+
+
+class CrashAtUnpickle(object):
+    """Bad object that triggers a segfault at unpickling time."""
+    def __reduce__(self):
+        return _crash, ()
+
+
+class ExitAtPickle(object):
+    """Bad object that triggers a process exit at pickling time."""
+    def __reduce__(self):
+        _exit()
+
+
+class ExitAtUnpickle(object):
+    """Bad object that triggers a process exit at unpickling time."""
+    def __reduce__(self):
+        return _exit, ()
+
+
+class ErrorAtPickle(object):
+    """Bad object that triggers an error at pickling time."""
+    def __reduce__(self):
+        from pickle import PicklingError
+        raise PicklingError("Error in pickle")
+
+
+class ErrorAtUnpickle(object):
+    """Bad object that triggers an error at unpickling time."""
+    def __reduce__(self):
+        from pickle import UnpicklingError
+        return _raise_error, (UnpicklingError, )
+
+
+class ExecutorDeadlockTest:
+    TIMEOUT = 15
+
+    @classmethod
+    def _sleep_id(cls, x, delay):
+        time.sleep(delay)
+        return x
+
+    def _fail_on_deadlock(self, executor):
+        # If we did not recover before TIMEOUT seconds, consider that the
+        # executor is in a deadlock state and forcefully clean all its
+        # composants.
+        import faulthandler
+        from tempfile import TemporaryFile
+        with TemporaryFile(mode="w+") as f:
+            faulthandler.dump_traceback(file=f)
+            f.seek(0)
+            tb = f.read()
+        for p in executor._processes.values():
+            p.terminate()
+        # This should be safe to call executor.shutdown here as all possible
+        # deadlocks should have been broken.
+        executor.shutdown(wait=True)
+        print(f"\nTraceback:\n {tb}", file=sys.__stderr__)
+        self.fail(f"Executor deadlock:\n\n{tb}")
+
+
+    def test_crash(self):
+        # extensive testing for deadlock caused by crashes in a pool.
+        self.executor.shutdown(wait=True)
+        crash_cases = [
+            # Check problem occuring while pickling a task in
+            # the task_handler thread
+            (id, (ErrorAtPickle(),), PicklingError, "error at task pickle"),
+            # Check problem occuring while unpickling a task on workers
+            (id, (ExitAtUnpickle(),), BrokenProcessPool,
+             "exit at task unpickle"),
+            (id, (ErrorAtUnpickle(),), BrokenProcessPool,
+             "error at task unpickle"),
+            (id, (CrashAtUnpickle(),), BrokenProcessPool,
+             "crash at task unpickle"),
+            # Check problem occuring during func execution on workers
+            (_crash, (), BrokenProcessPool,
+             "crash during func execution on worker"),
+            (_exit, (), SystemExit,
+             "exit during func execution on worker"),
+            (_raise_error, (RuntimeError, ), RuntimeError,
+             "error during func execution on worker"),
+            # Check problem occuring while pickling a task result
+            # on workers
+            (_return_instance, (CrashAtPickle,), BrokenProcessPool,
+             "crash during result pickle on worker"),
+            (_return_instance, (ExitAtPickle,), SystemExit,
+             "exit during result pickle on worker"),
+            (_return_instance, (ErrorAtPickle,), PicklingError,
+             "error during result pickle on worker"),
+            # Check problem occuring while unpickling a task in
+            # the result_handler thread
+            (_return_instance, (ErrorAtUnpickle,), BrokenProcessPool,
+             "error during result unpickle in result_handler"),
+            (_return_instance, (ExitAtUnpickle,), BrokenProcessPool,
+             "exit during result unpickle in result_handler")
+        ]
+        for func, args, error, name in crash_cases:
+            with self.subTest(name):
+                # The captured_stderr reduces the noise in the test report
+                with test.support.captured_stderr():
+                    executor = self.executor_type(
+                        max_workers=2, mp_context=get_context(self.ctx))
+                    res = executor.submit(func, *args)
+                    with self.assertRaises(error):
+                        try:
+                            res.result(timeout=self.TIMEOUT)
+                        except futures.TimeoutError:
+                            # If we did not recover before TIMEOUT seconds,
+                            # consider that the executor is in a deadlock state
+                            self._fail_on_deadlock(executor)
+                    executor.shutdown(wait=True)
+
+    def test_shutdown_deadlock(self):
+        # Test that the pool calling shutdown do not cause deadlock
+        # if a worker fails after the shutdown call.
+        self.executor.shutdown(wait=True)
+        with self.executor_type(max_workers=2,
+                                mp_context=get_context(self.ctx)) as executor:
+            self.executor = executor  # Allow clean up in fail_on_deadlock
+            f = executor.submit(_crash, delay=.1)
+            executor.shutdown(wait=True)
+            with self.assertRaises(BrokenProcessPool):
+                f.result()
+
+
+create_executor_tests(ExecutorDeadlockTest,
+                      executor_mixins=(ProcessPoolForkMixin,
+                                       ProcessPoolForkserverMixin,
+                                       ProcessPoolSpawnMixin))
+
 
 class FutureTests(BaseTestCase):
     def test_done_callback_with_result(self):
diff --git a/Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst b/Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst
new file mode 100644
index 00000000000..49cbbb3b920
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst
@@ -0,0 +1,4 @@
+Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when
+task arguments or results cause pickling or unpickling errors.
+This should make sure that calls to the :class:`ProcessPoolExecutor` API
+always eventually return.


From webhook-mailer at python.org  Fri Jan  5 07:02:05 2018
From: webhook-mailer at python.org (xdegaye)
Date: Fri, 05 Jan 2018 12:02:05 -0000
Subject: [Python-checkins] bpo-32390: Fix compilation failure on AIX after
 f_fsid was added to os.statvfs() (#4972)
Message-ID: <mailman.3.1515153727.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/502d551c6d782963d26957a9e5ff1588946f233f
commit: 502d551c6d782963d26957a9e5ff1588946f233f
branch: master
author: Michael Felt <aixtools at users.noreply.github.com>
committer: xdegaye <xdegaye at gmail.com>
date: 2018-01-05T13:01:58+01:00
summary:

bpo-32390: Fix compilation failure on AIX after f_fsid was added to os.statvfs() (#4972)

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst
M Modules/posixmodule.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst
new file mode 100644
index 00000000000..0e9c1bff540
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst	
@@ -0,0 +1 @@
+Fix the compilation failure on AIX after the f_fsid field has been added to the object returned by os.statvfs() (issue #32143). Original patch by Michael Felt.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 47b79fcc798..b0e48dabbd5 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -9336,7 +9336,13 @@ _pystatvfs_fromstructstatvfs(struct statvfs st) {
     PyStructSequence_SET_ITEM(v, 8, PyLong_FromLong((long) st.f_flag));
     PyStructSequence_SET_ITEM(v, 9, PyLong_FromLong((long) st.f_namemax));
 #endif
+/* The _ALL_SOURCE feature test macro defines f_fsid as a structure
+ * (issue #32390). */
+#if defined(_AIX) && defined(_ALL_SOURCE)
+    PyStructSequence_SET_ITEM(v, 10, PyLong_FromUnsignedLong(st.f_fsid.val[0]));
+#else
     PyStructSequence_SET_ITEM(v, 10, PyLong_FromUnsignedLong(st.f_fsid));
+#endif
     if (PyErr_Occurred()) {
         Py_DECREF(v);
         return NULL;


From lp_benchmark_robot at intel.com  Fri Jan  5 19:21:56 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Fri, 5 Jan 2018 16:21:56 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-05
Message-ID: <51d53f93-fbbd-4b69-a4b4-50d75f9b9ec9@orsmsx102.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-05 03:04:11-08:00.
  - commit:		94459fd
  - previous commit:	78e24d4
  - revision date:	2018-01-05 11:15:54+01:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.864% |  -1.048%   |  +7.068%   |  +7.453%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 0.637% |  -1.039%   |  +21.662%  |  +11.220%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 0.371% |  -1.213%   |  +23.821%  |  +9.994%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.450% |  -1.248%   |  +21.403%  |  +9.771%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 3.279% |  -1.787%   |  +7.706%   |  +13.773%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.758% |  -0.181%   |  +9.314%   |  +12.805%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.560% |  -1.177%   |  +4.821%   |  +12.284%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.716% |  +0.006%   |  -0.718%   |  +8.480%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 2.442% |  -2.020%   |  +9.631%   |  +17.588%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.758% |  -1.983%   |  +12.224%  |  +15.307%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.234% |  -0.081%   |  +3.339%   |  +6.863%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.666% |  -0.021%   |  +4.655%   |  +5.399%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.697% |  +0.221%   |  +2.602%   |  +7.150%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.180% |  -0.216%   |  +12.043%  |  +11.046%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.396% |  +1.045%   |  +8.979%   |  +10.103%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.933% |  +0.173%   |  +6.379%   |  +11.780%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 1.287% |  -0.586%   |  +9.973%   |  +11.274%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.763% |  -0.993%   |  +6.239%   |  +11.801%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.987% |  +0.726%   |  +2.872%   |  +10.298%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 1.232% |  -0.382%   |  -0.566%   |  +13.569%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.543% |  +0.474%   |  +8.504%   |  +10.989%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.740% |  -0.629%   |  +46.843%  |  +12.828%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.657% |  +0.213%   |  +9.415%   |  +12.412%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.802% |  +0.371%   |  +17.429%  |  +13.394%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.885% |  -0.002%   |  +11.730%  |  +6.939%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 1.807% |  -1.723%   |  +2.417%   |  +6.687%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.795% |  -1.184%   |  -5.068%   |  +5.783%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 2.168% |  +0.020%   |  +4.328%   |  +8.456%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.338% |  +0.315%   |  +1.034%   |  +12.977%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.138% |  -0.351%   |  +0.929%   |  +19.859%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.551% |  -0.350%   |  +1.728%   |  +20.824%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.677% |  -0.266%   |  +3.487%   |  +21.061%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 1.090% |  +0.907%   |  +12.046%  |  +10.328%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.252% |  +0.041%   |  -0.039%   |  +10.884%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.143% |  +0.038%   |  +10.729%  |  +5.242%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.115% |  +0.039%   |  +2.063%   |  +5.290%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.782% |  -0.145%   |  +10.305%  |  +13.857%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.997% |  -3.427%   |  +1.418%   |  +13.637%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.533% |  +1.468%   |  -0.985%   |  +9.388%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.198% |  +1.173%   |  -4.058%   |  +1.978%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 2.424% |  +1.405%   |  +5.133%   |  +4.866%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.219% |  -0.091%   |  +8.727%   |  +14.729%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.843% |  -0.576%   |  -3.083%   |  +5.794%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.655% |  -1.013%   |  +25.971%  |  +10.630%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.279% |  +0.715%   |  +4.874%   |  +6.938%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.303% |  +0.081%   |  +14.374%  |  +9.510%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 1.143% |  -2.008%   |  -5.470%   |  -0.788%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.401% |  +0.660%   |  +5.978%   |  +2.501%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.146% |  -1.150%   |  +6.143%   |  +8.080%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.589% |  -2.078%   |  +5.223%   |  +6.500%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 6.116% |  -1.086%   |  +18.443%  |  +10.180%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.945% |  -1.420%   |  +11.129%  |  +7.136%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.250% |  -0.612%   |  +8.325%   |  +8.001%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 4.463% |  -0.862%   |  +11.475%  |  +9.276%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 5.670% |  +1.937%   |  +11.079%  |  +9.735%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 2.877% |  +0.285%   |  +19.522%  |  +11.200%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.235% |  -0.272%   |  +6.069%   |  +7.393%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 7.344% |  -0.834%   |  +1.532%   |  +1.665%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 5.504% |  -0.252%   |  +6.512%   |  +22.397%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 2.451% |  -1.121%   |  -4.920%   |  +19.375%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 1.619% |  +0.054%   |  +8.572%   |  +7.640%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.234% |  -0.707%   |  +1.470%   |  +12.707%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 1.840% |  -0.598%   |  +2.949%   |  +8.183%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 3.556% |  +0.091%   |  -8.284%   |  +13.168%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.287% |  -0.482%   |  +3.955%   |  +12.228%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-05

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Sat Jan  6 02:12:17 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Sat, 06 Jan 2018 07:12:17 -0000
Subject: [Python-checkins] advance copyright years to 2018 (GH-5094) (#5104)
Message-ID: <mailman.4.1515222737.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/e29573910c2c442c39def7396b7a54ae1487e686
commit: e29573910c2c442c39def7396b7a54ae1487e686
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2018-01-05T23:12:14-08:00
summary:

advance copyright years to 2018 (GH-5094) (#5104)

(cherry picked from commit 65f2a6dcc2bc28a8566b74c8e9273f982331ec48)

files:
M Doc/copyright.rst
M Doc/license.rst
M LICENSE
M Mac/IDLE/IDLE.app/Contents/Info.plist
M Mac/PythonLauncher/Info.plist.in
M Mac/Resources/app/Info.plist.in
M Mac/Resources/framework/Info.plist.in
M Python/getcopyright.c
M README.rst

diff --git a/Doc/copyright.rst b/Doc/copyright.rst
index 2b5400cd9fb..540ff5ef059 100644
--- a/Doc/copyright.rst
+++ b/Doc/copyright.rst
@@ -4,7 +4,7 @@ Copyright
 
 Python and this documentation is:
 
-Copyright ? 2001-2017 Python Software Foundation. All rights reserved.
+Copyright ? 2001-2018 Python Software Foundation. All rights reserved.
 
 Copyright ? 2000 BeOpen.com. All rights reserved.
 
diff --git a/Doc/license.rst b/Doc/license.rst
index 49d29cee314..e8dbeb1ef28 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -87,7 +87,7 @@ PSF LICENSE AGREEMENT FOR PYTHON |release|
       analyze, test, perform and/or display publicly, prepare derivative works,
       distribute, and otherwise use Python |release| alone or in any derivative
       version, provided, however, that PSF's License Agreement and PSF's notice of
-      copyright, i.e., "Copyright ? 2001-2017 Python Software Foundation; All Rights
+      copyright, i.e., "Copyright ? 2001-2018 Python Software Foundation; All Rights
       Reserved" are retained in Python |release| alone or in any derivative version
       prepared by Licensee.
 
diff --git a/LICENSE b/LICENSE
index 529349e4b38..1afbedba92b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -73,9 +73,9 @@ analyze, test, perform and/or display publicly, prepare derivative works,
 distribute, and otherwise use Python alone or in any derivative version,
 provided, however, that PSF's License Agreement and PSF's notice of copyright,
 i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
-2011, 2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation; All Rights
-Reserved" are retained in Python alone or in any derivative version prepared by
-Licensee.
+2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation; All
+Rights Reserved" are retained in Python alone or in any derivative version
+prepared by Licensee.
 
 3. In the event Licensee prepares a derivative work that is based on
 or incorporates Python or any part thereof, and wants to make
diff --git a/Mac/IDLE/IDLE.app/Contents/Info.plist b/Mac/IDLE/IDLE.app/Contents/Info.plist
index 5507687e90e..826d3793f75 100644
--- a/Mac/IDLE/IDLE.app/Contents/Info.plist
+++ b/Mac/IDLE/IDLE.app/Contents/Info.plist
@@ -36,7 +36,7 @@
 	<key>CFBundleExecutable</key>
 	<string>IDLE</string>
 	<key>CFBundleGetInfoString</key>
-	<string>%version%, ? 2001-2017 Python Software Foundation</string>
+	<string>%version%, ? 2001-2018 Python Software Foundation</string>
 	<key>CFBundleIconFile</key>
 	<string>IDLE.icns</string>
 	<key>CFBundleIdentifier</key>
diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in
index f1ab79f3d44..5fa346ed4d3 100644
--- a/Mac/PythonLauncher/Info.plist.in
+++ b/Mac/PythonLauncher/Info.plist.in
@@ -40,7 +40,7 @@
 	<key>CFBundleExecutable</key>
 	<string>Python Launcher</string>
 	<key>CFBundleGetInfoString</key>
-	<string>%VERSION%, ? 2001-2017 Python Software Foundation</string>
+	<string>%VERSION%, ? 2001-2018 Python Software Foundation</string>
 	<key>CFBundleIconFile</key>
 	<string>PythonLauncher.icns</string>
 	<key>CFBundleIdentifier</key>
diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in
index a23166e6d32..abe9ae23e34 100644
--- a/Mac/Resources/app/Info.plist.in
+++ b/Mac/Resources/app/Info.plist.in
@@ -37,7 +37,7 @@
 	<key>CFBundleInfoDictionaryVersion</key>
 	<string>6.0</string>
 	<key>CFBundleLongVersionString</key>
-	<string>%version%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%version%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleName</key>
 	<string>Python</string>
 	<key>CFBundlePackageType</key>
diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in
index 7a64619e295..c1ea9f68892 100644
--- a/Mac/Resources/framework/Info.plist.in
+++ b/Mac/Resources/framework/Info.plist.in
@@ -17,9 +17,9 @@
 	<key>CFBundlePackageType</key>
 	<string>FMWK</string>
 	<key>CFBundleShortVersionString</key>
-	<string>%VERSION%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleLongVersionString</key>
-	<string>%VERSION%, (c) 2001-2017 Python Software Foundation.</string>
+	<string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string>
 	<key>CFBundleSignature</key>
 	<string>????</string>
 	<key>CFBundleVersion</key>
diff --git a/Python/getcopyright.c b/Python/getcopyright.c
index cac647c3ce4..51772ecef31 100644
--- a/Python/getcopyright.c
+++ b/Python/getcopyright.c
@@ -4,7 +4,7 @@
 
 static const char cprt[] =
 "\
-Copyright (c) 2001-2017 Python Software Foundation.\n\
+Copyright (c) 2001-2018 Python Software Foundation.\n\
 All Rights Reserved.\n\
 \n\
 Copyright (c) 2000 BeOpen.com.\n\
diff --git a/README.rst b/README.rst
index 05c31da8f2a..30e694e6462 100644
--- a/README.rst
+++ b/README.rst
@@ -14,7 +14,7 @@ This is Python version 3.6.4
    :target: https://codecov.io/gh/python/cpython
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
-2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation.  All rights
+2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation.  All rights
 reserved.
 
 See the end of this file for further copyright and license information.
@@ -232,7 +232,7 @@ Copyright and License Information
 ---------------------------------
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
-2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation.  All rights
+2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation.  All rights
 reserved.
 
 Copyright (c) 2000 BeOpen.com.  All rights reserved.


From webhook-mailer at python.org  Sat Jan  6 02:15:37 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Sat, 06 Jan 2018 07:15:37 -0000
Subject: [Python-checkins] bpo-29137: Remove fpectl module (#4789)
Message-ID: <mailman.5.1515222938.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/735ae8d139a673b30b321dc10acfd3d14f0d633b
commit: 735ae8d139a673b30b321dc10acfd3d14f0d633b
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2018-01-05T23:15:34-08:00
summary:

bpo-29137: Remove fpectl module (#4789)

This module has never been enabled by default, never worked correctly
on x86-64, and caused ABI problems that caused C extension
compatibility. See bpo-29137 for details/discussion.

files:
A Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst
D Doc/library/fpectl.rst
D Modules/fpectlmodule.c
D Modules/fpetestmodule.c
M Doc/library/exceptions.rst
M Doc/library/python.rst
M Doc/license.rst
M Include/pyfpe.h
M Modules/Setup.dist
M Python/pyfpe.c
M aclocal.m4
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
index a6b20a5ac95..c8d32cff4e0 100644
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -154,10 +154,7 @@ The following exceptions are the exceptions that are usually raised.
 
 .. exception:: FloatingPointError
 
-   Raised when a floating point operation fails.  This exception is always defined,
-   but can only be raised when Python is configured with the
-   ``--with-fpectl`` option, or the :const:`WANT_SIGFPE_HANDLER` symbol is
-   defined in the :file:`pyconfig.h` file.
+   Not currently used.
 
 
 .. exception:: GeneratorExit
diff --git a/Doc/library/fpectl.rst b/Doc/library/fpectl.rst
deleted file mode 100644
index 96607165ba4..00000000000
--- a/Doc/library/fpectl.rst
+++ /dev/null
@@ -1,121 +0,0 @@
-:mod:`fpectl` --- Floating point exception control
-==================================================
-
-.. module:: fpectl
-   :platform: Unix
-   :synopsis: Provide control for floating point exception handling.
-
-.. moduleauthor:: Lee Busby <busby1 at llnl.gov>
-.. sectionauthor:: Lee Busby <busby1 at llnl.gov>
-
-.. note::
-
-   The :mod:`fpectl` module is not built by default, and its usage is discouraged
-   and may be dangerous except in the hands of experts.  See also the section
-   :ref:`fpectl-limitations` on limitations for more details.
-
-.. index:: single: IEEE-754
-
---------------
-
-Most computers carry out floating point operations in conformance with the
-so-called IEEE-754 standard. On any real computer, some floating point
-operations produce results that cannot be expressed as a normal floating point
-value. For example, try ::
-
-   >>> import math
-   >>> math.exp(1000)
-   inf
-   >>> math.exp(1000) / math.exp(1000)
-   nan
-
-(The example above will work on many platforms. DEC Alpha may be one exception.)
-"Inf" is a special, non-numeric value in IEEE-754 that stands for "infinity",
-and "nan" means "not a number." Note that, other than the non-numeric results,
-nothing special happened when you asked Python to carry out those calculations.
-That is in fact the default behaviour prescribed in the IEEE-754 standard, and
-if it works for you, stop reading now.
-
-In some circumstances, it would be better to raise an exception and stop
-processing at the point where the faulty operation was attempted. The
-:mod:`fpectl` module is for use in that situation. It provides control over
-floating point units from several hardware manufacturers, allowing the user to
-turn on the generation of :const:`SIGFPE` whenever any of the IEEE-754
-exceptions Division by Zero, Overflow, or Invalid Operation occurs. In tandem
-with a pair of wrapper macros that are inserted into the C code comprising your
-python system, :const:`SIGFPE` is trapped and converted into the Python
-:exc:`FloatingPointError` exception.
-
-The :mod:`fpectl` module defines the following functions and may raise the given
-exception:
-
-
-.. function:: turnon_sigfpe()
-
-   Turn on the generation of :const:`SIGFPE`, and set up an appropriate signal
-   handler.
-
-
-.. function:: turnoff_sigfpe()
-
-   Reset default handling of floating point exceptions.
-
-
-.. exception:: FloatingPointError
-
-   After :func:`turnon_sigfpe` has been executed, a floating point operation that
-   raises one of the IEEE-754 exceptions Division by Zero, Overflow, or Invalid
-   operation will in turn raise this standard Python exception.
-
-
-.. _fpectl-example:
-
-Example
--------
-
-The following example demonstrates how to start up and test operation of the
-:mod:`fpectl` module. ::
-
-   >>> import fpectl
-   >>> import fpetest
-   >>> fpectl.turnon_sigfpe()
-   >>> fpetest.test()
-   overflow        PASS
-   FloatingPointError: Overflow
-
-   div by 0        PASS
-   FloatingPointError: Division by zero
-     [ more output from test elided ]
-   >>> import math
-   >>> math.exp(1000)
-   Traceback (most recent call last):
-     File "<stdin>", line 1, in <module>
-   FloatingPointError: in math_1
-
-
-.. _fpectl-limitations:
-
-Limitations and other considerations
-------------------------------------
-
-Setting up a given processor to trap IEEE-754 floating point errors currently
-requires custom code on a per-architecture basis. You may have to modify
-:mod:`fpectl` to control your particular hardware.
-
-Conversion of an IEEE-754 exception to a Python exception requires that the
-wrapper macros ``PyFPE_START_PROTECT`` and ``PyFPE_END_PROTECT`` be inserted
-into your code in an appropriate fashion.  Python itself has been modified to
-support the :mod:`fpectl` module, but many other codes of interest to numerical
-analysts have not.
-
-The :mod:`fpectl` module is not thread-safe.
-
-
-.. seealso::
-
-   Some files in the source distribution may be interesting in learning more about
-   how this module operates. The include file :file:`Include/pyfpe.h` discusses the
-   implementation of this module at some length. :file:`Modules/fpetestmodule.c`
-   gives several examples of use. Many additional examples can be found in
-   :file:`Objects/floatobject.c`.
-
diff --git a/Doc/library/python.rst b/Doc/library/python.rst
index f307d7db6db..440dc6632b8 100644
--- a/Doc/library/python.rst
+++ b/Doc/library/python.rst
@@ -24,4 +24,3 @@ overview:
    gc.rst
    inspect.rst
    site.rst
-   fpectl.rst
diff --git a/Doc/license.rst b/Doc/license.rst
index aa4b1450e91..5f78240002f 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -349,48 +349,6 @@ Project, http://www.wide.ad.jp/. ::
    SUCH DAMAGE.
 
 
-Floating point exception control
---------------------------------
-
-The source for the :mod:`fpectl` module includes the following notice::
-
-     ---------------------------------------------------------------------
-    /                       Copyright (c) 1996.                           \
-   |          The Regents of the University of California.                 |
-   |                        All rights reserved.                           |
-   |                                                                       |
-   |   Permission to use, copy, modify, and distribute this software for   |
-   |   any purpose without fee is hereby granted, provided that this en-   |
-   |   tire notice is included in all copies of any software which is or   |
-   |   includes  a  copy  or  modification  of  this software and in all   |
-   |   copies of the supporting documentation for such software.           |
-   |                                                                       |
-   |   This  work was produced at the University of California, Lawrence   |
-   |   Livermore National Laboratory under  contract  no.  W-7405-ENG-48   |
-   |   between  the  U.S.  Department  of  Energy and The Regents of the   |
-   |   University of California for the operation of UC LLNL.              |
-   |                                                                       |
-   |                              DISCLAIMER                               |
-   |                                                                       |
-   |   This  software was prepared as an account of work sponsored by an   |
-   |   agency of the United States Government. Neither the United States   |
-   |   Government  nor the University of California nor any of their em-   |
-   |   ployees, makes any warranty, express or implied, or  assumes  any   |
-   |   liability  or  responsibility  for the accuracy, completeness, or   |
-   |   usefulness of any information,  apparatus,  product,  or  process   |
-   |   disclosed,   or  represents  that  its  use  would  not  infringe   |
-   |   privately-owned rights. Reference herein to any specific  commer-   |
-   |   cial  products,  process,  or  service  by trade name, trademark,   |
-   |   manufacturer, or otherwise, does not  necessarily  constitute  or   |
-   |   imply  its endorsement, recommendation, or favoring by the United   |
-   |   States Government or the University of California. The views  and   |
-   |   opinions  of authors expressed herein do not necessarily state or   |
-   |   reflect those of the United States Government or  the  University   |
-   |   of  California,  and shall not be used for advertising or product   |
-    \  endorsement purposes.                                              /
-     ---------------------------------------------------------------------
-
-
 Asynchronous socket services
 ----------------------------
 
diff --git a/Include/pyfpe.h b/Include/pyfpe.h
index d86cb7486ce..5a99e397931 100644
--- a/Include/pyfpe.h
+++ b/Include/pyfpe.h
@@ -1,176 +1,12 @@
 #ifndef Py_PYFPE_H
 #define Py_PYFPE_H
-#ifdef __cplusplus
-extern "C" {
-#endif
-/*
-     ---------------------------------------------------------------------
-    /                       Copyright (c) 1996.                           \
-   |          The Regents of the University of California.                 |
-   |                        All rights reserved.                           |
-   |                                                                       |
-   |   Permission to use, copy, modify, and distribute this software for   |
-   |   any purpose without fee is hereby granted, provided that this en-   |
-   |   tire notice is included in all copies of any software which is or   |
-   |   includes  a  copy  or  modification  of  this software and in all   |
-   |   copies of the supporting documentation for such software.           |
-   |                                                                       |
-   |   This  work was produced at the University of California, Lawrence   |
-   |   Livermore National Laboratory under  contract  no.  W-7405-ENG-48   |
-   |   between  the  U.S.  Department  of  Energy and The Regents of the   |
-   |   University of California for the operation of UC LLNL.              |
-   |                                                                       |
-   |                              DISCLAIMER                               |
-   |                                                                       |
-   |   This  software was prepared as an account of work sponsored by an   |
-   |   agency of the United States Government. Neither the United States   |
-   |   Government  nor the University of California nor any of their em-   |
-   |   ployees, makes any warranty, express or implied, or  assumes  any   |
-   |   liability  or  responsibility  for the accuracy, completeness, or   |
-   |   usefulness of any information,  apparatus,  product,  or  process   |
-   |   disclosed,   or  represents  that  its  use  would  not  infringe   |
-   |   privately-owned rights. Reference herein to any specific  commer-   |
-   |   cial  products,  process,  or  service  by trade name, trademark,   |
-   |   manufacturer, or otherwise, does not  necessarily  constitute  or   |
-   |   imply  its endorsement, recommendation, or favoring by the United   |
-   |   States Government or the University of California. The views  and   |
-   |   opinions  of authors expressed herein do not necessarily state or   |
-   |   reflect those of the United States Government or  the  University   |
-   |   of  California,  and shall not be used for advertising or product   |
-    \  endorsement purposes.                                              /
-     ---------------------------------------------------------------------
-*/
 
-/*
- *       Define macros for handling SIGFPE.
- *       Lee Busby, LLNL, November, 1996
- *       busby1 at llnl.gov
- *
- *********************************************
- * Overview of the system for handling SIGFPE:
- *
- * This file (Include/pyfpe.h) defines a couple of "wrapper" macros for
- * insertion into your Python C code of choice. Their proper use is
- * discussed below. The file Python/pyfpe.c defines a pair of global
- * variables PyFPE_jbuf and PyFPE_counter which are used by the signal
- * handler for SIGFPE to decide if a particular exception was protected
- * by the macros. The signal handler itself, and code for enabling the
- * generation of SIGFPE in the first place, is in a (new) Python module
- * named fpectl. This module is standard in every respect. It can be loaded
- * either statically or dynamically as you choose, and like any other
- * Python module, has no effect until you import it.
- *
- * In the general case, there are three steps toward handling SIGFPE in any
- * Python code:
- *
- * 1) Add the *_PROTECT macros to your C code as required to protect
- *    dangerous floating point sections.
- *
- * 2) Turn on the inclusion of the code by adding the ``--with-fpectl''
- *    flag at the time you run configure.  If the fpectl or other modules
- *    which use the *_PROTECT macros are to be dynamically loaded, be
- *    sure they are compiled with WANT_SIGFPE_HANDLER defined.
- *
- * 3) When python is built and running, import fpectl, and execute
- *    fpectl.turnon_sigfpe(). This sets up the signal handler and enables
- *    generation of SIGFPE whenever an exception occurs. From this point
- *    on, any properly trapped SIGFPE should result in the Python
- *    FloatingPointError exception.
- *
- * Step 1 has been done already for the Python kernel code, and should be
- * done soon for the NumPy array package.  Step 2 is usually done once at
- * python install time. Python's behavior with respect to SIGFPE is not
- * changed unless you also do step 3. Thus you can control this new
- * facility at compile time, or run time, or both.
- *
- ********************************
- * Using the macros in your code:
- *
- * static PyObject *foobar(PyObject *self,PyObject *args)
- * {
- *     ....
- *     PyFPE_START_PROTECT("Error in foobar", return 0)
- *     result = dangerous_op(somearg1, somearg2, ...);
- *     PyFPE_END_PROTECT(result)
- *     ....
- * }
- *
- * If a floating point error occurs in dangerous_op, foobar returns 0 (NULL),
- * after setting the associated value of the FloatingPointError exception to
- * "Error in foobar". ``Dangerous_op'' can be a single operation, or a block
- * of code, function calls, or any combination, so long as no alternate
- * return is possible before the PyFPE_END_PROTECT macro is reached.
- *
- * The macros can only be used in a function context where an error return
- * can be recognized as signaling a Python exception. (Generally, most
- * functions that return a PyObject * will qualify.)
- *
- * Guido's original design suggestion for PyFPE_START_PROTECT and
- * PyFPE_END_PROTECT had them open and close a local block, with a locally
- * defined jmp_buf and jmp_buf pointer. This would allow recursive nesting
- * of the macros. The Ansi C standard makes it clear that such local
- * variables need to be declared with the "volatile" type qualifier to keep
- * setjmp from corrupting their values. Some current implementations seem
- * to be more restrictive. For example, the HPUX man page for setjmp says
- *
- *   Upon the return from a setjmp() call caused by a longjmp(), the
- *   values of any non-static local variables belonging to the routine
- *   from which setjmp() was called are undefined. Code which depends on
- *   such values is not guaranteed to be portable.
- *
- * I therefore decided on a more limited form of nesting, using a counter
- * variable (PyFPE_counter) to keep track of any recursion.  If an exception
- * occurs in an ``inner'' pair of macros, the return will apparently
- * come from the outermost level.
- *
+/* These macros used to do something when Python was built with --with-fpectl,
+ * but support for that was dropped in 3.7. We continue to define them though,
+ * to avoid breaking API users.
  */
 
-#ifdef WANT_SIGFPE_HANDLER
-#include <signal.h>
-#include <setjmp.h>
-#include <math.h>
-extern jmp_buf PyFPE_jbuf;
-extern int PyFPE_counter;
-extern double PyFPE_dummy(void *);
-
-#define PyFPE_START_PROTECT(err_string, leave_stmt) \
-if (!PyFPE_counter++ && setjmp(PyFPE_jbuf)) { \
-        PyErr_SetString(PyExc_FloatingPointError, err_string); \
-        PyFPE_counter = 0; \
-        leave_stmt; \
-}
-
-/*
- * This (following) is a heck of a way to decrement a counter. However,
- * unless the macro argument is provided, code optimizers will sometimes move
- * this statement so that it gets executed *before* the unsafe expression
- * which we're trying to protect.  That pretty well messes things up,
- * of course.
- *
- * If the expression(s) you're trying to protect don't happen to return a
- * value, you will need to manufacture a dummy result just to preserve the
- * correct ordering of statements.  Note that the macro passes the address
- * of its argument (so you need to give it something which is addressable).
- * If your expression returns multiple results, pass the last such result
- * to PyFPE_END_PROTECT.
- *
- * Note that PyFPE_dummy returns a double, which is cast to int.
- * This seeming insanity is to tickle the Floating Point Unit (FPU).
- * If an exception has occurred in a preceding floating point operation,
- * some architectures (notably Intel 80x86) will not deliver the interrupt
- * until the *next* floating point operation.  This is painful if you've
- * already decremented PyFPE_counter.
- */
-#define PyFPE_END_PROTECT(v) PyFPE_counter -= (int)PyFPE_dummy(&(v));
-
-#else
-
 #define PyFPE_START_PROTECT(err_string, leave_stmt)
 #define PyFPE_END_PROTECT(v)
 
-#endif
-
-#ifdef __cplusplus
-}
-#endif
 #endif /* !Py_PYFPE_H */
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst b/Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst
new file mode 100644
index 00000000000..85f3df5c8f5
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst
@@ -0,0 +1,3 @@
+The ``fpectl`` library has been removed. It was never enabled by default,
+never worked correctly on x86-64, and it changed the Python ABI in ways that
+caused unexpected breakage of C extensions.
diff --git a/Modules/Setup.dist b/Modules/Setup.dist
index 9c3e1bed30c..1f2d56c0655 100644
--- a/Modules/Setup.dist
+++ b/Modules/Setup.dist
@@ -334,20 +334,6 @@ _symtable symtablemodule.c
 #parser parsermodule.c
 
 
-# Lee Busby's SIGFPE modules.
-# The library to link fpectl with is platform specific.
-# Choose *one* of the options below for fpectl:
-
-# For Solaris with SunPro compiler (tested on Solaris 2.5 with SunPro C 4.2):
-# (Without the compiler you don't have -lsunmath.)
-#fpectl fpectlmodule.c -R/opt/SUNWspro/lib -lsunmath -lm
-
-# For other systems: see instructions in fpectlmodule.c.
-#fpectl fpectlmodule.c ...
-
-# Test module for fpectl.  No extra libraries needed.
-#fpetest fpetestmodule.c
-
 # Andrew Kuchling's zlib module.
 # This require zlib 1.1.3 (or later).
 # See http://www.gzip.org/zlib/
diff --git a/Modules/fpectlmodule.c b/Modules/fpectlmodule.c
deleted file mode 100644
index 42ef0f6072e..00000000000
--- a/Modules/fpectlmodule.c
+++ /dev/null
@@ -1,255 +0,0 @@
-/*
-     ---------------------------------------------------------------------
-    /                       Copyright (c) 1996.                           \
-   |          The Regents of the University of California.                 |
-   |                        All rights reserved.                           |
-   |                                                                       |
-   |   Permission to use, copy, modify, and distribute this software for   |
-   |   any purpose without fee is hereby granted, provided that this en-   |
-   |   tire notice is included in all copies of any software which is or   |
-   |   includes  a  copy  or  modification  of  this software and in all   |
-   |   copies of the supporting documentation for such software.           |
-   |                                                                       |
-   |   This  work was produced at the University of California, Lawrence   |
-   |   Livermore National Laboratory under  contract  no.  W-7405-ENG-48   |
-   |   between  the  U.S.  Department  of  Energy and The Regents of the   |
-   |   University of California for the operation of UC LLNL.              |
-   |                                                                       |
-   |                              DISCLAIMER                               |
-   |                                                                       |
-   |   This  software was prepared as an account of work sponsored by an   |
-   |   agency of the United States Government. Neither the United States   |
-   |   Government  nor the University of California nor any of their em-   |
-   |   ployees, makes any warranty, express or implied, or  assumes  any   |
-   |   liability  or  responsibility  for the accuracy, completeness, or   |
-   |   usefulness of any information,  apparatus,  product,  or  process   |
-   |   disclosed,   or  represents  that  its  use  would  not  infringe   |
-   |   privately-owned rights. Reference herein to any specific  commer-   |
-   |   cial  products,  process,  or  service  by trade name, trademark,   |
-   |   manufacturer, or otherwise, does not  necessarily  constitute  or   |
-   |   imply  its endorsement, recommendation, or favoring by the United   |
-   |   States Government or the University of California. The views  and   |
-   |   opinions  of authors expressed herein do not necessarily state or   |
-   |   reflect those of the United States Government or  the  University   |
-   |   of  California,  and shall not be used for advertising or product   |
-    \  endorsement purposes.                                              /
-     ---------------------------------------------------------------------
-*/
-
-/*
-                  Floating point exception control module.
-
-   This Python module provides bare-bones control over floating point
-   units from several hardware manufacturers.  Specifically, it allows
-   the user to turn on the generation of SIGFPE whenever any of the
-   three serious IEEE 754 exceptions (Division by Zero, Overflow,
-   Invalid Operation) occurs.  We currently ignore Underflow and
-   Inexact Result exceptions, although those could certainly be added
-   if desired.
-
-   The module also establishes a signal handler for SIGFPE during
-   initialization.  This builds on code found in the Python
-   distribution at Include/pyfpe.h and Python/pyfpe.c.  If those files
-   are not in your Python distribution, find them in a patch at
-   ftp://icf.llnl.gov/pub/python/busby/patches.961108.tgz.
-
-   This module is only useful to you if it happens to include code
-   specific for your hardware and software environment.  If you can
-   contribute OS-specific code for new platforms, or corrections for
-   the code provided, it will be greatly appreciated.
-
-   ** Version 1.0: September 20, 1996.  Lee Busby, LLNL.
- */
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-#include "Python.h"
-#include <signal.h>
-
-#if defined(__FreeBSD__)
-#  include <ieeefp.h>
-#endif
-
-#ifndef WANT_SIGFPE_HANDLER
-/* Define locally if they are not defined in Python.  This gives only
- * the limited control to induce a core dump in case of an exception.
- */
-#include <setjmp.h>
-static jmp_buf PyFPE_jbuf;
-static int PyFPE_counter = 0;
-#endif
-
-typedef void Sigfunc(int);
-static Sigfunc sigfpe_handler;
-static void fpe_reset(Sigfunc *);
-
-static PyObject *fpe_error;
-
-PyMODINIT_FUNC PyInit_fpectl(void);
-static PyObject *turnon_sigfpe            (PyObject *self,PyObject *args);
-static PyObject *turnoff_sigfpe           (PyObject *self,PyObject *args);
-
-static PyMethodDef fpectl_methods[] = {
-    {"turnon_sigfpe",            (PyCFunction) turnon_sigfpe,            METH_VARARGS},
-    {"turnoff_sigfpe",           (PyCFunction) turnoff_sigfpe,           METH_VARARGS},
-    {0,0}
-};
-
-static PyObject *turnon_sigfpe(PyObject *self,PyObject *args)
-{
-    /* Do any architecture-specific one-time only initialization here. */
-
-    fpe_reset(sigfpe_handler);
-    Py_RETURN_NONE;
-}
-
-static void fpe_reset(Sigfunc *handler)
-{
-    /* Reset the exception handling machinery, and reset the signal
-     * handler for SIGFPE to the given handler.
-     */
-
-/*-- SunOS and Solaris ----------------------------------------------------*/
-#if defined(sun)
-    /* References: ieee_handler, ieee_sun, ieee_functions, and ieee_flags
-       man pages (SunOS or Solaris)
-       cc -c -I/usr/local/python/include fpectlmodule.c
-       ld -G -o fpectlmodule.so -L/opt/SUNWspro/lib fpectlmodule.o -lsunmath -lm
-     */
-#include <math.h>
-#ifndef _SUNMATH_H
-    extern void nonstandard_arithmetic(void);
-    extern int ieee_flags(const char*, const char*, const char*, char **);
-    extern long ieee_handler(const char*, const char*, sigfpe_handler_type);
-#endif
-
-    const char *mode="exception", *in="all";
-    char *out;
-    (void) nonstandard_arithmetic();
-    (void) ieee_flags("clearall",mode,in,&out);
-    (void) ieee_handler("set","common",(sigfpe_handler_type)handler);
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- HPUX -----------------------------------------------------------------*/
-#elif defined(__hppa) || defined(hppa)
-    /* References:   fpsetmask man page */
-    /* cc -Aa +z -c -I/usr/local/python/include fpectlmodule.c */
-    /* ld -b -o fpectlmodule.sl fpectlmodule.o -lm */
-#include <math.h>
-    fpsetdefaults();
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- IBM AIX --------------------------------------------------------------*/
-#elif defined(__AIX) || defined(_AIX)
-    /* References:   fp_trap, fp_enable man pages */
-#include <fptrap.h>
-    fp_trap(FP_TRAP_SYNC);
-    fp_enable(TRP_INVALID | TRP_DIV_BY_ZERO | TRP_OVERFLOW);
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- DEC ALPHA LINUX ------------------------------------------------------*/
-#elif defined(__alpha) && defined(linux)
-#include <asm/fpu.h>
-    unsigned long fp_control =
-    IEEE_TRAP_ENABLE_INV | IEEE_TRAP_ENABLE_DZE | IEEE_TRAP_ENABLE_OVF;
-    ieee_set_fp_control(fp_control);
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- Cray Unicos ----------------------------------------------------------*/
-#elif defined(cray)
-    /* UNICOS delivers SIGFPE by default, but no matherr */
-#ifdef HAS_LIBMSET
-    libmset(-1);
-#endif
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- FreeBSD ----------------------------------------------------------------*/
-#elif defined(__FreeBSD__)
-    fpresetsticky(fpgetsticky());
-    fpsetmask(FP_X_INV | FP_X_DZ | FP_X_OFL);
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- Linux ----------------------------------------------------------------*/
-#elif defined(linux)
-#ifdef __GLIBC__
-#include <fpu_control.h>
-#else
-#include <i386/fpu_control.h>
-#endif
-#ifdef _FPU_SETCW
-    {
-        fpu_control_t cw = 0x1372;
-        _FPU_SETCW(cw);
-    }
-#else
-    __setfpucw(0x1372);
-#endif
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- Microsoft Windows, NT ------------------------------------------------*/
-#elif defined(_MSC_VER)
-    /* Reference: Visual C++ Books Online 4.2,
-       Run-Time Library Reference, _control87, _controlfp */
-#include <float.h>
-    unsigned int cw = _EM_INVALID | _EM_ZERODIVIDE | _EM_OVERFLOW;
-    (void)_controlfp(0, cw);
-    PyOS_setsig(SIGFPE, handler);
-
-/*-- Give Up --------------------------------------------------------------*/
-#else
-    fputs("Operation not implemented\n", stderr);
-#endif
-
-}
-
-static PyObject *turnoff_sigfpe(PyObject *self,PyObject *args)
-{
-#ifdef __FreeBSD__
-    fpresetsticky(fpgetsticky());
-    fpsetmask(0);
-#else
-    fputs("Operation not implemented\n", stderr);
-#endif
-    Py_RETURN_NONE;
-}
-
-static void sigfpe_handler(int signo)
-{
-    fpe_reset(sigfpe_handler);
-    if(PyFPE_counter) {
-        longjmp(PyFPE_jbuf, 1);
-    } else {
-        Py_FatalError("Unprotected floating point exception");
-    }
-}
-
-static struct PyModuleDef fpectlmodule = {
-        PyModuleDef_HEAD_INIT,
-        "fpectl",
-        NULL,
-        -1,
-        fpectl_methods,
-        NULL,
-        NULL,
-        NULL,
-        NULL
-};
-
-PyMODINIT_FUNC PyInit_fpectl(void)
-{
-    PyObject *m, *d;
-    m = PyModule_Create(&fpectlmodule);
-    if (m == NULL)
-        return NULL;
-    d = PyModule_GetDict(m);
-    fpe_error = PyErr_NewException("fpectl.error", NULL, NULL);
-    if (fpe_error != NULL)
-        PyDict_SetItemString(d, "error", fpe_error);
-    return m;
-}
-
-#ifdef __cplusplus
-}
-#endif
diff --git a/Modules/fpetestmodule.c b/Modules/fpetestmodule.c
deleted file mode 100644
index 1c232c98316..00000000000
--- a/Modules/fpetestmodule.c
+++ /dev/null
@@ -1,199 +0,0 @@
-/*
-     ---------------------------------------------------------------------
-    /                       Copyright (c) 1996.                           \
-   |          The Regents of the University of California.                 |
-   |                        All rights reserved.                           |
-   |                                                                       |
-   |   Permission to use, copy, modify, and distribute this software for   |
-   |   any purpose without fee is hereby granted, provided that this en-   |
-   |   tire notice is included in all copies of any software which is or   |
-   |   includes  a  copy  or  modification  of  this software and in all   |
-   |   copies of the supporting documentation for such software.           |
-   |                                                                       |
-   |   This  work was produced at the University of California, Lawrence   |
-   |   Livermore National Laboratory under  contract  no.  W-7405-ENG-48   |
-   |   between  the  U.S.  Department  of  Energy and The Regents of the   |
-   |   University of California for the operation of UC LLNL.              |
-   |                                                                       |
-   |                              DISCLAIMER                               |
-   |                                                                       |
-   |   This  software was prepared as an account of work sponsored by an   |
-   |   agency of the United States Government. Neither the United States   |
-   |   Government  nor the University of California nor any of their em-   |
-   |   ployees, makes any warranty, express or implied, or  assumes  any   |
-   |   liability  or  responsibility  for the accuracy, completeness, or   |
-   |   usefulness of any information,  apparatus,  product,  or  process   |
-   |   disclosed,   or  represents  that  its  use  would  not  infringe   |
-   |   privately-owned rights. Reference herein to any specific  commer-   |
-   |   cial  products,  process,  or  service  by trade name, trademark,   |
-   |   manufacturer, or otherwise, does not  necessarily  constitute  or   |
-   |   imply  its endorsement, recommendation, or favoring by the United   |
-   |   States Government or the University of California. The views  and   |
-   |   opinions  of authors expressed herein do not necessarily state or   |
-   |   reflect those of the United States Government or  the  University   |
-   |   of  California,  and shall not be used for advertising or product   |
-    \  endorsement purposes.                                              /
-     ---------------------------------------------------------------------
-*/
-
-/*
-                  Floating point exception test module.
-
- */
-
-#include "Python.h"
-
-static PyObject *fpe_error;
-
-PyMODINIT_FUNC PyInit_fpetest(void);
-static PyObject *test(PyObject *self,PyObject *args);
-static double db0(double);
-static double overflow(double);
-static double nest1(int, double);
-static double nest2(int, double);
-static double nest3(double);
-static void printerr(double);
-
-static PyMethodDef fpetest_methods[] = {
-    {"test",             (PyCFunction) test,             METH_VARARGS},
-    {0,0}
-};
-
-static PyObject *test(PyObject *self,PyObject *args)
-{
-    double r;
-
-    fprintf(stderr,"overflow");
-    r = overflow(1.e160);
-    printerr(r);
-
-    fprintf(stderr,"\ndiv by 0");
-    r = db0(0.0);
-    printerr(r);
-
-    fprintf(stderr,"\nnested outer");
-    r = nest1(0, 0.0);
-    printerr(r);
-
-    fprintf(stderr,"\nnested inner");
-    r = nest1(1, 1.0);
-    printerr(r);
-
-    fprintf(stderr,"\ntrailing outer");
-    r = nest1(2, 2.0);
-    printerr(r);
-
-    fprintf(stderr,"\nnested prior");
-    r = nest2(0, 0.0);
-    printerr(r);
-
-    fprintf(stderr,"\nnested interior");
-    r = nest2(1, 1.0);
-    printerr(r);
-
-    fprintf(stderr,"\nnested trailing");
-    r = nest2(2, 2.0);
-    printerr(r);
-
-    Py_RETURN_NONE;
-}
-
-static void printerr(double r)
-{
-    if(r == 3.1416){
-      fprintf(stderr,"\tPASS\n");
-      PyErr_Print();
-    }else{
-      fprintf(stderr,"\tFAIL\n");
-    }
-    PyErr_Clear();
-}
-
-static double nest1(int i, double x)
-{
-  double a = 1.0;
-
-  PyFPE_START_PROTECT("Division by zero, outer zone", return 3.1416)
-  if(i == 0){
-    a = 1./x;
-  }else if(i == 1){
-    /* This (following) message is never seen. */
-    PyFPE_START_PROTECT("Division by zero, inner zone", return 3.1416)
-    a = 1./(1. - x);
-    PyFPE_END_PROTECT(a)
-  }else if(i == 2){
-    a = 1./(2. - x);
-  }
-  PyFPE_END_PROTECT(a)
-
-  return a;
-}
-
-static double nest2(int i, double x)
-{
-  double a = 1.0;
-  PyFPE_START_PROTECT("Division by zero, prior error", return 3.1416)
-  if(i == 0){
-    a = 1./x;
-  }else if(i == 1){
-    a = nest3(x);
-  }else if(i == 2){
-    a = 1./(2. - x);
-  }
-  PyFPE_END_PROTECT(a)
-  return a;
-}
-
-static double nest3(double x)
-{
-  double result;
-  /* This (following) message is never seen. */
-  PyFPE_START_PROTECT("Division by zero, nest3 error", return 3.1416)
-  result = 1./(1. - x);
-  PyFPE_END_PROTECT(result)
-  return result;
-}
-
-static double db0(double x)
-{
-  double a;
-  PyFPE_START_PROTECT("Division by zero", return 3.1416)
-  a = 1./x;
-  PyFPE_END_PROTECT(a)
-  return a;
-}
-
-static double overflow(double b)
-{
-  double a;
-  PyFPE_START_PROTECT("Overflow", return 3.1416)
-  a = b*b;
-  PyFPE_END_PROTECT(a)
-  return a;
-}
-
-static struct PyModuleDef fpetestmodule = {
-    PyModuleDef_HEAD_INIT,
-    "fpetest",
-    NULL,
-    -1,
-    fpetest_methods,
-    NULL,
-    NULL,
-    NULL,
-    NULL
-};
-
-PyMODINIT_FUNC PyInit_fpetest(void)
-{
-    PyObject *m, *d;
-
-    m = PyModule_Create(&fpetestmodule);
-    if (m == NULL)
-    return NULL;
-    d = PyModule_GetDict(m);
-    fpe_error = PyErr_NewException("fpetest.error", NULL, NULL);
-    if (fpe_error != NULL)
-        PyDict_SetItemString(d, "error", fpe_error);
-    return m;
-}
diff --git a/Python/pyfpe.c b/Python/pyfpe.c
index 925fa4654d9..31ef5d73b70 100644
--- a/Python/pyfpe.c
+++ b/Python/pyfpe.c
@@ -1,20 +1,12 @@
-#include "pyconfig.h"
-#include "pyfpe.h"
-/*
- * The signal handler for SIGFPE is actually declared in an external
- * module fpectl, or as preferred by the user.  These variable
- * definitions are required in order to compile Python without
- * getting missing externals, but to actually handle SIGFPE requires
- * defining a handler and enabling generation of SIGFPE.
+/* These variables used to be used when Python was built with --with-fpectl,
+ * but support for that was dropped in 3.7. We continue to define them,
+ * though, because they may be referenced by extensions using the stable ABI.
  */
 
-#ifdef WANT_SIGFPE_HANDLER
-jmp_buf PyFPE_jbuf;
-int PyFPE_counter = 0;
-#endif
+#include "setjmp.h"
 
-/* Have this outside the above #ifdef, since some picky ANSI compilers issue a
-   warning when compiling an empty file. */
+jmp_buf PyFPE_jbuf;
+int PyFPE_counter;
 
 double
 PyFPE_dummy(void *dummy)
diff --git a/aclocal.m4 b/aclocal.m4
index 4eb2dc3bbe2..61b2539d2e9 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -1,6 +1,6 @@
-# generated automatically by aclocal 1.15 -*- Autoconf -*-
+# generated automatically by aclocal 1.15.1 -*- Autoconf -*-
 
-# Copyright (C) 1996-2014 Free Software Foundation, Inc.
+# Copyright (C) 1996-2017 Free Software Foundation, Inc.
 
 # This file is free software; the Free Software Foundation
 # gives unlimited permission to copy and/or distribute it,
@@ -13,7 +13,7 @@
 
 m4_ifndef([AC_CONFIG_MACRO_DIRS], [m4_defun([_AM_CONFIG_MACRO_DIRS], [])m4_defun([AC_CONFIG_MACRO_DIRS], [_AM_CONFIG_MACRO_DIRS($@)])])
 dnl pkg.m4 - Macros to locate and utilise pkg-config.   -*- Autoconf -*-
-dnl serial 11 (pkg-config-0.29.1)
+dnl serial 11 (pkg-config-0.29)
 dnl
 dnl Copyright ? 2004 Scott James Remnant <scott at netsplit.com>.
 dnl Copyright ? 2012-2015 Dan Nicholson <dbn.lists at gmail.com>
@@ -55,7 +55,7 @@ dnl
 dnl See the "Since" comment for each macro you use to see what version
 dnl of the macros you require.
 m4_defun([PKG_PREREQ],
-[m4_define([PKG_MACROS_VERSION], [0.29.1])
+[m4_define([PKG_MACROS_VERSION], [0.29])
 m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1,
     [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])])
 ])dnl PKG_PREREQ
@@ -170,7 +170,7 @@ if test $pkg_failed = yes; then
         _PKG_SHORT_ERRORS_SUPPORTED
         if test $_pkg_short_errors_supported = yes; then
 	        $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1`
-        else
+        else 
 	        $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1`
         fi
 	# Put the nasty error message in config.log where it belongs
diff --git a/configure b/configure
index da2c43fe7a2..afffc190751 100755
--- a/configure
+++ b/configure
@@ -832,7 +832,6 @@ with_c_locale_coercion
 with_c_locale_warning
 with_valgrind
 with_dtrace
-with_fpectl
 with_libm
 with_libc
 enable_big_digits
@@ -1539,7 +1538,6 @@ Optional Packages:
                           locale
   --with-valgrind         Enable Valgrind support
   --with(out)-dtrace      disable/enable DTrace support
-  --with-fpectl           enable SIGFPE catching
   --with-libm=STRING      math library
   --with-libc=STRING      C library
   --with(out)-computed-gotos
@@ -13691,29 +13689,6 @@ fi
 fi
 
 
-# Check for --with-fpectl
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-fpectl" >&5
-$as_echo_n "checking for --with-fpectl... " >&6; }
-
-# Check whether --with-fpectl was given.
-if test "${with_fpectl+set}" = set; then :
-  withval=$with_fpectl;
-if test "$withval" != no
-then
-
-$as_echo "#define WANT_SIGFPE_HANDLER 1" >>confdefs.h
-
-  { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
-$as_echo "yes" >&6; }
-else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
-$as_echo "no" >&6; }
-fi
-else
-  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
-$as_echo "no" >&6; }
-fi
-
-
 # check for --with-libm=...
 
 case $ac_sys_system in
diff --git a/configure.ac b/configure.ac
index 9a84e90e34c..0a48e41d5c3 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4135,20 +4135,6 @@ AC_CHECK_FUNC(__fpu_control,
   [AC_CHECK_LIB(ieee, __fpu_control)
 ])
 
-# Check for --with-fpectl
-AC_MSG_CHECKING(for --with-fpectl)
-AC_ARG_WITH(fpectl,
-            AS_HELP_STRING([--with-fpectl], [enable SIGFPE catching]),
-[
-if test "$withval" != no
-then
-  AC_DEFINE(WANT_SIGFPE_HANDLER, 1,
-  [Define if you want SIGFPE handled (see Include/pyfpe.h).])
-  AC_MSG_RESULT(yes)
-else AC_MSG_RESULT(no)
-fi],
-[AC_MSG_RESULT(no)])
-
 # check for --with-libm=...
 AC_SUBST(LIBM)
 case $ac_sys_system in
diff --git a/pyconfig.h.in b/pyconfig.h.in
index 4cf2f035c23..7d366562b0c 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -500,7 +500,7 @@
 /* Define to 1 if you have the <ieeefp.h> header file. */
 #undef HAVE_IEEEFP_H
 
-/* Define to 1 if you have the 'if_nameindex' function. */
+/* Define to 1 if you have the `if_nameindex' function. */
 #undef HAVE_IF_NAMEINDEX
 
 /* Define if you have the 'inet_aton' function. */
@@ -1413,9 +1413,6 @@
 #endif
 
 
-/* Define if you want SIGFPE handled (see Include/pyfpe.h). */
-#undef WANT_SIGFPE_HANDLER
-
 /* Define if WINDOW in curses.h offers a field _flags. */
 #undef WINDOW_HAS_FLAGS
 


From webhook-mailer at python.org  Sat Jan  6 02:53:10 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sat, 06 Jan 2018 07:53:10 -0000
Subject: [Python-checkins] bpo-32206: Pdb can now run modules (GH-4752)
Message-ID: <mailman.6.1515225191.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/9f1e5f1b7f074e026843a5d70834233a95a6bf9d
commit: 9f1e5f1b7f074e026843a5d70834233a95a6bf9d
branch: master
author: Mario Corchero <mariocj89 at gmail.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-06T17:53:05+10:00
summary:

bpo-32206: Pdb can now run modules (GH-4752)

Add a new argument "-m" to the pdb module to allow
users to run `python -m pdb -m my_module_name`.

This relies on private APIs in the runpy module to work,
but we can get away with that since they're both part of
the standard library and can be updated together if
the runpy internals get refactored.

files:
A Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst
M Doc/library/pdb.rst
M Doc/whatsnew/3.7.rst
M Lib/pdb.py
M Lib/test/test_pdb.py

diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst
index 4f3148fb5c3..e81c195782d 100644
--- a/Doc/library/pdb.rst
+++ b/Doc/library/pdb.rst
@@ -61,6 +61,12 @@ useful than quitting the debugger upon program's exit.
    :file:`pdb.py` now accepts a ``-c`` option that executes commands as if given
    in a :file:`.pdbrc` file, see :ref:`debugger-commands`.
 
+.. versionadded:: 3.7
+   :file:`pdb.py` now accepts a ``-m`` option that execute modules similar to the way
+   ``python3 -m`` does. As with a script, the debugger will pause execution just
+   before the first line of the module.
+
+
 The typical usage to break into the debugger from a running program is to
 insert ::
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 1311e9e2016..9785d599dd3 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -426,6 +426,10 @@ pdb
 argument.  If given, this is printed to the console just before debugging
 begins.  (Contributed by Barry Warsaw in :issue:`31389`.)
 
+pdb command line now accepts `-m module_name` as an alternative to
+script file. (Contributed by Mario Corchero in :issue:`32206`.)
+
+
 re
 --
 
diff --git a/Lib/pdb.py b/Lib/pdb.py
index 8dd4dedb220..d1a74bb7d7b 100755
--- a/Lib/pdb.py
+++ b/Lib/pdb.py
@@ -1521,6 +1521,24 @@ def lookupmodule(self, filename):
                 return fullname
         return None
 
+    def _runmodule(self, module_name):
+        self._wait_for_mainpyfile = True
+        self._user_requested_quit = False
+        import runpy
+        mod_name, mod_spec, code = runpy._get_module_details(module_name)
+        self.mainpyfile = self.canonic(code.co_filename)
+        import __main__
+        __main__.__dict__.clear()
+        __main__.__dict__.update({
+            "__name__": "__main__",
+            "__file__": self.mainpyfile,
+            "__package__": module_name,
+            "__loader__": mod_spec.loader,
+            "__spec__": mod_spec,
+            "__builtins__": __builtins__,
+        })
+        self.run(code)
+
     def _runscript(self, filename):
         # The script has to run in __main__ namespace (or imports from
         # __main__ will break).
@@ -1635,29 +1653,33 @@ def help():
 def main():
     import getopt
 
-    opts, args = getopt.getopt(sys.argv[1:], 'hc:', ['--help', '--command='])
+    opts, args = getopt.getopt(sys.argv[1:], 'mhc:', ['--help', '--command='])
 
     if not args:
         print(_usage)
         sys.exit(2)
 
     commands = []
+    run_as_module = False
     for opt, optarg in opts:
         if opt in ['-h', '--help']:
             print(_usage)
             sys.exit()
         elif opt in ['-c', '--command']:
             commands.append(optarg)
+        elif opt in ['-m']:
+            run_as_module = True
 
     mainpyfile = args[0]     # Get script filename
-    if not os.path.exists(mainpyfile):
+    if not run_as_module and not os.path.exists(mainpyfile):
         print('Error:', mainpyfile, 'does not exist')
         sys.exit(1)
 
     sys.argv[:] = args      # Hide "pdb.py" and pdb options from argument list
 
     # Replace pdb's dir with script's dir in front of module search path.
-    sys.path[0] = os.path.dirname(mainpyfile)
+    if not run_as_module:
+        sys.path[0] = os.path.dirname(mainpyfile)
 
     # Note on saving/restoring sys.argv: it's a good idea when sys.argv was
     # modified by the script being debugged. It's a bad idea when it was
@@ -1667,7 +1689,10 @@ def main():
     pdb.rcLines.extend(commands)
     while True:
         try:
-            pdb._runscript(mainpyfile)
+            if run_as_module:
+                pdb._runmodule(mainpyfile)
+            else:
+                pdb._runscript(mainpyfile)
             if pdb._user_requested_quit:
                 break
             print("The program finished and will be restarted")
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 71d8203fc56..0cd235e98ae 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -938,26 +938,47 @@ def test_pdb_issue_20766():
     pdb 2: <built-in function default_int_handler>
     """
 
+
 class PdbTestCase(unittest.TestCase):
+    def tearDown(self):
+        support.unlink(support.TESTFN)
 
-    def run_pdb(self, script, commands):
-        """Run 'script' lines with pdb and the pdb 'commands'."""
-        filename = 'main.py'
-        with open(filename, 'w') as f:
-            f.write(textwrap.dedent(script))
-        self.addCleanup(support.unlink, filename)
+    def _run_pdb(self, pdb_args, commands):
         self.addCleanup(support.rmtree, '__pycache__')
-        cmd = [sys.executable, '-m', 'pdb', filename]
-        stdout = stderr = None
-        with subprocess.Popen(cmd, stdout=subprocess.PIPE,
-                                   stdin=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT,
-                                   ) as proc:
+        cmd = [sys.executable, '-m', 'pdb'] + pdb_args
+        with subprocess.Popen(
+                cmd,
+                stdout=subprocess.PIPE,
+                stdin=subprocess.PIPE,
+                stderr=subprocess.STDOUT,
+        ) as proc:
             stdout, stderr = proc.communicate(str.encode(commands))
         stdout = stdout and bytes.decode(stdout)
         stderr = stderr and bytes.decode(stderr)
         return stdout, stderr
 
+    def run_pdb_script(self, script, commands):
+        """Run 'script' lines with pdb and the pdb 'commands'."""
+        filename = 'main.py'
+        with open(filename, 'w') as f:
+            f.write(textwrap.dedent(script))
+        self.addCleanup(support.unlink, filename)
+        return self._run_pdb([filename], commands)
+
+    def run_pdb_module(self, script, commands):
+        """Runs the script code as part of a module"""
+        self.module_name = 't_main'
+        support.rmtree(self.module_name)
+        main_file = self.module_name + '/__main__.py'
+        init_file = self.module_name + '/__init__.py'
+        os.mkdir(self.module_name)
+        with open(init_file, 'w') as f:
+            pass
+        with open(main_file, 'w') as f:
+            f.write(textwrap.dedent(script))
+        self.addCleanup(support.rmtree, self.module_name)
+        return self._run_pdb(['-m', self.module_name], commands)
+
     def _assert_find_function(self, file_content, func_name, expected):
         file_content = textwrap.dedent(file_content)
 
@@ -1034,7 +1055,7 @@ def bar():
         with open('bar.py', 'w') as f:
             f.write(textwrap.dedent(bar))
         self.addCleanup(support.unlink, 'bar.py')
-        stdout, stderr = self.run_pdb(script, commands)
+        stdout, stderr = self.run_pdb_script(script, commands)
         self.assertTrue(
             any('main.py(5)foo()->None' in l for l in stdout.splitlines()),
             'Fail to step into the caller after a return')
@@ -1071,7 +1092,7 @@ def test_issue16180(self):
         script = "def f: pass\n"
         commands = ''
         expected = "SyntaxError:"
-        stdout, stderr = self.run_pdb(script, commands)
+        stdout, stderr = self.run_pdb_script(script, commands)
         self.assertIn(expected, stdout,
             '\n\nExpected:\n{}\nGot:\n{}\n'
             'Fail to handle a syntax error in the debuggee.'
@@ -1119,13 +1140,119 @@ def test_header(self):
             pdb.set_trace(header=header)
         self.assertEqual(stdout.getvalue(), header + '\n')
 
-    def tearDown(self):
-        support.unlink(support.TESTFN)
+    def test_run_module(self):
+        script = """print("SUCCESS")"""
+        commands = """
+            continue
+            quit
+        """
+        stdout, stderr = self.run_pdb_module(script, commands)
+        self.assertTrue(any("SUCCESS" in l for l in stdout.splitlines()), stdout)
+
+    def test_module_is_run_as_main(self):
+        script = """
+            if __name__ == '__main__':
+                print("SUCCESS")
+        """
+        commands = """
+            continue
+            quit
+        """
+        stdout, stderr = self.run_pdb_module(script, commands)
+        self.assertTrue(any("SUCCESS" in l for l in stdout.splitlines()), stdout)
+
+    def test_breakpoint(self):
+        script = """
+            if __name__ == '__main__':
+                pass
+                print("SUCCESS")
+                pass
+        """
+        commands = """
+            b 3
+            quit
+        """
+        stdout, stderr = self.run_pdb_module(script, commands)
+        self.assertTrue(any("Breakpoint 1 at" in l for l in stdout.splitlines()), stdout)
+        self.assertTrue(all("SUCCESS" not in l for l in stdout.splitlines()), stdout)
+
+    def test_run_pdb_with_pdb(self):
+        commands = """
+            c
+            quit
+        """
+        stdout, stderr = self._run_pdb(["-m", "pdb"], commands)
+        self.assertIn("Debug the Python program given by pyfile.", stdout.splitlines())
+
+    def test_module_without_a_main(self):
+        module_name = 't_main'
+        support.rmtree(module_name)
+        init_file = module_name + '/__init__.py'
+        os.mkdir(module_name)
+        with open(init_file, 'w') as f:
+            pass
+        self.addCleanup(support.rmtree, module_name)
+        stdout, stderr = self._run_pdb(['-m', module_name], "")
+        self.assertIn("ImportError: No module named t_main.__main__",
+                      stdout.splitlines())
+
+    def test_blocks_at_first_code_line(self):
+        script = """
+                #This is a comment, on line 2
+
+                print("SUCCESS")
+        """
+        commands = """
+            quit
+        """
+        stdout, stderr = self.run_pdb_module(script, commands)
+        self.assertTrue(any("__main__.py(4)<module>()"
+                            in l for l in stdout.splitlines()), stdout)
+
+    def test_relative_imports(self):
+        self.module_name = 't_main'
+        support.rmtree(self.module_name)
+        main_file = self.module_name + '/__main__.py'
+        init_file = self.module_name + '/__init__.py'
+        module_file = self.module_name + '/module.py'
+        self.addCleanup(support.rmtree, self.module_name)
+        os.mkdir(self.module_name)
+        with open(init_file, 'w') as f:
+            f.write(textwrap.dedent("""
+                top_var = "VAR from top"
+            """))
+        with open(main_file, 'w') as f:
+            f.write(textwrap.dedent("""
+                from . import top_var
+                from .module import var
+                from . import module
+                pass # We'll stop here and print the vars
+            """))
+        with open(module_file, 'w') as f:
+            f.write(textwrap.dedent("""
+                var = "VAR from module"
+                var2 = "second var"
+            """))
+        commands = """
+            b 5
+            c
+            p top_var
+            p var
+            p module.var2
+            quit
+        """
+        stdout, _ = self._run_pdb(['-m', self.module_name], commands)
+        self.assertTrue(any("VAR from module" in l for l in stdout.splitlines()))
+        self.assertTrue(any("VAR from top" in l for l in stdout.splitlines()))
+        self.assertTrue(any("second var" in l for l in stdout.splitlines()))
 
 
 def load_tests(*args):
     from test import test_pdb
-    suites = [unittest.makeSuite(PdbTestCase), doctest.DocTestSuite(test_pdb)]
+    suites = [
+        unittest.makeSuite(PdbTestCase),
+        doctest.DocTestSuite(test_pdb)
+    ]
     return unittest.TestSuite(suites)
 
 
diff --git a/Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst b/Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst
new file mode 100644
index 00000000000..20d7eeace87
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst
@@ -0,0 +1 @@
+Add support to run modules with pdb


From solipsis at pitrou.net  Sat Jan  6 04:12:43 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 06 Jan 2018 09:12:43 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=15
Message-ID: <20180106091243.1.CC6245E9C269F732@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [7, -7, 8] memory blocks, sum=8
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [3, -1, 1] memory blocks, sum=3
test_multiprocessing_forkserver leaked [0, -2, 2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogvj9Fba', '--timeout', '7200']

From webhook-mailer at python.org  Sat Jan  6 08:25:42 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Sat, 06 Jan 2018 13:25:42 -0000
Subject: [Python-checkins] Cleanup grammar in unittest.mock.seal
 documentation (#5107)
Message-ID: <mailman.7.1515245143.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/85ac726a40707ae68a23d568c322868e353217ce
commit: 85ac726a40707ae68a23d568c322868e353217ce
branch: master
author: Paul Ganssle <pganssle at users.noreply.github.com>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-06T15:25:34+02:00
summary:

Cleanup grammar in unittest.mock.seal documentation (#5107)

files:
M Doc/library/unittest.mock.rst

diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst
index b6eb8ccb59d..ac9dd3b6f79 100644
--- a/Doc/library/unittest.mock.rst
+++ b/Doc/library/unittest.mock.rst
@@ -2370,12 +2370,12 @@ Sealing mocks
 
 .. function:: seal(mock)
 
-    Seal will disable the creation of mock children by preventing to get or set
-    any new attribute on the sealed mock. The sealing process is performed recursively.
+    Seal will disable the creation of mock children by preventing getting or setting
+    of any new attribute on the sealed mock. The sealing process is performed recursively.
 
     If a mock instance is assigned to an attribute instead of being dynamically created
-    it won't be considered in the sealing chain. This allows to prevent seal from fixing
-    part of the mock object.
+    it won't be considered in the sealing chain. This allows one to prevent seal from
+    fixing part of the mock object.
 
         >>> mock = Mock()
         >>> mock.submock.attribute1 = 2


From webhook-mailer at python.org  Sat Jan  6 10:18:57 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sat, 06 Jan 2018 15:18:57 -0000
Subject: [Python-checkins] bpo-31993: Do not allocate large temporary
 buffers in pickle dump. (#4353)
Message-ID: <mailman.8.1515251939.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/3cd7c6e6eb43dbd7d7180503265772a67953e682
commit: 3cd7c6e6eb43dbd7d7180503265772a67953e682
branch: master
author: Olivier Grisel <olivier.grisel at ensta.org>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-06T17:18:54+02:00
summary:

bpo-31993: Do not allocate large temporary buffers in pickle dump. (#4353)

The picklers do no longer allocate temporary memory when dumping large
bytes and str objects into a file object. Instead the data is
directly streamed into the underlying file object.

Previously the C implementation would buffer all content and issue a
single call to file.write() at the end of the dump. With protocol 4
this behavior has changed to issue one call to file.write() per frame.

The Python pickler with protocol 4 now dumps each frame content as a
memoryview to an IOBytes instance that is never reused and the
memoryview is no longer released after the call to write. This makes it
possible for the file object to delay access to the memoryview of
previous frames without forcing any additional memory copy as was
already possible with the C pickler.

files:
A Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst
M Lib/pickle.py
M Lib/pickletools.py
M Lib/test/pickletester.py
M Lib/test/test_pickletools.py
M Modules/_pickle.c

diff --git a/Lib/pickle.py b/Lib/pickle.py
index 350d4a46c06..301e8cf5589 100644
--- a/Lib/pickle.py
+++ b/Lib/pickle.py
@@ -201,14 +201,24 @@ def commit_frame(self, force=False):
         if self.current_frame:
             f = self.current_frame
             if f.tell() >= self._FRAME_SIZE_TARGET or force:
-                with f.getbuffer() as data:
-                    n = len(data)
-                    write = self.file_write
-                    write(FRAME)
-                    write(pack("<Q", n))
-                    write(data)
-                f.seek(0)
-                f.truncate()
+                data = f.getbuffer()
+                write = self.file_write
+                # Issue a single call to the write method of the underlying
+                # file object for the frame opcode with the size of the
+                # frame. The concatenation is expected to be less expensive
+                # than issuing an additional call to write.
+                write(FRAME + pack("<Q", len(data)))
+
+                # Issue a separate call to write to append the frame
+                # contents without concatenation to the above to avoid a
+                # memory copy.
+                write(data)
+
+                # Start the new frame with a new io.BytesIO instance so that
+                # the file object can have delayed access to the previous frame
+                # contents via an unreleased memoryview of the previous
+                # io.BytesIO instance.
+                self.current_frame = io.BytesIO()
 
     def write(self, data):
         if self.current_frame:
@@ -216,6 +226,21 @@ def write(self, data):
         else:
             return self.file_write(data)
 
+    def write_large_bytes(self, header, payload):
+        write = self.file_write
+        if self.current_frame:
+            # Terminate the current frame and flush it to the file.
+            self.commit_frame(force=True)
+
+        # Perform direct write of the header and payload of the large binary
+        # object. Be careful not to concatenate the header and the payload
+        # prior to calling 'write' as we do not want to allocate a large
+        # temporary bytes object.
+        # We intentionally do not insert a protocol 4 frame opcode to make
+        # it possible to optimize file.read calls in the loader.
+        write(header)
+        write(payload)
+
 
 class _Unframer:
 
@@ -379,6 +404,7 @@ def __init__(self, file, protocol=None, *, fix_imports=True):
             raise TypeError("file must have a 'write' attribute")
         self.framer = _Framer(self._file_write)
         self.write = self.framer.write
+        self._write_large_bytes = self.framer.write_large_bytes
         self.memo = {}
         self.proto = int(protocol)
         self.bin = protocol >= 1
@@ -699,7 +725,9 @@ def save_bytes(self, obj):
         if n <= 0xff:
             self.write(SHORT_BINBYTES + pack("<B", n) + obj)
         elif n > 0xffffffff and self.proto >= 4:
-            self.write(BINBYTES8 + pack("<Q", n) + obj)
+            self._write_large_bytes(BINBYTES8 + pack("<Q", n), obj)
+        elif n >= self.framer._FRAME_SIZE_TARGET:
+            self._write_large_bytes(BINBYTES + pack("<I", n), obj)
         else:
             self.write(BINBYTES + pack("<I", n) + obj)
         self.memoize(obj)
@@ -712,7 +740,9 @@ def save_str(self, obj):
             if n <= 0xff and self.proto >= 4:
                 self.write(SHORT_BINUNICODE + pack("<B", n) + encoded)
             elif n > 0xffffffff and self.proto >= 4:
-                self.write(BINUNICODE8 + pack("<Q", n) + encoded)
+                self._write_large_bytes(BINUNICODE8 + pack("<Q", n), encoded)
+            elif n >= self.framer._FRAME_SIZE_TARGET:
+                self._write_large_bytes(BINUNICODE + pack("<I", n), encoded)
             else:
                 self.write(BINUNICODE + pack("<I", n) + encoded)
         else:
diff --git a/Lib/pickletools.py b/Lib/pickletools.py
index 408c2ac1360..8486cbf8436 100644
--- a/Lib/pickletools.py
+++ b/Lib/pickletools.py
@@ -2279,7 +2279,7 @@ def optimize(p):
             if arg > proto:
                 proto = arg
             if pos == 0:
-                protoheader = p[pos: end_pos]
+                protoheader = p[pos:end_pos]
             else:
                 opcodes.append((pos, end_pos))
         else:
@@ -2295,6 +2295,7 @@ def optimize(p):
         pickler.framer.start_framing()
     idx = 0
     for op, arg in opcodes:
+        frameless = False
         if op is put:
             if arg not in newids:
                 continue
@@ -2305,8 +2306,12 @@ def optimize(p):
             data = pickler.get(newids[arg])
         else:
             data = p[op:arg]
-        pickler.framer.commit_frame()
-        pickler.write(data)
+            frameless = len(data) > pickler.framer._FRAME_SIZE_TARGET
+        pickler.framer.commit_frame(force=frameless)
+        if frameless:
+            pickler.framer.file_write(data)
+        else:
+            pickler.write(data)
     pickler.framer.end_framing()
     return out.getvalue()
 
diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
index bf6116b2dfb..5d983eb617f 100644
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -2042,21 +2042,40 @@ def test_setitems_on_non_dicts(self):
     def check_frame_opcodes(self, pickled):
         """
         Check the arguments of FRAME opcodes in a protocol 4+ pickle.
+
+        Note that binary objects that are larger than FRAME_SIZE_TARGET are not
+        framed by default and are therefore considered a frame by themselves in
+        the following consistency check.
         """
-        frame_opcode_size = 9
-        last_arg = last_pos = None
+        last_arg = last_pos = last_frame_opcode_size = None
+        frameless_opcode_sizes = {
+            'BINBYTES': 5,
+            'BINUNICODE': 5,
+            'BINBYTES8': 9,
+            'BINUNICODE8': 9,
+        }
         for op, arg, pos in pickletools.genops(pickled):
-            if op.name != 'FRAME':
+            if op.name in frameless_opcode_sizes:
+                if len(arg) > self.FRAME_SIZE_TARGET:
+                    frame_opcode_size = frameless_opcode_sizes[op.name]
+                    arg = len(arg)
+                else:
+                    continue
+            elif op.name == 'FRAME':
+                frame_opcode_size = 9
+            else:
                 continue
+
             if last_pos is not None:
                 # The previous frame's size should be equal to the number
                 # of bytes up to the current frame.
-                frame_size = pos - last_pos - frame_opcode_size
+                frame_size = pos - last_pos - last_frame_opcode_size
                 self.assertEqual(frame_size, last_arg)
             last_arg, last_pos = arg, pos
+            last_frame_opcode_size = frame_opcode_size
         # The last frame's size should be equal to the number of bytes up
         # to the pickle's end.
-        frame_size = len(pickled) - last_pos - frame_opcode_size
+        frame_size = len(pickled) - last_pos - last_frame_opcode_size
         self.assertEqual(frame_size, last_arg)
 
     def test_framing_many_objects(self):
@@ -2076,15 +2095,36 @@ def test_framing_many_objects(self):
 
     def test_framing_large_objects(self):
         N = 1024 * 1024
-        obj = [b'x' * N, b'y' * N, b'z' * N]
+        obj = [b'x' * N, b'y' * N, 'z' * N]
         for proto in range(4, pickle.HIGHEST_PROTOCOL + 1):
-            with self.subTest(proto=proto):
-                pickled = self.dumps(obj, proto)
-                unpickled = self.loads(pickled)
-                self.assertEqual(obj, unpickled)
-                n_frames = count_opcode(pickle.FRAME, pickled)
-                self.assertGreaterEqual(n_frames, len(obj))
-                self.check_frame_opcodes(pickled)
+            for fast in [True, False]:
+                with self.subTest(proto=proto, fast=fast):
+                    if hasattr(self, 'pickler'):
+                        buf = io.BytesIO()
+                        pickler = self.pickler(buf, protocol=proto)
+                        pickler.fast = fast
+                        pickler.dump(obj)
+                        pickled = buf.getvalue()
+                    elif fast:
+                        continue
+                    else:
+                        # Fallback to self.dumps when fast=False and
+                        # self.pickler is not available.
+                        pickled = self.dumps(obj, proto)
+                    unpickled = self.loads(pickled)
+                    # More informative error message in case of failure.
+                    self.assertEqual([len(x) for x in obj],
+                                     [len(x) for x in unpickled])
+                    # Perform full equality check if the lengths match.
+                    self.assertEqual(obj, unpickled)
+                    n_frames = count_opcode(pickle.FRAME, pickled)
+                    if not fast:
+                        # One frame per memoize for each large object.
+                        self.assertGreaterEqual(n_frames, len(obj))
+                    else:
+                        # One frame at the beginning and one at the end.
+                        self.assertGreaterEqual(n_frames, 2)
+                    self.check_frame_opcodes(pickled)
 
     def test_optional_frames(self):
         if pickle.HIGHEST_PROTOCOL < 4:
@@ -2125,6 +2165,71 @@ def remove_frames(pickled, keep_frame=None):
                             count_opcode(pickle.FRAME, pickled))
             self.assertEqual(obj, self.loads(some_frames_pickle))
 
+    def test_framed_write_sizes_with_delayed_writer(self):
+        class ChunkAccumulator:
+            """Accumulate pickler output in a list of raw chunks."""
+
+            def __init__(self):
+                self.chunks = []
+
+            def write(self, chunk):
+                self.chunks.append(chunk)
+
+            def concatenate_chunks(self):
+                # Some chunks can be memoryview instances, we need to convert
+                # them to bytes to be able to call join
+                return b"".join([c.tobytes() if hasattr(c, 'tobytes') else c
+                                 for c in self.chunks])
+
+        small_objects = [(str(i).encode('ascii'), i % 42, {'i': str(i)})
+                         for i in range(int(1e4))]
+
+        for proto in range(4, pickle.HIGHEST_PROTOCOL + 1):
+            # Protocol 4 packs groups of small objects into frames and issues
+            # calls to write only once or twice per frame:
+            # The C pickler issues one call to write per-frame (header and
+            # contents) while Python pickler issues two calls to write: one for
+            # the frame header and one for the frame binary contents.
+            writer = ChunkAccumulator()
+            self.pickler(writer, proto).dump(small_objects)
+
+            # Actually read the binary content of the chunks after the end
+            # of the call to dump: ant memoryview passed to write should not
+            # be released otherwise this delayed access would not be possible.
+            pickled = writer.concatenate_chunks()
+            reconstructed = self.loads(pickled)
+            self.assertEqual(reconstructed, small_objects)
+            self.assertGreater(len(writer.chunks), 1)
+
+            n_frames, remainder = divmod(len(pickled), self.FRAME_SIZE_TARGET)
+            if remainder > 0:
+                n_frames += 1
+
+            # There should be at least one call to write per frame
+            self.assertGreaterEqual(len(writer.chunks), n_frames)
+
+            # but not too many either: there can be one for the proto,
+            # one per-frame header and one per frame for the actual contents.
+            self.assertGreaterEqual(2 * n_frames + 1, len(writer.chunks))
+
+            chunk_sizes = [len(c) for c in writer.chunks[:-1]]
+            large_sizes = [s for s in chunk_sizes
+                           if s >= self.FRAME_SIZE_TARGET]
+            small_sizes = [s for s in chunk_sizes
+                           if s < self.FRAME_SIZE_TARGET]
+
+            # Large chunks should not be too large:
+            for chunk_size in large_sizes:
+                self.assertGreater(2 * self.FRAME_SIZE_TARGET, chunk_size)
+
+            last_chunk_size = len(writer.chunks[-1])
+            self.assertGreater(2 * self.FRAME_SIZE_TARGET, last_chunk_size)
+
+            # Small chunks (if any) should be very small
+            # (only proto and frame headers)
+            for chunk_size in small_sizes:
+                self.assertGreaterEqual(9, chunk_size)
+
     def test_nested_names(self):
         global Nested
         class Nested:
diff --git a/Lib/test/test_pickletools.py b/Lib/test/test_pickletools.py
index b3cab0e4fb3..e40a958f7eb 100644
--- a/Lib/test/test_pickletools.py
+++ b/Lib/test/test_pickletools.py
@@ -15,6 +15,9 @@ def loads(self, buf, **kwds):
     # Test relies on precise output of dumps()
     test_pickle_to_2x = None
 
+    # Test relies on writing by chunks into a file object.
+    test_framed_write_sizes_with_delayed_writer = None
+
     def test_optimize_long_binget(self):
         data = [str(i) for i in range(257)]
         data.append(data[-1])
diff --git a/Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst b/Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst
new file mode 100644
index 00000000000..b453e21c80c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst
@@ -0,0 +1,14 @@
+The picklers do no longer allocate temporary memory when dumping large
+``bytes`` and ``str`` objects into a file object. Instead the data is
+directly streamed into the underlying file object.
+
+Previously the C implementation would buffer all content and issue a
+single call to ``file.write`` at the end of the dump. With protocol 4
+this behavior has changed to issue one call to ``file.write`` per frame.
+
+The Python pickler with protocol 4 now dumps each frame content as a
+memoryview to an IOBytes instance that is never reused and the
+memoryview is no longer released after the call to write. This makes it
+possible for the file object to delay access to the memoryview of
+previous frames without forcing any additional memory copy as was
+already possible with the C pickler.
diff --git a/Modules/_pickle.c b/Modules/_pickle.c
index da915efd831..5cb1fba6cc2 100644
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -971,20 +971,6 @@ _Pickler_CommitFrame(PicklerObject *self)
     return 0;
 }
 
-static int
-_Pickler_OpcodeBoundary(PicklerObject *self)
-{
-    Py_ssize_t frame_len;
-
-    if (!self->framing || self->frame_start == -1)
-        return 0;
-    frame_len = self->output_len - self->frame_start - FRAME_HEADER_SIZE;
-    if (frame_len >= FRAME_SIZE_TARGET)
-        return _Pickler_CommitFrame(self);
-    else
-        return 0;
-}
-
 static PyObject *
 _Pickler_GetString(PicklerObject *self)
 {
@@ -1019,6 +1005,38 @@ _Pickler_FlushToFile(PicklerObject *self)
     return (result == NULL) ? -1 : 0;
 }
 
+static int
+_Pickler_OpcodeBoundary(PicklerObject *self)
+{
+    Py_ssize_t frame_len;
+
+    if (!self->framing || self->frame_start == -1) {
+        return 0;
+    }
+    frame_len = self->output_len - self->frame_start - FRAME_HEADER_SIZE;
+    if (frame_len >= FRAME_SIZE_TARGET) {
+        if(_Pickler_CommitFrame(self)) {
+            return -1;
+        }
+        /* Flush the content of the commited frame to the underlying
+         * file and reuse the pickler buffer for the next frame so as
+         * to limit memory usage when dumping large complex objects to
+         * a file.
+         *
+         * self->write is NULL when called via dumps.
+         */
+        if (self->write != NULL) {
+            if (_Pickler_FlushToFile(self) < 0) {
+                return -1;
+            }
+            if (_Pickler_ClearBuffer(self) < 0) {
+                return -1;
+            }
+        }
+    }
+    return 0;
+}
+
 static Py_ssize_t
 _Pickler_Write(PicklerObject *self, const char *s, Py_ssize_t data_len)
 {
@@ -2124,6 +2142,51 @@ save_float(PicklerObject *self, PyObject *obj)
     return 0;
 }
 
+/* No-copy code-path to write large contiguous data directly into the
+   underlying file object, bypassing the output_buffer of the Pickler. */
+static int
+_Pickler_write_large_bytes(
+    PicklerObject *self, const char *header, Py_ssize_t header_size,
+    PyObject *payload)
+{
+    assert(self->output_buffer != NULL);
+    assert(self->write != NULL);
+    PyObject *result;
+
+    /* Commit the previous frame. */
+    if (_Pickler_CommitFrame(self)) {
+        return -1;
+    }
+    /* Disable frameing temporarily */
+    self->framing = 0;
+
+    if (_Pickler_Write(self, header, header_size) < 0) {
+        return -1;
+    }
+    /* Dump the output buffer to the file. */
+    if (_Pickler_FlushToFile(self) < 0) {
+        return -1;
+    }
+
+    /* Stream write the payload into the file without going through the
+       output buffer. */
+    result = PyObject_CallFunctionObjArgs(self->write, payload, NULL);
+    if (result == NULL) {
+        return -1;
+    }
+    Py_DECREF(result);
+
+    /* Reinitialize the buffer for subsequent calls to _Pickler_Write. */
+    if (_Pickler_ClearBuffer(self) < 0) {
+        return -1;
+    }
+
+    /* Re-enable framing for subsequent calls to _Pickler_Write. */
+    self->framing = 1;
+
+    return 0;
+}
+
 static int
 save_bytes(PicklerObject *self, PyObject *obj)
 {
@@ -2202,11 +2265,21 @@ save_bytes(PicklerObject *self, PyObject *obj)
             return -1;          /* string too large */
         }
 
-        if (_Pickler_Write(self, header, len) < 0)
-            return -1;
-
-        if (_Pickler_Write(self, PyBytes_AS_STRING(obj), size) < 0)
-            return -1;
+        if (size < FRAME_SIZE_TARGET || self->write == NULL) {
+            if (_Pickler_Write(self, header, len) < 0) {
+                return -1;
+            }
+            if (_Pickler_Write(self, PyBytes_AS_STRING(obj), size) < 0) {
+                return -1;
+            }
+        }
+        else {
+            /* Bypass the in-memory buffer to directly stream large data
+               into the underlying file object. */
+            if (_Pickler_write_large_bytes(self, header, len, obj) < 0) {
+                return -1;
+            }
+        }
 
         if (memo_put(self, obj) < 0)
             return -1;
@@ -2291,6 +2364,7 @@ write_utf8(PicklerObject *self, const char *data, Py_ssize_t size)
 {
     char header[9];
     Py_ssize_t len;
+    PyObject *mem;
 
     assert(size >= 0);
     if (size <= 0xff && self->proto >= 4) {
@@ -2317,11 +2391,27 @@ write_utf8(PicklerObject *self, const char *data, Py_ssize_t size)
         return -1;
     }
 
-    if (_Pickler_Write(self, header, len) < 0)
-        return -1;
-    if (_Pickler_Write(self, data, size) < 0)
-        return -1;
-
+    if (size < FRAME_SIZE_TARGET || self->write == NULL) {
+        if (_Pickler_Write(self, header, len) < 0) {
+            return -1;
+        }
+        if (_Pickler_Write(self, data, size) < 0) {
+            return -1;
+        }
+    }
+    else {
+        /* Bypass the in-memory buffer to directly stream large data
+           into the underlying file object. */
+        mem = PyMemoryView_FromMemory((char *) data, size, PyBUF_READ);
+        if (mem == NULL) {
+            return -1;
+        }
+        if (_Pickler_write_large_bytes(self, header, len, mem) < 0) {
+            Py_DECREF(mem);
+            return -1;
+        }
+        Py_DECREF(mem);
+    }
     return 0;
 }
 


From webhook-mailer at python.org  Sat Jan  6 12:41:55 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sat, 06 Jan 2018 17:41:55 -0000
Subject: [Python-checkins] bpo-32499: Add dataclasses.is_dataclass(obj),
 which returns True if obj is a dataclass or an instance of one.
 (#5113)
Message-ID: <mailman.9.1515260517.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/e7ba013d870012157f695ead7e3645c2828a7fc5
commit: e7ba013d870012157f695ead7e3645c2828a7fc5
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-06T12:41:53-05:00
summary:

bpo-32499: Add dataclasses.is_dataclass(obj), which returns True if obj is a dataclass or an instance of one. (#5113)

files:
A Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst
M Lib/dataclasses.py
M Lib/test/test_dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index eaaed63ef28..b4786bf502e 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -16,6 +16,7 @@
            'astuple',
            'make_dataclass',
            'replace',
+           'is_dataclass',
            ]
 
 # Raised when an attempt is made to modify a frozen class.
@@ -615,11 +616,17 @@ def fields(class_or_instance):
     return tuple(f for f in fields.values() if f._field_type is _FIELD)
 
 
-def _isdataclass(obj):
+def _is_dataclass_instance(obj):
     """Returns True if obj is an instance of a dataclass."""
     return not isinstance(obj, type) and hasattr(obj, _MARKER)
 
 
+def is_dataclass(obj):
+    """Returns True if obj is a dataclass or an instance of a
+    dataclass."""
+    return hasattr(obj, _MARKER)
+
+
 def asdict(obj, *, dict_factory=dict):
     """Return the fields of a dataclass instance as a new dictionary mapping
     field names to field values.
@@ -639,12 +646,12 @@ class C:
     dataclass instances. This will also look into built-in containers:
     tuples, lists, and dicts.
     """
-    if not _isdataclass(obj):
+    if not _is_dataclass_instance(obj):
         raise TypeError("asdict() should be called on dataclass instances")
     return _asdict_inner(obj, dict_factory)
 
 def _asdict_inner(obj, dict_factory):
-    if _isdataclass(obj):
+    if _is_dataclass_instance(obj):
         result = []
         for f in fields(obj):
             value = _asdict_inner(getattr(obj, f.name), dict_factory)
@@ -678,12 +685,12 @@ class C:
     tuples, lists, and dicts.
     """
 
-    if not _isdataclass(obj):
+    if not _is_dataclass_instance(obj):
         raise TypeError("astuple() should be called on dataclass instances")
     return _astuple_inner(obj, tuple_factory)
 
 def _astuple_inner(obj, tuple_factory):
-    if _isdataclass(obj):
+    if _is_dataclass_instance(obj):
         result = []
         for f in fields(obj):
             value = _astuple_inner(getattr(obj, f.name), tuple_factory)
@@ -751,7 +758,7 @@ class C:
     # We're going to mutate 'changes', but that's okay because it's a new
     #  dict, even if called with 'replace(obj, **my_changes)'.
 
-    if not _isdataclass(obj):
+    if not _is_dataclass_instance(obj):
         raise TypeError("replace() should be called on dataclass instances")
 
     # It's an error to have init=False fields in 'changes'.
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index ed695639882..fca384d8c3c 100755
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -1,6 +1,6 @@
 from dataclasses import (
     dataclass, field, FrozenInstanceError, fields, asdict, astuple,
-    make_dataclass, replace, InitVar, Field, MISSING
+    make_dataclass, replace, InitVar, Field, MISSING, is_dataclass,
 )
 
 import pickle
@@ -1365,27 +1365,32 @@ class C:
 
         self.assertIs(C().x, int)
 
-    def test_isdataclass(self):
-        # There is no isdataclass() helper any more, but the PEP
-        #  describes how to write it, so make sure that works.  Note
-        #  that this version returns True for both classes and
-        #  instances.
-        def isdataclass(obj):
-            try:
-                fields(obj)
-                return True
-            except TypeError:
-                return False
+    def test_is_dataclass(self):
+        class NotDataClass:
+            pass
 
-        self.assertFalse(isdataclass(0))
-        self.assertFalse(isdataclass(int))
+        self.assertFalse(is_dataclass(0))
+        self.assertFalse(is_dataclass(int))
+        self.assertFalse(is_dataclass(NotDataClass))
+        self.assertFalse(is_dataclass(NotDataClass()))
 
         @dataclass
         class C:
             x: int
 
-        self.assertTrue(isdataclass(C))
-        self.assertTrue(isdataclass(C(0)))
+        @dataclass
+        class D:
+            d: C
+            e: int
+
+        c = C(10)
+        d = D(c, 4)
+
+        self.assertTrue(is_dataclass(C))
+        self.assertTrue(is_dataclass(c))
+        self.assertFalse(is_dataclass(c.x))
+        self.assertTrue(is_dataclass(d.d))
+        self.assertFalse(is_dataclass(d.e))
 
     def test_helper_fields_with_class_instance(self):
         # Check that we can call fields() on either a class or instance,
diff --git a/Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst b/Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst
new file mode 100644
index 00000000000..bf3e99c8d86
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst
@@ -0,0 +1,2 @@
+Add dataclasses.is_dataclass(obj), which returns True if obj is a dataclass
+or an instance of one.


From webhook-mailer at python.org  Sat Jan  6 16:14:06 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sat, 06 Jan 2018 21:14:06 -0000
Subject: [Python-checkins] bpo-32278: Allow dataclasses.make_dataclass() to
 omit type information. (gh-5115)
Message-ID: <mailman.10.1515273248.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/ed7d429ebb591f65cef558760fb4ebdc4fc8f8b0
commit: ed7d429ebb591f65cef558760fb4ebdc4fc8f8b0
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-06T16:14:03-05:00
summary:

bpo-32278: Allow dataclasses.make_dataclass() to omit type information. (gh-5115)

files:
A Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst
M Lib/dataclasses.py
M Lib/test/test_dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index b4786bf502e..c4e94b8f667 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -708,9 +708,10 @@ def _astuple_inner(obj, tuple_factory):
 def make_dataclass(cls_name, fields, *, bases=(), namespace=None):
     """Return a new dynamically created dataclass.
 
-    The dataclass name will be 'cls_name'.  'fields' is an interable
-    of either (name, type) or (name, type, Field) objects. Field
-    objects are created by calling 'field(name, type [, Field])'.
+    The dataclass name will be 'cls_name'.  'fields' is an iterable
+    of either (name), (name, type) or (name, type, Field) objects. If type is
+    omitted, use the string 'typing.Any'.  Field objects are created by
+    calling 'field(name, type [, Field])'.
 
       C = make_class('C', [('a', int', ('b', int, Field(init=False))], bases=Base)
 
@@ -730,12 +731,19 @@ class C(Base):
         # Copy namespace since we're going to mutate it.
         namespace = namespace.copy()
 
-    anns = collections.OrderedDict((name, tp) for name, tp, *_ in fields)
-    namespace['__annotations__'] = anns
+    anns = collections.OrderedDict()
     for item in fields:
-        if len(item) == 3:
+        if isinstance(item, str):
+            name = item
+            tp = 'typing.Any'
+        elif len(item) == 2:
+            name, tp, = item
+        elif len(item) == 3:
             name, tp, spec = item
             namespace[name] = spec
+        anns[name] = tp
+
+    namespace['__annotations__'] = anns
     cls = type(cls_name, bases, namespace)
     return dataclass(cls)
 
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index fca384d8c3c..c44c53d039d 100755
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -2033,6 +2033,20 @@ def test_helper_make_dataclass_class_var(self):
         self.assertEqual(C.y, 10)
         self.assertEqual(C.z, 20)
 
+    def test_helper_make_dataclass_no_types(self):
+        C = make_dataclass('Point', ['x', 'y', 'z'])
+        c = C(1, 2, 3)
+        self.assertEqual(vars(c), {'x': 1, 'y': 2, 'z': 3})
+        self.assertEqual(C.__annotations__, {'x': 'typing.Any',
+                                             'y': 'typing.Any',
+                                             'z': 'typing.Any'})
+
+        C = make_dataclass('Point', ['x', ('y', int), 'z'])
+        c = C(1, 2, 3)
+        self.assertEqual(vars(c), {'x': 1, 'y': 2, 'z': 3})
+        self.assertEqual(C.__annotations__, {'x': 'typing.Any',
+                                             'y': int,
+                                             'z': 'typing.Any'})
 
 class TestDocString(unittest.TestCase):
     def assertDocStrEqual(self, a, b):
diff --git a/Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst b/Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst
new file mode 100644
index 00000000000..c627468072e
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst
@@ -0,0 +1,2 @@
+Make type information optional on dataclasses.make_dataclass(). If omitted,
+the string 'typing.Any' is used.


From webhook-mailer at python.org  Sat Jan  6 17:10:01 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sat, 06 Jan 2018 22:10:01 -0000
Subject: [Python-checkins] bpo-32279: Add additional params to
 make_dataclass(), pass through to dataclass(). (gh-5117)
Message-ID: <mailman.11.1515276602.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/d80b443f02e087dd289ffefd04179c675304d76d
commit: d80b443f02e087dd289ffefd04179c675304d76d
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-06T17:09:58-05:00
summary:

 bpo-32279: Add additional params to make_dataclass(), pass through to dataclass(). (gh-5117)

files:
A Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst
M Lib/dataclasses.py
M Lib/test/test_dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index c4e94b8f667..62b7c5ed70a 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -705,7 +705,8 @@ def _astuple_inner(obj, tuple_factory):
         return deepcopy(obj)
 
 
-def make_dataclass(cls_name, fields, *, bases=(), namespace=None):
+def make_dataclass(cls_name, fields, *, bases=(), namespace=None, init=True,
+                   repr=True, eq=True, order=False, hash=None, frozen=False):
     """Return a new dynamically created dataclass.
 
     The dataclass name will be 'cls_name'.  'fields' is an iterable
@@ -723,6 +724,9 @@ class C(Base):
           b: int = field(init=False)
 
     For the bases and namespace paremeters, see the builtin type() function.
+
+    The parameters init, repr, eq, order, hash, and frozen are passed to
+    dataclass().
     """
 
     if namespace is None:
@@ -745,8 +749,8 @@ class C(Base):
 
     namespace['__annotations__'] = anns
     cls = type(cls_name, bases, namespace)
-    return dataclass(cls)
-
+    return dataclass(cls, init=init, repr=repr, eq=eq, order=order,
+                     hash=hash, frozen=frozen)
 
 def replace(obj, **changes):
     """Return a new object replacing specified fields with new values.
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index c44c53d039d..69819ea4507 100755
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -2033,6 +2033,23 @@ def test_helper_make_dataclass_class_var(self):
         self.assertEqual(C.y, 10)
         self.assertEqual(C.z, 20)
 
+    def test_helper_make_dataclass_other_params(self):
+        C = make_dataclass('C',
+                           [('x', int),
+                            ('y', ClassVar[int], 10),
+                            ('z', ClassVar[int], field(default=20)),
+                            ],
+                           init=False)
+        # Make sure we have a repr, but no init.
+        self.assertNotIn('__init__', vars(C))
+        self.assertIn('__repr__', vars(C))
+
+        # Make sure random other params don't work.
+        with self.assertRaisesRegex(TypeError, 'unexpected keyword argument'):
+            C = make_dataclass('C',
+                               [],
+                               xxinit=False)
+
     def test_helper_make_dataclass_no_types(self):
         C = make_dataclass('Point', ['x', 'y', 'z'])
         c = C(1, 2, 3)
diff --git a/Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst b/Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst
new file mode 100644
index 00000000000..b7ef8c80f5a
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst
@@ -0,0 +1,2 @@
+Add params to dataclasses.make_dataclasses(): init, repr, eq, order, hash,
+and frozen.  Pass them through to dataclass().


From webhook-mailer at python.org  Sat Jan  6 21:05:21 2018
From: webhook-mailer at python.org (Steve Dower)
Date: Sun, 07 Jan 2018 02:05:21 -0000
Subject: [Python-checkins] bpo-32282: Remove unnecessary check for
 `VersionHelpers.h` in `socketmodule.c` on Windows
Message-ID: <mailman.12.1515290721.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/af11a15c586e980a157c04ee60b6e33dc7228f3f
commit: af11a15c586e980a157c04ee60b6e33dc7228f3f
branch: master
author: Max B?langer <aeromax at gmail.com>
committer: Steve Dower <steve.dower at microsoft.com>
date: 2018-01-07T13:05:18+11:00
summary:

bpo-32282: Remove unnecessary check for `VersionHelpers.h` in `socketmodule.c` on Windows

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst
M Modules/socketmodule.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst
new file mode 100644
index 00000000000..1c833c5439b
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst	
@@ -0,0 +1,2 @@
+Fix an unnecessary ifdef in the include of VersionHelpers.h in socketmodule
+on Windows.
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index d52d9db743a..d75a51af9a6 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -297,10 +297,8 @@ if_indextoname(index) -- return the corresponding interface name\n\
 #  include <fcntl.h>
 # endif
 
-#if defined(_MSC_VER) && _MSC_VER >= 1800
 /* Provides the IsWindows7SP1OrGreater() function */
 #include <VersionHelpers.h>
-#endif
 
 #endif
 
@@ -6552,15 +6550,7 @@ PyInit__socket(void)
 
 #ifdef MS_WINDOWS
     if (support_wsa_no_inherit == -1) {
-#if defined(_MSC_VER) && _MSC_VER >= 1800
         support_wsa_no_inherit = IsWindows7SP1OrGreater();
-#else
-        DWORD version = GetVersion();
-        DWORD major = (DWORD)LOBYTE(LOWORD(version));
-        DWORD minor = (DWORD)HIBYTE(LOWORD(version));
-        /* need Windows 7 SP1, 2008 R2 SP1 or later */
-        support_wsa_no_inherit = major > 6 || (major == 6 && minor >= 1);
-#endif
     }
 #endif
 


From webhook-mailer at python.org  Sat Jan  6 21:41:20 2018
From: webhook-mailer at python.org (Zachary Ware)
Date: Sun, 07 Jan 2018 02:41:20 -0000
Subject: [Python-checkins] Kill the AppVeyor file whitelist (GH-5121)
Message-ID: <mailman.13.1515292881.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/7f7de371f947dc38e67505601927e9bc58fa268a
commit: 7f7de371f947dc38e67505601927e9bc58fa268a
branch: master
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-06T20:41:17-06:00
summary:

Kill the AppVeyor file whitelist (GH-5121)

It's more trouble than it's worth, since AppVeyor only checks the HEAD commit of a PR rather than the full diff against the base branch to decide which files changed.

files:
M .github/appveyor.yml

diff --git a/.github/appveyor.yml b/.github/appveyor.yml
index b052b28f8aa..b7a01335d61 100644
--- a/.github/appveyor.yml
+++ b/.github/appveyor.yml
@@ -16,20 +16,3 @@ environment:
   HOST_PYTHON: C:\Python36\python.exe
 image:
 - Visual Studio 2017
-
-# Only trigger AppVeyor if actual code or its configuration changes
-only_commits:
-  files:
-    - .github/appveyor.yml
-    - .gitattributes
-    - Grammar/
-    - Include/
-    - Lib/
-    - Modules/
-    - Objects/
-    - PC/
-    - PCbuild/
-    - Parser/
-    - Programs/
-    - Python/
-    - Tools/


From webhook-mailer at python.org  Sat Jan  6 22:16:37 2018
From: webhook-mailer at python.org (Zachary Ware)
Date: Sun, 07 Jan 2018 03:16:37 -0000
Subject: [Python-checkins] [3.6] Kill the AppVeyor file whitelist (GH-5122)
Message-ID: <mailman.14.1515294998.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/157be7ca1de0d046a092e2d44d6d7200de6add8b
commit: 157be7ca1de0d046a092e2d44d6d7200de6add8b
branch: 3.6
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-06T21:16:33-06:00
summary:

[3.6] Kill the AppVeyor file whitelist (GH-5122)

It's more trouble than it's worth, since AppVeyor only checks the HEAD commit of a PR rather than the full diff against the base branch to decide which files changed.
(cherry picked from commit 7f7de371f947dc38e67505601927e9bc58fa268a)

files:
M .github/appveyor.yml

diff --git a/.github/appveyor.yml b/.github/appveyor.yml
index 96674ce84d8..b8a01c313f6 100644
--- a/.github/appveyor.yml
+++ b/.github/appveyor.yml
@@ -17,20 +17,3 @@ environment:
 image:
 - Visual Studio 2015
 - Visual Studio 2017
-
-# Only trigger AppVeyor if actual code or its configuration changes
-only_commits:
-  files:
-    - .github/appveyor.yml
-    - .gitattributes
-    - Grammar/
-    - Include/
-    - Lib/
-    - Modules/
-    - Objects/
-    - PC/
-    - PCBuild/
-    - Parser/
-    - Programs/
-    - Python/
-    - Tools/


From webhook-mailer at python.org  Sat Jan  6 22:16:59 2018
From: webhook-mailer at python.org (Zachary Ware)
Date: Sun, 07 Jan 2018 03:16:59 -0000
Subject: [Python-checkins] [2.7] Kill the AppVeyor file whitelist (GH-5123)
Message-ID: <mailman.15.1515295021.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/cd02003b9e1479ac250d6dec4ce75a2ec3d12a71
commit: cd02003b9e1479ac250d6dec4ce75a2ec3d12a71
branch: 2.7
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-06T21:16:56-06:00
summary:

[2.7] Kill the AppVeyor file whitelist (GH-5123)

It's more trouble than it's worth, since AppVeyor only checks the HEAD commit of a PR rather than the full diff against the base branch to decide which files changed.
(cherry picked from commit 7f7de371f947dc38e67505601927e9bc58fa268a)

files:
M .github/appveyor.yml

diff --git a/.github/appveyor.yml b/.github/appveyor.yml
index 2cd9a42feb8..43bd7ed9665 100644
--- a/.github/appveyor.yml
+++ b/.github/appveyor.yml
@@ -14,20 +14,3 @@ test_script:
 - cmd: PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest -j2
 environment:
   HOST_PYTHON: C:\Python36\python.exe
-
-# Only trigger AppVeyor if actual code or its configuration changes
-only_commits:
-  files:
-    - .github/appveyor.yml
-    - .gitattributes
-    - Grammar/
-    - Include/
-    - Lib/
-    - Modules/
-    - Objects/
-    - PC/
-    - PCBuild/
-    - Parser/
-    - Programs/
-    - Python/
-    - Tools/


From solipsis at pitrou.net  Sun Jan  7 04:17:04 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 07 Jan 2018 09:17:04 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=79
Message-ID: <20180107091704.1.C9DC0181CE360C68@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, 7] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, 0, 44] references, sum=44
test_multiprocessing_fork leaked [0, 0, 22] memory blocks, sum=22
test_multiprocessing_fork leaked [0, 0, 2] file descriptors, sum=2
test_multiprocessing_forkserver leaked [2, -1, -1] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogskyxRz', '--timeout', '7200']

From webhook-mailer at python.org  Sun Jan  7 08:19:53 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sun, 07 Jan 2018 13:19:53 -0000
Subject: [Python-checkins] Fix dataclasses docstring typo. (gh-5118)
Message-ID: <mailman.16.1515331194.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/d327ae6ba157fb5118df28a86975c5ca523f05e2
commit: d327ae6ba157fb5118df28a86975c5ca523f05e2
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-07T08:19:45-05:00
summary:

Fix dataclasses docstring typo. (gh-5118)

files:
M Lib/dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 62b7c5ed70a..9e186c3c7b7 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -712,9 +712,9 @@ def make_dataclass(cls_name, fields, *, bases=(), namespace=None, init=True,
     The dataclass name will be 'cls_name'.  'fields' is an iterable
     of either (name), (name, type) or (name, type, Field) objects. If type is
     omitted, use the string 'typing.Any'.  Field objects are created by
-    calling 'field(name, type [, Field])'.
+    the equivalent of calling 'field(name, type [, Field-info])'.
 
-      C = make_class('C', [('a', int', ('b', int, Field(init=False))], bases=Base)
+      C = make_class('C', ['x', ('y', int'), ('z', int, Field(init=False))], bases=[Base])
 
     is equivalent to:
 


From webhook-mailer at python.org  Sun Jan  7 08:30:21 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 07 Jan 2018 13:30:21 -0000
Subject: [Python-checkins] bpo-30579: Allow TracebackType creation and
 tb_next mutation from Python (GH-4793)
Message-ID: <mailman.17.1515331821.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/e46a8af450210ee5c7f0459ad6beddbc626ae60f
commit: e46a8af450210ee5c7f0459ad6beddbc626ae60f
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-07T23:30:18+10:00
summary:

bpo-30579: Allow TracebackType creation and tb_next mutation from Python (GH-4793)

Third party projects may wish to hide their own internal machinery in
order to present more comprehensible tracebacks to end users
(e.g. Jinja2 and Trio both do this).

Previously such projects have had to rely on ctypes to do so:

  https://github.com/pallets/jinja/blob/fe3dadacdf4cf411d0a5b6bbd4d5234697a28af2/jinja2/debug.py#L345
  https://github.com/python-trio/trio/blob/1e86b1aee8c0c759f6f239ae53a05d0d3963c629/trio/_core/_multierror.py#L296

This provides a Python level API for creating and modifying real
Traceback objects, allowing tracebacks to be edited at runtime.

Patch by Nathaniel Smith.

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst
A Python/clinic/traceback.c.h
M Lib/test/test_raise.py
M Python/traceback.c

diff --git a/Lib/test/test_raise.py b/Lib/test/test_raise.py
index 103f6086d20..c1ef154a9a9 100644
--- a/Lib/test/test_raise.py
+++ b/Lib/test/test_raise.py
@@ -228,6 +228,72 @@ def test_accepts_traceback(self):
             self.fail("No exception raised")
 
 
+class TestTracebackType(unittest.TestCase):
+
+    def raiser(self):
+        raise ValueError
+
+    def test_attrs(self):
+        try:
+            self.raiser()
+        except Exception as exc:
+            tb = exc.__traceback__
+
+        self.assertIsInstance(tb.tb_next, types.TracebackType)
+        self.assertIs(tb.tb_frame, sys._getframe())
+        self.assertIsInstance(tb.tb_lasti, int)
+        self.assertIsInstance(tb.tb_lineno, int)
+
+        self.assertIs(tb.tb_next.tb_next, None)
+
+        # Invalid assignments
+        with self.assertRaises(TypeError):
+            del tb.tb_next
+
+        with self.assertRaises(TypeError):
+            tb.tb_next = "asdf"
+
+        # Loops
+        with self.assertRaises(ValueError):
+            tb.tb_next = tb
+
+        with self.assertRaises(ValueError):
+            tb.tb_next.tb_next = tb
+
+        # Valid assignments
+        tb.tb_next = None
+        self.assertIs(tb.tb_next, None)
+
+        new_tb = get_tb()
+        tb.tb_next = new_tb
+        self.assertIs(tb.tb_next, new_tb)
+
+    def test_constructor(self):
+        other_tb = get_tb()
+        frame = sys._getframe()
+
+        tb = types.TracebackType(other_tb, frame, 1, 2)
+        self.assertEqual(tb.tb_next, other_tb)
+        self.assertEqual(tb.tb_frame, frame)
+        self.assertEqual(tb.tb_lasti, 1)
+        self.assertEqual(tb.tb_lineno, 2)
+
+        tb = types.TracebackType(None, frame, 1, 2)
+        self.assertEqual(tb.tb_next, None)
+
+        with self.assertRaises(TypeError):
+            types.TracebackType("no", frame, 1, 2)
+
+        with self.assertRaises(TypeError):
+            types.TracebackType(other_tb, "no", 1, 2)
+
+        with self.assertRaises(TypeError):
+            types.TracebackType(other_tb, frame, "no", 2)
+
+        with self.assertRaises(TypeError):
+            types.TracebackType(other_tb, frame, 1, "nuh-uh")
+
+
 class TestContext(unittest.TestCase):
     def test_instance_context_instance_raise(self):
         context = IndexError()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst
new file mode 100644
index 00000000000..392ebf6a79e
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst	
@@ -0,0 +1,2 @@
+Implement TracebackType.__new__ to allow Python-level creation of
+traceback objects, and make TracebackType.tb_next mutable.
diff --git a/Python/clinic/traceback.c.h b/Python/clinic/traceback.c.h
new file mode 100644
index 00000000000..d9daccbbb74
--- /dev/null
+++ b/Python/clinic/traceback.c.h
@@ -0,0 +1,35 @@
+/*[clinic input]
+preserve
+[clinic start generated code]*/
+
+PyDoc_STRVAR(tb_new__doc__,
+"TracebackType(tb_next, tb_frame, tb_lasti, tb_lineno)\n"
+"--\n"
+"\n"
+"Create a new traceback object.");
+
+static PyObject *
+tb_new_impl(PyTypeObject *type, PyObject *tb_next, PyFrameObject *tb_frame,
+            int tb_lasti, int tb_lineno);
+
+static PyObject *
+tb_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"tb_next", "tb_frame", "tb_lasti", "tb_lineno", NULL};
+    static _PyArg_Parser _parser = {"OO!ii:TracebackType", _keywords, 0};
+    PyObject *tb_next;
+    PyFrameObject *tb_frame;
+    int tb_lasti;
+    int tb_lineno;
+
+    if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser,
+        &tb_next, &PyFrame_Type, &tb_frame, &tb_lasti, &tb_lineno)) {
+        goto exit;
+    }
+    return_value = tb_new_impl(type, tb_next, tb_frame, tb_lasti, tb_lineno);
+
+exit:
+    return return_value;
+}
+/*[clinic end generated code: output=0133130d7d19556f input=a9049054013a1b77]*/
diff --git a/Python/traceback.c b/Python/traceback.c
index 831b4f26249..b00864b06e4 100644
--- a/Python/traceback.c
+++ b/Python/traceback.c
@@ -27,6 +27,65 @@ _Py_IDENTIFIER(close);
 _Py_IDENTIFIER(open);
 _Py_IDENTIFIER(path);
 
+/*[clinic input]
+class TracebackType "PyTracebackObject *" "&PyTraceback_Type"
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=928fa06c10151120]*/
+
+#include "clinic/traceback.c.h"
+
+static PyObject *
+tb_create_raw(PyTracebackObject *next, PyFrameObject *frame, int lasti,
+              int lineno)
+{
+    PyTracebackObject *tb;
+    if ((next != NULL && !PyTraceBack_Check(next)) ||
+                    frame == NULL || !PyFrame_Check(frame)) {
+        PyErr_BadInternalCall();
+        return NULL;
+    }
+    tb = PyObject_GC_New(PyTracebackObject, &PyTraceBack_Type);
+    if (tb != NULL) {
+        Py_XINCREF(next);
+        tb->tb_next = next;
+        Py_XINCREF(frame);
+        tb->tb_frame = frame;
+        tb->tb_lasti = lasti;
+        tb->tb_lineno = lineno;
+        PyObject_GC_Track(tb);
+    }
+    return (PyObject *)tb;
+}
+
+/*[clinic input]
+ at classmethod
+TracebackType.__new__ as tb_new
+
+  tb_next: object
+  tb_frame: object(type='PyFrameObject *', subclass_of='&PyFrame_Type')
+  tb_lasti: int
+  tb_lineno: int
+
+Create a new traceback object.
+[clinic start generated code]*/
+
+static PyObject *
+tb_new_impl(PyTypeObject *type, PyObject *tb_next, PyFrameObject *tb_frame,
+            int tb_lasti, int tb_lineno)
+/*[clinic end generated code: output=fa077debd72d861a input=01cbe8ec8783fca7]*/
+{
+    if (tb_next == Py_None) {
+        tb_next = NULL;
+    } else if (!PyTraceBack_Check(tb_next)) {
+        return PyErr_Format(PyExc_TypeError,
+                            "expected traceback object or None, got '%s'",
+                            Py_TYPE(tb_next)->tp_name);
+    }
+
+    return tb_create_raw((PyTracebackObject *)tb_next, tb_frame, tb_lasti,
+                         tb_lineno);
+}
+
 static PyObject *
 tb_dir(PyTracebackObject *self)
 {
@@ -34,19 +93,72 @@ tb_dir(PyTracebackObject *self)
                                    "tb_lasti", "tb_lineno");
 }
 
+static PyObject *
+tb_next_get(PyTracebackObject *self, void *Py_UNUSED(_))
+{
+    PyObject* ret = (PyObject*)self->tb_next;
+    if (!ret) {
+        ret = Py_None;
+    }
+    Py_INCREF(ret);
+    return ret;
+}
+
+static int
+tb_next_set(PyTracebackObject *self, PyObject *new_next, void *Py_UNUSED(_))
+{
+    if (!new_next) {
+        PyErr_Format(PyExc_TypeError, "can't delete tb_next attribute");
+        return -1;
+    }
+
+    /* We accept None or a traceback object, and map None -> NULL (inverse of
+       tb_next_get) */
+    if (new_next == Py_None) {
+        new_next = NULL;
+    } else if (!PyTraceBack_Check(new_next)) {
+        PyErr_Format(PyExc_TypeError,
+                     "expected traceback object, got '%s'",
+                     Py_TYPE(new_next)->tp_name);
+        return -1;
+    }
+
+    /* Check for loops */
+    PyTracebackObject *cursor = (PyTracebackObject *)new_next;
+    while (cursor) {
+        if (cursor == self) {
+            PyErr_Format(PyExc_ValueError, "traceback loop detected");
+            return -1;
+        }
+        cursor = cursor->tb_next;
+    }
+
+    PyObject *old_next = (PyObject*)self->tb_next;
+    Py_XINCREF(new_next);
+    self->tb_next = (PyTracebackObject *)new_next;
+    Py_XDECREF(old_next);
+
+    return 0;
+}
+
+
 static PyMethodDef tb_methods[] = {
    {"__dir__", (PyCFunction)tb_dir, METH_NOARGS},
    {NULL, NULL, 0, NULL},
 };
 
 static PyMemberDef tb_memberlist[] = {
-    {"tb_next",         T_OBJECT,       OFF(tb_next),   READONLY},
     {"tb_frame",        T_OBJECT,       OFF(tb_frame),  READONLY},
     {"tb_lasti",        T_INT,          OFF(tb_lasti),  READONLY},
     {"tb_lineno",       T_INT,          OFF(tb_lineno), READONLY},
     {NULL}      /* Sentinel */
 };
 
+static PyGetSetDef tb_getsetters[] = {
+    {"tb_next", (getter)tb_next_get, (setter)tb_next_set, NULL, NULL},
+    {NULL}      /* Sentinel */
+};
+
 static void
 tb_dealloc(PyTracebackObject *tb)
 {
@@ -94,7 +206,7 @@ PyTypeObject PyTraceBack_Type = {
     0,                  /* tp_setattro */
     0,                                          /* tp_as_buffer */
     Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
-    0,                                          /* tp_doc */
+    tb_new__doc__,                              /* tp_doc */
     (traverseproc)tb_traverse,                  /* tp_traverse */
     (inquiry)tb_clear,                          /* tp_clear */
     0,                                          /* tp_richcompare */
@@ -103,39 +215,24 @@ PyTypeObject PyTraceBack_Type = {
     0,                                          /* tp_iternext */
     tb_methods,         /* tp_methods */
     tb_memberlist,      /* tp_members */
-    0,                                          /* tp_getset */
+    tb_getsetters,                              /* tp_getset */
     0,                                          /* tp_base */
     0,                                          /* tp_dict */
+    0,                                          /* tp_descr_get */
+    0,                                          /* tp_descr_set */
+    0,                                          /* tp_dictoffset */
+    0,                                          /* tp_init */
+    0,                                          /* tp_alloc */
+    tb_new,                                     /* tp_new */
 };
 
-static PyTracebackObject *
-newtracebackobject(PyTracebackObject *next, PyFrameObject *frame)
-{
-    PyTracebackObject *tb;
-    if ((next != NULL && !PyTraceBack_Check(next)) ||
-                    frame == NULL || !PyFrame_Check(frame)) {
-        PyErr_BadInternalCall();
-        return NULL;
-    }
-    tb = PyObject_GC_New(PyTracebackObject, &PyTraceBack_Type);
-    if (tb != NULL) {
-        Py_XINCREF(next);
-        tb->tb_next = next;
-        Py_XINCREF(frame);
-        tb->tb_frame = frame;
-        tb->tb_lasti = frame->f_lasti;
-        tb->tb_lineno = PyFrame_GetLineNumber(frame);
-        PyObject_GC_Track(tb);
-    }
-    return tb;
-}
-
 int
 PyTraceBack_Here(PyFrameObject *frame)
 {
     PyObject *exc, *val, *tb, *newtb;
     PyErr_Fetch(&exc, &val, &tb);
-    newtb = (PyObject *)newtracebackobject((PyTracebackObject *)tb, frame);
+    newtb = tb_create_raw((PyTracebackObject *)tb, frame, frame->f_lasti,
+                          PyFrame_GetLineNumber(frame));
     if (newtb == NULL) {
         _PyErr_ChainExceptions(exc, val, tb);
         return -1;


From webhook-mailer at python.org  Sun Jan  7 10:54:37 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 07 Jan 2018 15:54:37 -0000
Subject: [Python-checkins] bpo-31802: Fix importing native path module
 before importing os. (#4017)
Message-ID: <mailman.18.1515340478.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/3460198f6ba40a839f105c381f07179aba1e8c61
commit: 3460198f6ba40a839f105c381f07179aba1e8c61
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-07T17:54:31+02:00
summary:

bpo-31802: Fix importing native path module before importing os. (#4017)

files:
A Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
M Lib/macpath.py
M Lib/ntpath.py
M Lib/posixpath.py
M Lib/test/test_genericpath.py

diff --git a/Lib/macpath.py b/Lib/macpath.py
index f85a91435b9..aacf7235b01 100644
--- a/Lib/macpath.py
+++ b/Lib/macpath.py
@@ -1,5 +1,17 @@
 """Pathname and path-related operations for the Macintosh."""
 
+# strings representing various path-related bits and pieces
+# These are primarily for export; internally, they are hardcoded.
+# Should be set before imports for resolving cyclic dependency.
+curdir = ':'
+pardir = '::'
+extsep = '.'
+sep = ':'
+pathsep = '\n'
+defpath = ':'
+altsep = None
+devnull = 'Dev:Null'
+
 import os
 from stat import *
 import genericpath
@@ -16,17 +28,6 @@
            "curdir","pardir","sep","pathsep","defpath","altsep","extsep",
            "devnull","realpath","supports_unicode_filenames"]
 
-# strings representing various path-related bits and pieces
-# These are primarily for export; internally, they are hardcoded.
-curdir = ':'
-pardir = '::'
-extsep = '.'
-sep = ':'
-pathsep = '\n'
-defpath = ':'
-altsep = None
-devnull = 'Dev:Null'
-
 def _get_colon(path):
     if isinstance(path, bytes):
         return b':'
diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index 10d3f2dc35b..2182ec776cc 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -5,6 +5,18 @@
 module as os.path.
 """
 
+# strings representing various path-related bits and pieces
+# These are primarily for export; internally, they are hardcoded.
+# Should be set before imports for resolving cyclic dependency.
+curdir = '.'
+pardir = '..'
+extsep = '.'
+sep = '\\'
+pathsep = ';'
+altsep = '/'
+defpath = '.;C:\\bin'
+devnull = 'nul'
+
 import os
 import sys
 import stat
@@ -19,17 +31,6 @@
            "extsep","devnull","realpath","supports_unicode_filenames","relpath",
            "samefile", "sameopenfile", "samestat", "commonpath"]
 
-# strings representing various path-related bits and pieces
-# These are primarily for export; internally, they are hardcoded.
-curdir = '.'
-pardir = '..'
-extsep = '.'
-sep = '\\'
-pathsep = ';'
-altsep = '/'
-defpath = '.;C:\\bin'
-devnull = 'nul'
-
 def _get_bothseps(path):
     if isinstance(path, bytes):
         return b'\\/'
diff --git a/Lib/posixpath.py b/Lib/posixpath.py
index 6dbdab27497..e92186c64e0 100644
--- a/Lib/posixpath.py
+++ b/Lib/posixpath.py
@@ -10,6 +10,18 @@
 for manipulation of the pathname component of URLs.
 """
 
+# Strings representing various path-related bits and pieces.
+# These are primarily for export; internally, they are hardcoded.
+# Should be set before imports for resolving cyclic dependency.
+curdir = '.'
+pardir = '..'
+extsep = '.'
+sep = '/'
+pathsep = ':'
+defpath = ':/bin:/usr/bin'
+altsep = None
+devnull = '/dev/null'
+
 import os
 import sys
 import stat
@@ -25,16 +37,6 @@
            "devnull","realpath","supports_unicode_filenames","relpath",
            "commonpath"]
 
-# Strings representing various path-related bits and pieces.
-# These are primarily for export; internally, they are hardcoded.
-curdir = '.'
-pardir = '..'
-extsep = '.'
-sep = '/'
-pathsep = ':'
-defpath = ':/bin:/usr/bin'
-altsep = None
-devnull = '/dev/null'
 
 def _get_sep(path):
     if isinstance(path, bytes):
diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py
index 01e11da0980..ad5a59c44ed 100644
--- a/Lib/test/test_genericpath.py
+++ b/Lib/test/test_genericpath.py
@@ -8,6 +8,7 @@
 import unittest
 import warnings
 from test import support
+from test.support.script_helper import assert_python_ok
 
 
 def create_file(filename, data=b'foo'):
@@ -486,6 +487,9 @@ def test_relpath_errors(self):
             with self.assertRaisesRegex(TypeError, 'bytearray'):
                 self.pathmodule.relpath(bytearray(b'foo'), bytearray(b'bar'))
 
+    def test_import(self):
+        assert_python_ok('-S', '-c', 'import ' + self.pathmodule.__name__)
+
 
 class PathLikeTests(unittest.TestCase):
 
diff --git a/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst b/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
new file mode 100644
index 00000000000..77df6bbe695
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
@@ -0,0 +1,2 @@
+Importing native path module (``posixpath``, ``ntpath``) now works even if
+the ``os`` module still is not imported.


From webhook-mailer at python.org  Sun Jan  7 11:35:30 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 07 Jan 2018 16:35:30 -0000
Subject: [Python-checkins] bpo-31802: Fix importing native path module
 before importing os. (GH-4017) (#5129)
Message-ID: <mailman.19.1515342932.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/94a3facc0e5ce02ef34d9fa595703cd96cb95984
commit: 94a3facc0e5ce02ef34d9fa595703cd96cb95984
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-07T18:35:28+02:00
summary:

bpo-31802: Fix importing native path module before importing os. (GH-4017) (#5129)

(cherry picked from commit 3460198f6ba40a839f105c381f07179aba1e8c61)

files:
A Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
M Lib/macpath.py
M Lib/ntpath.py
M Lib/posixpath.py
M Lib/test/test_genericpath.py

diff --git a/Lib/macpath.py b/Lib/macpath.py
index a90d1053bc8..d15b7e1e9e2 100644
--- a/Lib/macpath.py
+++ b/Lib/macpath.py
@@ -1,5 +1,17 @@
 """Pathname and path-related operations for the Macintosh."""
 
+# strings representing various path-related bits and pieces
+# These are primarily for export; internally, they are hardcoded.
+# Should be set before imports for resolving cyclic dependency.
+curdir = ':'
+pardir = '::'
+extsep = '.'
+sep = ':'
+pathsep = '\n'
+defpath = ':'
+altsep = None
+devnull = 'Dev:Null'
+
 import os
 from stat import *
 import genericpath
@@ -12,17 +24,6 @@
            "curdir","pardir","sep","pathsep","defpath","altsep","extsep",
            "devnull","realpath","supports_unicode_filenames"]
 
-# strings representing various path-related bits and pieces
-# These are primarily for export; internally, they are hardcoded.
-curdir = ':'
-pardir = '::'
-extsep = '.'
-sep = ':'
-pathsep = '\n'
-defpath = ':'
-altsep = None
-devnull = 'Dev:Null'
-
 def _get_colon(path):
     if isinstance(path, bytes):
         return b':'
diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index a8f4b37f648..a5e79ba6637 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -5,6 +5,18 @@
 module as os.path.
 """
 
+# strings representing various path-related bits and pieces
+# These are primarily for export; internally, they are hardcoded.
+# Should be set before imports for resolving cyclic dependency.
+curdir = '.'
+pardir = '..'
+extsep = '.'
+sep = '\\'
+pathsep = ';'
+altsep = '/'
+defpath = '.;C:\\bin'
+devnull = 'nul'
+
 import os
 import sys
 import stat
@@ -19,17 +31,6 @@
            "extsep","devnull","realpath","supports_unicode_filenames","relpath",
            "samefile", "sameopenfile", "samestat", "commonpath"]
 
-# strings representing various path-related bits and pieces
-# These are primarily for export; internally, they are hardcoded.
-curdir = '.'
-pardir = '..'
-extsep = '.'
-sep = '\\'
-pathsep = ';'
-altsep = '/'
-defpath = '.;C:\\bin'
-devnull = 'nul'
-
 def _get_bothseps(path):
     if isinstance(path, bytes):
         return b'\\/'
diff --git a/Lib/posixpath.py b/Lib/posixpath.py
index 6dbdab27497..e92186c64e0 100644
--- a/Lib/posixpath.py
+++ b/Lib/posixpath.py
@@ -10,6 +10,18 @@
 for manipulation of the pathname component of URLs.
 """
 
+# Strings representing various path-related bits and pieces.
+# These are primarily for export; internally, they are hardcoded.
+# Should be set before imports for resolving cyclic dependency.
+curdir = '.'
+pardir = '..'
+extsep = '.'
+sep = '/'
+pathsep = ':'
+defpath = ':/bin:/usr/bin'
+altsep = None
+devnull = '/dev/null'
+
 import os
 import sys
 import stat
@@ -25,16 +37,6 @@
            "devnull","realpath","supports_unicode_filenames","relpath",
            "commonpath"]
 
-# Strings representing various path-related bits and pieces.
-# These are primarily for export; internally, they are hardcoded.
-curdir = '.'
-pardir = '..'
-extsep = '.'
-sep = '/'
-pathsep = ':'
-defpath = ':/bin:/usr/bin'
-altsep = None
-devnull = '/dev/null'
 
 def _get_sep(path):
     if isinstance(path, bytes):
diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py
index 01e11da0980..ad5a59c44ed 100644
--- a/Lib/test/test_genericpath.py
+++ b/Lib/test/test_genericpath.py
@@ -8,6 +8,7 @@
 import unittest
 import warnings
 from test import support
+from test.support.script_helper import assert_python_ok
 
 
 def create_file(filename, data=b'foo'):
@@ -486,6 +487,9 @@ def test_relpath_errors(self):
             with self.assertRaisesRegex(TypeError, 'bytearray'):
                 self.pathmodule.relpath(bytearray(b'foo'), bytearray(b'bar'))
 
+    def test_import(self):
+        assert_python_ok('-S', '-c', 'import ' + self.pathmodule.__name__)
+
 
 class PathLikeTests(unittest.TestCase):
 
diff --git a/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst b/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
new file mode 100644
index 00000000000..77df6bbe695
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
@@ -0,0 +1,2 @@
+Importing native path module (``posixpath``, ``ntpath``) now works even if
+the ``os`` module still is not imported.


From webhook-mailer at python.org  Sun Jan  7 13:59:51 2018
From: webhook-mailer at python.org (Zachary Ware)
Date: Sun, 07 Jan 2018 18:59:51 -0000
Subject: [Python-checkins] Fix version in AppVeyor config (GH-5124)
Message-ID: <mailman.20.1515351592.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/b216a253129b41d7792417bcd7737b2ebdb26c9a
commit: b216a253129b41d7792417bcd7737b2ebdb26c9a
branch: master
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-07T12:59:48-06:00
summary:

Fix version in AppVeyor config (GH-5124)

files:
M .github/appveyor.yml

diff --git a/.github/appveyor.yml b/.github/appveyor.yml
index b7a01335d61..b7d40787318 100644
--- a/.github/appveyor.yml
+++ b/.github/appveyor.yml
@@ -1,4 +1,4 @@
-version: 3.7.0a0.{build}
+version: 3.7build{build}
 clone_depth: 5
 branches:
   only:


From webhook-mailer at python.org  Sun Jan  7 14:00:36 2018
From: webhook-mailer at python.org (Zachary Ware)
Date: Sun, 07 Jan 2018 19:00:36 -0000
Subject: [Python-checkins] Fix version in AppVeyor config (GH-5125)
Message-ID: <mailman.21.1515351638.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/9c99fd163d5ca9bcc0b7ddd0d1e3b8717a63237c
commit: 9c99fd163d5ca9bcc0b7ddd0d1e3b8717a63237c
branch: 3.6
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-07T13:00:33-06:00
summary:

Fix version in AppVeyor config (GH-5125)

files:
M .github/appveyor.yml

diff --git a/.github/appveyor.yml b/.github/appveyor.yml
index b8a01c313f6..162a7d8a701 100644
--- a/.github/appveyor.yml
+++ b/.github/appveyor.yml
@@ -1,4 +1,4 @@
-version: 3.6.1+.{build}
+version: 3.6build{build}
 clone_depth: 5
 branches:
   only:


From webhook-mailer at python.org  Sun Jan  7 14:00:59 2018
From: webhook-mailer at python.org (Zachary Ware)
Date: Sun, 07 Jan 2018 19:00:59 -0000
Subject: [Python-checkins] Fix version in AppVeyor config (GH-5126)
Message-ID: <mailman.22.1515351659.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/0ebf0ae6a2b055f6a2373d9d1976f6b243ed8b00
commit: 0ebf0ae6a2b055f6a2373d9d1976f6b243ed8b00
branch: 2.7
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-07T13:00:56-06:00
summary:

Fix version in AppVeyor config (GH-5126)

files:
M .github/appveyor.yml

diff --git a/.github/appveyor.yml b/.github/appveyor.yml
index 43bd7ed9665..ddd5b15c5aa 100644
--- a/.github/appveyor.yml
+++ b/.github/appveyor.yml
@@ -1,4 +1,4 @@
-version: 2.7.13+.{build}
+version: 2.7build{build}
 clone_depth: 5
 branches:
   only:


From webhook-mailer at python.org  Sun Jan  7 14:30:19 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sun, 07 Jan 2018 19:30:19 -0000
Subject: [Python-checkins] bpo-32506: Change dataclasses from OrderedDict to
 plain dict. (gh-5131)
Message-ID: <mailman.23.1515353420.21073.python-checkins@python.org>

https://github.com/python/cpython/commit/d13889214a4c81b78fa8683d35bdbd17ff22f4fe
commit: d13889214a4c81b78fa8683d35bdbd17ff22f4fe
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-07T14:30:17-05:00
summary:

bpo-32506: Change dataclasses from OrderedDict to plain dict. (gh-5131)

files:
A Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst
M Lib/dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 9e186c3c7b7..d80054920ce 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -1,7 +1,6 @@
 import sys
 import types
 from copy import deepcopy
-import collections
 import inspect
 
 __all__ = ['dataclass',
@@ -448,11 +447,11 @@ def _set_attribute(cls, name, value):
 
 
 def _process_class(cls, repr, eq, order, hash, init, frozen):
-    # Use an OrderedDict because:
-    #  - Order matters!
-    #  - Derived class fields overwrite base class fields, but the
-    #    order is defined by the base class, which is found first.
-    fields = collections.OrderedDict()
+    # Now that dicts retain insertion order, there's no reason to use
+    #  an ordered dict.  I am leveraging that ordering here, because
+    #  derived class fields overwrite base class fields, but the order
+    #  is defined by the base class, which is found first.
+    fields = {}
 
     # Find our base classes in reverse MRO order, and exclude
     #  ourselves.  In reversed order so that more derived classes
@@ -612,7 +611,8 @@ def fields(class_or_instance):
     except AttributeError:
         raise TypeError('must be called with a dataclass type or instance')
 
-    # Exclude pseudo-fields.
+    # Exclude pseudo-fields.  Note that fields is sorted by insertion
+    #  order, so the order of the tuple is as the fields were defined.
     return tuple(f for f in fields.values() if f._field_type is _FIELD)
 
 
@@ -735,7 +735,7 @@ class C(Base):
         # Copy namespace since we're going to mutate it.
         namespace = namespace.copy()
 
-    anns = collections.OrderedDict()
+    anns = {}
     for item in fields:
         if isinstance(item, str):
             name = item
diff --git a/Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst b/Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst
new file mode 100644
index 00000000000..e524769c6df
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst
@@ -0,0 +1,2 @@
+Now that dict is defined as keeping insertion order, drop OrderedDict and
+just use plain dict.


From webhook-mailer at python.org  Sun Jan  7 21:45:05 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Mon, 08 Jan 2018 02:45:05 -0000
Subject: [Python-checkins] bpo-31975 (PEP 565): Show DeprecationWarning in
 __main__ (GH-4458)
Message-ID: <mailman.2.1515379506.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/9b99747386b690007027c3be2a5d7cfe3d3634f5
commit: 9b99747386b690007027c3be2a5d7cfe3d3634f5
branch: master
author: Nick Coghlan <ncoghlan at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-08T12:45:02+10:00
summary:

bpo-31975 (PEP 565): Show DeprecationWarning in __main__ (GH-4458)

- primary change is to add a new default filter entry for
  'default::DeprecationWarning:__main__'
- secondary change is an internal one to cope with plain
  strings in the warning module's internal filter list
  (this avoids the need to create a compiled regex object
  early on during interpreter startup)
- assorted documentation updates, including many more
  examples of configuring the warnings settings
- additional tests to ensure that both the pure Python and
  the C accelerated warnings modules have the expected
  default configuration

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst
M Doc/library/exceptions.rst
M Doc/library/warnings.rst
M Doc/tools/susp-ignored.csv
M Doc/using/cmdline.rst
M Doc/whatsnew/3.7.rst
M Lib/test/test_cmd_line.py
M Lib/test/test_warnings/__init__.py
M Lib/warnings.py
M Python/_warnings.c

diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
index c8d32cff4e0..aa3141053b3 100644
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -661,11 +661,13 @@ depending on the system error code.
    :pep:`3151` - Reworking the OS and IO exception hierarchy
 
 
+.. _warning-categories-as-exceptions:
+
 Warnings
 --------
 
-The following exceptions are used as warning categories; see the :mod:`warnings`
-module for more information.
+The following exceptions are used as warning categories; see the
+:ref:`warning-categories` documentation for more details.
 
 .. exception:: Warning
 
@@ -679,12 +681,14 @@ module for more information.
 
 .. exception:: DeprecationWarning
 
-   Base class for warnings about deprecated features.
+   Base class for warnings about deprecated features when those warnings are
+   intended for other Python developers.
 
 
 .. exception:: PendingDeprecationWarning
 
-   Base class for warnings about features which will be deprecated in the future.
+   Base class for warnings about features which will be deprecated in the
+   future.
 
 
 .. exception:: SyntaxWarning
@@ -699,8 +703,8 @@ module for more information.
 
 .. exception:: FutureWarning
 
-   Base class for warnings about constructs that will change semantically in the
-   future.
+   Base class for warnings about deprecated features when those warnings are
+   intended for end users of applications that are written in Python.
 
 
 .. exception:: ImportWarning
@@ -720,7 +724,8 @@ module for more information.
 
 .. exception:: ResourceWarning
 
-   Base class for warnings related to resource usage.
+   Base class for warnings related to resource usage. Ignored by the default
+   warning filters.
 
    .. versionadded:: 3.2
 
diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst
index f67f4bc24c4..b04bd79e4bb 100644
--- a/Doc/library/warnings.rst
+++ b/Doc/library/warnings.rst
@@ -51,8 +51,17 @@ Warning Categories
 ------------------
 
 There are a number of built-in exceptions that represent warning categories.
-This categorization is useful to be able to filter out groups of warnings.  The
-following warnings category classes are currently defined:
+This categorization is useful to be able to filter out groups of warnings.
+
+While these are technically
+:ref:`built-in exceptions <warning-categories-as-exceptions>`, they are
+documented here, because conceptually they belong to the warnings mechanism.
+
+User code can define additional warning categories by subclassing one of the
+standard warning categories.  A warning category must always be a subclass of
+the :exc:`Warning` class.
+
+The following warnings category classes are currently defined:
 
 .. tabularcolumns:: |l|p{0.6\linewidth}|
 
@@ -66,7 +75,9 @@ following warnings category classes are currently defined:
 | :exc:`UserWarning`               | The default category for :func:`warn`.        |
 +----------------------------------+-----------------------------------------------+
 | :exc:`DeprecationWarning`        | Base category for warnings about deprecated   |
-|                                  | features (ignored by default).                |
+|                                  | features when those warnings are intended for |
+|                                  | other Python developers (ignored by default,  |
+|                                  | unless triggered by code in ``__main__``).    |
 +----------------------------------+-----------------------------------------------+
 | :exc:`SyntaxWarning`             | Base category for warnings about dubious      |
 |                                  | syntactic features.                           |
@@ -74,8 +85,10 @@ following warnings category classes are currently defined:
 | :exc:`RuntimeWarning`            | Base category for warnings about dubious      |
 |                                  | runtime features.                             |
 +----------------------------------+-----------------------------------------------+
-| :exc:`FutureWarning`             | Base category for warnings about constructs   |
-|                                  | that will change semantically in the future.  |
+| :exc:`FutureWarning`             | Base category for warnings about deprecated   |
+|                                  | features when those warnings are intended for |
+|                                  | end users of applications that are written in |
+|                                  | Python.                                       |
 +----------------------------------+-----------------------------------------------+
 | :exc:`PendingDeprecationWarning` | Base category for warnings about features     |
 |                                  | that will be deprecated in the future         |
@@ -95,13 +108,12 @@ following warnings category classes are currently defined:
 |                                  | resource usage.                               |
 +----------------------------------+-----------------------------------------------+
 
-
-While these are technically built-in exceptions, they are documented here,
-because conceptually they belong to the warnings mechanism.
-
-User code can define additional warning categories by subclassing one of the
-standard warning categories.  A warning category must always be a subclass of
-the :exc:`Warning` class.
+.. versionchanged:: 3.7
+  Previously :exc:`DeprecationWarning` and :exc:`FutureWarning` were
+  distinguished based on whether a feature was being removed entirely or
+  changing its behaviour. They are now distinguished based on their
+  intended audience and the way they're handled by the default warnings
+  filters.
 
 
 .. _warning-filter:
@@ -114,7 +126,7 @@ into errors (raising an exception).
 
 Conceptually, the warnings filter maintains an ordered list of filter
 specifications; any specific warning is matched against each filter
-specification in the list in turn until a match is found; the match determines
+specification in the list in turn until a match is found; the filter determines
 the disposition of the match.  Each entry is a tuple of the form (*action*,
 *message*, *category*, *module*, *lineno*), where:
 
@@ -123,19 +135,19 @@ the disposition of the match.  Each entry is a tuple of the form (*action*,
   +---------------+----------------------------------------------+
   | Value         | Disposition                                  |
   +===============+==============================================+
+  | ``"default"`` | print the first occurrence of matching       |
+  |               | warnings for each location (module +         |
+  |               | line number) where the warning is issued     |
+  +---------------+----------------------------------------------+
   | ``"error"``   | turn matching warnings into exceptions       |
   +---------------+----------------------------------------------+
   | ``"ignore"``  | never print matching warnings                |
   +---------------+----------------------------------------------+
   | ``"always"``  | always print matching warnings               |
   +---------------+----------------------------------------------+
-  | ``"default"`` | print the first occurrence of matching       |
-  |               | warnings for each location where the warning |
-  |               | is issued                                    |
-  +---------------+----------------------------------------------+
   | ``"module"``  | print the first occurrence of matching       |
   |               | warnings for each module where the warning   |
-  |               | is issued                                    |
+  |               | is issued (regardless of line number)        |
   +---------------+----------------------------------------------+
   | ``"once"``    | print only the first occurrence of matching  |
   |               | warnings, regardless of location             |
@@ -157,33 +169,119 @@ the disposition of the match.  Each entry is a tuple of the form (*action*,
 Since the :exc:`Warning` class is derived from the built-in :exc:`Exception`
 class, to turn a warning into an error we simply raise ``category(message)``.
 
+If a warning is reported and doesn't match any registered filter then the
+"default" action is applied (hence its name).
+
+
+.. _describing-warning-filters:
+
+Describing Warning Filters
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 The warnings filter is initialized by :option:`-W` options passed to the Python
-interpreter command line.  The interpreter saves the arguments for all
-:option:`-W` options without interpretation in ``sys.warnoptions``; the
-:mod:`warnings` module parses these when it is first imported (invalid options
-are ignored, after printing a message to ``sys.stderr``).
+interpreter command line and the :envvar:`PYTHONWARNINGS` environment variable.
+The interpreter saves the arguments for all supplied entries without
+interpretation in ``sys.warnoptions``; the :mod:`warnings` module parses these
+when it is first imported (invalid options are ignored, after printing a
+message to ``sys.stderr``).
+
+Individual warnings filters are specified as a sequence of fields separated by
+colons::
 
+   action:message:category:module:line
 
-Default Warning Filters
-~~~~~~~~~~~~~~~~~~~~~~~
+The meaning of each of these fields is as described in :ref:`warning-filter`.
+When listing multiple filters on a single line (as for
+:envvar:`PYTHONWARNINGS`), the individual filters are separated by commas,and
+the filters listed later take precedence over those listed before them (as
+they're applied left-to-right, and the most recently applied filters take
+precedence over earlier ones).
+
+Commonly used warning filters apply to either all warnings, warnings in a
+particular category, or warnings raised by particular modules or packages.
+Some examples::
+
+   default                      # Show all warnings (even those ignored by default)
+   ignore                       # Ignore all warnings
+   error                        # Convert all warnings to errors
+   error::ResourceWarning       # Treat ResourceWarning messages as errors
+   default::DeprecationWarning  # Show DeprecationWarning messages
+   ignore,default:::mymodule    # Only report warnings triggered by "mymodule"
+   error:::mymodule[.*]         # Convert warnings to errors in "mymodule"
+                                # and any subpackages of "mymodule"
+
+
+.. _default-warning-filter:
+
+Default Warning Filter
+~~~~~~~~~~~~~~~~~~~~~~
 
 By default, Python installs several warning filters, which can be overridden by
-the command-line options passed to :option:`-W` and calls to
-:func:`filterwarnings`.
+the :option:`-W` command-line option, the :envvar:`PYTHONWARNINGS` environment
+variable and calls to :func:`filterwarnings`.
 
-* :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, and
-  :exc:`ImportWarning` are ignored.
+In regular release builds, the default warning filter has the following entries
+(in order of precedence)::
 
-* :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or
-  twice; in this case this warning is either printed (``-b``) or turned into an
-  exception (``-bb``).
+    default::DeprecationWarning:__main__
+    ignore::DeprecationWarning
+    ignore::PendingDeprecationWarning
+    ignore::ImportWarning
+    ignore::ResourceWarning
 
-* :exc:`ResourceWarning` is ignored unless Python was built in debug mode.
+In debug builds, the list of default warning filters is empty.
 
 .. versionchanged:: 3.2
    :exc:`DeprecationWarning` is now ignored by default in addition to
    :exc:`PendingDeprecationWarning`.
 
+.. versionchanged:: 3.7
+  :exc:`DeprecationWarning` is once again shown by default when triggered
+  directly by code in ``__main__``.
+
+.. versionchanged:: 3.7
+  :exc:`BytesWarning` no longer appears in the default filter list and is
+  instead configured via :data:`sys.warnoptions` when :option:`-b` is specified
+  twice.
+
+
+.. _warning-disable:
+
+Overriding the default filter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Developers of applications written in Python may wish to hide *all* Python level
+warnings from their users by default, and only display them when running tests
+or otherwise working on the application. The :data:`sys.warnoptions` attribute
+used to pass filter configurations to the interpreter can be used as a marker to
+indicate whether or not warnings should be disabled::
+
+    import sys
+
+    if not sys.warnoptions:
+        import warnings
+        warnings.simplefilter("ignore")
+
+Developers of test runners for Python code are advised to instead ensure that
+*all* warnings are displayed by default for the code under test, using code
+like::
+
+    import sys
+
+    if not sys.warnoptions:
+        import os, warnings
+        warnings.simplefilter("default") # Change the filter in this process
+        os.environ["PYTHONWARNINGS"] = "default" # Also affect subprocesses
+
+Finally, developers of interactive shells that run user code in a namespace
+other than ``__main__`` are advised to ensure that :exc:`DeprecationWarning`
+messages are made visible by default, using code like the following (where
+``user_ns`` is the module used to execute code entered interactively)::
+
+    import warnings
+    warnings.filterwarnings("default", category=DeprecationWarning,
+                                       module=user_ns.get("__name__"))
+
 
 .. _warning-suppress:
 
@@ -191,7 +289,8 @@ Temporarily Suppressing Warnings
 --------------------------------
 
 If you are using code that you know will raise a warning, such as a deprecated
-function, but do not want to see the warning, then it is possible to suppress
+function, but do not want to see the warning (even when warnings have been
+explicitly configured via the command line), then it is possible to suppress
 the warning using the :class:`catch_warnings` context manager::
 
     import warnings
@@ -261,38 +360,30 @@ entries from the warnings list before each new operation).
 
 .. _warning-ignored:
 
-Updating Code For New Versions of Python
-----------------------------------------
-
-Warnings that are only of interest to the developer are ignored by default. As
-such you should make sure to test your code with typically ignored warnings
-made visible. You can do this from the command-line by passing :option:`-Wd <-W>`
-to the interpreter (this is shorthand for :option:`!-W default`).  This enables
-default handling for all warnings, including those that are ignored by default.
-To change what action is taken for encountered warnings you simply change what
-argument is passed to :option:`-W`, e.g. :option:`!-W error`. See the
-:option:`-W` flag for more details on what is possible.
-
-To programmatically do the same as :option:`!-Wd`, use::
-
-  warnings.simplefilter('default')
-
-Make sure to execute this code as soon as possible. This prevents the
-registering of what warnings have been raised from unexpectedly influencing how
-future warnings are treated.
-
-Having certain warnings ignored by default is done to prevent a user from
-seeing warnings that are only of interest to the developer. As you do not
-necessarily have control over what interpreter a user uses to run their code,
-it is possible that a new version of Python will be released between your
-release cycles.  The new interpreter release could trigger new warnings in your
-code that were not there in an older interpreter, e.g.
-:exc:`DeprecationWarning` for a module that you are using. While you as a
-developer want to be notified that your code is using a deprecated module, to a
-user this information is essentially noise and provides no benefit to them.
-
-The :mod:`unittest` module has been also updated to use the ``'default'``
-filter while running tests.
+Updating Code For New Versions of Dependencies
+----------------------------------------------
+
+Warning categories that are primarily of interest to Python developers (rather
+than end users of applications written in Python) are ignored by default.
+
+Notably, this "ignored by default" list includes :exc:`DeprecationWarning`
+(for every module except ``__main__``), which means developers should make sure
+to test their code with typically ignored warnings made visible in order to
+receive timely notifications of future breaking API changes (whether in the
+standard library or third party packages).
+
+In the ideal case, the code will have a suitable test suite, and the test runner
+will take care of implicitly enabling all warnings when running tests
+(the test runner provided by the :mod:`unittest` module does this).
+
+In less ideal cases, applications can be checked for use of deprecated
+interfaces by passing :option:`-Wd <-W>` to the Python interpreter (this is
+shorthand for :option:`!-W default`) or setting ``PYTHONWARNINGS=default`` in
+the environment. This enables default handling for all warnings, including those
+that are ignored by default. To change what action is taken for encountered
+warnings you can change what argument is passed to :option:`-W` (e.g.
+:option:`!-W error`). See the :option:`-W` flag for more details on what is
+possible.
 
 
 .. _warning-functions:
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index 48dd53f85d2..cfdd5266c51 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -259,12 +259,8 @@ tutorial/stdlib2,,:start,extra = data[start:start+extra_size]
 tutorial/stdlib2,,:start,"fields = struct.unpack('<IIIHH', data[start:start+16])"
 tutorial/stdlib2,,:start,filename = data[start:start+filenamesize]
 tutorial/stdlib2,,:Warning,WARNING:root:Warning:config file server.conf not found
-using/cmdline,,:category,action:message:category:module:line
 using/cmdline,,:errorhandler,:errorhandler
-using/cmdline,,:line,action:message:category:module:line
 using/cmdline,,:line,file:line: category: message
-using/cmdline,,:message,action:message:category:module:line
-using/cmdline,,:module,action:message:category:module:line
 using/unix,,:Packaging,https://en.opensuse.org/Portal:Packaging
 whatsnew/2.0,,:len,
 whatsnew/2.3,,::,
@@ -302,6 +298,20 @@ whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf
 library/re,,`,!#$%&'*+-.^_`|~:
 library/re,,`,!\#\$%\&'\*\+\-\.\^_`\|\~:
 library/tarfile,,:xz,'x:xz'
+library/warnings,,:message,action:message:category:module:line
+library/warnings,,:category,action:message:category:module:line
+library/warnings,,:module,action:message:category:module:line
+library/warnings,,:line,action:message:category:module:line
+library/warnings,,::,error::ResourceWarning
+library/warnings,,::,default::DeprecationWarning
+library/warnings,,::,default:::mymodule
+library/warnings,,:mymodule,default:::mymodule
+library/warnings,,::,error:::mymodule
+library/warnings,,:mymodule,error:::mymodule
+library/warnings,,::,ignore::DeprecationWarning
+library/warnings,,::,ignore::PendingDeprecationWarning
+library/warnings,,::,ignore::ImportWarning
+library/warnings,,::,ignore::ResourceWarning
 library/xml.etree.elementtree,,:sometag,prefix:sometag
 library/xml.etree.elementtree,,:fictional,"<actors xmlns:fictional=""http://characters.example.com"""
 library/xml.etree.elementtree,,:character,<fictional:character>Lancelot</fictional:character>
@@ -330,3 +340,4 @@ whatsnew/3.7,,`,'`'
 whatsnew/3.7,,::,error::BytesWarning
 whatsnew/changelog,,::,error::BytesWarning
 whatsnew/changelog,,::,default::BytesWarning
+whatsnew/changelog,,::,default::DeprecationWarning
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
index b1bd47fa6b9..1e9ed6e645a 100644
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -356,49 +356,27 @@ Miscellaneous options
    :option:`-W` options are ignored (though, a warning message is printed about
    invalid options when the first warning is issued).
 
-   Warnings can also be controlled from within a Python program using the
+   Warnings can also be controlled using the :envvar:`PYTHONWARNINGS`
+   environment variable and from within a Python program using the
    :mod:`warnings` module.
 
-   The simplest form of argument is one of the following action strings (or a
-   unique abbreviation):
-
-   ``ignore``
-      Ignore all warnings.
-   ``default``
-      Explicitly request the default behavior (printing each warning once per
-      source line).
-   ``all``
-      Print a warning each time it occurs (this may generate many messages if a
-      warning is triggered repeatedly for the same source line, such as inside a
-      loop).
-   ``module``
-      Print each warning only the first time it occurs in each module.
-   ``once``
-      Print each warning only the first time it occurs in the program.
-   ``error``
-      Raise an exception instead of printing a warning message.
-
-   The full form of argument is::
-
-       action:message:category:module:line
-
-   Here, *action* is as explained above but only applies to messages that match
-   the remaining fields.  Empty fields match all values; trailing empty fields
-   may be omitted.  The *message* field matches the start of the warning message
-   printed; this match is case-insensitive.  The *category* field matches the
-   warning category.  This must be a class name; the match tests whether the
-   actual warning category of the message is a subclass of the specified warning
-   category.  The full class name must be given.  The *module* field matches the
-   (fully-qualified) module name; this match is case-sensitive.  The *line*
-   field matches the line number, where zero matches all line numbers and is
-   thus equivalent to an omitted line number.
+   The simplest settings apply a particular action unconditionally to all
+   warnings emitted by a process (even those that are otherwise ignored by
+   default)::
 
-   .. seealso::
-      :mod:`warnings` -- the warnings module
+       -Wdefault  # Warn once per call location
+       -Werror    # Convert to exceptions
+       -Walways   # Warn every time
+       -Wmodule   # Warn once per calling module
+       -Wonce     # Warn once per Python process
+       -Wignore   # Never warn
 
-      :pep:`230` -- Warning framework
+   The action names can be abbreviated as desired (e.g. ``-Wi``, ``-Wd``,
+   ``-Wa``, ``-We``) and the interpreter will resolve them to the appropriate
+   action name.
 
-      :envvar:`PYTHONWARNINGS`
+   See :ref:`warning-filter` and :ref:`describing-warning-filters` for more
+   details.
 
 
 .. cmdoption:: -x
@@ -659,7 +637,23 @@ conflict.
 
    This is equivalent to the :option:`-W` option. If set to a comma
    separated string, it is equivalent to specifying :option:`-W` multiple
-   times.
+   times, with filters later in the list taking precedence over those earlier
+   in the list.
+
+   The simplest settings apply a particular action unconditionally to all
+   warnings emitted by a process (even those that are otherwise ignored by
+   default)::
+
+       PYTHONWARNINGS=default  # Warn once per call location
+       PYTHONWARNINGS=error    # Convert to exceptions
+       PYTHONWARNINGS=always   # Warn every time
+       PYTHONWARNINGS=module   # Warn once per calling module
+       PYTHONWARNINGS=once     # Warn once per Python process
+       PYTHONWARNINGS=ignore   # Never warn
+
+   See :ref:`warning-filter` and :ref:`describing-warning-filters` for more
+   details.
+
 
 .. envvar:: PYTHONFAULTHANDLER
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 9785d599dd3..992d9ba6e58 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -70,6 +70,7 @@ Summary -- Release highlights
 New Features
 ============
 
+
 .. _whatsnew37-pep538:
 
 PEP 538: Legacy C Locale Coercion
@@ -107,6 +108,7 @@ locale remains active when the core interpreter is initialized.
     :pep:`538` -- Coercing the legacy C locale to a UTF-8 based locale
        PEP written and implemented by Nick Coghlan.
 
+
 .. _whatsnew37-pep553:
 
 PEP 553: Built-in breakpoint()
@@ -203,6 +205,44 @@ resolution on Linux and Windows.
        PEP written and implemented by Victor Stinner
 
 
+.. _whatsnew37-pep565:
+
+PEP 565: Show DeprecationWarning in ``__main__``
+------------------------------------------------
+
+The default handling of :exc:`DeprecationWarning` has been changed such that
+these warnings are once more shown by default, but only when the code
+triggering them is running directly in the ``__main__`` module. As a result,
+developers of single file scripts and those using Python interactively should
+once again start seeing deprecation warnings for the APIs they use, but
+deprecation warnings triggered by imported application, library and framework
+modules will continue to be hidden by default.
+
+As a result of this change, the standard library now allows developers to choose
+between three different deprecation warning behaviours:
+
+* :exc:`FutureWarning`: always displayed by default, recommended for warnings
+  intended to be seen by application end users (e.g. for deprecated application
+  configuration settings).
+* :exc:`DeprecationWarning`: displayed by default only in ``__main__`` and when
+  running tests, recommended for warnings intended to be seen by other Python
+  developers where a version upgrade may result in changed behaviour or an
+  error.
+* :exc:`PendingDeprecationWarning`: displayed by default only when running
+  tests, intended for cases where a future version upgrade will change the
+  warning category to :exc:`DeprecationWarning` or :exc:`FutureWarning`.
+
+Previously both :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`
+were only visible when running tests, which meant that developers primarily
+writing single file scripts or using Python interactively could be surprised
+by breaking changes in the APIs they used.
+
+.. seealso::
+
+    :pep:`565` -- Show DeprecationWarning in ``__main__``
+      PEP written and implemented by Nick Coghlan
+
+
 PEP 540: Add a new UTF-8 mode
 -----------------------------
 
diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
index 54ea3773a06..a6b663403f4 100644
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -558,6 +558,7 @@ def test_xdev(self):
             expected_filters = "default::Warning"
         else:
             expected_filters = ("default::Warning "
+                                "default::DeprecationWarning "
                                 "ignore::DeprecationWarning "
                                 "ignore::PendingDeprecationWarning "
                                 "ignore::ImportWarning "
@@ -626,6 +627,7 @@ def test_warnings_filter_precedence(self):
                             "always::UserWarning")
         if not Py_DEBUG:
             expected_filters += (" "
+                                 "default::DeprecationWarning "
                                  "ignore::DeprecationWarning "
                                  "ignore::PendingDeprecationWarning "
                                  "ignore::ImportWarning "
diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py
index 039c96e02c9..31ab94b7471 100644
--- a/Lib/test/test_warnings/__init__.py
+++ b/Lib/test/test_warnings/__init__.py
@@ -16,6 +16,8 @@
 py_warnings = support.import_fresh_module('warnings', blocked=['_warnings'])
 c_warnings = support.import_fresh_module('warnings', fresh=['_warnings'])
 
+Py_DEBUG = hasattr(sys, 'gettotalrefcount')
+
 @contextmanager
 def warnings_state(module):
     """Use a specific warnings implementation in warning_tests."""
@@ -320,6 +322,7 @@ def test_simplefilter_duplicate_filters(self):
                 self.module.filters[0][0], "error",
                 "simplefilter did not promote filter to the beginning of list"
             )
+
     def test_append_duplicate(self):
         with original_warnings.catch_warnings(module=self.module,
                 record=True) as w:
@@ -1143,6 +1146,37 @@ def test_conflicting_envvar_and_command_line(self):
              b"  File \"<string>\", line 1, in <module>",
              b"DeprecationWarning: Message"])
 
+    def test_default_filter_configuration(self):
+        pure_python_api = self.module is py_warnings
+        if Py_DEBUG:
+            expected_default_filters = []
+        else:
+            if pure_python_api:
+                main_module_filter = re.compile("__main__")
+            else:
+                main_module_filter = "__main__"
+            expected_default_filters = [
+                ('default', None, DeprecationWarning, main_module_filter, 0),
+                ('ignore', None, DeprecationWarning, None, 0),
+                ('ignore', None, PendingDeprecationWarning, None, 0),
+                ('ignore', None, ImportWarning, None, 0),
+                ('ignore', None, ResourceWarning, None, 0),
+            ]
+        expected_output = [str(f).encode() for f in expected_default_filters]
+
+        if pure_python_api:
+            # Disable the warnings acceleration module in the subprocess
+            code = "import sys; sys.modules.pop('warnings', None); sys.modules['_warnings'] = None; "
+        else:
+            code = ""
+        code += "import warnings; [print(f) for f in warnings.filters]"
+
+        rc, stdout, stderr = assert_python_ok("-c", code, __isolated=True)
+        stdout_lines = [line.strip() for line in stdout.splitlines()]
+        self.maxDiff = None
+        self.assertEqual(stdout_lines, expected_output)
+
+
     @unittest.skipUnless(sys.getfilesystemencoding() != 'ascii',
                          'requires non-ascii filesystemencoding')
     def test_nonascii(self):
@@ -1192,7 +1226,7 @@ def __del__(self):
         rc, out, err = assert_python_ok("-c", code)
         # note: "__main__" filename is not correct, it should be the name
         # of the script
-        self.assertEqual(err, b'__main__:7: UserWarning: test')
+        self.assertEqual(err.decode(), '__main__:7: UserWarning: test')
 
     def test_late_resource_warning(self):
         # Issue #21925: Emitting a ResourceWarning late during the Python
diff --git a/Lib/warnings.py b/Lib/warnings.py
index f4331c8ac38..76ad4dac018 100644
--- a/Lib/warnings.py
+++ b/Lib/warnings.py
@@ -519,8 +519,10 @@ def _filters_mutated():
 # Module initialization
 _processoptions(sys.warnoptions)
 if not _warnings_defaults:
-    # Several warning categories are ignored by default in Py_DEBUG builds
+    # Several warning categories are ignored by default in regular builds
     if not hasattr(sys, 'gettotalrefcount'):
+        filterwarnings("default", category=DeprecationWarning,
+                       module="__main__", append=1)
         simplefilter("ignore", category=DeprecationWarning, append=1)
         simplefilter("ignore", category=PendingDeprecationWarning, append=1)
         simplefilter("ignore", category=ImportWarning, append=1)
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst
new file mode 100644
index 00000000000..98cfae0dd8c
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst	
@@ -0,0 +1,4 @@
+The default warning filter list now starts with a
+"default::DeprecationWarning:__main__" entry, so deprecation warnings are
+once again shown by default in single-file scripts and at the interactive
+prompt.
diff --git a/Python/_warnings.c b/Python/_warnings.c
index be8370da06d..c286364dda0 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -12,6 +12,7 @@ MODULE_NAME " provides basic warning filtering support.\n"
 _Py_IDENTIFIER(argv);
 _Py_IDENTIFIER(stderr);
 #ifndef Py_DEBUG
+_Py_IDENTIFIER(default);
 _Py_IDENTIFIER(ignore);
 #endif
 
@@ -22,8 +23,20 @@ check_matched(PyObject *obj, PyObject *arg)
     _Py_IDENTIFIER(match);
     int rc;
 
+    /* A 'None' filter always matches */
     if (obj == Py_None)
         return 1;
+
+    /* An internal plain text default filter must match exactly */
+    if (PyUnicode_CheckExact(obj)) {
+        int cmp_result = PyUnicode_Compare(obj, arg);
+        if (cmp_result == -1 && PyErr_Occurred()) {
+            return -1;
+        }
+        return !cmp_result;
+    }
+
+    /* Otherwise assume a regex filter and call its match() method */
     result = _PyObject_CallMethodIdObjArgs(obj, &PyId_match, arg, NULL);
     if (result == NULL)
         return -1;
@@ -1158,16 +1171,27 @@ static PyMethodDef warnings_functions[] = {
 
 #ifndef Py_DEBUG
 static PyObject *
-create_filter(PyObject *category, _Py_Identifier *id)
+create_filter(PyObject *category, _Py_Identifier *id, const char *modname)
 {
+    PyObject *modname_obj = NULL;
     PyObject *action_str = _PyUnicode_FromId(id);
     if (action_str == NULL) {
         return NULL;
     }
 
+    /* Default to "no module name" for initial filter set */
+    if (modname != NULL) {
+        modname_obj = PyUnicode_InternFromString(modname);
+        if (modname_obj == NULL) {
+            return NULL;
+        }
+    } else {
+        modname_obj = Py_None;
+    }
+
     /* This assumes the line number is zero for now. */
     return PyTuple_Pack(5, action_str, Py_None,
-                        category, Py_None, _PyLong_Zero);
+                        category, modname_obj, _PyLong_Zero);
 }
 #endif
 
@@ -1180,20 +1204,22 @@ init_filters(void)
     return PyList_New(0);
 #else
     /* Other builds ignore a number of warning categories by default */
-    PyObject *filters = PyList_New(4);
+    PyObject *filters = PyList_New(5);
     if (filters == NULL) {
         return NULL;
     }
 
     size_t pos = 0;  /* Post-incremented in each use. */
     PyList_SET_ITEM(filters, pos++,
-                    create_filter(PyExc_DeprecationWarning, &PyId_ignore));
+                    create_filter(PyExc_DeprecationWarning, &PyId_default, "__main__"));
+    PyList_SET_ITEM(filters, pos++,
+                    create_filter(PyExc_DeprecationWarning, &PyId_ignore, NULL));
     PyList_SET_ITEM(filters, pos++,
-                    create_filter(PyExc_PendingDeprecationWarning, &PyId_ignore));
+                    create_filter(PyExc_PendingDeprecationWarning, &PyId_ignore, NULL));
     PyList_SET_ITEM(filters, pos++,
-                    create_filter(PyExc_ImportWarning, &PyId_ignore));
+                    create_filter(PyExc_ImportWarning, &PyId_ignore, NULL));
     PyList_SET_ITEM(filters, pos++,
-                    create_filter(PyExc_ResourceWarning, &PyId_ignore));
+                    create_filter(PyExc_ResourceWarning, &PyId_ignore, NULL));
 
     for (size_t x = 0; x < pos; x++) {
         if (PyList_GET_ITEM(filters, x) == NULL) {


From solipsis at pitrou.net  Mon Jan  8 04:13:38 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 08 Jan 2018 09:13:38 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=4
Message-ID: <20180108091338.1.02FF286F2B4319F8@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 7, -7] memory blocks, sum=0
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogUb3VGl', '--timeout', '7200']

From lp_benchmark_robot at intel.com  Mon Jan  8 19:12:10 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Mon, 8 Jan 2018 16:12:10 -0800
Subject: [Python-checkins] [1 down, 1 up,
 63 flat] Results for Python (master branch) 2018-01-08
Message-ID: <3abfad68-7c49-4267-ba45-24c9ff258047@orsmsx102.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-08 03:04:20-08:00.
  - commit:		9b99747
  - previous commit:	94459fd
  - revision date:	2018-01-08 12:45:02+10:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.877% |  -0.468%   |  +6.634%   |  +7.355%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 0.902% |  +0.593%   |  +22.126%  |  +10.038%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.801% |  +0.481%   |  +24.188%  |  +9.773%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 1.106% |  +0.687%   |  +21.943%  |  +8.379%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 3.284% |  +0.922%   |  +8.557%   |  +12.401%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.392% |  +0.148%   |  +9.448%   |  +13.009%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.010% |  +0.638%   |  +5.428%   |  +11.480%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.633% |  -0.203%   |  -0.922%   |  +7.934%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 3.357% |  +0.092%   |  +9.715%   |  +17.606%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 4.004% |  +0.391%   |  +12.568%  |  +13.919%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.010% |  +0.505%   |  +3.827%   |  +6.857%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.661% |  -0.325%   |  +4.345%   |  +5.370%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.544% |  -0.081%   |  +2.524%   |  +6.957%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.276% |  +0.050%   |  +12.087%  |  +10.860%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.701% |  -1.640%   |  +7.486%   |  +11.358%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.966% |  +0.267%   |  +6.629%   |  +11.498%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.546% |  +0.437%   |  +10.366%  |  +10.286%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.189% |  +1.888%   |  +8.009%   |  +10.683%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.315% |  +0.366%   |  +3.227%   |  +9.403%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 4.750% |  -0.051%   |  -0.618%   |  +12.792%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.483% |  -0.336%   |  +8.196%   |  +12.579%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.899% |  -0.893%   |  +46.368%  |  +12.253%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.525% |  -0.604%   |  +8.868%   |  +14.074%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.950% |  -0.485%   |  +17.029%  |  +13.877%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 4.237% |  -0.641%   |  +11.164%  |  +7.549%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.967% |  +1.376%   |  +3.759%   |  +6.018%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.760% |  +0.160%   |  -4.900%   |  +1.717%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.875% |  -0.129%   |  +4.204%   |  +7.350%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.638% |  +2.027%   |  +3.040%   |  +10.370%  |
+-----+------------------------+--------+------------+------------+------------+
| :-( |                  pickle| 0.805% |  -2.969%   |  -2.012%   |  +23.179%  |
+-----+------------------------+--------+------------+------------+------------+
| :-) |             pickle_dict| 0.147% |  +1.726%   |  +3.424%   |  +17.158%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 1.840% |  +1.285%   |  +4.727%   |  +17.140%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 2.336% |  -0.031%   |  +12.019%  |  +8.932%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.090% |  -0.068%   |  -0.107%   |  +10.187%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.159% |  -0.153%   |  +10.593%  |  +5.432%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.129% |  -0.158%   |  +1.909%   |  +5.496%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.866% |  +0.074%   |  +10.372%  |  +13.845%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 6.122% |  -0.466%   |  +0.959%   |  +11.910%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.251% |  +0.407%   |  -0.574%   |  +10.239%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 1.926% |  -0.750%   |  -4.838%   |  +7.192%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 2.440% |  +0.630%   |  +5.731%   |  +4.391%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.562% |  -0.055%   |  +8.677%   |  +15.131%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.800% |  +0.221%   |  -2.854%   |  +5.201%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 3.310% |  +0.467%   |  +26.317%  |  +8.063%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.134% |  -0.125%   |  +4.755%   |  +5.065%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.914% |  +0.360%   |  +14.682%  |  +8.204%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.690% |  -1.118%   |  -6.649%   |  +1.040%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.430% |  -0.994%   |  +5.043%   |  +4.038%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.310% |  -0.197%   |  +5.958%   |  +6.834%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.252% |  +0.621%   |  +5.812%   |  +5.297%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 3.597% |  +0.772%   |  +19.072%  |  +7.331%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.092% |  +0.554%   |  +11.622%  |  +7.130%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.515% |  -0.281%   |  +8.068%   |  +8.008%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.648% |  +0.165%   |  +11.621%  |  +8.536%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.177% |  -0.862%   |  +10.313%  |  +9.834%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 5.259% |  -2.292%   |  +17.677%  |  +13.728%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.903% |  -0.190%   |  +5.890%   |  +6.610%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 2.419% |  +0.604%   |  +2.127%   |  -1.123%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 7.309% |  -2.272%   |  +4.388%   |  +24.216%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 1.615% |  +2.433%   |  -2.367%   |  +17.246%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 1.495% |  -0.781%   |  +7.857%   |  +6.644%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.099% |  -0.983%   |  +0.501%   |  +13.741%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.306% |  -1.053%   |  +1.928%   |  +9.173%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 4.331% |  -1.302%   |  -9.694%   |  +14.885%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.306% |  -1.169%   |  +2.832%   |  +12.798%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-down-1-up-63-flat-results-for-python-master-branch-2018-01-08

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Mon Jan  8 21:08:02 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 09 Jan 2018 02:08:02 -0000
Subject: [Python-checkins] bpo-32248: install new test_importlib directories
 (#5138)
Message-ID: <mailman.3.1515463683.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/ca0c5f26563349ed761af13590acef80e1064eab
commit: ca0c5f26563349ed761af13590acef80e1064eab
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-08T21:07:59-05:00
summary:

bpo-32248: install new test_importlib directories (#5138)

files:
M Makefile.pre.in

diff --git a/Makefile.pre.in b/Makefile.pre.in
index 4fe9affe0d2..59ba9d4c1f1 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1250,9 +1250,26 @@ LIBSUBDIRS=	tkinter tkinter/test tkinter/test/test_tkinter \
 		test/test_import/data/circular_imports/subpkg \
 		test/test_import/data/package \
 		test/test_import/data/package2 \
+		importlib \
+		test/test_importlib \
+		test/test_importlib/builtin \
+		test/test_importlib/data01 \
+		test/test_importlib/data01/subdirectory \
+		test/test_importlib/data02 \
+		test/test_importlib/data02/one \
+		test/test_importlib/data02/two \
+		test/test_importlib/data03 \
+		test/test_importlib/data03/namespace \
+		test/test_importlib/data03/namespace/portion1 \
+		test/test_importlib/data03/namespace/portion2 \
+		test/test_importlib/extension \
+		test/test_importlib/frozen \
+		test/test_importlib/import_ \
 		test/test_importlib/namespace_pkgs \
 		test/test_importlib/namespace_pkgs/both_portions \
 		test/test_importlib/namespace_pkgs/both_portions/foo \
+		test/test_importlib/namespace_pkgs/module_and_namespace_package \
+		test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test \
 		test/test_importlib/namespace_pkgs/not_a_namespace_pkg \
 		test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo \
 		test/test_importlib/namespace_pkgs/portion1 \
@@ -1268,8 +1285,9 @@ LIBSUBDIRS=	tkinter tkinter/test tkinter/test/test_tkinter \
 		test/test_importlib/namespace_pkgs/project3 \
 		test/test_importlib/namespace_pkgs/project3/parent \
 		test/test_importlib/namespace_pkgs/project3/parent/child \
-                test/test_importlib/namespace_pkgs/module_and_namespace_package \
-                test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test \
+		test/test_importlib/source \
+		test/test_importlib/zipdata01 \
+		test/test_importlib/zipdata02 \
 		asyncio \
 		test/test_asyncio \
 		collections concurrent concurrent/futures encodings \
@@ -1284,9 +1302,6 @@ LIBSUBDIRS=	tkinter tkinter/test tkinter/test/test_tkinter \
 		ctypes ctypes/test ctypes/macholib \
 		idlelib idlelib/Icons idlelib/idle_test \
 		distutils distutils/command distutils/tests $(XMLLIBSUBDIRS) \
-		importlib test/test_importlib test/test_importlib/builtin \
-		test/test_importlib/extension test/test_importlib/frozen \
-		test/test_importlib/import_ test/test_importlib/source \
 		test/test_tools test/test_warnings test/test_warnings/data \
 		turtledemo \
 		multiprocessing multiprocessing/dummy \


From webhook-mailer at python.org  Tue Jan  9 03:14:51 2018
From: webhook-mailer at python.org (Steve Dower)
Date: Tue, 09 Jan 2018 08:14:51 -0000
Subject: [Python-checkins] bpo-32507: Change Windows install to include
 app-local UCRT (#5119)
Message-ID: <mailman.4.1515485692.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/d135f20ae8887acc7716561bc8f4c7eb6d58d24c
commit: d135f20ae8887acc7716561bc8f4c7eb6d58d24c
branch: master
author: Steve Dower <steve.dower at microsoft.com>
committer: GitHub <noreply at github.com>
date: 2018-01-09T19:14:46+11:00
summary:

bpo-32507: Change Windows install to include app-local UCRT (#5119)

files:
A Misc/NEWS.d/next/Windows/2018-01-07-12-32-49.bpo-32507.vB4gxk.rst
A Misc/NEWS.d/next/Windows/2018-01-07-12-33-21.bpo-29911.ewSJKb.rst
A Tools/msi/ucrt/ucrt.wixproj
A Tools/msi/ucrt/ucrt.wxs
A Tools/msi/ucrt/ucrt_en-US.wxl
D PC/dllbase_nt.txt
M PCbuild/_bz2.vcxproj
M PCbuild/_ctypes.vcxproj
M PCbuild/_decimal.vcxproj
M PCbuild/_elementtree.vcxproj
M PCbuild/_msi.vcxproj
M PCbuild/_multiprocessing.vcxproj
M PCbuild/_overlapped.vcxproj
M PCbuild/_socket.vcxproj
M PCbuild/_sqlite3.vcxproj
M PCbuild/_testbuffer.vcxproj
M PCbuild/_testcapi.vcxproj
M PCbuild/_testimportmultiple.vcxproj
M PCbuild/python.vcxproj
M PCbuild/pythoncore.vcxproj
M PCbuild/pythonw.vcxproj
M PCbuild/select.vcxproj
M PCbuild/unicodedata.vcxproj
M PCbuild/xxlimited.vcxproj
M Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
M Tools/msi/bundle/bundle.targets
M Tools/msi/bundle/packagegroups/crt.wxs
M Tools/msi/exe/exe.wxs
M Tools/msi/get_externals.bat
M Tools/msi/msi.props
M Tools/msi/uploadrelease.proj

diff --git a/Misc/NEWS.d/next/Windows/2018-01-07-12-32-49.bpo-32507.vB4gxk.rst b/Misc/NEWS.d/next/Windows/2018-01-07-12-32-49.bpo-32507.vB4gxk.rst
new file mode 100644
index 00000000000..010271ca24e
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2018-01-07-12-32-49.bpo-32507.vB4gxk.rst
@@ -0,0 +1,2 @@
+Use app-local UCRT install rather than the proper update for old versions of
+Windows.
diff --git a/Misc/NEWS.d/next/Windows/2018-01-07-12-33-21.bpo-29911.ewSJKb.rst b/Misc/NEWS.d/next/Windows/2018-01-07-12-33-21.bpo-29911.ewSJKb.rst
new file mode 100644
index 00000000000..7f311efef46
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2018-01-07-12-33-21.bpo-29911.ewSJKb.rst
@@ -0,0 +1 @@
+Ensure separate Modify and Uninstall buttons are displayed.
diff --git a/PC/dllbase_nt.txt b/PC/dllbase_nt.txt
deleted file mode 100644
index 93a15d12bf8..00000000000
--- a/PC/dllbase_nt.txt
+++ /dev/null
@@ -1,77 +0,0 @@
-In Win32, DLL's are "pre-linked" using a specified base address.
-When the DLL is loaded, an attempt is made to place it at
-that address.  If that address is already in use, a new base address
-is selected, and the DLL subject to fixups.  Apparently, these
-fixups are very slow, and significant performance gains can be
-made by selecting a good base address.
-
-This document is to allocate base addresses to core Python
-and Python .PYD files, to give a better change of optimal performance.
-This base address is passed to the linker using the /BASE
-command line switch.
-
-
-Python.exe/Pythonw.exe     - 1d000000 - 1e000000 (-1)
-Python.dll                 - 1e000000 - 1e100000 (-1)
-
-Standard Extension Modules 1e100000 - 1e200000  ""
- - _symtable                 1e100000 - 1e110000    pyd removed in 2.4
- - bsddb                     1e180000 - 1e188000    pyd removed in 3.0
- - _tkinter                  1e190000 - 1e1A0000
- - parser                    1e1A0000 - 1e1B0000    pyd removed in 2.4
- - zlib                      1e1B0000 - 1e1C0000
- - winreg                    1e1C0000 - 1e1D0000    pyd removed in 2.4
- - _socket                   1e1D0000 - 1e1E0000
- - _sre                      1e1E0000 - 1e1F0000    pyd removed in 2.4
- - mmap                      1e1F0000 - 1e1FFFFF    pyd removed in 2.4
-
-More standard extensions 1D100000 - 1e000000
- - pyexpat                   1D100000 - 1D110000
- - select                    1D110000 - 1D120000
- - unicodedata               1D120000 - 1D160000
- - winsound                  1D160000 - 1D170000
- - bZ2                       1D170000 - 1D180000
- - datetime                  1D180000 - 1D190000    pyd removed in 2.4
- - _csv                      1D190000 - 1D1A0000    pyd removed in 2.4
- - _ctypes                   1D1A0000 - 1D1B0000
-
-Other extension modules
- - win32api                  1e200000 - 1e220000
- - win32ras                  1e220000 - 1e230000
- - win32lz                   1e230000 - 1e240000
- - timer                     1e240000 - 1e250000
- - mmapfile                  1e250000 - 1e260000
- - win32pipe                 1e260000 - 1e270000
- - avl                       1e270000 - 1e270000
- - dbhash                    1e280000 - 1e290000
- - win32net                  1e290000 - 1e2A0000
- - win32security             1e2A0000 - 1e2B0000
- - win32print                1e2B0000 - 1e2c0000
- - <unused>                  1e2d0000 - 1e2e0000
- - win32gui                  1e2e0000 - 1e2f0000
- - _imaging                  1e2f0000 - 1e300000
- - multiarray                1e300000 - 1e310000
- - win32help                 1e310000 - 1e320000
- - win32clipboard            1e320000 - 1e330000
- - win2kras                  1e330000 - 1e340000
- - pythoncom                 1e340000 - 1e400000
- - win32ui                   1e400000 - 1e500000
- - win32uiole                1e500000 - 1e600000
- - pywintypes                1e600000 - 1e700000
- - win32process              1e700000 - 1e800000
- - odbc                      1e710000 - 1e720000
- - dbi                       1e720000 - 1e730000
- - win32file                 1e730000 - 1e740000
- - win32wnet                 1e740000 - 1e750000
- - win32com.shell            1e750000 - 1e760000
- - win32com.internet         1e760000 - 1e770000
- - win32com.exchange         1e770000 - 1e780000
- - win32com.exchdapi         1e780000 - 1e790000
- - win32com.axscript         1e790000 - 1e7a0000
- - win32com.axdebug          1e7b0000 - 1e7c0000
- - win32com.adsi             1e7f0000 - 1e800000
- - win32event                1e810000 - 1e820000
- - win32evtlog               1e820000 - 1e830000
- - win32com.axcontrol        1e830000 - 1e840000
-
-
diff --git a/PCbuild/_bz2.vcxproj b/PCbuild/_bz2.vcxproj
index b3ab378c6ed..3c21848ae28 100644
--- a/PCbuild/_bz2.vcxproj
+++ b/PCbuild/_bz2.vcxproj
@@ -66,9 +66,6 @@
       <PreprocessorDefinitions>WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <DisableSpecificWarnings>4244;4267;%(DisableSpecificWarnings)</DisableSpecificWarnings>
     </ClCompile>
-    <Link>
-      <BaseAddress>0x1D170000</BaseAddress>
-    </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
     <ClCompile Include="..\Modules\_bz2module.c" />
diff --git a/PCbuild/_ctypes.vcxproj b/PCbuild/_ctypes.vcxproj
index 43d51f82965..81670a75bd6 100644
--- a/PCbuild/_ctypes.vcxproj
+++ b/PCbuild/_ctypes.vcxproj
@@ -64,7 +64,6 @@
       <AdditionalIncludeDirectories>..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
     </ClCompile>
     <Link>
-      <BaseAddress>0x1D1A0000</BaseAddress>
       <AdditionalOptions>/EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions)</AdditionalOptions>
     </Link>
   </ItemDefinitionGroup>
diff --git a/PCbuild/_decimal.vcxproj b/PCbuild/_decimal.vcxproj
index 9c9c19abbe3..b14f31093ff 100644
--- a/PCbuild/_decimal.vcxproj
+++ b/PCbuild/_decimal.vcxproj
@@ -66,9 +66,6 @@
       <PreprocessorDefinitions Condition="'$(Platform)' == 'x64'">CONFIG_64;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
     </ClCompile>
-    <Link>
-      <BaseAddress>0x1D1A0000</BaseAddress>
-    </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
     <ClInclude Include="..\Modules\_decimal\libmpdec\basearith.h" />
diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj
index 0db61f20818..2338af4505b 100644
--- a/PCbuild/_elementtree.vcxproj
+++ b/PCbuild/_elementtree.vcxproj
@@ -64,9 +64,6 @@
       <AdditionalIncludeDirectories>..\Modules\expat;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;USE_PYEXPAT_CAPI;XML_STATIC;%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
-    <Link>
-      <BaseAddress>0x1D100000</BaseAddress>
-    </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
     <ClInclude Include="..\Modules\expat\ascii.h" />
diff --git a/PCbuild/_msi.vcxproj b/PCbuild/_msi.vcxproj
index 3895d450f35..6be83ca6860 100644
--- a/PCbuild/_msi.vcxproj
+++ b/PCbuild/_msi.vcxproj
@@ -62,7 +62,6 @@
   <ItemDefinitionGroup>
     <Link>
       <AdditionalDependencies>cabinet.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <BaseAddress>0x1D160000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/_multiprocessing.vcxproj b/PCbuild/_multiprocessing.vcxproj
index bb2bb4114c9..0e3f376eaf9 100644
--- a/PCbuild/_multiprocessing.vcxproj
+++ b/PCbuild/_multiprocessing.vcxproj
@@ -62,7 +62,6 @@
   <ItemDefinitionGroup>
     <Link>
       <AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <BaseAddress>0x1e1D0000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/_overlapped.vcxproj b/PCbuild/_overlapped.vcxproj
index 26b2d214c35..829cc81b623 100644
--- a/PCbuild/_overlapped.vcxproj
+++ b/PCbuild/_overlapped.vcxproj
@@ -62,7 +62,6 @@
   <ItemDefinitionGroup>
     <Link>
       <AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <BaseAddress>0x1D110000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/_socket.vcxproj b/PCbuild/_socket.vcxproj
index d5c4d1b5b6f..9e27891081e 100644
--- a/PCbuild/_socket.vcxproj
+++ b/PCbuild/_socket.vcxproj
@@ -62,7 +62,6 @@
   <ItemDefinitionGroup>
     <Link>
       <AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <BaseAddress>0x1e1D0000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj
index 5456bb5b238..1f1a1c8cada 100644
--- a/PCbuild/_sqlite3.vcxproj
+++ b/PCbuild/_sqlite3.vcxproj
@@ -64,9 +64,6 @@
       <AdditionalIncludeDirectories>$(sqlite3Dir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <PreprocessorDefinitions>MODULE_NAME="sqlite3";%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
-    <Link>
-      <BaseAddress>0x1e180000</BaseAddress>
-    </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
     <ClInclude Include="..\Modules\_sqlite\cache.h" />
diff --git a/PCbuild/_testbuffer.vcxproj b/PCbuild/_testbuffer.vcxproj
index 1f45b298102..fd2f320d797 100644
--- a/PCbuild/_testbuffer.vcxproj
+++ b/PCbuild/_testbuffer.vcxproj
@@ -60,11 +60,6 @@
   <PropertyGroup>
     <_ProjectFileVersion>10.0.40219.1</_ProjectFileVersion>
   </PropertyGroup>
-  <ItemDefinitionGroup>
-    <Link>
-      <BaseAddress>0x1e1F0000</BaseAddress>
-    </Link>
-  </ItemDefinitionGroup>
   <ItemGroup>
     <ClCompile Include="..\Modules\_testbuffer.c" />
   </ItemGroup>
diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj
index 365b07c5c5c..e668000bf2b 100644
--- a/PCbuild/_testcapi.vcxproj
+++ b/PCbuild/_testcapi.vcxproj
@@ -60,11 +60,6 @@
   <PropertyGroup>
     <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
   </PropertyGroup>
-  <ItemDefinitionGroup>
-    <Link>
-      <BaseAddress>0x1e1F0000</BaseAddress>
-    </Link>
-  </ItemDefinitionGroup>
   <ItemGroup>
     <ClCompile Include="..\Modules\_testcapimodule.c" />
   </ItemGroup>
diff --git a/PCbuild/_testimportmultiple.vcxproj b/PCbuild/_testimportmultiple.vcxproj
index 37c1a64ac8c..56da66cf327 100644
--- a/PCbuild/_testimportmultiple.vcxproj
+++ b/PCbuild/_testimportmultiple.vcxproj
@@ -60,11 +60,6 @@
   <PropertyGroup>
     <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
   </PropertyGroup>
-  <ItemDefinitionGroup>
-    <Link>
-      <BaseAddress>0x1e1F0000</BaseAddress>
-    </Link>
-  </ItemDefinitionGroup>
   <ItemGroup>
     <ClCompile Include="..\Modules\_testimportmultiple.c" />
   </ItemGroup>
diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj
index 423e68c57b2..8b64e364f18 100644
--- a/PCbuild/python.vcxproj
+++ b/PCbuild/python.vcxproj
@@ -62,7 +62,6 @@
     <Link>
       <SubSystem>Console</SubSystem>
       <StackReserveSize>2000000</StackReserveSize>
-      <BaseAddress>0x1d000000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
index 25651d00049..c8a5d7d01b4 100644
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -76,7 +76,6 @@
     <Link>
       <AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(PySourcePath)PC\external\$(PlatformToolset)\$(ArchName)</AdditionalLibraryDirectories>
-      <BaseAddress>0x1e000000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/pythonw.vcxproj b/PCbuild/pythonw.vcxproj
index e40f66c856b..19d64a332b2 100644
--- a/PCbuild/pythonw.vcxproj
+++ b/PCbuild/pythonw.vcxproj
@@ -58,7 +58,6 @@
   <ItemDefinitionGroup>
     <Link>
       <StackReserveSize>2000000</StackReserveSize>
-      <BaseAddress>0x1d000000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/select.vcxproj b/PCbuild/select.vcxproj
index d6112ab6fe3..ea75705ae49 100644
--- a/PCbuild/select.vcxproj
+++ b/PCbuild/select.vcxproj
@@ -61,7 +61,6 @@
   <ItemDefinitionGroup>
     <Link>
       <AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <BaseAddress>0x1D110000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/PCbuild/unicodedata.vcxproj b/PCbuild/unicodedata.vcxproj
index 317c1a8a56d..b80da53af33 100644
--- a/PCbuild/unicodedata.vcxproj
+++ b/PCbuild/unicodedata.vcxproj
@@ -59,11 +59,6 @@
   <PropertyGroup>
     <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
   </PropertyGroup>
-  <ItemDefinitionGroup>
-    <Link>
-      <BaseAddress>0x1D120000</BaseAddress>
-    </Link>
-  </ItemDefinitionGroup>
   <ItemGroup>
     <ClInclude Include="..\Modules\unicodedata_db.h" />
     <ClInclude Include="..\Modules\unicodename_db.h" />
diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj
index 9dbdc77335d..a796255c820 100644
--- a/PCbuild/xxlimited.vcxproj
+++ b/PCbuild/xxlimited.vcxproj
@@ -66,7 +66,6 @@
     </ClCompile>
     <Link>
       <AdditionalDependencies>wsock32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <BaseAddress>0x1D110000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
index e88981fc3ab..2e468b7e57b 100644
--- a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
+++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
@@ -281,10 +281,6 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
         case ID_INSTALL_BUTTON:
             SavePageSettings();
 
-            if (!WillElevate() && !QueryElevateForCrtInstall()) {
-                break;
-            }
-
             hr = BalGetNumericVariable(L"InstallAllUsers", &installAllUsers);
             ExitOnFailure(hr, L"Failed to get install scope");
 
@@ -332,10 +328,6 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
                 ReleaseStr(targetDir);
             }
 
-            if (!WillElevate() && !QueryElevateForCrtInstall()) {
-                break;
-            }
-
             OnPlan(_command.action);
             break;
 
@@ -2642,30 +2634,6 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
         return result;
     }
 
-    BOOL QueryElevateForCrtInstall() {
-        // Called to prompt the user that even though they think they won't need
-        // to elevate, they actually will because of the CRT install.
-        if (IsCrtInstalled()) {
-            // CRT is already installed - no need to prompt
-            return TRUE;
-        }
-        
-        LONGLONG elevated;
-        HRESULT hr = BalGetNumericVariable(L"WixBundleElevated", &elevated);
-        if (SUCCEEDED(hr) && elevated) {
-            // Already elevated - no need to prompt
-            return TRUE;
-        }
-
-        LOC_STRING *locStr;
-        hr = LocGetString(_wixLoc, L"#(loc.ElevateForCRTInstall)", &locStr);
-        if (FAILED(hr)) {
-            BalLogError(hr, "Failed to get ElevateForCRTInstall string");
-            return FALSE;
-        }
-        return ::MessageBoxW(_hWnd, locStr->wzText, _theme->sczCaption, MB_YESNO) != IDNO;
-    }
-
     HRESULT EvaluateConditions() {
         HRESULT hr = S_OK;
         BOOL result = FALSE;
diff --git a/Tools/msi/bundle/bundle.targets b/Tools/msi/bundle/bundle.targets
index e0dae21202e..f882d2ee1f0 100644
--- a/Tools/msi/bundle/bundle.targets
+++ b/Tools/msi/bundle/bundle.targets
@@ -71,6 +71,7 @@
         <Package Include="..\tcltk\tcltk*.wixproj" />
         <Package Include="..\test\test*.wixproj" />
         <Package Include="..\tools\tools*.wixproj" />
+        <Package Include="..\ucrt\ucrt*.wixproj" />
     </ItemGroup>
     
     <PropertyGroup>
diff --git a/Tools/msi/bundle/packagegroups/crt.wxs b/Tools/msi/bundle/packagegroups/crt.wxs
index dc4047590e8..61b756fc255 100644
--- a/Tools/msi/bundle/packagegroups/crt.wxs
+++ b/Tools/msi/bundle/packagegroups/crt.wxs
@@ -2,48 +2,22 @@
 <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
     <Fragment>
         <PackageGroup Id="crt">
-            <PackageGroupRef Id="crt_14.0_v6.0" />
-            <PackageGroupRef Id="crt_14.0_v6.1" />
-            <PackageGroupRef Id="crt_14.0_v6.2" />
-            <PackageGroupRef Id="crt_14.0_v6.3" />
-        </PackageGroup>
-    </Fragment>
-
-    <?foreach ver in v6.0;v6.1;v6.2;v6.3 ?>
-    <?if "$(var.ver)" = "v6.0" ?>
-    <?define msuver=6.0 ?>
-    <?elseif "$(var.ver)" = "v6.1" ?>
-    <?define msuver=6.1 ?>
-    <?elseif "$(var.ver)" = "v6.2" ?>
-    <?define msuver=8-RT ?>
-    <?elseif "$(var.ver)" = "v6.3" ?>
-    <?define msuver=8.1 ?>
-    <?else ?>
-    <?error unknown version $(var.ver) ?>
-    <?endif ?>
-
-    <Fragment>
-        <PackageGroup Id="crt_14.0_$(var.ver)">
-            <MsuPackage Id="crt_14.0_$(var.ver)_x86"
-                        KB="2999226"
-                        SourceFile="!(bindpath.redist)\Windows$(var.msuver)-KB2999226-x86.msu"
-                        DisplayName="!(loc.CRTDescription)"
-                        Description="!(loc.CRTDescription)"
+            <MsiPackage Id="ucrt_AllUsers"
+                        SourceFile="ucrt.msi"
                         Compressed="$(var.CompressMSI)"
                         DownloadUrl="$(var.DownloadUrl)"
-                        InstallCondition="not CRTInstalled and VersionNT = $(var.ver) and not VersionNT64 and (Include_core or Include_exe or Include_launcher or Include_pip) and not LauncherOnly" />
-
-            <MsuPackage Id="crt_14.0_$(var.ver)_x64"
-                        KB="2999226"
-                        SourceFile="!(bindpath.redist)\Windows$(var.msuver)-KB2999226-x64.msu"
-                        DisplayName="!(loc.CRTDescription)"
-                        Description="!(loc.CRTDescription)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and not CRTInstalled and (Include_core or Include_exe or Include_pip) and not LauncherOnly">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="ucrt_JustForMe"
+                        SourceFile="ucrt.msi"
                         Compressed="$(var.CompressMSI)"
                         DownloadUrl="$(var.DownloadUrl)"
-                        InstallCondition="not CRTInstalled and VersionNT64 = $(var.ver) and (Include_core or Include_exe or Include_launcher or Include_pip) and not LauncherOnly" />
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and not CRTInstalled and (Include_core or Include_exe or Include_pip) and not LauncherOnly">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
         </PackageGroup>
     </Fragment>
-
-    <?undef msuver ?>
-    <?endforeach ?>
 </Wix>
\ No newline at end of file
diff --git a/Tools/msi/exe/exe.wxs b/Tools/msi/exe/exe.wxs
index 03d43c60324..98d31b3a77b 100644
--- a/Tools/msi/exe/exe.wxs
+++ b/Tools/msi/exe/exe.wxs
@@ -3,10 +3,10 @@
     <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
         <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
         <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
-        
+
         <PropertyRef Id="UpgradeTable" />
         <PropertyRef Id="REGISTRYKEY" />
-        
+
         <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
             <ComponentGroupRef Id="exe_python" Primary="yes" />
             <ComponentGroupRef Id="exe_reg" Primary="yes" />
@@ -14,7 +14,7 @@
             <ComponentGroupRef Id="exe_icons" />
             <ComponentRef Id="OptionalFeature" />
         </Feature>
-            
+
         <Feature Id="Shortcuts" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
             <ComponentGroupRef Id="exe_python" />
             <Component Id="exe_shortcut" Directory="MenuDir" Guid="*">
diff --git a/Tools/msi/get_externals.bat b/Tools/msi/get_externals.bat
index 913512a778f..d66781d7c42 100644
--- a/Tools/msi/get_externals.bat
+++ b/Tools/msi/get_externals.bat
@@ -55,7 +55,7 @@ set binaries=%binaries%     binutils
 set binaries=%binaries%     gpg
 set binaries=%binaries%     htmlhelp
 set binaries=%binaries%     nuget
-set binaries=%binaries%     redist
+set binaries=%binaries%     redist-1
 set binaries=%binaries%     wix
 
 for %%b in (%binaries%) do (
diff --git a/Tools/msi/msi.props b/Tools/msi/msi.props
index 0318f7aa8eb..0fe822af931 100644
--- a/Tools/msi/msi.props
+++ b/Tools/msi/msi.props
@@ -54,8 +54,8 @@
         <OutputPath Condition="!HasTrailingSlash($(OutputPath))">$(OutputPath)\</OutputPath>
         <OutDir>$(OutputPath)</OutDir>
         <ReuseCabinetCache>true</ReuseCabinetCache>
-        <CRTRedist Condition="'$(CRTRedist)' == ''">$(ExternalsDir)\windows-installer\redist</CRTRedist>
-        <CRTRedist Condition="!Exists($(CRTRedist))"></CRTRedist>
+        <CRTRedist Condition="'$(CRTRedist)' == ''">$(ExternalsDir)\windows-installer\redist-1\$(Platform)</CRTRedist>
+        <CRTRedist>$([System.IO.Path]::GetFullPath($(CRTRedist)))</CRTRedist>
         <DocFilename>python$(MajorVersionNumber)$(MinorVersionNumber)$(MicroVersionNumber)$(ReleaseLevelName).chm</DocFilename>
 
         <InstallerVersion>$(MajorVersionNumber).$(MinorVersionNumber).$(Field3Value).0</InstallerVersion>
@@ -123,10 +123,7 @@
         <LinkerBindInputPaths Include="$(tcltkDir)">
             <BindName>tcltk</BindName>
         </LinkerBindInputPaths>
-        <LinkerBindInputPaths Include="$(CRTRedist)" Condition="'$(CRTRedist)' != ''">
-            <BindName>redist</BindName>
-        </LinkerBindInputPaths>
-        <LinkerBindInputPaths Include="$(VS140COMNTOOLS)\..\..\VC\redist\$(Platform)\Microsoft.VC140.CRT">
+        <LinkerBindInputPaths Include="$(CRTRedist)">
             <BindName>redist</BindName>
         </LinkerBindInputPaths>
         <LinkerBindInputPaths Include="$(BuildPath32)">
diff --git a/Tools/msi/ucrt/ucrt.wixproj b/Tools/msi/ucrt/ucrt.wixproj
new file mode 100644
index 00000000000..52a7e0e672b
--- /dev/null
+++ b/Tools/msi/ucrt/ucrt.wixproj
@@ -0,0 +1,28 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{E233091D-2EE3-49D8-A7F1-6E266CE1B6B1}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>ucrt</OutputName>
+        <OutputType>Package</OutputType>
+    </PropertyGroup>
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="ucrt.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    <ItemGroup>
+        <InstallFiles Include="$(CRTRedist)\ucrt\*.dll">
+            <SourceBase>$(CRTRedist)</SourceBase>
+            <Source>!(bindpath.redist)\</Source>
+            <TargetBase>$(CRTRedist)\ucrt</TargetBase>
+            <Target_>InstallDirectory\</Target_>
+            <Group>ucrt_dll</Group>
+            <IncludeInCat>false</IncludeInCat>
+        </InstallFiles>
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/ucrt/ucrt.wxs b/Tools/msi/ucrt/ucrt.wxs
new file mode 100644
index 00000000000..76e56820c53
--- /dev/null
+++ b/Tools/msi/ucrt/ucrt.wxs
@@ -0,0 +1,14 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="ucrt_dll" />
+        </Feature>
+    </Product>
+</Wix>
diff --git a/Tools/msi/ucrt/ucrt_en-US.wxl b/Tools/msi/ucrt/ucrt_en-US.wxl
new file mode 100644
index 00000000000..f42dd5fe44c
--- /dev/null
+++ b/Tools/msi/ucrt/ucrt_en-US.wxl
@@ -0,0 +1,5 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">C Runtime Library</String>
+    <String Id="ShortDescriptor">ucrt</String>
+</WixLocalization>
diff --git a/Tools/msi/uploadrelease.proj b/Tools/msi/uploadrelease.proj
index 96b99a4ff63..e70ae1de22f 100644
--- a/Tools/msi/uploadrelease.proj
+++ b/Tools/msi/uploadrelease.proj
@@ -21,7 +21,7 @@
     </PropertyGroup>
     
     <ItemGroup>
-        <File Include="$(OutputPath)\*.msi;$(OutputPath)\*.msu">
+        <File Include="$(OutputPath)\*.msi">
             <CopyTo>$(MSITarget)</CopyTo>
         </File>
         <File Include="$(OutputPath)\*.exe;$(OutputPath)\*.zip">


From solipsis at pitrou.net  Tue Jan  9 04:13:30 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 09 Jan 2018 09:13:30 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=3
Message-ID: <20180109091330.1.9FCFEB1C37E63A1E@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [-1, -1, 2] memory blocks, sum=0
test_multiprocessing_spawn leaked [1, 0, -2] memory blocks, sum=-1


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogMSSfrq', '--timeout', '7200']

From webhook-mailer at python.org  Tue Jan  9 10:22:40 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 09 Jan 2018 15:22:40 -0000
Subject: [Python-checkins] Update docs for 3.7.0a4
Message-ID: <mailman.5.1515511361.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/3b43bfaae665ab784ef67cefc3a3f95a75aafdc4
commit: 3b43bfaae665ab784ef67cefc3a3f95a75aafdc4
branch: master
author: Ned Deily <nad at python.org>
committer: Ned Deily <nad at python.org>
date: 2018-01-08T21:57:13-05:00
summary:

Update docs for 3.7.0a4

files:
A Misc/NEWS.d/3.7.0a4.rst
D Misc/NEWS.d/next/C API/2017-12-07-15-58-15.bpo-32241.LbyQt6.rst
D Misc/NEWS.d/next/C API/2017-12-12-23-09-46.bpo-32264.ahRlOI.rst
D Misc/NEWS.d/next/C API/2017-12-16-09-59-35.bpo-29084.ZGJ-LJ.rst
D Misc/NEWS.d/next/C API/2017-12-20-23-22-32.bpo-32030.d1dcwh.rst
D Misc/NEWS.d/next/Core and Builtins/2017-07-26-00-20-15.bpo-29469.potmyI.rst
D Misc/NEWS.d/next/Core and Builtins/2017-11-26-14-38-44.bpo-31650.JWf_Im.rst
D Misc/NEWS.d/next/Core and Builtins/2017-11-28-15-04-14.bpo-31901.mDeCLK.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-33-47.bpo-32225.ucKjvw.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-42-58.bpo-32226.G8fqb6.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-05-23-10-58.bpo-29240.qpJP5l.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-06-20-18-34.bpo-32230.PgGQaB.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-07-17-22-30.bpo-20361.zQUmbi.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-07-23-44-29.bpo-31506.j1U2fU.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-09-11-03-51.bpo-32260.1DAO-p.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-14-11-48-19.bpo-30416.hlHo_9.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-15-00-13-04.bpo-32329.q47IN2.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-15-11-50-06.bpo-27169.VO84fQ.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-16-14-30-21.bpo-32259.GoOJiX.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-19-21-14-41.bpo-32379.B7mOmI.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst
D Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-38-17.bpo-32399.wlH12z.rst
D Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst
D Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst
D Misc/NEWS.d/next/Documentation/2017-12-24-17-29-37.bpo-32418.eZe-ID.rst
D Misc/NEWS.d/next/Library/2017-09-16-02-56-33.bpo-15216.lqXCTT.rst
D Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst
D Misc/NEWS.d/next/Library/2017-10-10-18-56-46.bpo-30241.F_go20.rst
D Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
D Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst
D Misc/NEWS.d/next/Library/2017-11-02-11-57-41.bpo-27456.snzyTC.rst
D Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst
D Misc/NEWS.d/next/Library/2017-11-24-08-35-43.bpo-26439.IC45_f.rst
D Misc/NEWS.d/next/Library/2017-11-26-17-28-26.bpo-32143.o7YdXL.rst
D Misc/NEWS.d/next/Library/2017-12-04-12-23-26.bpo-32199.nGof4v.rst
D Misc/NEWS.d/next/Library/2017-12-04-17-41-40.bpo-15873.-T4TRK.rst
D Misc/NEWS.d/next/Library/2017-12-05-13-25-15.bpo-32227.3vnWFS.rst
D Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst
D Misc/NEWS.d/next/Library/2017-12-08-11-02-26.bpo-32193.NJe_TQ.rst
D Misc/NEWS.d/next/Library/2017-12-08-15-09-41.bpo-22589.8ouqI6.rst
D Misc/NEWS.d/next/Library/2017-12-09-11-30-35.bpo-32253.TQHSYF.rst
D Misc/NEWS.d/next/Library/2017-12-10-00-57-51.bpo-32265.kELtTE.rst
D Misc/NEWS.d/next/Library/2017-12-10-12-30-13.bpo-32269.Q85pKj.rst
D Misc/NEWS.d/next/Library/2017-12-10-18-59-13.bpo-32272.Mu84Am.rst
D Misc/NEWS.d/next/Library/2017-12-10-19-14-55.bpo-32273.5KKlCv.rst
D Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst
D Misc/NEWS.d/next/Library/2017-12-10-23-44-56.bpo-30050.4SZ3lY.rst
D Misc/NEWS.d/next/Library/2017-12-11-09-53-14.bpo-32277.jkKiVC.rst
D Misc/NEWS.d/next/Library/2017-12-12-07-29-06.bpo-32255.2bfNmM.rst
D Misc/NEWS.d/next/Library/2017-12-12-16-58-20.bpo-32250.UljTa0.rst
D Misc/NEWS.d/next/Library/2017-12-12-18-01-01.bpo-32296.bwscHz.rst
D Misc/NEWS.d/next/Library/2017-12-13-00-00-37.bpo-17852.Q8BP8N.rst
D Misc/NEWS.d/next/Library/2017-12-13-16-47-38.bpo-32314.W4_U2j.rst
D Misc/NEWS.d/next/Library/2017-12-13-19-02-38.bpo-29970.uxVOpk.rst
D Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst
D Misc/NEWS.d/next/Library/2017-12-13-22-10-36.bpo-29711.hJjghA.rst
D Misc/NEWS.d/next/Library/2017-12-13-22-38-08.bpo-32302.othtTr.rst
D Misc/NEWS.d/next/Library/2017-12-14-10-10-10.bpo-32323.ideco.rst
D Misc/NEWS.d/next/Library/2017-12-14-16-00-25.bpo-32327.bbkSxA.rst
D Misc/NEWS.d/next/Library/2017-12-14-17-28-54.bpo-32311.DL5Ytn.rst
D Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
D Misc/NEWS.d/next/Library/2017-12-15-23-48-43.bpo-32331.fIg1Uc.rst
D Misc/NEWS.d/next/Library/2017-12-16-18-50-57.bpo-32348.5j__he.rst
D Misc/NEWS.d/next/Library/2017-12-17-14-23-23.bpo-32351.95fh2K.rst
D Misc/NEWS.d/next/Library/2017-12-17-21-42-24.bpo-32355.tbaTWA.rst
D Misc/NEWS.d/next/Library/2017-12-17-22-50-51.bpo-32356.roZJpA.rst
D Misc/NEWS.d/next/Library/2017-12-18-00-36-41.bpo-32357.t1F3sn.rst
D Misc/NEWS.d/next/Library/2017-12-19-00-37-28.bpo-32363.YTeGU0.rst
D Misc/NEWS.d/next/Library/2017-12-21-11-08-42.bpo-26133.mt81QV.rst
D Misc/NEWS.d/next/Library/2017-12-23-12-45-00.bpo-32415.YufXTU.rst
D Misc/NEWS.d/next/Library/2017-12-25-11-09-46.bpo-31721.5gM972.rst
D Misc/NEWS.d/next/Library/2017-12-25-20-22-47.bpo-32422.5H3Wq2.rst
D Misc/NEWS.d/next/Library/2017-12-28-21-30-40.bpo-32441.LqlboJ.rst
D Misc/NEWS.d/next/Library/2017-12-29-00-44-42.bpo-23749.QL1Cxd.rst
D Misc/NEWS.d/next/Library/2017-12-31-20-32-58.bpo-32468.YBs__0.rst
D Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst
D Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst
D Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst
D Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst
D Misc/NEWS.d/next/Tests/2017-11-11-16-35-18.bpo-32002.itDxIo.rst
D Misc/NEWS.d/next/Tests/2017-12-11-13-31-33.bpo-32252.YnFw7J.rst
D Misc/NEWS.d/next/Tools-Demos/2017-12-22-09-25-51.bpo-24960.TGdAgO.rst
D Misc/NEWS.d/next/Windows/2017-08-18-18-00-24.bpo-19764.ODpc9y.rst
M Lib/pydoc_data/topics.py

diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
index bb9c1efbd0d..dc9591adc78 100644
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Autogenerated by Sphinx on Tue Dec  5 03:14:53 2017
+# Autogenerated by Sphinx on Mon Jan  8 21:23:03 2018
 topics = {'assert': 'The "assert" statement\n'
            '**********************\n'
            '\n'
@@ -571,6 +571,65 @@
                      '   sorts it.\n'
                      '\n'
                      '\n'
+                     'Customizing module attribute access\n'
+                     '===================================\n'
+                     '\n'
+                     'Special names "__getattr__" and "__dir__" can be also '
+                     'used to\n'
+                     'customize access to module attributes. The "__getattr__" '
+                     'function at\n'
+                     'the module level should accept one argument which is the '
+                     'name of an\n'
+                     'attribute and return the computed value or raise an '
+                     '"AttributeError".\n'
+                     'If an attribute is not found on a module object through '
+                     'the normal\n'
+                     'lookup, i.e. "object.__getattribute__()", then '
+                     '"__getattr__" is\n'
+                     'searched in the module "__dict__" before raising an '
+                     '"AttributeError".\n'
+                     'If found, it is called with the attribute name and the '
+                     'result is\n'
+                     'returned.\n'
+                     '\n'
+                     'The "__dir__" function should accept no arguments, and '
+                     'return a list\n'
+                     'of strings that represents the names accessible on '
+                     'module. If present,\n'
+                     'this function overrides the standard "dir()" search on a '
+                     'module.\n'
+                     '\n'
+                     'For a more fine grained customization of the module '
+                     'behavior (setting\n'
+                     'attributes, properties, etc.), one can set the '
+                     '"__class__" attribute\n'
+                     'of a module object to a subclass of "types.ModuleType". '
+                     'For example:\n'
+                     '\n'
+                     '   import sys\n'
+                     '   from types import ModuleType\n'
+                     '\n'
+                     '   class VerboseModule(ModuleType):\n'
+                     '       def __repr__(self):\n'
+                     "           return f'Verbose {self.__name__}'\n"
+                     '\n'
+                     '       def __setattr__(self, attr, value):\n'
+                     "           print(f'Setting {attr}...')\n"
+                     '           setattr(self, attr, value)\n'
+                     '\n'
+                     '   sys.modules[__name__].__class__ = VerboseModule\n'
+                     '\n'
+                     'Note: Defining module "__getattr__" and setting module '
+                     '"__class__"\n'
+                     '  only affect lookups made using the attribute access '
+                     'syntax --\n'
+                     '  directly accessing the module globals (whether by code '
+                     'within the\n'
+                     "  module, or via a reference to the module's globals "
+                     'dictionary) is\n'
+                     '  unaffected.\n'
+                     '\n'
+                     '\n'
                      'Implementing Descriptors\n'
                      '========================\n'
                      '\n'
@@ -2905,63 +2964,52 @@
                   '\n'
                   '   Called when the instance is about to be destroyed.  This '
                   'is also\n'
-                  '   called a destructor.  If a base class has a "__del__()" '
-                  'method, the\n'
-                  '   derived class\'s "__del__()" method, if any, must '
-                  'explicitly call it\n'
-                  '   to ensure proper deletion of the base class part of the '
-                  'instance.\n'
-                  '   Note that it is possible (though not recommended!) for '
+                  '   called a finalizer or (improperly) a destructor.  If a '
+                  'base class\n'
+                  '   has a "__del__()" method, the derived class\'s '
+                  '"__del__()" method,\n'
+                  '   if any, must explicitly call it to ensure proper '
+                  'deletion of the\n'
+                  '   base class part of the instance.\n'
+                  '\n'
+                  '   It is possible (though not recommended!) for the '
+                  '"__del__()" method\n'
+                  '   to postpone destruction of the instance by creating a '
+                  'new reference\n'
+                  '   to it.  This is called object *resurrection*.  It is\n'
+                  '   implementation-dependent whether "__del__()" is called a '
+                  'second\n'
+                  '   time when a resurrected object is about to be destroyed; '
                   'the\n'
-                  '   "__del__()" method to postpone destruction of the '
-                  'instance by\n'
-                  '   creating a new reference to it.  It may then be called '
-                  'at a later\n'
-                  '   time when this new reference is deleted.  It is not '
-                  'guaranteed that\n'
-                  '   "__del__()" methods are called for objects that still '
-                  'exist when\n'
-                  '   the interpreter exits.\n'
+                  '   current *CPython* implementation only calls it once.\n'
+                  '\n'
+                  '   It is not guaranteed that "__del__()" methods are called '
+                  'for\n'
+                  '   objects that still exist when the interpreter exits.\n'
                   '\n'
                   '   Note: "del x" doesn\'t directly call "x.__del__()" --- '
                   'the former\n'
                   '     decrements the reference count for "x" by one, and the '
                   'latter is\n'
-                  '     only called when "x"\'s reference count reaches zero.  '
-                  'Some common\n'
-                  '     situations that may prevent the reference count of an '
-                  'object from\n'
-                  '     going to zero include: circular references between '
-                  'objects (e.g.,\n'
-                  '     a doubly-linked list or a tree data structure with '
-                  'parent and\n'
-                  '     child pointers); a reference to the object on the '
-                  'stack frame of\n'
-                  '     a function that caught an exception (the traceback '
-                  'stored in\n'
-                  '     "sys.exc_info()[2]" keeps the stack frame alive); or a '
+                  '     only called when "x"\'s reference count reaches zero.\n'
+                  '\n'
+                  '   **CPython implementation detail:** It is possible for a '
                   'reference\n'
-                  '     to the object on the stack frame that raised an '
-                  'unhandled\n'
-                  '     exception in interactive mode (the traceback stored '
-                  'in\n'
-                  '     "sys.last_traceback" keeps the stack frame alive).  '
-                  'The first\n'
-                  '     situation can only be remedied by explicitly breaking '
-                  'the cycles;\n'
-                  '     the second can be resolved by freeing the reference to '
-                  'the\n'
-                  '     traceback object when it is no longer useful, and the '
-                  'third can\n'
-                  '     be resolved by storing "None" in "sys.last_traceback". '
-                  'Circular\n'
-                  '     references which are garbage are detected and cleaned '
-                  'up when the\n'
-                  "     cyclic garbage collector is enabled (it's on by "
-                  'default). Refer\n'
-                  '     to the documentation for the "gc" module for more '
-                  'information\n'
-                  '     about this topic.\n'
+                  '   cycle to prevent the reference count of an object from '
+                  'going to\n'
+                  '   zero.  In this case, the cycle will be later detected '
+                  'and deleted\n'
+                  '   by the *cyclic garbage collector*.  A common cause of '
+                  'reference\n'
+                  '   cycles is when an exception has been caught in a local '
+                  'variable.\n'
+                  "   The frame's locals then reference the exception, which "
+                  'references\n'
+                  '   its own traceback, which references the locals of all '
+                  'frames caught\n'
+                  '   in the traceback.\n'
+                  '\n'
+                  '   See also: Documentation for the "gc" module.\n'
                   '\n'
                   '   Warning: Due to the precarious circumstances under '
                   'which\n'
@@ -2969,29 +3017,35 @@
                   'during\n'
                   '     their execution are ignored, and a warning is printed '
                   'to\n'
-                  '     "sys.stderr" instead. Also, when "__del__()" is '
-                  'invoked in\n'
-                  '     response to a module being deleted (e.g., when '
-                  'execution of the\n'
-                  '     program is done), other globals referenced by the '
+                  '     "sys.stderr" instead. In particular:\n'
+                  '\n'
+                  '     * "__del__()" can be invoked when arbitrary code is '
+                  'being\n'
+                  '       executed, including from any arbitrary thread.  If '
                   '"__del__()"\n'
-                  '     method may already have been deleted or in the process '
-                  'of being\n'
-                  '     torn down (e.g. the import machinery shutting down).  '
-                  'For this\n'
-                  '     reason, "__del__()" methods should do the absolute '
-                  'minimum needed\n'
-                  '     to maintain external invariants.  Starting with '
-                  'version 1.5,\n'
-                  '     Python guarantees that globals whose name begins with '
-                  'a single\n'
-                  '     underscore are deleted from their module before other '
-                  'globals are\n'
-                  '     deleted; if no other references to such globals exist, '
-                  'this may\n'
-                  '     help in assuring that imported modules are still '
-                  'available at the\n'
-                  '     time when the "__del__()" method is called.\n'
+                  '       needs to take a lock or invoke any other blocking '
+                  'resource, it\n'
+                  '       may deadlock as the resource may already be taken by '
+                  'the code\n'
+                  '       that gets interrupted to execute "__del__()".\n'
+                  '\n'
+                  '     * "__del__()" can be executed during interpreter '
+                  'shutdown.  As\n'
+                  '       a consequence, the global variables it needs to '
+                  'access\n'
+                  '       (including other modules) may already have been '
+                  'deleted or set\n'
+                  '       to "None". Python guarantees that globals whose name '
+                  'begins\n'
+                  '       with a single underscore are deleted from their '
+                  'module before\n'
+                  '       other globals are deleted; if no other references to '
+                  'such\n'
+                  '       globals exist, this may help in assuring that '
+                  'imported modules\n'
+                  '       are still available at the time when the "__del__()" '
+                  'method is\n'
+                  '       called.\n'
                   '\n'
                   'object.__repr__(self)\n'
                   '\n'
@@ -3346,6 +3400,13 @@
              'executes\n'
              'commands as if given in a ".pdbrc" file, see Debugger Commands.\n'
              '\n'
+             'New in version 3.7: "pdb.py" now accepts a "-m" option that '
+             'execute\n'
+             'modules similar to the way "python3 -m" does. As with a script, '
+             'the\n'
+             'debugger will pause execution just before the first line of the\n'
+             'module.\n'
+             '\n'
              'The typical usage to break into the debugger from a running '
              'program is\n'
              'to insert\n'
@@ -7563,91 +7624,87 @@
                  '\n'
                  '   Called when the instance is about to be destroyed.  This '
                  'is also\n'
-                 '   called a destructor.  If a base class has a "__del__()" '
-                 'method, the\n'
-                 '   derived class\'s "__del__()" method, if any, must '
-                 'explicitly call it\n'
-                 '   to ensure proper deletion of the base class part of the '
-                 'instance.\n'
-                 '   Note that it is possible (though not recommended!) for '
+                 '   called a finalizer or (improperly) a destructor.  If a '
+                 'base class\n'
+                 '   has a "__del__()" method, the derived class\'s '
+                 '"__del__()" method,\n'
+                 '   if any, must explicitly call it to ensure proper deletion '
+                 'of the\n'
+                 '   base class part of the instance.\n'
+                 '\n'
+                 '   It is possible (though not recommended!) for the '
+                 '"__del__()" method\n'
+                 '   to postpone destruction of the instance by creating a new '
+                 'reference\n'
+                 '   to it.  This is called object *resurrection*.  It is\n'
+                 '   implementation-dependent whether "__del__()" is called a '
+                 'second\n'
+                 '   time when a resurrected object is about to be destroyed; '
                  'the\n'
-                 '   "__del__()" method to postpone destruction of the '
-                 'instance by\n'
-                 '   creating a new reference to it.  It may then be called at '
-                 'a later\n'
-                 '   time when this new reference is deleted.  It is not '
-                 'guaranteed that\n'
-                 '   "__del__()" methods are called for objects that still '
-                 'exist when\n'
-                 '   the interpreter exits.\n'
+                 '   current *CPython* implementation only calls it once.\n'
+                 '\n'
+                 '   It is not guaranteed that "__del__()" methods are called '
+                 'for\n'
+                 '   objects that still exist when the interpreter exits.\n'
                  '\n'
                  '   Note: "del x" doesn\'t directly call "x.__del__()" --- '
                  'the former\n'
                  '     decrements the reference count for "x" by one, and the '
                  'latter is\n'
-                 '     only called when "x"\'s reference count reaches zero.  '
-                 'Some common\n'
-                 '     situations that may prevent the reference count of an '
-                 'object from\n'
-                 '     going to zero include: circular references between '
-                 'objects (e.g.,\n'
-                 '     a doubly-linked list or a tree data structure with '
-                 'parent and\n'
-                 '     child pointers); a reference to the object on the stack '
-                 'frame of\n'
-                 '     a function that caught an exception (the traceback '
-                 'stored in\n'
-                 '     "sys.exc_info()[2]" keeps the stack frame alive); or a '
+                 '     only called when "x"\'s reference count reaches zero.\n'
+                 '\n'
+                 '   **CPython implementation detail:** It is possible for a '
                  'reference\n'
-                 '     to the object on the stack frame that raised an '
-                 'unhandled\n'
-                 '     exception in interactive mode (the traceback stored in\n'
-                 '     "sys.last_traceback" keeps the stack frame alive).  The '
-                 'first\n'
-                 '     situation can only be remedied by explicitly breaking '
-                 'the cycles;\n'
-                 '     the second can be resolved by freeing the reference to '
-                 'the\n'
-                 '     traceback object when it is no longer useful, and the '
-                 'third can\n'
-                 '     be resolved by storing "None" in "sys.last_traceback". '
-                 'Circular\n'
-                 '     references which are garbage are detected and cleaned '
-                 'up when the\n'
-                 "     cyclic garbage collector is enabled (it's on by "
-                 'default). Refer\n'
-                 '     to the documentation for the "gc" module for more '
-                 'information\n'
-                 '     about this topic.\n'
+                 '   cycle to prevent the reference count of an object from '
+                 'going to\n'
+                 '   zero.  In this case, the cycle will be later detected and '
+                 'deleted\n'
+                 '   by the *cyclic garbage collector*.  A common cause of '
+                 'reference\n'
+                 '   cycles is when an exception has been caught in a local '
+                 'variable.\n'
+                 "   The frame's locals then reference the exception, which "
+                 'references\n'
+                 '   its own traceback, which references the locals of all '
+                 'frames caught\n'
+                 '   in the traceback.\n'
+                 '\n'
+                 '   See also: Documentation for the "gc" module.\n'
                  '\n'
                  '   Warning: Due to the precarious circumstances under which\n'
                  '     "__del__()" methods are invoked, exceptions that occur '
                  'during\n'
                  '     their execution are ignored, and a warning is printed '
                  'to\n'
-                 '     "sys.stderr" instead. Also, when "__del__()" is invoked '
-                 'in\n'
-                 '     response to a module being deleted (e.g., when '
-                 'execution of the\n'
-                 '     program is done), other globals referenced by the '
+                 '     "sys.stderr" instead. In particular:\n'
+                 '\n'
+                 '     * "__del__()" can be invoked when arbitrary code is '
+                 'being\n'
+                 '       executed, including from any arbitrary thread.  If '
                  '"__del__()"\n'
-                 '     method may already have been deleted or in the process '
-                 'of being\n'
-                 '     torn down (e.g. the import machinery shutting down).  '
-                 'For this\n'
-                 '     reason, "__del__()" methods should do the absolute '
-                 'minimum needed\n'
-                 '     to maintain external invariants.  Starting with version '
-                 '1.5,\n'
-                 '     Python guarantees that globals whose name begins with a '
-                 'single\n'
-                 '     underscore are deleted from their module before other '
-                 'globals are\n'
-                 '     deleted; if no other references to such globals exist, '
-                 'this may\n'
-                 '     help in assuring that imported modules are still '
-                 'available at the\n'
-                 '     time when the "__del__()" method is called.\n'
+                 '       needs to take a lock or invoke any other blocking '
+                 'resource, it\n'
+                 '       may deadlock as the resource may already be taken by '
+                 'the code\n'
+                 '       that gets interrupted to execute "__del__()".\n'
+                 '\n'
+                 '     * "__del__()" can be executed during interpreter '
+                 'shutdown.  As\n'
+                 '       a consequence, the global variables it needs to '
+                 'access\n'
+                 '       (including other modules) may already have been '
+                 'deleted or set\n'
+                 '       to "None". Python guarantees that globals whose name '
+                 'begins\n'
+                 '       with a single underscore are deleted from their '
+                 'module before\n'
+                 '       other globals are deleted; if no other references to '
+                 'such\n'
+                 '       globals exist, this may help in assuring that '
+                 'imported modules\n'
+                 '       are still available at the time when the "__del__()" '
+                 'method is\n'
+                 '       called.\n'
                  '\n'
                  'object.__repr__(self)\n'
                  '\n'
@@ -8031,6 +8088,65 @@
                  '   sorts it.\n'
                  '\n'
                  '\n'
+                 'Customizing module attribute access\n'
+                 '-----------------------------------\n'
+                 '\n'
+                 'Special names "__getattr__" and "__dir__" can be also used '
+                 'to\n'
+                 'customize access to module attributes. The "__getattr__" '
+                 'function at\n'
+                 'the module level should accept one argument which is the '
+                 'name of an\n'
+                 'attribute and return the computed value or raise an '
+                 '"AttributeError".\n'
+                 'If an attribute is not found on a module object through the '
+                 'normal\n'
+                 'lookup, i.e. "object.__getattribute__()", then "__getattr__" '
+                 'is\n'
+                 'searched in the module "__dict__" before raising an '
+                 '"AttributeError".\n'
+                 'If found, it is called with the attribute name and the '
+                 'result is\n'
+                 'returned.\n'
+                 '\n'
+                 'The "__dir__" function should accept no arguments, and '
+                 'return a list\n'
+                 'of strings that represents the names accessible on module. '
+                 'If present,\n'
+                 'this function overrides the standard "dir()" search on a '
+                 'module.\n'
+                 '\n'
+                 'For a more fine grained customization of the module behavior '
+                 '(setting\n'
+                 'attributes, properties, etc.), one can set the "__class__" '
+                 'attribute\n'
+                 'of a module object to a subclass of "types.ModuleType". For '
+                 'example:\n'
+                 '\n'
+                 '   import sys\n'
+                 '   from types import ModuleType\n'
+                 '\n'
+                 '   class VerboseModule(ModuleType):\n'
+                 '       def __repr__(self):\n'
+                 "           return f'Verbose {self.__name__}'\n"
+                 '\n'
+                 '       def __setattr__(self, attr, value):\n'
+                 "           print(f'Setting {attr}...')\n"
+                 '           setattr(self, attr, value)\n'
+                 '\n'
+                 '   sys.modules[__name__].__class__ = VerboseModule\n'
+                 '\n'
+                 'Note: Defining module "__getattr__" and setting module '
+                 '"__class__"\n'
+                 '  only affect lookups made using the attribute access syntax '
+                 '--\n'
+                 '  directly accessing the module globals (whether by code '
+                 'within the\n'
+                 "  module, or via a reference to the module's globals "
+                 'dictionary) is\n'
+                 '  unaffected.\n'
+                 '\n'
+                 '\n'
                  'Implementing Descriptors\n'
                  '------------------------\n'
                  '\n'
@@ -12211,18 +12327,18 @@
              '   sequence concatenation or repetition.\n'
              '\n'
              '8. "index" raises "ValueError" when *x* is not found in *s*. '
-             'When\n'
-             '   supported, the additional arguments to the index method '
-             'allow\n'
-             '   efficient searching of subsections of the sequence. Passing '
-             'the\n'
-             '   extra arguments is roughly equivalent to using '
-             '"s[i:j].index(x)",\n'
-             '   only without copying any data and with the returned index '
-             'being\n'
-             '   relative to the start of the sequence rather than the start '
-             'of the\n'
-             '   slice.\n'
+             'Not\n'
+             '   all implementations support passing the additional arguments '
+             '*i*\n'
+             '   and *j*. These arguments allow efficient searching of '
+             'subsections\n'
+             '   of the sequence. Passing the extra arguments is roughly '
+             'equivalent\n'
+             '   to using "s[i:j].index(x)", only without copying any data and '
+             'with\n'
+             '   the returned index being relative to the start of the '
+             'sequence\n'
+             '   rather than the start of the slice.\n'
              '\n'
              '\n'
              'Immutable Sequence Types\n'
diff --git a/Misc/NEWS.d/3.7.0a4.rst b/Misc/NEWS.d/3.7.0a4.rst
new file mode 100644
index 00000000000..cc90bd68de0
--- /dev/null
+++ b/Misc/NEWS.d/3.7.0a4.rst
@@ -0,0 +1,847 @@
+.. bpo: 31975
+.. date: 2018-01-05-20-54-27
+.. nonce: AmftlU
+.. release date: 2018-01-08
+.. section: Core and Builtins
+
+The default warning filter list now starts with a
+"default::DeprecationWarning:__main__" entry, so deprecation warnings are
+once again shown by default in single-file scripts and at the interactive
+prompt.
+
+..
+
+.. bpo: 32226
+.. date: 2018-01-04-15-06-15
+.. nonce: 7cAvRG
+.. section: Core and Builtins
+
+``__class_getitem__`` is now an automatic class method.
+
+..
+
+.. bpo: 32399
+.. date: 2017-12-22-13-38-17
+.. nonce: wlH12z
+.. section: Core and Builtins
+
+Add AIX uuid library support for RFC4122 using uuid_create() in libc.a
+
+..
+
+.. bpo: 32390
+.. date: 2017-12-22-13-28-07
+.. nonce: QPj083
+.. section: Core and Builtins
+
+Fix the compilation failure on AIX after the f_fsid field has been added to
+the object returned by os.statvfs() (issue #32143). Original patch by
+Michael Felt.
+
+..
+
+.. bpo: 32379
+.. date: 2017-12-19-21-14-41
+.. nonce: B7mOmI
+.. section: Core and Builtins
+
+Make MRO computation faster when a class inherits from a single base.
+
+..
+
+.. bpo: 32259
+.. date: 2017-12-16-14-30-21
+.. nonce: GoOJiX
+.. section: Core and Builtins
+
+The error message of a TypeError raised when unpack non-iterable is now more
+specific.
+
+..
+
+.. bpo: 27169
+.. date: 2017-12-15-11-50-06
+.. nonce: VO84fQ
+.. section: Core and Builtins
+
+The ``__debug__`` constant is now optimized out at compile time. This fixes
+also bpo-22091.
+
+..
+
+.. bpo: 32329
+.. date: 2017-12-15-00-13-04
+.. nonce: q47IN2
+.. section: Core and Builtins
+
+The :option:`-R` option now turns on hash randomization when the
+:envvar:`PYTHONHASHSEED` environment variable is set to ``0``. Previously,
+the option was ignored. Moreover, ``sys.flags.hash_randomization`` is now
+properly set to 0 when hash randomization is turned off by
+``PYTHONHASHSEED=0``.
+
+..
+
+.. bpo: 30416
+.. date: 2017-12-14-11-48-19
+.. nonce: hlHo_9
+.. section: Core and Builtins
+
+The optimizer is now protected from spending much time doing complex
+calculations and consuming much memory for creating large constants in
+constant folding. Increased limits for constants that can be produced in
+constant folding.
+
+..
+
+.. bpo: 32282
+.. date: 2017-12-12-14-02-28
+.. nonce: xFVMTn
+.. section: Core and Builtins
+
+Fix an unnecessary ifdef in the include of VersionHelpers.h in socketmodule
+on Windows.
+
+..
+
+.. bpo: 30579
+.. date: 2017-12-11-01-52-42
+.. nonce: X6cEzf
+.. section: Core and Builtins
+
+Implement TracebackType.__new__ to allow Python-level creation of traceback
+objects, and make TracebackType.tb_next mutable.
+
+..
+
+.. bpo: 32260
+.. date: 2017-12-09-11-03-51
+.. nonce: 1DAO-p
+.. section: Core and Builtins
+
+Don't byte swap the input keys to the SipHash algorithm on big-endian
+platforms. This should ensure siphash gives consistent results across
+platforms.
+
+..
+
+.. bpo: 31506
+.. date: 2017-12-07-23-44-29
+.. nonce: j1U2fU
+.. section: Core and Builtins
+
+Improve the error message logic for object.__new__ and object.__init__.
+Patch by Sanyam Khurana.
+
+..
+
+.. bpo: 20361
+.. date: 2017-12-07-17-22-30
+.. nonce: zQUmbi
+.. section: Core and Builtins
+
+``-b`` and ``-bb`` now inject ``'default::BytesWarning'`` and
+``error::BytesWarning`` entries into ``sys.warnoptions``, ensuring that they
+take precedence over any other warning filters configured via the ``-W``
+option or the ``PYTHONWARNINGS`` environment variable.
+
+..
+
+.. bpo: 32230
+.. date: 2017-12-06-20-18-34
+.. nonce: PgGQaB
+.. section: Core and Builtins
+
+`-X dev` now injects a ``'default'`` entry into sys.warnoptions, ensuring
+that it behaves identically to actually passing ``-Wdefault`` at the command
+line.
+
+..
+
+.. bpo: 29240
+.. date: 2017-12-05-23-10-58
+.. nonce: qpJP5l
+.. section: Core and Builtins
+
+Add a new UTF-8 mode: implementation of the :pep:`540`.
+
+..
+
+.. bpo: 32226
+.. date: 2017-12-05-21-42-58
+.. nonce: G8fqb6
+.. section: Core and Builtins
+
+PEP 560: Add support for __mro_entries__ and __class_getitem__. Implemented
+by Ivan Levkivskyi.
+
+..
+
+.. bpo: 32225
+.. date: 2017-12-05-21-33-47
+.. nonce: ucKjvw
+.. section: Core and Builtins
+
+PEP 562: Add support for module ``__getattr__`` and ``__dir__``. Implemented
+by Ivan Levkivskyi.
+
+..
+
+.. bpo: 31901
+.. date: 2017-11-28-15-04-14
+.. nonce: mDeCLK
+.. section: Core and Builtins
+
+The `atexit` module now has its callback stored per interpreter.
+
+..
+
+.. bpo: 31650
+.. date: 2017-11-26-14-38-44
+.. nonce: JWf_Im
+.. section: Core and Builtins
+
+Implement PEP 552 (Deterministic pycs). Python now supports invalidating
+bytecode cache files bashed on a source content hash rather than source
+last-modified time.
+
+..
+
+.. bpo: 29469
+.. date: 2017-07-26-00-20-15
+.. nonce: potmyI
+.. section: Core and Builtins
+
+Move constant folding from bytecode layer to AST layer. Original patch by
+Eugene Toder.
+
+..
+
+.. bpo: 32506
+.. date: 2018-01-07-11-32-42
+.. nonce: MaT-zU
+.. section: Library
+
+Now that dict is defined as keeping insertion order, drop OrderedDict and
+just use plain dict.
+
+..
+
+.. bpo: 32279
+.. date: 2018-01-06-16-50-11
+.. nonce: 1xOpU8
+.. section: Library
+
+Add params to dataclasses.make_dataclasses(): init, repr, eq, order, hash,
+and frozen.  Pass them through to dataclass().
+
+..
+
+.. bpo: 32278
+.. date: 2018-01-06-15-15-34
+.. nonce: bGnGc0
+.. section: Library
+
+Make type information optional on dataclasses.make_dataclass(). If omitted,
+the string 'typing.Any' is used.
+
+..
+
+.. bpo: 32499
+.. date: 2018-01-06-10-54-16
+.. nonce: koyY-4
+.. section: Library
+
+Add dataclasses.is_dataclass(obj), which returns True if obj is a dataclass
+or an instance of one.
+
+..
+
+.. bpo: 32468
+.. date: 2017-12-31-20-32-58
+.. nonce: YBs__0
+.. section: Library
+
+Improve frame repr() to mention filename, code name and current line number.
+
+..
+
+.. bpo: 23749
+.. date: 2017-12-29-00-44-42
+.. nonce: QL1Cxd
+.. section: Library
+
+asyncio: Implement loop.start_tls()
+
+..
+
+.. bpo: 32441
+.. date: 2017-12-28-21-30-40
+.. nonce: LqlboJ
+.. section: Library
+
+Return the new file descriptor (i.e., the second argument) from ``os.dup2``.
+Previously, ``None`` was always returned.
+
+..
+
+.. bpo: 32422
+.. date: 2017-12-25-20-22-47
+.. nonce: 5H3Wq2
+.. section: Library
+
+``functools.lru_cache`` uses less memory (3 words for each cached key) and
+takes about 1/3 time for cyclic GC.
+
+..
+
+.. bpo: 31721
+.. date: 2017-12-25-11-09-46
+.. nonce: 5gM972
+.. section: Library
+
+Prevent Python crash from happening when Future._log_traceback is set to
+True manually.  Now it can only be set to False, or a ValueError is raised.
+
+..
+
+.. bpo: 32415
+.. date: 2017-12-23-12-45-00
+.. nonce: YufXTU
+.. section: Library
+
+asyncio: Add Task.get_loop() and Future.get_loop()
+
+..
+
+.. bpo: 26133
+.. date: 2017-12-21-11-08-42
+.. nonce: mt81QV
+.. section: Library
+
+Don't unsubscribe signals in asyncio UNIX event loop on interpreter
+shutdown.
+
+..
+
+.. bpo: 32363
+.. date: 2017-12-19-00-37-28
+.. nonce: YTeGU0
+.. section: Library
+
+Make asyncio.Task.set_exception() and set_result() raise
+NotImplementedError. Task._step() and Future.__await__() raise proper
+exceptions when they are in an invalid state, instead of raising an
+AssertionError.
+
+..
+
+.. bpo: 32357
+.. date: 2017-12-18-00-36-41
+.. nonce: t1F3sn
+.. section: Library
+
+Optimize asyncio.iscoroutine() and loop.create_task() for non-native
+coroutines (e.g. async/await compiled with Cython).
+
+'loop.create_task(python_coroutine)' used to be 20% faster than
+'loop.create_task(cython_coroutine)'.  Now, the latter is as fast.
+
+..
+
+.. bpo: 32356
+.. date: 2017-12-17-22-50-51
+.. nonce: roZJpA
+.. section: Library
+
+asyncio.transport.resume_reading() and pause_reading() are now idempotent.
+New transport.is_reading() method is added.
+
+..
+
+.. bpo: 32355
+.. date: 2017-12-17-21-42-24
+.. nonce: tbaTWA
+.. section: Library
+
+Optimize asyncio.gather(); now up to 15% faster.
+
+..
+
+.. bpo: 32351
+.. date: 2017-12-17-14-23-23
+.. nonce: 95fh2K
+.. section: Library
+
+Use fastpath in asyncio.sleep if delay<0 (2x boost)
+
+..
+
+.. bpo: 32348
+.. date: 2017-12-16-18-50-57
+.. nonce: 5j__he
+.. section: Library
+
+Optimize asyncio.Future schedule/add/remove callback.  The optimization
+shows 3-6% performance improvements of async/await code.
+
+..
+
+.. bpo: 32331
+.. date: 2017-12-15-23-48-43
+.. nonce: fIg1Uc
+.. section: Library
+
+Fix socket.settimeout() and socket.setblocking() to keep socket.type as is.
+Fix socket.socket() constructor to reset any bit flags applied to socket's
+type.  This change only affects OSes that have SOCK_NONBLOCK and/or
+SOCK_CLOEXEC.
+
+..
+
+.. bpo: 32248
+.. date: 2017-12-15-15-34-12
+.. nonce: zmO8G2
+.. section: Library
+
+Add :class:`importlib.abc.ResourceReader` as an ABC for loaders to provide a
+unified API for reading resources contained within packages.  Also add
+:mod:`importlib.resources` as the port of ``importlib_resources``.
+
+..
+
+.. bpo: 32311
+.. date: 2017-12-14-17-28-54
+.. nonce: DL5Ytn
+.. section: Library
+
+Implement asyncio.create_task(coro) shortcut
+
+..
+
+.. bpo: 32327
+.. date: 2017-12-14-16-00-25
+.. nonce: bbkSxA
+.. section: Library
+
+Convert asyncio functions that were documented as coroutines to coroutines.
+Affected functions: loop.sock_sendall, loop.sock_recv, loop.sock_accept,
+loop.run_in_executor, loop.getaddrinfo, loop.getnameinfo.
+
+..
+
+.. bpo: 32323
+.. date: 2017-12-14-10-10-10
+.. nonce: ideco
+.. section: Library
+
+:func:`urllib.parse.urlsplit()` does not convert zone-id (scope) to lower
+case for scoped IPv6 addresses in hostnames now.
+
+..
+
+.. bpo: 32302
+.. date: 2017-12-13-22-38-08
+.. nonce: othtTr
+.. section: Library
+
+Fix bdist_wininst of distutils for CRT v142: it binary compatible with CRT
+v140.
+
+..
+
+.. bpo: 29711
+.. date: 2017-12-13-22-10-36
+.. nonce: hJjghA
+.. section: Library
+
+Fix ``stop_serving`` in asyncio proactor loop kill all listening servers
+
+..
+
+.. bpo: 32308
+.. date: 2017-12-13-20-31-30
+.. nonce: CUbsb2
+.. section: Library
+
+:func:`re.sub()` now replaces empty matches adjacent to a previous non-empty
+match.
+
+..
+
+.. bpo: 29970
+.. date: 2017-12-13-19-02-38
+.. nonce: uxVOpk
+.. section: Library
+
+Abort asyncio SSLProtocol connection if handshake not complete within 10s
+
+..
+
+.. bpo: 32314
+.. date: 2017-12-13-16-47-38
+.. nonce: W4_U2j
+.. section: Library
+
+Implement asyncio.run().
+
+..
+
+.. bpo: 17852
+.. date: 2017-12-13-00-00-37
+.. nonce: Q8BP8N
+.. section: Library
+
+Revert incorrect fix based on misunderstanding of _Py_PyAtExit() semantics.
+
+..
+
+.. bpo: 32296
+.. date: 2017-12-12-18-01-01
+.. nonce: bwscHz
+.. section: Library
+
+Implement asyncio._get_running_loop() and get_event_loop() in C. This makes
+them 4x faster.
+
+..
+
+.. bpo: 32250
+.. date: 2017-12-12-16-58-20
+.. nonce: UljTa0
+.. section: Library
+
+Implement ``asyncio.current_task()`` and ``asyncio.all_tasks()``. Add
+helpers intended to be used by alternative task implementations:
+``asyncio._register_task``, ``asyncio._enter_task``, ``asyncio._leave_task``
+and ``asyncio._unregister_task``. Deprecate ``asyncio.Task.current_task()``
+and ``asyncio.Task.all_tasks()``.
+
+..
+
+.. bpo: 32255
+.. date: 2017-12-12-07-29-06
+.. nonce: 2bfNmM
+.. section: Library
+
+A single empty field is now always quoted when written into a CSV file. This
+allows to distinguish an empty row from a row consisting of a single empty
+field. Patch by Licht Takeuchi.
+
+..
+
+.. bpo: 32277
+.. date: 2017-12-11-09-53-14
+.. nonce: jkKiVC
+.. section: Library
+
+Raise ``NotImplementedError`` instead of ``SystemError`` on platforms where
+``chmod(..., follow_symlinks=False)`` is not supported.  Patch by Anthony
+Sottile.
+
+..
+
+.. bpo: 30050
+.. date: 2017-12-10-23-44-56
+.. nonce: 4SZ3lY
+.. section: Library
+
+New argument warn_on_full_buffer to signal.set_wakeup_fd lets you control
+whether Python prints a warning on stderr when the wakeup fd buffer
+overflows.
+
+..
+
+.. bpo: 29137
+.. date: 2017-12-10-21-19-14
+.. nonce: CFcON1
+.. section: Library
+
+The ``fpectl`` library has been removed. It was never enabled by default,
+never worked correctly on x86-64, and it changed the Python ABI in ways that
+caused unexpected breakage of C extensions.
+
+..
+
+.. bpo: 32273
+.. date: 2017-12-10-19-14-55
+.. nonce: 5KKlCv
+.. section: Library
+
+Move asyncio.test_utils to test.test_asyncio.
+
+..
+
+.. bpo: 32272
+.. date: 2017-12-10-18-59-13
+.. nonce: Mu84Am
+.. section: Library
+
+Remove asyncio.async() function.
+
+..
+
+.. bpo: 32269
+.. date: 2017-12-10-12-30-13
+.. nonce: Q85pKj
+.. section: Library
+
+Add asyncio.get_running_loop() function.
+
+..
+
+.. bpo: 32265
+.. date: 2017-12-10-00-57-51
+.. nonce: kELtTE
+.. section: Library
+
+All class and static methods of builtin types now are correctly classified
+by inspect.classify_class_attrs() and grouped in pydoc ouput. Added
+types.ClassMethodDescriptorType for unbound class methods of builtin types.
+
+..
+
+.. bpo: 32253
+.. date: 2017-12-09-11-30-35
+.. nonce: TQHSYF
+.. section: Library
+
+Deprecate ``yield from lock``, ``await lock``, ``with (yield from lock)``
+and ``with await lock`` for asyncio synchronization primitives.
+
+..
+
+.. bpo: 22589
+.. date: 2017-12-08-15-09-41
+.. nonce: 8ouqI6
+.. section: Library
+
+Changed MIME type of .bmp from 'image/x-ms-bmp' to 'image/bmp'
+
+..
+
+.. bpo: 32193
+.. date: 2017-12-08-11-02-26
+.. nonce: NJe_TQ
+.. section: Library
+
+Convert asyncio to use *async/await* syntax. Old styled ``yield from`` is
+still supported too.
+
+..
+
+.. bpo: 32206
+.. date: 2017-12-07-13-14-40
+.. nonce: obm4OM
+.. section: Library
+
+Add support to run modules with pdb
+
+..
+
+.. bpo: 32227
+.. date: 2017-12-05-13-25-15
+.. nonce: 3vnWFS
+.. section: Library
+
+``functools.singledispatch`` now supports registering implementations using
+type annotations.
+
+..
+
+.. bpo: 15873
+.. date: 2017-12-04-17-41-40
+.. nonce: -T4TRK
+.. section: Library
+
+Added new alternate constructors :meth:`datetime.datetime.fromisoformat`,
+:meth:`datetime.time.fromisoformat` and :meth:`datetime.date.fromisoformat`
+as the inverse operation of each classes's respective ``isoformat`` methods.
+
+..
+
+.. bpo: 32199
+.. date: 2017-12-04-12-23-26
+.. nonce: nGof4v
+.. section: Library
+
+The getnode() ip getter now uses 'ip link' instead of 'ip link list'.
+
+..
+
+.. bpo: 32143
+.. date: 2017-11-26-17-28-26
+.. nonce: o7YdXL
+.. section: Library
+
+os.statvfs() includes the f_fsid field from statvfs(2)
+
+..
+
+.. bpo: 26439
+.. date: 2017-11-24-08-35-43
+.. nonce: IC45_f
+.. section: Library
+
+Fix ctypes.util.find_library() for AIX by implementing
+ctypes._aix.find_library() Patch by: Michael Felt
+
+..
+
+.. bpo: 31993
+.. date: 2017-11-10-00-05-08
+.. nonce: -OMNg8
+.. section: Library
+
+The picklers no longer allocate temporary memory when dumping large
+``bytes`` and ``str`` objects into a file object. Instead the data is
+directly streamed into the underlying file object.
+
+..
+
+.. bpo: 27456
+.. date: 2017-11-02-11-57-41
+.. nonce: snzyTC
+.. section: Library
+
+Ensure TCP_NODELAY is set on Linux. Tests by Victor Stinner.
+
+..
+
+.. bpo: 31778
+.. date: 2017-10-18-17-29-30
+.. nonce: B6vAkP
+.. section: Library
+
+ast.literal_eval() is now more strict. Addition and subtraction of arbitrary
+numbers no longer allowed.
+
+..
+
+.. bpo: 31802
+.. date: 2017-10-17-14-52-14
+.. nonce: sYj2Zv
+.. section: Library
+
+Importing native path module (``posixpath``, ``ntpath``) now works even if
+the ``os`` module still is not imported.
+
+..
+
+.. bpo: 30241
+.. date: 2017-10-10-18-56-46
+.. nonce: F_go20
+.. section: Library
+
+Add contextlib.AbstractAsyncContextManager. Patch by Jelle Zijlstra.
+
+..
+
+.. bpo: 31699
+.. date: 2017-10-05-11-06-32
+.. nonce: MF47Y6
+.. section: Library
+
+Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task
+arguments or results cause pickling or unpickling errors. This should make
+sure that calls to the :class:`ProcessPoolExecutor` API always eventually
+return.
+
+..
+
+.. bpo: 15216
+.. date: 2017-09-16-02-56-33
+.. nonce: lqXCTT
+.. section: Library
+
+``TextIOWrapper.reconfigure()`` supports changing *encoding*, *errors*, and
+*newline*.
+
+..
+
+.. bpo: 32418
+.. date: 2017-12-24-17-29-37
+.. nonce: eZe-ID
+.. section: Documentation
+
+Add get_loop() method to Server and AbstractServer classes.
+
+..
+
+.. bpo: 32252
+.. date: 2017-12-11-13-31-33
+.. nonce: YnFw7J
+.. section: Tests
+
+Fix faulthandler_suppress_crash_report() used to prevent core dump files
+when testing crashes. getrlimit() returns zero on success.
+
+..
+
+.. bpo: 32002
+.. date: 2017-11-11-16-35-18
+.. nonce: itDxIo
+.. section: Tests
+
+Adjust C locale coercion testing for the empty locale and POSIX locale cases
+to more readily adjust to platform dependent behaviour.
+
+..
+
+.. bpo: 19764
+.. date: 2017-08-18-18-00-24
+.. nonce: ODpc9y
+.. section: Windows
+
+Implement support for `subprocess.Popen(close_fds=True)` on Windows. Patch
+by Segev Finer.
+
+..
+
+.. bpo: 24960
+.. date: 2017-12-22-09-25-51
+.. nonce: TGdAgO
+.. section: Tools/Demos
+
+2to3 and lib2to3 can now read pickled grammar files using pkgutil.get_data()
+rather than probing the filesystem. This lets 2to3 and lib2to3 work when run
+from a zipfile.
+
+..
+
+.. bpo: 32030
+.. date: 2017-12-20-23-22-32
+.. nonce: d1dcwh
+.. section: C API
+
+Py_Initialize() doesn't reset the memory allocators to default if the
+``PYTHONMALLOC`` environment variable is not set.
+
+..
+
+.. bpo: 29084
+.. date: 2017-12-16-09-59-35
+.. nonce: ZGJ-LJ
+.. section: C API
+
+Undocumented C API for OrderedDict has been excluded from the limited C API.
+It was added by mistake and actually never worked in the limited C API.
+
+..
+
+.. bpo: 32264
+.. date: 2017-12-12-23-09-46
+.. nonce: ahRlOI
+.. section: C API
+
+Moved the pygetopt.h header into internal/, since it has no public APIs.
+
+..
+
+.. bpo: 32241
+.. date: 2017-12-07-15-58-15
+.. nonce: LbyQt6
+.. section: C API
+
+:c:func:`Py_SetProgramName` and :c:func:`Py_SetPythonHome` now take the
+``const wchar *`` arguments instead of ``wchar *``.
diff --git a/Misc/NEWS.d/next/C API/2017-12-07-15-58-15.bpo-32241.LbyQt6.rst b/Misc/NEWS.d/next/C API/2017-12-07-15-58-15.bpo-32241.LbyQt6.rst
deleted file mode 100644
index 71fa8baa807..00000000000
--- a/Misc/NEWS.d/next/C API/2017-12-07-15-58-15.bpo-32241.LbyQt6.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-:c:func:`Py_SetProgramName` and :c:func:`Py_SetPythonHome` now take the
-``const wchar *`` arguments instead of ``wchar *``.
diff --git a/Misc/NEWS.d/next/C API/2017-12-12-23-09-46.bpo-32264.ahRlOI.rst b/Misc/NEWS.d/next/C API/2017-12-12-23-09-46.bpo-32264.ahRlOI.rst
deleted file mode 100644
index 5033d61a676..00000000000
--- a/Misc/NEWS.d/next/C API/2017-12-12-23-09-46.bpo-32264.ahRlOI.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Moved the pygetopt.h header into internal/, since it has no public APIs.
diff --git a/Misc/NEWS.d/next/C API/2017-12-16-09-59-35.bpo-29084.ZGJ-LJ.rst b/Misc/NEWS.d/next/C API/2017-12-16-09-59-35.bpo-29084.ZGJ-LJ.rst
deleted file mode 100644
index 940f7160772..00000000000
--- a/Misc/NEWS.d/next/C API/2017-12-16-09-59-35.bpo-29084.ZGJ-LJ.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Undocumented C API for OrderedDict has been excluded from the limited C API.
-It was added by mistake and actually never worked in the limited C API.
diff --git a/Misc/NEWS.d/next/C API/2017-12-20-23-22-32.bpo-32030.d1dcwh.rst b/Misc/NEWS.d/next/C API/2017-12-20-23-22-32.bpo-32030.d1dcwh.rst
deleted file mode 100644
index 8e8b430e247..00000000000
--- a/Misc/NEWS.d/next/C API/2017-12-20-23-22-32.bpo-32030.d1dcwh.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Py_Initialize() doesn't reset the memory allocators to default if the
-``PYTHONMALLOC`` environment variable is not set.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-07-26-00-20-15.bpo-29469.potmyI.rst b/Misc/NEWS.d/next/Core and Builtins/2017-07-26-00-20-15.bpo-29469.potmyI.rst
deleted file mode 100644
index 138a913a4f8..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-07-26-00-20-15.bpo-29469.potmyI.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Move constant folding from bytecode layer to AST layer.
-Original patch by Eugene Toder.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-11-26-14-38-44.bpo-31650.JWf_Im.rst b/Misc/NEWS.d/next/Core and Builtins/2017-11-26-14-38-44.bpo-31650.JWf_Im.rst
deleted file mode 100644
index b653f0b27d2..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-11-26-14-38-44.bpo-31650.JWf_Im.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-Implement PEP 552 (Deterministic pycs). Python now supports invalidating
-bytecode cache files bashed on a source content hash rather than source
-last-modified time.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-11-28-15-04-14.bpo-31901.mDeCLK.rst b/Misc/NEWS.d/next/Core and Builtins/2017-11-28-15-04-14.bpo-31901.mDeCLK.rst
deleted file mode 100644
index 613eee0bf58..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-11-28-15-04-14.bpo-31901.mDeCLK.rst	
+++ /dev/null
@@ -1 +0,0 @@
-The `atexit` module now has its callback stored per interpreter.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-33-47.bpo-32225.ucKjvw.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-33-47.bpo-32225.ucKjvw.rst
deleted file mode 100644
index 5cde073dd63..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-33-47.bpo-32225.ucKjvw.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-PEP 562: Add support for module ``__getattr__`` and ``__dir__``. Implemented by Ivan
-Levkivskyi.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-42-58.bpo-32226.G8fqb6.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-42-58.bpo-32226.G8fqb6.rst
deleted file mode 100644
index 97954fd1bb0..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-05-21-42-58.bpo-32226.G8fqb6.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-PEP 560: Add support for __mro_entries__ and __class_getitem__. Implemented
-by Ivan Levkivskyi.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-05-23-10-58.bpo-29240.qpJP5l.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-05-23-10-58.bpo-29240.qpJP5l.rst
deleted file mode 100644
index dbd9d619411..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-05-23-10-58.bpo-29240.qpJP5l.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Add a new UTF-8 mode: implementation of the :pep:`540`.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-06-20-18-34.bpo-32230.PgGQaB.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-06-20-18-34.bpo-32230.PgGQaB.rst
deleted file mode 100644
index b8656e4f25a..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-06-20-18-34.bpo-32230.PgGQaB.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-`-X dev` now injects a ``'default'`` entry into sys.warnoptions, ensuring
-that it behaves identically to actually passing ``-Wdefault`` at the command
-line.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-07-17-22-30.bpo-20361.zQUmbi.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-07-17-22-30.bpo-20361.zQUmbi.rst
deleted file mode 100644
index df7aed7f07a..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-07-17-22-30.bpo-20361.zQUmbi.rst	
+++ /dev/null
@@ -1,4 +0,0 @@
-``-b`` and ``-bb`` now inject ``'default::BytesWarning'`` and
-``error::BytesWarning`` entries into ``sys.warnoptions``, ensuring that they
-take precedence over any other warning filters configured via the ``-W``
-option or the ``PYTHONWARNINGS`` environment variable.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-07-23-44-29.bpo-31506.j1U2fU.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-07-23-44-29.bpo-31506.j1U2fU.rst
deleted file mode 100644
index ceb9ee2ce8a..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-07-23-44-29.bpo-31506.j1U2fU.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Improve the error message logic for object.__new__ and object.__init__.
-Patch by Sanyam Khurana.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-09-11-03-51.bpo-32260.1DAO-p.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-09-11-03-51.bpo-32260.1DAO-p.rst
deleted file mode 100644
index 523ffe0a611..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-09-11-03-51.bpo-32260.1DAO-p.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-Don't byte swap the input keys to the SipHash algorithm on big-endian
-platforms. This should ensure siphash gives consistent results across
-platforms.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst
deleted file mode 100644
index 392ebf6a79e..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-11-01-52-42.bpo-30579.X6cEzf.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Implement TracebackType.__new__ to allow Python-level creation of
-traceback objects, and make TracebackType.tb_next mutable.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst
deleted file mode 100644
index 1c833c5439b..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-12-14-02-28.bpo-32282.xFVMTn.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix an unnecessary ifdef in the include of VersionHelpers.h in socketmodule
-on Windows.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-14-11-48-19.bpo-30416.hlHo_9.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-14-11-48-19.bpo-30416.hlHo_9.rst
deleted file mode 100644
index 532700e2481..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-14-11-48-19.bpo-30416.hlHo_9.rst	
+++ /dev/null
@@ -1,4 +0,0 @@
-The optimizer is now protected from spending much time doing complex
-calculations and consuming much memory for creating large constants in
-constant folding. Increased limits for constants that can be produced in
-constant folding.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-15-00-13-04.bpo-32329.q47IN2.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-15-00-13-04.bpo-32329.q47IN2.rst
deleted file mode 100644
index 86bcf234477..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-15-00-13-04.bpo-32329.q47IN2.rst	
+++ /dev/null
@@ -1,5 +0,0 @@
-The :option:`-R` option now turns on hash randomization when the
-:envvar:`PYTHONHASHSEED` environment variable is set to ``0``. Previously,
-the option was ignored. Moreover, ``sys.flags.hash_randomization`` is now
-properly set to 0 when hash randomization is turned off by
-``PYTHONHASHSEED=0``.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-15-11-50-06.bpo-27169.VO84fQ.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-15-11-50-06.bpo-27169.VO84fQ.rst
deleted file mode 100644
index 81d1d15cbbd..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-15-11-50-06.bpo-27169.VO84fQ.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-The ``__debug__`` constant is now optimized out at compile time. This fixes also
-bpo-22091.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-16-14-30-21.bpo-32259.GoOJiX.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-16-14-30-21.bpo-32259.GoOJiX.rst
deleted file mode 100644
index 1129c759f9f..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-16-14-30-21.bpo-32259.GoOJiX.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-The error message of a TypeError raised when unpack non-iterable is now more
-specific.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-19-21-14-41.bpo-32379.B7mOmI.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-19-21-14-41.bpo-32379.B7mOmI.rst
deleted file mode 100644
index 1050c61b9ec..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-19-21-14-41.bpo-32379.B7mOmI.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Make MRO computation faster when a class inherits from a single base.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst
deleted file mode 100644
index 0e9c1bff540..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-28-07.bpo-32390.QPj083.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Fix the compilation failure on AIX after the f_fsid field has been added to the object returned by os.statvfs() (issue #32143). Original patch by Michael Felt.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-38-17.bpo-32399.wlH12z.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-38-17.bpo-32399.wlH12z.rst
deleted file mode 100644
index ab18f268c81..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2017-12-22-13-38-17.bpo-32399.wlH12z.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Add AIX uuid library support for RFC4122 using uuid_create() in libc.a
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst
deleted file mode 100644
index 842775f3b93..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2018-01-04-15-06-15.bpo-32226.7cAvRG.rst	
+++ /dev/null
@@ -1 +0,0 @@
-``__class_getitem__`` is now an automatic class method.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst
deleted file mode 100644
index 98cfae0dd8c..00000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2018-01-05-20-54-27.bpo-31975.AmftlU.rst	
+++ /dev/null
@@ -1,4 +0,0 @@
-The default warning filter list now starts with a
-"default::DeprecationWarning:__main__" entry, so deprecation warnings are
-once again shown by default in single-file scripts and at the interactive
-prompt.
diff --git a/Misc/NEWS.d/next/Documentation/2017-12-24-17-29-37.bpo-32418.eZe-ID.rst b/Misc/NEWS.d/next/Documentation/2017-12-24-17-29-37.bpo-32418.eZe-ID.rst
deleted file mode 100644
index 9441b74ef84..00000000000
--- a/Misc/NEWS.d/next/Documentation/2017-12-24-17-29-37.bpo-32418.eZe-ID.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add get_loop() method to Server and AbstractServer classes.
diff --git a/Misc/NEWS.d/next/Library/2017-09-16-02-56-33.bpo-15216.lqXCTT.rst b/Misc/NEWS.d/next/Library/2017-09-16-02-56-33.bpo-15216.lqXCTT.rst
deleted file mode 100644
index 0e9fd558ff0..00000000000
--- a/Misc/NEWS.d/next/Library/2017-09-16-02-56-33.bpo-15216.lqXCTT.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-``TextIOWrapper.reconfigure()`` supports changing *encoding*, *errors*, and
-*newline*.
diff --git a/Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst b/Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst
deleted file mode 100644
index 49cbbb3b920..00000000000
--- a/Misc/NEWS.d/next/Library/2017-10-05-11-06-32.bpo-31699.MF47Y6.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when
-task arguments or results cause pickling or unpickling errors.
-This should make sure that calls to the :class:`ProcessPoolExecutor` API
-always eventually return.
diff --git a/Misc/NEWS.d/next/Library/2017-10-10-18-56-46.bpo-30241.F_go20.rst b/Misc/NEWS.d/next/Library/2017-10-10-18-56-46.bpo-30241.F_go20.rst
deleted file mode 100644
index 9b6c6f6e422..00000000000
--- a/Misc/NEWS.d/next/Library/2017-10-10-18-56-46.bpo-30241.F_go20.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add contextlib.AbstractAsyncContextManager. Patch by Jelle Zijlstra.
diff --git a/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst b/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
deleted file mode 100644
index 77df6bbe695..00000000000
--- a/Misc/NEWS.d/next/Library/2017-10-17-14-52-14.bpo-31802.sYj2Zv.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Importing native path module (``posixpath``, ``ntpath``) now works even if
-the ``os`` module still is not imported.
diff --git a/Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst b/Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst
deleted file mode 100644
index 452ad6e4bd2..00000000000
--- a/Misc/NEWS.d/next/Library/2017-10-18-17-29-30.bpo-31778.B6vAkP.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-ast.literal_eval() is now more strict. Addition and subtraction of
-arbitrary numbers no longer allowed.
diff --git a/Misc/NEWS.d/next/Library/2017-11-02-11-57-41.bpo-27456.snzyTC.rst b/Misc/NEWS.d/next/Library/2017-11-02-11-57-41.bpo-27456.snzyTC.rst
deleted file mode 100644
index fa7b5616bb3..00000000000
--- a/Misc/NEWS.d/next/Library/2017-11-02-11-57-41.bpo-27456.snzyTC.rst
+++ /dev/null
@@ -1 +0,0 @@
-Ensure TCP_NODELAY is set on Linux. Tests by Victor Stinner.
diff --git a/Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst b/Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst
deleted file mode 100644
index b453e21c80c..00000000000
--- a/Misc/NEWS.d/next/Library/2017-11-10-00-05-08.bpo-31993.-OMNg8.rst
+++ /dev/null
@@ -1,14 +0,0 @@
-The picklers do no longer allocate temporary memory when dumping large
-``bytes`` and ``str`` objects into a file object. Instead the data is
-directly streamed into the underlying file object.
-
-Previously the C implementation would buffer all content and issue a
-single call to ``file.write`` at the end of the dump. With protocol 4
-this behavior has changed to issue one call to ``file.write`` per frame.
-
-The Python pickler with protocol 4 now dumps each frame content as a
-memoryview to an IOBytes instance that is never reused and the
-memoryview is no longer released after the call to write. This makes it
-possible for the file object to delay access to the memoryview of
-previous frames without forcing any additional memory copy as was
-already possible with the C pickler.
diff --git a/Misc/NEWS.d/next/Library/2017-11-24-08-35-43.bpo-26439.IC45_f.rst b/Misc/NEWS.d/next/Library/2017-11-24-08-35-43.bpo-26439.IC45_f.rst
deleted file mode 100644
index 91c9a45b503..00000000000
--- a/Misc/NEWS.d/next/Library/2017-11-24-08-35-43.bpo-26439.IC45_f.rst
+++ /dev/null
@@ -1,12 +0,0 @@
-Fix ctypes.util.find_library() for AIX
-by implementing ctypes._aix.find_library()
-Patch by: Michael Felt aka aixtools
-
-ctypes.util.find_library has always returned None on a standard AIX.
-With this patch there is support for both AIX and svr4 shared libraries.
-None is returned only when there is nothinbg found. Normal behavior is now:
-on AIX find_library("FOO") returns either libFOO.a(libFOO.so) or libFOO.so
-while legacy names e.g., find_library("c") returns libc.a(shr.o)
-or libc.a(shr_64.o) - depending on 32 or 64-bit operations.
-Include RTLD_MEMBER to mode to support AIX legacy library(member) names
-(Modules/_ctype/posixmodule.c), ctypes/__init__.py and configure.ac)
diff --git a/Misc/NEWS.d/next/Library/2017-11-26-17-28-26.bpo-32143.o7YdXL.rst b/Misc/NEWS.d/next/Library/2017-11-26-17-28-26.bpo-32143.o7YdXL.rst
deleted file mode 100644
index f416ec504ec..00000000000
--- a/Misc/NEWS.d/next/Library/2017-11-26-17-28-26.bpo-32143.o7YdXL.rst
+++ /dev/null
@@ -1 +0,0 @@
-os.statvfs() includes the f_fsid field from statvfs(2)
diff --git a/Misc/NEWS.d/next/Library/2017-12-04-12-23-26.bpo-32199.nGof4v.rst b/Misc/NEWS.d/next/Library/2017-12-04-12-23-26.bpo-32199.nGof4v.rst
deleted file mode 100644
index d5f88cef5c7..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-04-12-23-26.bpo-32199.nGof4v.rst
+++ /dev/null
@@ -1 +0,0 @@
-The getnode() ip getter now uses 'ip link' instead of 'ip link list'.
diff --git a/Misc/NEWS.d/next/Library/2017-12-04-17-41-40.bpo-15873.-T4TRK.rst b/Misc/NEWS.d/next/Library/2017-12-04-17-41-40.bpo-15873.-T4TRK.rst
deleted file mode 100644
index 98a841597af..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-04-17-41-40.bpo-15873.-T4TRK.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Added new alternate constructors :meth:`datetime.datetime.fromisoformat`,
-:meth:`datetime.time.fromisoformat` and :meth:`datetime.date.fromisoformat`
-as the inverse operation of each classes's respective ``isoformat`` methods.
diff --git a/Misc/NEWS.d/next/Library/2017-12-05-13-25-15.bpo-32227.3vnWFS.rst b/Misc/NEWS.d/next/Library/2017-12-05-13-25-15.bpo-32227.3vnWFS.rst
deleted file mode 100644
index 4dbc7ba907b..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-05-13-25-15.bpo-32227.3vnWFS.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-``functools.singledispatch`` now supports registering implementations using
-type annotations.
diff --git a/Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst b/Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst
deleted file mode 100644
index 20d7eeace87..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-07-13-14-40.bpo-32206.obm4OM.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add support to run modules with pdb
diff --git a/Misc/NEWS.d/next/Library/2017-12-08-11-02-26.bpo-32193.NJe_TQ.rst b/Misc/NEWS.d/next/Library/2017-12-08-11-02-26.bpo-32193.NJe_TQ.rst
deleted file mode 100644
index 6982f1e1ac5..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-08-11-02-26.bpo-32193.NJe_TQ.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Convert asyncio to use *async/await* syntax. Old styled ``yield from`` is
-still supported too.
diff --git a/Misc/NEWS.d/next/Library/2017-12-08-15-09-41.bpo-22589.8ouqI6.rst b/Misc/NEWS.d/next/Library/2017-12-08-15-09-41.bpo-22589.8ouqI6.rst
deleted file mode 100644
index e8469cba610..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-08-15-09-41.bpo-22589.8ouqI6.rst
+++ /dev/null
@@ -1 +0,0 @@
-Changed MIME type of .bmp from 'image/x-ms-bmp' to 'image/bmp'
diff --git a/Misc/NEWS.d/next/Library/2017-12-09-11-30-35.bpo-32253.TQHSYF.rst b/Misc/NEWS.d/next/Library/2017-12-09-11-30-35.bpo-32253.TQHSYF.rst
deleted file mode 100644
index 2916410e457..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-09-11-30-35.bpo-32253.TQHSYF.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Deprecate ``yield from lock``, ``await lock``, ``with (yield from lock)``
-and ``with await lock`` for asyncio synchronization primitives.
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-00-57-51.bpo-32265.kELtTE.rst b/Misc/NEWS.d/next/Library/2017-12-10-00-57-51.bpo-32265.kELtTE.rst
deleted file mode 100644
index 16686c1b418..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-10-00-57-51.bpo-32265.kELtTE.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-All class and static methods of builtin types now are correctly classified
-by inspect.classify_class_attrs() and grouped in pydoc ouput. Added
-types.ClassMethodDescriptorType for unbound class methods of builtin types.
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-12-30-13.bpo-32269.Q85pKj.rst b/Misc/NEWS.d/next/Library/2017-12-10-12-30-13.bpo-32269.Q85pKj.rst
deleted file mode 100644
index 9fa12446de2..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-10-12-30-13.bpo-32269.Q85pKj.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add asyncio.get_running_loop() function.
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-18-59-13.bpo-32272.Mu84Am.rst b/Misc/NEWS.d/next/Library/2017-12-10-18-59-13.bpo-32272.Mu84Am.rst
deleted file mode 100644
index 500e3c8c2a5..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-10-18-59-13.bpo-32272.Mu84Am.rst
+++ /dev/null
@@ -1 +0,0 @@
-Remove asyncio.async() function.
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-19-14-55.bpo-32273.5KKlCv.rst b/Misc/NEWS.d/next/Library/2017-12-10-19-14-55.bpo-32273.5KKlCv.rst
deleted file mode 100644
index 3bbec8f6e34..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-10-19-14-55.bpo-32273.5KKlCv.rst
+++ /dev/null
@@ -1 +0,0 @@
-Move asyncio.test_utils to test.test_asyncio.
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst b/Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst
deleted file mode 100644
index 85f3df5c8f5..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-10-21-19-14.bpo-29137.CFcON1.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-The ``fpectl`` library has been removed. It was never enabled by default,
-never worked correctly on x86-64, and it changed the Python ABI in ways that
-caused unexpected breakage of C extensions.
diff --git a/Misc/NEWS.d/next/Library/2017-12-10-23-44-56.bpo-30050.4SZ3lY.rst b/Misc/NEWS.d/next/Library/2017-12-10-23-44-56.bpo-30050.4SZ3lY.rst
deleted file mode 100644
index 76de12bbe80..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-10-23-44-56.bpo-30050.4SZ3lY.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-New argument warn_on_full_buffer to signal.set_wakeup_fd lets you control
-whether Python prints a warning on stderr when the wakeup fd buffer
-overflows.
diff --git a/Misc/NEWS.d/next/Library/2017-12-11-09-53-14.bpo-32277.jkKiVC.rst b/Misc/NEWS.d/next/Library/2017-12-11-09-53-14.bpo-32277.jkKiVC.rst
deleted file mode 100644
index 765527a6553..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-11-09-53-14.bpo-32277.jkKiVC.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Raise ``NotImplementedError`` instead of ``SystemError`` on platforms where
-``chmod(..., follow_symlinks=False)`` is not supported.  Patch by Anthony
-Sottile.
diff --git a/Misc/NEWS.d/next/Library/2017-12-12-07-29-06.bpo-32255.2bfNmM.rst b/Misc/NEWS.d/next/Library/2017-12-12-07-29-06.bpo-32255.2bfNmM.rst
deleted file mode 100644
index dafee67bdd0..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-12-07-29-06.bpo-32255.2bfNmM.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-A single empty field is now always quoted when written into a CSV file.
-This allows to distinguish an empty row from a row consisting of a single empty field.
-Patch by Licht Takeuchi.
diff --git a/Misc/NEWS.d/next/Library/2017-12-12-16-58-20.bpo-32250.UljTa0.rst b/Misc/NEWS.d/next/Library/2017-12-12-16-58-20.bpo-32250.UljTa0.rst
deleted file mode 100644
index f2d016df39a..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-12-16-58-20.bpo-32250.UljTa0.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Implement ``asyncio.current_task()`` and ``asyncio.all_tasks()``. Add
-helpers intended to be used by alternative task implementations:
-``asyncio._register_task``, ``asyncio._enter_task``, ``asyncio._leave_task``
-and ``asyncio._unregister_task``. Deprecate ``asyncio.Task.current_task()``
-and ``asyncio.Task.all_tasks()``.
diff --git a/Misc/NEWS.d/next/Library/2017-12-12-18-01-01.bpo-32296.bwscHz.rst b/Misc/NEWS.d/next/Library/2017-12-12-18-01-01.bpo-32296.bwscHz.rst
deleted file mode 100644
index 4100d48518e..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-12-18-01-01.bpo-32296.bwscHz.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Implement asyncio._get_running_loop() and get_event_loop() in C. This makes
-them 4x faster.
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-00-00-37.bpo-17852.Q8BP8N.rst b/Misc/NEWS.d/next/Library/2017-12-13-00-00-37.bpo-17852.Q8BP8N.rst
deleted file mode 100644
index 768108d5a54..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-13-00-00-37.bpo-17852.Q8BP8N.rst
+++ /dev/null
@@ -1 +0,0 @@
-Revert incorrect fix based on misunderstanding of _Py_PyAtExit() semantics.
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-16-47-38.bpo-32314.W4_U2j.rst b/Misc/NEWS.d/next/Library/2017-12-13-16-47-38.bpo-32314.W4_U2j.rst
deleted file mode 100644
index 416906c9662..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-13-16-47-38.bpo-32314.W4_U2j.rst
+++ /dev/null
@@ -1 +0,0 @@
-Implement asyncio.run().
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-19-02-38.bpo-29970.uxVOpk.rst b/Misc/NEWS.d/next/Library/2017-12-13-19-02-38.bpo-29970.uxVOpk.rst
deleted file mode 100644
index d3d9ae94b32..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-13-19-02-38.bpo-29970.uxVOpk.rst
+++ /dev/null
@@ -1 +0,0 @@
-Abort asyncio SSLProtocol connection if handshake not complete within 10s
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst b/Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst
deleted file mode 100644
index d760d6089ba..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-13-20-31-30.bpo-32308.CUbsb2.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-:func:`re.sub()` now replaces empty matches adjacent to a previous non-empty
-match.
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-22-10-36.bpo-29711.hJjghA.rst b/Misc/NEWS.d/next/Library/2017-12-13-22-10-36.bpo-29711.hJjghA.rst
deleted file mode 100644
index 6774172fb6b..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-13-22-10-36.bpo-29711.hJjghA.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix ``stop_serving`` in asyncio proactor loop kill all listening servers
diff --git a/Misc/NEWS.d/next/Library/2017-12-13-22-38-08.bpo-32302.othtTr.rst b/Misc/NEWS.d/next/Library/2017-12-13-22-38-08.bpo-32302.othtTr.rst
deleted file mode 100644
index abe51f094a8..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-13-22-38-08.bpo-32302.othtTr.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix bdist_wininst of distutils for CRT v142: it binary compatible with CRT
-v140.
diff --git a/Misc/NEWS.d/next/Library/2017-12-14-10-10-10.bpo-32323.ideco.rst b/Misc/NEWS.d/next/Library/2017-12-14-10-10-10.bpo-32323.ideco.rst
deleted file mode 100644
index 3077d7cdfef..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-14-10-10-10.bpo-32323.ideco.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-:func:`urllib.parse.urlsplit()` does not convert zone-id (scope) to lower case
-for scoped IPv6 addresses in hostnames now.
diff --git a/Misc/NEWS.d/next/Library/2017-12-14-16-00-25.bpo-32327.bbkSxA.rst b/Misc/NEWS.d/next/Library/2017-12-14-16-00-25.bpo-32327.bbkSxA.rst
deleted file mode 100644
index 5fd9fbca4b4..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-14-16-00-25.bpo-32327.bbkSxA.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Convert asyncio functions that were documented as coroutines to coroutines.
-Affected functions: loop.sock_sendall, loop.sock_recv, loop.sock_accept,
-loop.run_in_executor, loop.getaddrinfo, loop.getnameinfo.
diff --git a/Misc/NEWS.d/next/Library/2017-12-14-17-28-54.bpo-32311.DL5Ytn.rst b/Misc/NEWS.d/next/Library/2017-12-14-17-28-54.bpo-32311.DL5Ytn.rst
deleted file mode 100644
index e2d10959476..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-14-17-28-54.bpo-32311.DL5Ytn.rst
+++ /dev/null
@@ -1 +0,0 @@
-Implement asyncio.create_task(coro) shortcut
diff --git a/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst b/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
deleted file mode 100644
index 02b7e5fef11..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Add :class:`importlib.abc.ResourceReader` as an ABC for loaders to provide a
-unified API for reading resources contained within packages.  Also add
-:mod:`importlib.resources` as the port of ``importlib_resources``.
diff --git a/Misc/NEWS.d/next/Library/2017-12-15-23-48-43.bpo-32331.fIg1Uc.rst b/Misc/NEWS.d/next/Library/2017-12-15-23-48-43.bpo-32331.fIg1Uc.rst
deleted file mode 100644
index 53a262cf220..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-15-23-48-43.bpo-32331.fIg1Uc.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Fix socket.settimeout() and socket.setblocking() to keep socket.type
-as is. Fix socket.socket() constructor to reset any bit flags applied to
-socket's type.  This change only affects OSes that have SOCK_NONBLOCK
-and/or SOCK_CLOEXEC.
diff --git a/Misc/NEWS.d/next/Library/2017-12-16-18-50-57.bpo-32348.5j__he.rst b/Misc/NEWS.d/next/Library/2017-12-16-18-50-57.bpo-32348.5j__he.rst
deleted file mode 100644
index b3618db7c33..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-16-18-50-57.bpo-32348.5j__he.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Optimize asyncio.Future schedule/add/remove callback.  The optimization
-shows 3-6% performance improvements of async/await code.
diff --git a/Misc/NEWS.d/next/Library/2017-12-17-14-23-23.bpo-32351.95fh2K.rst b/Misc/NEWS.d/next/Library/2017-12-17-14-23-23.bpo-32351.95fh2K.rst
deleted file mode 100644
index 56f52d2a418..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-17-14-23-23.bpo-32351.95fh2K.rst
+++ /dev/null
@@ -1 +0,0 @@
-Use fastpath in asyncio.sleep if delay<0 (2x boost)
diff --git a/Misc/NEWS.d/next/Library/2017-12-17-21-42-24.bpo-32355.tbaTWA.rst b/Misc/NEWS.d/next/Library/2017-12-17-21-42-24.bpo-32355.tbaTWA.rst
deleted file mode 100644
index ca908e97803..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-17-21-42-24.bpo-32355.tbaTWA.rst
+++ /dev/null
@@ -1 +0,0 @@
-Optimize asyncio.gather(); now up to 15% faster.
diff --git a/Misc/NEWS.d/next/Library/2017-12-17-22-50-51.bpo-32356.roZJpA.rst b/Misc/NEWS.d/next/Library/2017-12-17-22-50-51.bpo-32356.roZJpA.rst
deleted file mode 100644
index 84b5381fad3..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-17-22-50-51.bpo-32356.roZJpA.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-asyncio.transport.resume_reading() and pause_reading() are now idempotent.
-New transport.is_reading() method is added.
diff --git a/Misc/NEWS.d/next/Library/2017-12-18-00-36-41.bpo-32357.t1F3sn.rst b/Misc/NEWS.d/next/Library/2017-12-18-00-36-41.bpo-32357.t1F3sn.rst
deleted file mode 100644
index f51eaf541dc..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-18-00-36-41.bpo-32357.t1F3sn.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Optimize asyncio.iscoroutine() and loop.create_task() for non-native
-coroutines (e.g. async/await compiled with Cython).
-
-'loop.create_task(python_coroutine)' used to be 20% faster than
-'loop.create_task(cython_coroutine)'.  Now, the latter is as fast.
diff --git a/Misc/NEWS.d/next/Library/2017-12-19-00-37-28.bpo-32363.YTeGU0.rst b/Misc/NEWS.d/next/Library/2017-12-19-00-37-28.bpo-32363.YTeGU0.rst
deleted file mode 100644
index 4c376c5ea91..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-19-00-37-28.bpo-32363.YTeGU0.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Make asyncio.Task.set_exception() and set_result() raise
-NotImplementedError. Task._step() and Future.__await__() raise proper
-exceptions when they are in an invalid state, instead of raising an
-AssertionError.
diff --git a/Misc/NEWS.d/next/Library/2017-12-21-11-08-42.bpo-26133.mt81QV.rst b/Misc/NEWS.d/next/Library/2017-12-21-11-08-42.bpo-26133.mt81QV.rst
deleted file mode 100644
index 0653d1960e5..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-21-11-08-42.bpo-26133.mt81QV.rst
+++ /dev/null
@@ -1 +0,0 @@
-Don't unsubscribe signals in asyncio UNIX event loop on interpreter shutdown.
diff --git a/Misc/NEWS.d/next/Library/2017-12-23-12-45-00.bpo-32415.YufXTU.rst b/Misc/NEWS.d/next/Library/2017-12-23-12-45-00.bpo-32415.YufXTU.rst
deleted file mode 100644
index f1f57376241..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-23-12-45-00.bpo-32415.YufXTU.rst
+++ /dev/null
@@ -1 +0,0 @@
-asyncio: Add Task.get_loop() and Future.get_loop()
diff --git a/Misc/NEWS.d/next/Library/2017-12-25-11-09-46.bpo-31721.5gM972.rst b/Misc/NEWS.d/next/Library/2017-12-25-11-09-46.bpo-31721.5gM972.rst
deleted file mode 100644
index a989a94a7df..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-25-11-09-46.bpo-31721.5gM972.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Prevent Python crash from happening when Future._log_traceback is set to
-True manually.  Now it can only be set to False, or a ValueError is raised.
diff --git a/Misc/NEWS.d/next/Library/2017-12-25-20-22-47.bpo-32422.5H3Wq2.rst b/Misc/NEWS.d/next/Library/2017-12-25-20-22-47.bpo-32422.5H3Wq2.rst
deleted file mode 100644
index 5918c322237..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-25-20-22-47.bpo-32422.5H3Wq2.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-``functools.lru_cache`` uses less memory (3 words for each cached key) and
-takes about 1/3 time for cyclic GC.
diff --git a/Misc/NEWS.d/next/Library/2017-12-28-21-30-40.bpo-32441.LqlboJ.rst b/Misc/NEWS.d/next/Library/2017-12-28-21-30-40.bpo-32441.LqlboJ.rst
deleted file mode 100644
index a0fe4f3cc6c..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-28-21-30-40.bpo-32441.LqlboJ.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Return the new file descriptor (i.e., the second argument) from ``os.dup2``.
-Previously, ``None`` was always returned.
diff --git a/Misc/NEWS.d/next/Library/2017-12-29-00-44-42.bpo-23749.QL1Cxd.rst b/Misc/NEWS.d/next/Library/2017-12-29-00-44-42.bpo-23749.QL1Cxd.rst
deleted file mode 100644
index d6de1fef901..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-29-00-44-42.bpo-23749.QL1Cxd.rst
+++ /dev/null
@@ -1 +0,0 @@
-asyncio: Implement loop.start_tls()
diff --git a/Misc/NEWS.d/next/Library/2017-12-31-20-32-58.bpo-32468.YBs__0.rst b/Misc/NEWS.d/next/Library/2017-12-31-20-32-58.bpo-32468.YBs__0.rst
deleted file mode 100644
index 135bbd816e8..00000000000
--- a/Misc/NEWS.d/next/Library/2017-12-31-20-32-58.bpo-32468.YBs__0.rst
+++ /dev/null
@@ -1 +0,0 @@
-Improve frame repr() to mention filename, code name and current line number.
diff --git a/Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst b/Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst
deleted file mode 100644
index bf3e99c8d86..00000000000
--- a/Misc/NEWS.d/next/Library/2018-01-06-10-54-16.bpo-32499.koyY-4.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Add dataclasses.is_dataclass(obj), which returns True if obj is a dataclass
-or an instance of one.
diff --git a/Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst b/Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst
deleted file mode 100644
index c627468072e..00000000000
--- a/Misc/NEWS.d/next/Library/2018-01-06-15-15-34.bpo-32278.bGnGc0.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Make type information optional on dataclasses.make_dataclass(). If omitted,
-the string 'typing.Any' is used.
diff --git a/Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst b/Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst
deleted file mode 100644
index b7ef8c80f5a..00000000000
--- a/Misc/NEWS.d/next/Library/2018-01-06-16-50-11.bpo-32279.1xOpU8.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Add params to dataclasses.make_dataclasses(): init, repr, eq, order, hash,
-and frozen.  Pass them through to dataclass().
diff --git a/Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst b/Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst
deleted file mode 100644
index e524769c6df..00000000000
--- a/Misc/NEWS.d/next/Library/2018-01-07-11-32-42.bpo-32506.MaT-zU.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Now that dict is defined as keeping insertion order, drop OrderedDict and
-just use plain dict.
diff --git a/Misc/NEWS.d/next/Tests/2017-11-11-16-35-18.bpo-32002.itDxIo.rst b/Misc/NEWS.d/next/Tests/2017-11-11-16-35-18.bpo-32002.itDxIo.rst
deleted file mode 100644
index 9cc7ec22c45..00000000000
--- a/Misc/NEWS.d/next/Tests/2017-11-11-16-35-18.bpo-32002.itDxIo.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Adjust C locale coercion testing for the empty locale and POSIX locale
-cases to more readily adjust to platform dependent behaviour.
diff --git a/Misc/NEWS.d/next/Tests/2017-12-11-13-31-33.bpo-32252.YnFw7J.rst b/Misc/NEWS.d/next/Tests/2017-12-11-13-31-33.bpo-32252.YnFw7J.rst
deleted file mode 100644
index ee4c56bb73f..00000000000
--- a/Misc/NEWS.d/next/Tests/2017-12-11-13-31-33.bpo-32252.YnFw7J.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix faulthandler_suppress_crash_report() used to prevent core dump files
-when testing crashes. getrlimit() returns zero on success.
diff --git a/Misc/NEWS.d/next/Tools-Demos/2017-12-22-09-25-51.bpo-24960.TGdAgO.rst b/Misc/NEWS.d/next/Tools-Demos/2017-12-22-09-25-51.bpo-24960.TGdAgO.rst
deleted file mode 100644
index 6f9d83c5772..00000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2017-12-22-09-25-51.bpo-24960.TGdAgO.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-2to3 and lib2to3 can now read pickled grammar files using pkgutil.get_data()
-rather than probing the filesystem. This lets 2to3 and lib2to3 work when run
-from a zipfile.
diff --git a/Misc/NEWS.d/next/Windows/2017-08-18-18-00-24.bpo-19764.ODpc9y.rst b/Misc/NEWS.d/next/Windows/2017-08-18-18-00-24.bpo-19764.ODpc9y.rst
deleted file mode 100644
index b5f5cae8323..00000000000
--- a/Misc/NEWS.d/next/Windows/2017-08-18-18-00-24.bpo-19764.ODpc9y.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Implement support for `subprocess.Popen(close_fds=True)` on Windows. Patch
-by Segev Finer.


From webhook-mailer at python.org  Tue Jan  9 10:28:44 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 09 Jan 2018 15:28:44 -0000
Subject: [Python-checkins] Post-release version bump to a4+
Message-ID: <mailman.6.1515511725.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/2ab3a8fd264f24baf321b861a997bf8848f232d5
commit: 2ab3a8fd264f24baf321b861a997bf8848f232d5
branch: master
author: Ned Deily <nad at python.org>
committer: Ned Deily <nad at python.org>
date: 2018-01-09T10:27:57-05:00
summary:

Post-release version bump to a4+

files:
M Include/patchlevel.h
M README.rst

diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index 49c76757674..cea35de9493 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -23,7 +23,7 @@
 #define PY_RELEASE_SERIAL       4
 
 /* Version as a string */
-#define PY_VERSION              "3.7.0a4"
+#define PY_VERSION              "3.7.0a4+"
 /*--end constants--*/
 
 /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
diff --git a/README.rst b/README.rst
index 41f809b641f..b6d744d19eb 100644
--- a/README.rst
+++ b/README.rst
@@ -1,5 +1,5 @@
-This is Python version 3.7.0 alpha 4
-====================================
+This is Python version 3.7.0 alpha 4+
+=====================================
 
 .. image:: https://travis-ci.org/python/cpython.svg?branch=master
    :alt: CPython build status on Travis CI


From webhook-mailer at python.org  Tue Jan  9 14:38:10 2018
From: webhook-mailer at python.org (Antoine Pitrou)
Date: Tue, 09 Jan 2018 19:38:10 -0000
Subject: [Python-checkins] bpo-32493: Not only AIX,
 but FreeBSD has uuid_create support (#5089)
Message-ID: <mailman.7.1515526691.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/b4ebaa7099c3413b42a97777581c4ca560fe7540
commit: b4ebaa7099c3413b42a97777581c4ca560fe7540
branch: master
author: David Carlier <dcarlier at afilias.info>
committer: Antoine Pitrou <pitrou at free.fr>
date: 2018-01-09T20:38:07+01:00
summary:

bpo-32493: Not only AIX, but FreeBSD has uuid_create support (#5089)

Allow building the _uuid extension module on FreeBSD and OpenBSD.

files:
A Misc/NEWS.d/next/Library/2018-01-08-15-53-37.bpo-32493.vTXxGN.rst
M Modules/_uuidmodule.c
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Misc/NEWS.d/next/Library/2018-01-08-15-53-37.bpo-32493.vTXxGN.rst b/Misc/NEWS.d/next/Library/2018-01-08-15-53-37.bpo-32493.vTXxGN.rst
new file mode 100644
index 00000000000..82d33aa55b7
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-08-15-53-37.bpo-32493.vTXxGN.rst
@@ -0,0 +1 @@
+UUID module fixes build for FreeBSD/OpenBSD
diff --git a/Modules/_uuidmodule.c b/Modules/_uuidmodule.c
index 1b37511c228..16aa09b04ff 100644
--- a/Modules/_uuidmodule.c
+++ b/Modules/_uuidmodule.c
@@ -19,10 +19,7 @@ py_uuid_generate_time_safe(void)
     res = uuid_generate_time_safe(uuid);
     return Py_BuildValue("y#i", (const char *) uuid, sizeof(uuid), res);
 #elif HAVE_UUID_CREATE
-/*
- * AIX support for uuid - RFC4122
- */
-    unsigned32 status;
+    uint32_t status;
     uuid_create(&uuid, &status);
     return Py_BuildValue("y#i", (const char *) &uuid, sizeof(uuid), (int) status);
 #else
diff --git a/configure b/configure
index afffc190751..9286443124a 100755
--- a/configure
+++ b/configure
@@ -9560,8 +9560,9 @@ fi
 rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
 
 # AIX provides support for RFC4122 (uuid) in libc.a starting with AIX 6.1 (anno 2007)
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for RFC4122 - uuid support on AIX" >&5
-$as_echo_n "checking for RFC4122 - uuid support on AIX... " >&6; }
+# FreeBSD and OpenBSD provides support as well
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for uuid_create" >&5
+$as_echo_n "checking for uuid_create... " >&6; }
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 #include <uuid.h>
diff --git a/configure.ac b/configure.ac
index 0a48e41d5c3..1894e21f304 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2696,13 +2696,14 @@ void *x = uuid_generate_time_safe
 )
 
 # AIX provides support for RFC4122 (uuid) in libc.a starting with AIX 6.1 (anno 2007)
-AC_MSG_CHECKING(for RFC4122 - uuid support on AIX)
+# FreeBSD and OpenBSD provides support as well
+AC_MSG_CHECKING(for uuid_create)
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <uuid.h>]], [[
 #ifndef uuid_create
 void *x = uuid_create
 #endif
 ]])],
-  [AC_DEFINE(HAVE_UUID_CREATE, 1, Define if uuid_create() exists. AIX support for uuid:RFC4122)
+  [AC_DEFINE(HAVE_UUID_CREATE, 1, Define if uuid_create() exists.)
    AC_MSG_RESULT(yes)],
   [AC_MSG_RESULT(no)]
 )
diff --git a/pyconfig.h.in b/pyconfig.h.in
index 7d366562b0c..ff1083ae0b0 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -1191,7 +1191,7 @@
 /* Define to 1 if you have the <utime.h> header file. */
 #undef HAVE_UTIME_H
 
-/* Define if uuid_create() exists. AIX support for uuid:RFC4122 */
+/* Define if uuid_create() exists. */
 #undef HAVE_UUID_CREATE
 
 /* Define if uuid_generate_time_safe() exists. */


From webhook-mailer at python.org  Tue Jan  9 14:54:56 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 09 Jan 2018 19:54:56 -0000
Subject: [Python-checkins] bpo-24340: Fix estimation of the code stack size.
 (#5076)
Message-ID: <mailman.8.1515527698.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/d4864c61e3e27e337762dc45e504977299bd5b46
commit: d4864c61e3e27e337762dc45e504977299bd5b46
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-09T21:54:52+02:00
summary:

bpo-24340: Fix estimation of the code stack size. (#5076)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-01-21-59-31.bpo-24340.hmKBvg.rst
M Lib/test/test_compile.py
M Lib/test/test_dis.py
M Python/compile.c
M Python/importlib.h
M Python/importlib_external.h

diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py
index ae3043af0bd..29aa362590f 100644
--- a/Lib/test/test_compile.py
+++ b/Lib/test/test_compile.py
@@ -672,7 +672,7 @@ def __fspath__(self):
         compile("42", PathLike("test_compile_pathlike"), "single")
 
 
-class TestStackSize(unittest.TestCase):
+class TestExpressionStackSize(unittest.TestCase):
     # These tests check that the computed stack size for a code object
     # stays within reasonable bounds (see issue #21523 for an example
     # dysfunction).
@@ -710,5 +710,294 @@ def test_func_and(self):
         self.check_stack_size(code)
 
 
+class TestStackSizeStability(unittest.TestCase):
+    # Check that repeating certain snippets doesn't increase the stack size
+    # beyond what a single snippet requires.
+
+    def check_stack_size(self, snippet, async_=False):
+        def compile_snippet(i):
+            ns = {}
+            script = """def func():\n""" + i * snippet
+            if async_:
+                script = "async " + script
+            code = compile(script, "<script>", "exec")
+            exec(code, ns, ns)
+            return ns['func'].__code__
+
+        sizes = [compile_snippet(i).co_stacksize for i in range(2, 5)]
+        if len(set(sizes)) != 1:
+            import dis, io
+            out = io.StringIO()
+            dis.dis(compile_snippet(1), file=out)
+            self.fail("stack sizes diverge with # of consecutive snippets: "
+                      "%s\n%s\n%s" % (sizes, snippet, out.getvalue()))
+
+    def test_if(self):
+        snippet = """
+            if x:
+                a
+            """
+        self.check_stack_size(snippet)
+
+    def test_if_else(self):
+        snippet = """
+            if x:
+                a
+            elif y:
+                b
+            else:
+                c
+            """
+        self.check_stack_size(snippet)
+
+    def test_try_except_bare(self):
+        snippet = """
+            try:
+                a
+            except:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_try_except_qualified(self):
+        snippet = """
+            try:
+                a
+            except ImportError:
+                b
+            except:
+                c
+            else:
+                d
+            """
+        self.check_stack_size(snippet)
+
+    def test_try_except_as(self):
+        snippet = """
+            try:
+                a
+            except ImportError as e:
+                b
+            except:
+                c
+            else:
+                d
+            """
+        self.check_stack_size(snippet)
+
+    def test_try_finally(self):
+        snippet = """
+                try:
+                    a
+                finally:
+                    b
+            """
+        self.check_stack_size(snippet)
+
+    def test_with(self):
+        snippet = """
+            with x as y:
+                a
+            """
+        self.check_stack_size(snippet)
+
+    def test_while_else(self):
+        snippet = """
+            while x:
+                a
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_for(self):
+        snippet = """
+            for x in y:
+                a
+            """
+        self.check_stack_size(snippet)
+
+    def test_for_else(self):
+        snippet = """
+            for x in y:
+                a
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_for_break_continue(self):
+        snippet = """
+            for x in y:
+                if z:
+                    break
+                elif u:
+                    continue
+                else:
+                    a
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_for_break_continue_inside_try_finally_block(self):
+        snippet = """
+            for x in y:
+                try:
+                    if z:
+                        break
+                    elif u:
+                        continue
+                    else:
+                        a
+                finally:
+                    f
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_for_break_inside_finally_block(self):
+        snippet = """
+            for x in y:
+                try:
+                    t
+                finally:
+                    if z:
+                        break
+                    else:
+                        a
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_for_break_continue_inside_except_block(self):
+        snippet = """
+            for x in y:
+                try:
+                    t
+                except:
+                    if z:
+                        break
+                    elif u:
+                        continue
+                    else:
+                        a
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_for_break_continue_inside_with_block(self):
+        snippet = """
+            for x in y:
+                with c:
+                    if z:
+                        break
+                    elif u:
+                        continue
+                    else:
+                        a
+            else:
+                b
+            """
+        self.check_stack_size(snippet)
+
+    def test_return_inside_try_finally_block(self):
+        snippet = """
+            try:
+                if z:
+                    return
+                else:
+                    a
+            finally:
+                f
+            """
+        self.check_stack_size(snippet)
+
+    def test_return_inside_finally_block(self):
+        snippet = """
+            try:
+                t
+            finally:
+                if z:
+                    return
+                else:
+                    a
+            """
+        self.check_stack_size(snippet)
+
+    def test_return_inside_except_block(self):
+        snippet = """
+            try:
+                t
+            except:
+                if z:
+                    return
+                else:
+                    a
+            """
+        self.check_stack_size(snippet)
+
+    def test_return_inside_with_block(self):
+        snippet = """
+            with c:
+                if z:
+                    return
+                else:
+                    a
+            """
+        self.check_stack_size(snippet)
+
+    def test_async_with(self):
+        snippet = """
+            async with x as y:
+                a
+            """
+        self.check_stack_size(snippet, async_=True)
+
+    def test_async_for(self):
+        snippet = """
+            async for x in y:
+                a
+            """
+        self.check_stack_size(snippet, async_=True)
+
+    def test_async_for_else(self):
+        snippet = """
+            async for x in y:
+                a
+            else:
+                b
+            """
+        self.check_stack_size(snippet, async_=True)
+
+    def test_for_break_continue_inside_async_with_block(self):
+        snippet = """
+            for x in y:
+                async with c:
+                    if z:
+                        break
+                    elif u:
+                        continue
+                    else:
+                        a
+            else:
+                b
+            """
+        self.check_stack_size(snippet, async_=True)
+
+    def test_return_inside_async_with_block(self):
+        snippet = """
+            async with c:
+                if z:
+                    return
+                else:
+                    a
+            """
+        self.check_stack_size(snippet, async_=True)
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
index bfbbee2794a..590f041aa92 100644
--- a/Lib/test/test_dis.py
+++ b/Lib/test/test_dis.py
@@ -284,18 +284,18 @@ def bug1333982(x=[]):
              22 POP_TOP
              24 STORE_FAST               0 (e)
              26 POP_TOP
-             28 SETUP_FINALLY           12 (to 42)
+             28 SETUP_FINALLY           10 (to 40)
 
 %3d          30 LOAD_FAST                0 (e)
              32 LOAD_ATTR                1 (__traceback__)
              34 STORE_FAST               1 (tb)
              36 POP_BLOCK
-             38 POP_EXCEPT
-             40 LOAD_CONST               0 (None)
-        >>   42 LOAD_CONST               0 (None)
-             44 STORE_FAST               0 (e)
-             46 DELETE_FAST              0 (e)
-             48 END_FINALLY
+             38 LOAD_CONST               0 (None)
+        >>   40 LOAD_CONST               0 (None)
+             42 STORE_FAST               0 (e)
+             44 DELETE_FAST              0 (e)
+             46 END_FINALLY
+             48 POP_EXCEPT
              50 JUMP_FORWARD             2 (to 54)
         >>   52 END_FINALLY
 
@@ -741,7 +741,7 @@ def f(c=c):
 Argument count:    0
 Kw-only arguments: 0
 Number of locals:  2
-Stack size:        17
+Stack size:        10
 Flags:             OPTIMIZED, NEWLOCALS, NOFREE, COROUTINE
 Constants:
    0: None
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-01-21-59-31.bpo-24340.hmKBvg.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-01-21-59-31.bpo-24340.hmKBvg.rst
new file mode 100644
index 00000000000..c4189752d50
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-01-21-59-31.bpo-24340.hmKBvg.rst	
@@ -0,0 +1 @@
+Fixed estimation of the code stack size.
diff --git a/Python/compile.c b/Python/compile.c
index cd039168eba..7ba2ff34df2 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -857,10 +857,22 @@ compiler_set_lineno(struct compiler *c, int off)
     b->b_instr[off].i_lineno = c->u->u_lineno;
 }
 
-int
-PyCompile_OpcodeStackEffect(int opcode, int oparg)
+/* Return the stack effect of opcode with argument oparg.
+
+   Some opcodes have different stack effect when jump to the target and
+   when not jump. The 'jump' parameter specifies the case:
+
+   * 0 -- when not jump
+   * 1 -- when jump
+   * -1 -- maximal
+ */
+/* XXX Make the stack effect of WITH_CLEANUP_START and
+   WITH_CLEANUP_FINISH deterministic. */
+static int
+stack_effect(int opcode, int oparg, int jump)
 {
     switch (opcode) {
+        /* Stack manipulation */
         case POP_TOP:
             return -1;
         case ROT_TWO:
@@ -871,6 +883,7 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
         case DUP_TOP_TWO:
             return 2;
 
+        /* Unary operators */
         case UNARY_POSITIVE:
         case UNARY_NEGATIVE:
         case UNARY_NOT:
@@ -883,6 +896,7 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
         case MAP_ADD:
             return -2;
 
+        /* Binary operators */
         case BINARY_POWER:
         case BINARY_MULTIPLY:
         case BINARY_MATRIX_MULTIPLY:
@@ -932,11 +946,16 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
         case BREAK_LOOP:
             return 0;
         case SETUP_WITH:
-            return 7;
+            /* 1 in the normal flow.
+             * Restore the stack position and push 6 values before jumping to
+             * the handler if an exception be raised. */
+            return jump ? 6 : 1;
         case WITH_CLEANUP_START:
-            return 1;
+            return 2; /* or 1, depending on TOS */
         case WITH_CLEANUP_FINISH:
-            return -1; /* XXX Sometimes more */
+            /* Pop a variable number of values pushed by WITH_CLEANUP_START
+             * + __exit__ or __aexit__. */
+            return -3;
         case RETURN_VALUE:
             return -1;
         case IMPORT_STAR:
@@ -950,9 +969,10 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
         case POP_BLOCK:
             return 0;
         case POP_EXCEPT:
-            return 0;  /* -3 except if bad bytecode */
+            return -3;
         case END_FINALLY:
-            return -1; /* or -2 or -3 if exception occurred */
+            /* Pop 6 values when an exception was raised. */
+            return -6;
 
         case STORE_NAME:
             return -1;
@@ -963,7 +983,8 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
         case UNPACK_EX:
             return (oparg&0xFF) + (oparg>>8);
         case FOR_ITER:
-            return 1; /* or -1, at end of iterator */
+            /* -1 at end of iterator, 1 if continue iterating. */
+            return jump > 0 ? -1 : 1;
 
         case STORE_ATTR:
             return -2;
@@ -1002,12 +1023,15 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
         case IMPORT_FROM:
             return 1;
 
+        /* Jumps */
         case JUMP_FORWARD:
-        case JUMP_IF_TRUE_OR_POP:  /* -1 if jump not taken */
-        case JUMP_IF_FALSE_OR_POP:  /*  "" */
         case JUMP_ABSOLUTE:
             return 0;
 
+        case JUMP_IF_TRUE_OR_POP:
+        case JUMP_IF_FALSE_OR_POP:
+            return jump ? 0 : -1;
+
         case POP_JUMP_IF_FALSE:
         case POP_JUMP_IF_TRUE:
             return -1;
@@ -1021,8 +1045,10 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
             return 0;
         case SETUP_EXCEPT:
         case SETUP_FINALLY:
-            return 6; /* can push 3 values for the new exception
-                + 3 others for the previous exception state */
+            /* 0 in the normal flow.
+             * Restore the stack position and push 6 values before jumping to
+             * the handler if an exception be raised. */
+            return jump ? 6 : 0;
 
         case LOAD_FAST:
             return 1;
@@ -1035,6 +1061,8 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
 
         case RAISE_VARARGS:
             return -oparg;
+
+        /* Functions and calls */
         case CALL_FUNCTION:
             return -oparg;
         case CALL_METHOD:
@@ -1052,6 +1080,7 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
             else
                 return -1;
 
+        /* Closures */
         case LOAD_CLOSURE:
             return 1;
         case LOAD_DEREF:
@@ -1061,10 +1090,16 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
             return -1;
         case DELETE_DEREF:
             return 0;
+
+        /* Iterators and generators */
         case GET_AWAITABLE:
             return 0;
         case SETUP_ASYNC_WITH:
-            return 6;
+            /* 0 in the normal flow.
+             * Restore the stack position to the position before the result
+             * of __aenter__ and push 6 values before jumping to the handler
+             * if an exception be raised. */
+            return jump ? -1 + 6 : 0;
         case BEFORE_ASYNC_WITH:
             return 1;
         case GET_AITER:
@@ -1085,6 +1120,12 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
     return PY_INVALID_STACK_EFFECT; /* not reachable */
 }
 
+int
+PyCompile_OpcodeStackEffect(int opcode, int oparg)
+{
+    return stack_effect(opcode, oparg, -1);
+}
+
 /* Add an opcode with no argument.
    Returns 0 on failure, 1 on success.
 */
@@ -2329,6 +2370,7 @@ compiler_async_for(struct compiler *c, stmt_ty s)
     ADDOP(c, POP_TOP);
     ADDOP(c, POP_TOP);
     ADDOP(c, POP_EXCEPT); /* for SETUP_EXCEPT */
+    ADDOP(c, POP_TOP); /* for correct calculation of stack effect */
     ADDOP(c, POP_BLOCK); /* for SETUP_LOOP */
     ADDOP_JABS(c, JUMP_ABSOLUTE, after_loop_else);
 
@@ -2611,7 +2653,6 @@ compiler_try_except(struct compiler *c, stmt_ty s)
             /* second # body */
             VISIT_SEQ(c, stmt, handler->v.ExceptHandler.body);
             ADDOP(c, POP_BLOCK);
-            ADDOP(c, POP_EXCEPT);
             compiler_pop_fblock(c, FINALLY_TRY, cleanup_body);
 
             /* finally: */
@@ -2628,6 +2669,7 @@ compiler_try_except(struct compiler *c, stmt_ty s)
             compiler_nameop(c, handler->v.ExceptHandler.name, Del);
 
             ADDOP(c, END_FINALLY);
+            ADDOP(c, POP_EXCEPT);
             compiler_pop_fblock(c, FINALLY_END, cleanup_end);
         }
         else {
@@ -4899,44 +4941,55 @@ dfs(struct compiler *c, basicblock *b, struct assembler *a)
 static int
 stackdepth_walk(struct compiler *c, basicblock *b, int depth, int maxdepth)
 {
-    int i, target_depth, effect;
+    int i, new_depth, target_depth, effect;
     struct instr *instr;
-    if (b->b_seen || b->b_startdepth >= depth)
+    assert(!b->b_seen || b->b_startdepth == depth);
+    if (b->b_seen || b->b_startdepth >= depth) {
         return maxdepth;
+    }
+    /* Guard against infinite recursion */
     b->b_seen = 1;
     b->b_startdepth = depth;
     for (i = 0; i < b->b_iused; i++) {
         instr = &b->b_instr[i];
-        effect = PyCompile_OpcodeStackEffect(instr->i_opcode, instr->i_oparg);
+        effect = stack_effect(instr->i_opcode, instr->i_oparg, 0);
         if (effect == PY_INVALID_STACK_EFFECT) {
             fprintf(stderr, "opcode = %d\n", instr->i_opcode);
             Py_FatalError("PyCompile_OpcodeStackEffect()");
         }
-        depth += effect;
-
-        if (depth > maxdepth)
-            maxdepth = depth;
-        assert(depth >= 0); /* invalid code or bug in stackdepth() */
+        new_depth = depth + effect;
+        if (new_depth > maxdepth) {
+            maxdepth = new_depth;
+        }
+        assert(new_depth >= 0); /* invalid code or bug in stackdepth() */
         if (instr->i_jrel || instr->i_jabs) {
-            target_depth = depth;
-            if (instr->i_opcode == FOR_ITER) {
-                target_depth = depth-2;
+            /* Recursively inspect jump target */
+            effect = stack_effect(instr->i_opcode, instr->i_oparg, 1);
+            assert(effect != PY_INVALID_STACK_EFFECT);
+            target_depth = depth + effect;
+            if (target_depth > maxdepth) {
+                maxdepth = target_depth;
             }
-            else if (instr->i_opcode == SETUP_FINALLY ||
-                     instr->i_opcode == SETUP_EXCEPT) {
-                target_depth = depth+3;
-                if (target_depth > maxdepth)
-                    maxdepth = target_depth;
+            assert(target_depth >= 0); /* invalid code or bug in stackdepth() */
+            if (instr->i_opcode == CONTINUE_LOOP) {
+                /* Pops a variable number of values from the stack,
+                 * but the target should be already proceeding.
+                 */
+                assert(instr->i_target->b_seen);
+                assert(instr->i_target->b_startdepth <= depth);
+                goto out; /* remaining code is dead */
             }
-            else if (instr->i_opcode == JUMP_IF_TRUE_OR_POP ||
-                     instr->i_opcode == JUMP_IF_FALSE_OR_POP)
-                depth = depth - 1;
             maxdepth = stackdepth_walk(c, instr->i_target,
                                        target_depth, maxdepth);
-            if (instr->i_opcode == JUMP_ABSOLUTE ||
-                instr->i_opcode == JUMP_FORWARD) {
-                goto out; /* remaining code is dead */
-            }
+        }
+        depth = new_depth;
+        if (instr->i_opcode == JUMP_ABSOLUTE ||
+            instr->i_opcode == JUMP_FORWARD ||
+            instr->i_opcode == RETURN_VALUE ||
+            instr->i_opcode == RAISE_VARARGS ||
+            instr->i_opcode == BREAK_LOOP)
+        {
+            goto out; /* remaining code is dead */
         }
     }
     if (b->b_next)
diff --git a/Python/importlib.h b/Python/importlib.h
index a12c8424f63..0636fdf0b34 100644
--- a/Python/importlib.h
+++ b/Python/importlib.h
@@ -133,7 +133,7 @@ const unsigned char _Py_M__importlib[] = {
     1,10,1,8,1,4,1,6,1,8,1,122,24,95,77,111,
     100,117,108,101,76,111,99,107,46,104,97,115,95,100,101,97,
     100,108,111,99,107,99,1,0,0,0,0,0,0,0,2,0,
-    0,0,16,0,0,0,67,0,0,0,115,168,0,0,0,116,
+    0,0,9,0,0,0,67,0,0,0,115,168,0,0,0,116,
     0,160,1,161,0,125,1,124,0,116,2,124,1,60,0,122,
     138,120,132,124,0,106,3,143,96,1,0,124,0,106,4,100,
     1,107,2,115,48,124,0,106,5,124,1,107,2,114,72,124,
@@ -169,7 +169,7 @@ const unsigned char _Py_M__importlib[] = {
     1,2,1,8,1,20,1,6,1,14,1,4,1,8,1,12,
     1,12,1,24,2,10,1,18,2,122,19,95,77,111,100,117,
     108,101,76,111,99,107,46,97,99,113,117,105,114,101,99,1,
-    0,0,0,0,0,0,0,2,0,0,0,10,0,0,0,67,
+    0,0,0,0,0,0,0,2,0,0,0,9,0,0,0,67,
     0,0,0,115,122,0,0,0,116,0,160,1,161,0,125,1,
     124,0,106,2,143,98,1,0,124,0,106,3,124,1,107,3,
     114,34,116,4,100,1,131,1,130,1,124,0,106,5,100,2,
@@ -300,7 +300,7 @@ const unsigned char _Py_M__importlib[] = {
     0,114,10,0,0,0,114,11,0,0,0,114,42,0,0,0,
     141,0,0,0,115,6,0,0,0,8,2,8,4,8,4,114,
     42,0,0,0,99,1,0,0,0,0,0,0,0,3,0,0,
-    0,18,0,0,0,67,0,0,0,115,130,0,0,0,116,0,
+    0,8,0,0,0,67,0,0,0,115,130,0,0,0,116,0,
     160,1,161,0,1,0,122,106,121,14,116,2,124,0,25,0,
     131,0,125,1,87,0,110,24,4,0,116,3,107,10,114,48,
     1,0,1,0,1,0,100,1,125,1,89,0,110,2,88,0,
@@ -318,7 +318,7 @@ const unsigned char _Py_M__importlib[] = {
     108,111,98,97,108,32,105,109,112,111,114,116,32,108,111,99,
     107,32,116,111,32,112,114,111,116,101,99,116,10,32,32,32,
     32,95,109,111,100,117,108,101,95,108,111,99,107,115,46,78,
-    99,2,0,0,0,0,0,0,0,2,0,0,0,11,0,0,
+    99,2,0,0,0,0,0,0,0,2,0,0,0,8,0,0,
     0,83,0,0,0,115,48,0,0,0,116,0,160,1,161,0,
     1,0,122,24,116,2,160,3,124,1,161,1,124,0,107,8,
     114,30,116,2,124,1,61,0,87,0,100,0,116,0,160,4,
@@ -340,7 +340,7 @@ const unsigned char _Py_M__importlib[] = {
     0,0,0,114,45,0,0,0,157,0,0,0,115,28,0,0,
     0,0,6,8,1,2,1,2,1,14,1,14,1,10,2,8,
     1,8,1,10,2,8,2,12,11,20,2,10,2,114,45,0,
-    0,0,99,1,0,0,0,0,0,0,0,2,0,0,0,11,
+    0,0,99,1,0,0,0,0,0,0,0,2,0,0,0,8,
     0,0,0,67,0,0,0,115,54,0,0,0,116,0,124,0,
     131,1,125,1,121,12,124,1,160,1,161,0,1,0,87,0,
     110,20,4,0,116,2,107,10,114,40,1,0,1,0,1,0,
@@ -488,7 +488,7 @@ const unsigned char _Py_M__importlib[] = {
     0,0,0,218,17,95,108,111,97,100,95,109,111,100,117,108,
     101,95,115,104,105,109,253,0,0,0,115,12,0,0,0,0,
     6,10,1,10,1,10,1,10,1,10,2,114,84,0,0,0,
-    99,1,0,0,0,0,0,0,0,5,0,0,0,36,0,0,
+    99,1,0,0,0,0,0,0,0,5,0,0,0,8,0,0,
     0,67,0,0,0,115,216,0,0,0,116,0,124,0,100,1,
     100,0,131,3,125,1,116,1,124,1,100,2,131,2,114,54,
     121,10,124,1,160,2,124,0,161,1,83,0,4,0,116,3,
@@ -548,7 +548,7 @@ const unsigned char _Py_M__importlib[] = {
     0,0,115,4,0,0,0,0,4,8,1,122,27,95,105,110,
     115,116,97,108,108,101,100,95,115,97,102,101,108,121,46,95,
     95,101,110,116,101,114,95,95,99,1,0,0,0,0,0,0,
-    0,3,0,0,0,17,0,0,0,71,0,0,0,115,98,0,
+    0,3,0,0,0,8,0,0,0,71,0,0,0,115,98,0,
     0,0,122,82,124,0,106,0,125,2,116,1,100,1,100,2,
     132,0,124,1,68,0,131,1,131,1,114,64,121,14,116,2,
     106,3,124,2,106,4,61,0,87,0,113,80,4,0,116,5,
@@ -723,7 +723,7 @@ const unsigned char _Py_M__importlib[] = {
     0,0,0,0,1,10,1,14,1,10,1,18,1,10,1,8,
     1,10,1,122,19,77,111,100,117,108,101,83,112,101,99,46,
     95,95,114,101,112,114,95,95,99,2,0,0,0,0,0,0,
-    0,3,0,0,0,11,0,0,0,67,0,0,0,115,102,0,
+    0,3,0,0,0,8,0,0,0,67,0,0,0,115,102,0,
     0,0,124,0,106,0,125,2,121,70,124,0,106,1,124,1,
     106,1,107,2,111,76,124,0,106,2,124,1,106,2,107,2,
     111,76,124,0,106,3,124,1,106,3,107,2,111,76,124,2,
@@ -793,7 +793,7 @@ const unsigned char _Py_M__importlib[] = {
     0,0,0,76,1,0,0,115,18,0,0,0,12,37,4,1,
     14,11,8,10,8,12,12,9,14,4,12,8,12,4,114,102,
     0,0,0,41,2,114,103,0,0,0,114,105,0,0,0,99,
-    2,0,0,0,2,0,0,0,6,0,0,0,14,0,0,0,
+    2,0,0,0,2,0,0,0,6,0,0,0,8,0,0,0,
     67,0,0,0,115,154,0,0,0,116,0,124,1,100,1,131,
     2,114,74,116,1,100,2,107,8,114,22,116,2,130,1,116,
     1,106,3,125,4,124,3,100,2,107,8,114,48,124,4,124,
@@ -821,7 +821,7 @@ const unsigned char _Py_M__importlib[] = {
     177,1,0,0,115,34,0,0,0,0,2,10,1,8,1,4,
     1,6,2,8,1,12,1,12,1,6,1,8,2,8,1,10,
     1,2,1,14,1,14,1,12,3,4,2,114,78,0,0,0,
-    99,3,0,0,0,0,0,0,0,8,0,0,0,53,0,0,
+    99,3,0,0,0,0,0,0,0,8,0,0,0,8,0,0,
     0,67,0,0,0,115,56,1,0,0,121,10,124,0,106,0,
     125,3,87,0,110,20,4,0,116,1,107,10,114,30,1,0,
     1,0,1,0,89,0,110,14,88,0,124,3,100,0,107,9,
@@ -860,7 +860,7 @@ const unsigned char _Py_M__importlib[] = {
     14,1,10,1,2,1,14,1,16,1,10,2,14,1,20,1,
     6,1,6,1,114,128,0,0,0,70,41,1,218,8,111,118,
     101,114,114,105,100,101,99,2,0,0,0,1,0,0,0,5,
-    0,0,0,59,0,0,0,67,0,0,0,115,212,1,0,0,
+    0,0,0,8,0,0,0,67,0,0,0,115,212,1,0,0,
     124,2,115,20,116,0,124,1,100,1,100,0,131,3,100,0,
     107,8,114,54,121,12,124,0,106,1,124,1,95,2,87,0,
     110,20,4,0,116,3,107,10,114,52,1,0,1,0,1,0,
@@ -956,7 +956,7 @@ const unsigned char _Py_M__importlib[] = {
     0,0,0,114,10,0,0,0,114,11,0,0,0,114,91,0,
     0,0,69,2,0,0,115,16,0,0,0,0,3,20,1,10,
     1,10,1,10,2,16,2,6,1,14,2,114,91,0,0,0,
-    99,2,0,0,0,0,0,0,0,4,0,0,0,12,0,0,
+    99,2,0,0,0,0,0,0,0,4,0,0,0,9,0,0,
     0,67,0,0,0,115,178,0,0,0,124,0,106,0,125,2,
     116,1,124,2,131,1,143,148,1,0,116,2,106,3,160,4,
     124,2,161,1,124,1,107,9,114,54,100,1,160,5,124,2,
@@ -988,7 +988,7 @@ const unsigned char _Py_M__importlib[] = {
     86,2,0,0,115,30,0,0,0,0,2,6,1,10,1,16,
     1,10,1,12,1,10,1,10,1,14,2,14,1,4,1,14,
     1,12,4,14,2,22,1,114,80,0,0,0,99,1,0,0,
-    0,0,0,0,0,2,0,0,0,27,0,0,0,67,0,0,
+    0,0,0,0,0,2,0,0,0,8,0,0,0,67,0,0,
     0,115,206,0,0,0,124,0,106,0,160,1,124,0,106,2,
     161,1,1,0,116,3,106,4,124,0,106,2,25,0,125,1,
     116,5,124,1,100,1,100,0,131,3,100,0,107,8,114,76,
@@ -1016,7 +1016,7 @@ const unsigned char _Py_M__importlib[] = {
     1,12,1,14,1,6,1,16,1,2,4,8,1,10,1,22,
     1,14,1,6,1,16,1,2,1,10,1,14,1,6,1,114,
     139,0,0,0,99,1,0,0,0,0,0,0,0,2,0,0,
-    0,11,0,0,0,67,0,0,0,115,118,0,0,0,124,0,
+    0,9,0,0,0,67,0,0,0,115,118,0,0,0,124,0,
     106,0,100,0,107,9,114,30,116,1,124,0,106,0,100,1,
     131,2,115,30,116,2,124,0,131,1,83,0,116,3,124,0,
     131,1,125,1,116,4,124,1,131,1,143,54,1,0,124,0,
@@ -1406,7 +1406,7 @@ const unsigned char _Py_M__importlib[] = {
     0,114,11,0,0,0,218,17,95,102,105,110,100,95,115,112,
     101,99,95,108,101,103,97,99,121,93,3,0,0,115,8,0,
     0,0,0,3,12,1,8,1,4,1,114,164,0,0,0,99,
-    3,0,0,0,0,0,0,0,10,0,0,0,27,0,0,0,
+    3,0,0,0,0,0,0,0,10,0,0,0,10,0,0,0,
     67,0,0,0,115,240,0,0,0,116,0,106,1,125,3,124,
     3,100,1,107,8,114,22,116,2,100,2,131,1,130,1,124,
     3,115,38,116,3,160,4,100,3,116,5,161,2,1,0,124,
@@ -1475,7 +1475,7 @@ const unsigned char _Py_M__importlib[] = {
     1,8,1,10,1,10,1,4,1,8,2,12,1,114,173,0,
     0,0,122,16,78,111,32,109,111,100,117,108,101,32,110,97,
     109,101,100,32,122,4,123,33,114,125,99,2,0,0,0,0,
-    0,0,0,8,0,0,0,13,0,0,0,67,0,0,0,115,
+    0,0,0,8,0,0,0,8,0,0,0,67,0,0,0,115,
     220,0,0,0,100,0,125,2,124,0,160,0,100,1,161,1,
     100,2,25,0,125,3,124,3,114,134,124,3,116,1,106,2,
     107,7,114,42,116,3,124,1,124,3,131,2,1,0,124,0,
@@ -1508,7 +1508,7 @@ const unsigned char _Py_M__importlib[] = {
     4,1,14,1,4,1,10,1,10,2,10,1,10,1,10,1,
     2,1,10,1,14,1,16,1,20,1,10,1,8,1,20,2,
     8,1,4,2,10,1,22,1,114,178,0,0,0,99,2,0,
-    0,0,0,0,0,0,4,0,0,0,11,0,0,0,67,0,
+    0,0,0,0,0,0,4,0,0,0,9,0,0,0,67,0,
     0,0,115,94,0,0,0,116,0,124,0,131,1,143,38,1,
     0,116,1,106,2,160,3,124,0,116,4,161,2,125,2,124,
     2,116,4,107,8,114,42,116,5,124,0,124,1,131,2,83,
@@ -1560,7 +1560,7 @@ const unsigned char _Py_M__importlib[] = {
     114,10,0,0,0,114,11,0,0,0,114,181,0,0,0,214,
     3,0,0,115,8,0,0,0,0,9,12,1,8,1,12,1,
     114,181,0,0,0,41,1,218,9,114,101,99,117,114,115,105,
-    118,101,99,3,0,0,0,1,0,0,0,8,0,0,0,17,
+    118,101,99,3,0,0,0,1,0,0,0,8,0,0,0,11,
     0,0,0,67,0,0,0,115,234,0,0,0,116,0,124,0,
     100,1,131,2,114,230,120,218,124,1,68,0,93,210,125,4,
     116,1,124,4,116,2,131,2,115,78,124,3,114,46,124,0,
@@ -1572,10 +1572,10 @@ const unsigned char _Py_M__importlib[] = {
     1,0,113,16,116,0,124,0,124,4,131,2,115,16,100,10,
     160,8,124,0,106,3,124,4,161,2,125,6,121,14,116,9,
     124,2,124,6,131,2,1,0,87,0,113,16,4,0,116,10,
-    107,10,114,224,1,0,125,7,1,0,122,38,124,7,106,11,
+    107,10,114,224,1,0,125,7,1,0,122,36,124,7,106,11,
     124,6,107,2,114,206,116,12,106,13,160,14,124,6,116,15,
-    161,2,100,11,107,9,114,206,119,16,130,0,87,0,89,0,
-    100,11,100,11,125,7,126,7,88,0,113,16,88,0,113,16,
+    161,2,100,11,107,9,114,206,119,16,130,0,87,0,100,11,
+    100,11,125,7,126,7,88,0,89,0,113,16,88,0,113,16,
     87,0,124,0,83,0,41,12,122,238,70,105,103,117,114,101,
     32,111,117,116,32,119,104,97,116,32,95,95,105,109,112,111,
     114,116,95,95,32,115,104,111,117,108,100,32,114,101,116,117,
diff --git a/Python/importlib_external.h b/Python/importlib_external.h
index ea6f75aa43a..902bef45e32 100644
--- a/Python/importlib_external.h
+++ b/Python/importlib_external.h
@@ -174,7 +174,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
     4,0,0,0,218,10,95,112,97,116,104,95,115,116,97,116,
     75,0,0,0,115,2,0,0,0,0,7,114,39,0,0,0,
-    99,2,0,0,0,0,0,0,0,3,0,0,0,11,0,0,
+    99,2,0,0,0,0,0,0,0,3,0,0,0,8,0,0,
     0,67,0,0,0,115,48,0,0,0,121,12,116,0,124,0,
     131,1,125,2,87,0,110,20,4,0,116,1,107,10,114,32,
     1,0,1,0,1,0,100,1,83,0,88,0,124,2,106,2,
@@ -208,7 +208,7 @@ const unsigned char _Py_M__importlib_external[] = {
     2,0,0,0,114,4,0,0,0,218,11,95,112,97,116,104,
     95,105,115,100,105,114,99,0,0,0,115,6,0,0,0,0,
     2,4,1,8,1,114,46,0,0,0,233,182,1,0,0,99,
-    3,0,0,0,0,0,0,0,6,0,0,0,17,0,0,0,
+    3,0,0,0,0,0,0,0,6,0,0,0,11,0,0,0,
     67,0,0,0,115,162,0,0,0,100,1,160,0,124,0,116,
     1,124,0,131,1,161,2,125,3,116,2,160,3,124,3,116,
     2,106,4,116,2,106,5,66,0,116,2,106,6,66,0,124,
@@ -426,7 +426,7 @@ const unsigned char _Py_M__importlib_external[] = {
     12,1,8,1,10,1,12,1,12,1,8,1,6,1,10,1,
     10,1,8,1,6,1,10,1,8,1,16,1,10,1,6,1,
     8,1,16,1,8,1,6,1,8,1,14,1,114,87,0,0,
-    0,99,1,0,0,0,0,0,0,0,5,0,0,0,12,0,
+    0,99,1,0,0,0,0,0,0,0,5,0,0,0,9,0,
     0,0,67,0,0,0,115,126,0,0,0,116,0,124,0,131,
     1,100,1,107,2,114,16,100,2,83,0,124,0,160,1,100,
     3,161,1,92,3,125,1,125,2,125,3,124,1,114,56,124,
@@ -459,7 +459,7 @@ const unsigned char _Py_M__importlib_external[] = {
     116,95,115,111,117,114,99,101,102,105,108,101,89,1,0,0,
     115,20,0,0,0,0,7,12,1,4,1,16,1,24,1,4,
     1,2,1,12,1,18,1,18,1,114,93,0,0,0,99,1,
-    0,0,0,0,0,0,0,1,0,0,0,11,0,0,0,67,
+    0,0,0,0,0,0,0,1,0,0,0,8,0,0,0,67,
     0,0,0,115,72,0,0,0,124,0,160,0,116,1,116,2,
     131,1,161,1,114,46,121,8,116,3,124,0,131,1,83,0,
     4,0,116,4,107,10,114,42,1,0,1,0,1,0,89,0,
@@ -472,7 +472,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,4,0,0,0,218,11,95,103,101,116,95,99,97,99,104,
     101,100,108,1,0,0,115,16,0,0,0,0,1,14,1,2,
     1,8,1,14,1,8,1,14,1,4,2,114,97,0,0,0,
-    99,1,0,0,0,0,0,0,0,2,0,0,0,11,0,0,
+    99,1,0,0,0,0,0,0,0,2,0,0,0,8,0,0,
     0,67,0,0,0,115,52,0,0,0,121,14,116,0,124,0,
     131,1,106,1,125,1,87,0,110,24,4,0,116,2,107,10,
     114,38,1,0,1,0,1,0,100,1,125,1,89,0,110,2,
@@ -486,7 +486,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,2,0,0,0,114,4,0,0,0,218,10,95,99,97,
     108,99,95,109,111,100,101,120,1,0,0,115,12,0,0,0,
     0,2,2,1,14,1,14,1,10,3,8,1,114,99,0,0,
-    0,99,1,0,0,0,0,0,0,0,3,0,0,0,11,0,
+    0,99,1,0,0,0,0,0,0,0,3,0,0,0,8,0,
     0,0,3,0,0,0,115,68,0,0,0,100,6,135,0,102,
     1,100,2,100,3,132,9,125,1,121,10,116,0,106,1,125,
     2,87,0,110,28,4,0,116,2,107,10,114,52,1,0,1,
@@ -834,7 +834,7 @@ const unsigned char _Py_M__importlib_external[] = {
     1,114,157,0,0,0,41,2,114,122,0,0,0,218,26,115,
     117,98,109,111,100,117,108,101,95,115,101,97,114,99,104,95,
     108,111,99,97,116,105,111,110,115,99,2,0,0,0,2,0,
-    0,0,9,0,0,0,19,0,0,0,67,0,0,0,115,18,
+    0,0,9,0,0,0,8,0,0,0,67,0,0,0,115,18,
     1,0,0,124,1,100,1,107,8,114,60,100,2,125,1,116,
     0,124,2,100,3,131,2,114,70,121,14,124,2,160,1,124,
     0,161,1,125,1,87,0,113,70,4,0,116,2,107,10,114,
@@ -917,7 +917,7 @@ const unsigned char _Py_M__importlib_external[] = {
     123,115,121,115,95,118,101,114,115,105,111,110,125,92,77,111,
     100,117,108,101,115,92,123,102,117,108,108,110,97,109,101,125,
     92,68,101,98,117,103,70,99,2,0,0,0,0,0,0,0,
-    2,0,0,0,11,0,0,0,67,0,0,0,115,50,0,0,
+    2,0,0,0,8,0,0,0,67,0,0,0,115,50,0,0,
     0,121,14,116,0,160,1,116,0,106,2,124,1,161,2,83,
     0,4,0,116,3,107,10,114,44,1,0,1,0,1,0,116,
     0,160,1,116,0,106,4,124,1,161,2,83,0,88,0,100,
@@ -932,7 +932,7 @@ const unsigned char _Py_M__importlib_external[] = {
     14,1,122,36,87,105,110,100,111,119,115,82,101,103,105,115,
     116,114,121,70,105,110,100,101,114,46,95,111,112,101,110,95,
     114,101,103,105,115,116,114,121,99,2,0,0,0,0,0,0,
-    0,6,0,0,0,17,0,0,0,67,0,0,0,115,112,0,
+    0,6,0,0,0,9,0,0,0,67,0,0,0,115,112,0,
     0,0,124,0,106,0,114,14,124,0,106,1,125,2,110,6,
     124,0,106,2,125,2,124,2,106,3,124,1,100,1,116,4,
     106,5,100,0,100,2,133,2,25,0,22,0,100,3,141,2,
@@ -959,7 +959,7 @@ const unsigned char _Py_M__importlib_external[] = {
     105,110,100,111,119,115,82,101,103,105,115,116,114,121,70,105,
     110,100,101,114,46,95,115,101,97,114,99,104,95,114,101,103,
     105,115,116,114,121,78,99,4,0,0,0,0,0,0,0,8,
-    0,0,0,14,0,0,0,67,0,0,0,115,120,0,0,0,
+    0,0,0,8,0,0,0,67,0,0,0,115,120,0,0,0,
     124,0,160,0,124,1,161,1,125,4,124,4,100,0,107,8,
     114,22,100,0,83,0,121,12,116,1,124,4,131,1,1,0,
     87,0,110,20,4,0,116,2,107,10,114,54,1,0,1,0,
@@ -1190,12 +1190,12 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,2,0,0,0,114,4,0,0,0,114,198,0,0,0,
     251,2,0,0,115,0,0,0,0,122,21,83,111,117,114,99,
     101,76,111,97,100,101,114,46,115,101,116,95,100,97,116,97,
-    99,2,0,0,0,0,0,0,0,5,0,0,0,16,0,0,
+    99,2,0,0,0,0,0,0,0,5,0,0,0,10,0,0,
     0,67,0,0,0,115,82,0,0,0,124,0,160,0,124,1,
     161,1,125,2,121,14,124,0,160,1,124,2,161,1,125,3,
     87,0,110,48,4,0,116,2,107,10,114,72,1,0,125,4,
-    1,0,122,20,116,3,100,1,124,1,100,2,141,2,124,4,
-    130,2,87,0,89,0,100,3,100,3,125,4,126,4,88,0,
+    1,0,122,18,116,3,100,1,124,1,100,2,141,2,124,4,
+    130,2,87,0,100,3,100,3,125,4,126,4,88,0,89,0,
     110,2,88,0,116,4,124,3,131,1,83,0,41,4,122,52,
     67,111,110,99,114,101,116,101,32,105,109,112,108,101,109,101,
     110,116,97,116,105,111,110,32,111,102,32,73,110,115,112,101,
@@ -1233,7 +1233,7 @@ const unsigned char _Py_M__importlib_external[] = {
     95,116,111,95,99,111,100,101,12,3,0,0,115,4,0,0,
     0,0,5,12,1,122,27,83,111,117,114,99,101,76,111,97,
     100,101,114,46,115,111,117,114,99,101,95,116,111,95,99,111,
-    100,101,99,2,0,0,0,0,0,0,0,15,0,0,0,43,
+    100,101,99,2,0,0,0,0,0,0,0,15,0,0,0,9,
     0,0,0,67,0,0,0,115,46,2,0,0,124,0,160,0,
     124,1,161,1,125,2,100,1,125,3,100,1,125,4,100,1,
     125,5,100,2,125,6,100,3,125,7,121,12,116,1,124,2,
@@ -1449,1104 +1449,1104 @@ const unsigned char _Py_M__importlib_external[] = {
     0,2,8,1,122,32,83,111,117,114,99,101,70,105,108,101,
     76,111,97,100,101,114,46,95,99,97,99,104,101,95,98,121,
     116,101,99,111,100,101,105,182,1,0,0,41,1,114,222,0,
-    0,0,99,3,0,0,0,1,0,0,0,9,0,0,0,18,
-    0,0,0,67,0,0,0,115,248,0,0,0,116,0,124,1,
+    0,0,99,3,0,0,0,1,0,0,0,9,0,0,0,11,
+    0,0,0,67,0,0,0,115,250,0,0,0,116,0,124,1,
     131,1,92,2,125,4,125,5,103,0,125,6,120,38,124,4,
     114,54,116,1,124,4,131,1,115,54,116,0,124,4,131,1,
     92,2,125,4,125,7,124,6,160,2,124,7,161,1,1,0,
-    113,18,87,0,120,108,116,3,124,6,131,1,68,0,93,96,
+    113,18,87,0,120,110,116,3,124,6,131,1,68,0,93,98,
     125,7,116,4,124,4,124,7,131,2,125,4,121,14,116,5,
     160,6,124,4,161,1,1,0,87,0,113,66,4,0,116,7,
     107,10,114,116,1,0,1,0,1,0,119,66,89,0,113,66,
-    4,0,116,8,107,10,114,160,1,0,125,8,1,0,122,18,
+    4,0,116,8,107,10,114,162,1,0,125,8,1,0,122,18,
     116,9,160,10,100,1,124,4,124,8,161,3,1,0,100,2,
-    83,0,100,2,125,8,126,8,88,0,113,66,88,0,113,66,
-    87,0,121,28,116,11,124,1,124,2,124,3,131,3,1,0,
-    116,9,160,10,100,3,124,1,161,2,1,0,87,0,110,48,
-    4,0,116,8,107,10,114,242,1,0,125,8,1,0,122,20,
-    116,9,160,10,100,1,124,1,124,8,161,3,1,0,87,0,
-    89,0,100,2,100,2,125,8,126,8,88,0,110,2,88,0,
-    100,2,83,0,41,4,122,27,87,114,105,116,101,32,98,121,
-    116,101,115,32,100,97,116,97,32,116,111,32,97,32,102,105,
-    108,101,46,122,27,99,111,117,108,100,32,110,111,116,32,99,
-    114,101,97,116,101,32,123,33,114,125,58,32,123,33,114,125,
-    78,122,12,99,114,101,97,116,101,100,32,123,33,114,125,41,
-    12,114,38,0,0,0,114,46,0,0,0,114,165,0,0,0,
-    114,33,0,0,0,114,28,0,0,0,114,1,0,0,0,90,
-    5,109,107,100,105,114,218,15,70,105,108,101,69,120,105,115,
-    116,115,69,114,114,111,114,114,40,0,0,0,114,116,0,0,
-    0,114,130,0,0,0,114,56,0,0,0,41,9,114,102,0,
-    0,0,114,35,0,0,0,114,54,0,0,0,114,222,0,0,
-    0,218,6,112,97,114,101,110,116,114,96,0,0,0,114,27,
-    0,0,0,114,23,0,0,0,114,201,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,114,198,0,0,
-    0,161,3,0,0,115,42,0,0,0,0,2,12,1,4,2,
-    14,1,12,1,14,2,14,1,10,1,2,1,14,1,14,2,
-    6,1,16,3,6,1,8,1,20,1,2,1,12,1,16,1,
-    16,2,8,1,122,25,83,111,117,114,99,101,70,105,108,101,
-    76,111,97,100,101,114,46,115,101,116,95,100,97,116,97,78,
-    41,7,114,107,0,0,0,114,106,0,0,0,114,108,0,0,
-    0,114,109,0,0,0,114,197,0,0,0,114,199,0,0,0,
-    114,198,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    2,0,0,0,114,4,0,0,0,114,220,0,0,0,147,3,
-    0,0,115,6,0,0,0,12,4,8,5,8,5,114,220,0,
-    0,0,99,0,0,0,0,0,0,0,0,0,0,0,0,2,
-    0,0,0,64,0,0,0,115,32,0,0,0,101,0,90,1,
-    100,0,90,2,100,1,90,3,100,2,100,3,132,0,90,4,
-    100,4,100,5,132,0,90,5,100,6,83,0,41,7,218,20,
-    83,111,117,114,99,101,108,101,115,115,70,105,108,101,76,111,
-    97,100,101,114,122,45,76,111,97,100,101,114,32,119,104,105,
-    99,104,32,104,97,110,100,108,101,115,32,115,111,117,114,99,
-    101,108,101,115,115,32,102,105,108,101,32,105,109,112,111,114,
-    116,115,46,99,2,0,0,0,0,0,0,0,5,0,0,0,
-    5,0,0,0,67,0,0,0,115,68,0,0,0,124,0,160,
-    0,124,1,161,1,125,2,124,0,160,1,124,2,161,1,125,
-    3,124,1,124,2,100,1,156,2,125,4,116,2,124,3,124,
-    1,124,4,131,3,1,0,116,3,116,4,124,3,131,1,100,
-    2,100,0,133,2,25,0,124,1,124,2,100,3,141,3,83,
-    0,41,4,78,41,2,114,100,0,0,0,114,35,0,0,0,
-    114,126,0,0,0,41,2,114,100,0,0,0,114,91,0,0,
-    0,41,5,114,159,0,0,0,114,200,0,0,0,114,133,0,
-    0,0,114,145,0,0,0,114,208,0,0,0,41,5,114,102,
-    0,0,0,114,121,0,0,0,114,35,0,0,0,114,54,0,
-    0,0,114,132,0,0,0,114,2,0,0,0,114,2,0,0,
-    0,114,4,0,0,0,114,188,0,0,0,196,3,0,0,115,
-    18,0,0,0,0,1,10,1,10,4,2,1,8,2,12,1,
-    2,1,14,1,2,1,122,29,83,111,117,114,99,101,108,101,
-    115,115,70,105,108,101,76,111,97,100,101,114,46,103,101,116,
+    83,0,100,2,125,8,126,8,88,0,89,0,113,66,88,0,
+    113,66,87,0,121,28,116,11,124,1,124,2,124,3,131,3,
+    1,0,116,9,160,10,100,3,124,1,161,2,1,0,87,0,
+    110,48,4,0,116,8,107,10,114,244,1,0,125,8,1,0,
+    122,18,116,9,160,10,100,1,124,1,124,8,161,3,1,0,
+    87,0,100,2,100,2,125,8,126,8,88,0,89,0,110,2,
+    88,0,100,2,83,0,41,4,122,27,87,114,105,116,101,32,
+    98,121,116,101,115,32,100,97,116,97,32,116,111,32,97,32,
+    102,105,108,101,46,122,27,99,111,117,108,100,32,110,111,116,
+    32,99,114,101,97,116,101,32,123,33,114,125,58,32,123,33,
+    114,125,78,122,12,99,114,101,97,116,101,100,32,123,33,114,
+    125,41,12,114,38,0,0,0,114,46,0,0,0,114,165,0,
+    0,0,114,33,0,0,0,114,28,0,0,0,114,1,0,0,
+    0,90,5,109,107,100,105,114,218,15,70,105,108,101,69,120,
+    105,115,116,115,69,114,114,111,114,114,40,0,0,0,114,116,
+    0,0,0,114,130,0,0,0,114,56,0,0,0,41,9,114,
+    102,0,0,0,114,35,0,0,0,114,54,0,0,0,114,222,
+    0,0,0,218,6,112,97,114,101,110,116,114,96,0,0,0,
+    114,27,0,0,0,114,23,0,0,0,114,201,0,0,0,114,
+    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,198,
+    0,0,0,161,3,0,0,115,42,0,0,0,0,2,12,1,
+    4,2,14,1,12,1,14,2,14,1,10,1,2,1,14,1,
+    14,2,6,1,16,3,6,1,8,1,22,1,2,1,12,1,
+    16,1,16,2,8,1,122,25,83,111,117,114,99,101,70,105,
+    108,101,76,111,97,100,101,114,46,115,101,116,95,100,97,116,
+    97,78,41,7,114,107,0,0,0,114,106,0,0,0,114,108,
+    0,0,0,114,109,0,0,0,114,197,0,0,0,114,199,0,
+    0,0,114,198,0,0,0,114,2,0,0,0,114,2,0,0,
+    0,114,2,0,0,0,114,4,0,0,0,114,220,0,0,0,
+    147,3,0,0,115,6,0,0,0,12,4,8,5,8,5,114,
+    220,0,0,0,99,0,0,0,0,0,0,0,0,0,0,0,
+    0,2,0,0,0,64,0,0,0,115,32,0,0,0,101,0,
+    90,1,100,0,90,2,100,1,90,3,100,2,100,3,132,0,
+    90,4,100,4,100,5,132,0,90,5,100,6,83,0,41,7,
+    218,20,83,111,117,114,99,101,108,101,115,115,70,105,108,101,
+    76,111,97,100,101,114,122,45,76,111,97,100,101,114,32,119,
+    104,105,99,104,32,104,97,110,100,108,101,115,32,115,111,117,
+    114,99,101,108,101,115,115,32,102,105,108,101,32,105,109,112,
+    111,114,116,115,46,99,2,0,0,0,0,0,0,0,5,0,
+    0,0,5,0,0,0,67,0,0,0,115,68,0,0,0,124,
+    0,160,0,124,1,161,1,125,2,124,0,160,1,124,2,161,
+    1,125,3,124,1,124,2,100,1,156,2,125,4,116,2,124,
+    3,124,1,124,4,131,3,1,0,116,3,116,4,124,3,131,
+    1,100,2,100,0,133,2,25,0,124,1,124,2,100,3,141,
+    3,83,0,41,4,78,41,2,114,100,0,0,0,114,35,0,
+    0,0,114,126,0,0,0,41,2,114,100,0,0,0,114,91,
+    0,0,0,41,5,114,159,0,0,0,114,200,0,0,0,114,
+    133,0,0,0,114,145,0,0,0,114,208,0,0,0,41,5,
+    114,102,0,0,0,114,121,0,0,0,114,35,0,0,0,114,
+    54,0,0,0,114,132,0,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,114,188,0,0,0,196,3,0,
+    0,115,18,0,0,0,0,1,10,1,10,4,2,1,8,2,
+    12,1,2,1,14,1,2,1,122,29,83,111,117,114,99,101,
+    108,101,115,115,70,105,108,101,76,111,97,100,101,114,46,103,
+    101,116,95,99,111,100,101,99,2,0,0,0,0,0,0,0,
+    2,0,0,0,1,0,0,0,67,0,0,0,115,4,0,0,
+    0,100,1,83,0,41,2,122,39,82,101,116,117,114,110,32,
+    78,111,110,101,32,97,115,32,116,104,101,114,101,32,105,115,
+    32,110,111,32,115,111,117,114,99,101,32,99,111,100,101,46,
+    78,114,2,0,0,0,41,2,114,102,0,0,0,114,121,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,202,0,0,0,212,3,0,0,115,2,0,0,0,0,
+    2,122,31,83,111,117,114,99,101,108,101,115,115,70,105,108,
+    101,76,111,97,100,101,114,46,103,101,116,95,115,111,117,114,
+    99,101,78,41,6,114,107,0,0,0,114,106,0,0,0,114,
+    108,0,0,0,114,109,0,0,0,114,188,0,0,0,114,202,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,114,225,0,0,0,192,3,0,0,
+    115,4,0,0,0,12,4,8,16,114,225,0,0,0,99,0,
+    0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,
+    0,0,0,115,92,0,0,0,101,0,90,1,100,0,90,2,
+    100,1,90,3,100,2,100,3,132,0,90,4,100,4,100,5,
+    132,0,90,5,100,6,100,7,132,0,90,6,100,8,100,9,
+    132,0,90,7,100,10,100,11,132,0,90,8,100,12,100,13,
+    132,0,90,9,100,14,100,15,132,0,90,10,100,16,100,17,
+    132,0,90,11,101,12,100,18,100,19,132,0,131,1,90,13,
+    100,20,83,0,41,21,218,19,69,120,116,101,110,115,105,111,
+    110,70,105,108,101,76,111,97,100,101,114,122,93,76,111,97,
+    100,101,114,32,102,111,114,32,101,120,116,101,110,115,105,111,
+    110,32,109,111,100,117,108,101,115,46,10,10,32,32,32,32,
+    84,104,101,32,99,111,110,115,116,114,117,99,116,111,114,32,
+    105,115,32,100,101,115,105,103,110,101,100,32,116,111,32,119,
+    111,114,107,32,119,105,116,104,32,70,105,108,101,70,105,110,
+    100,101,114,46,10,10,32,32,32,32,99,3,0,0,0,0,
+    0,0,0,3,0,0,0,2,0,0,0,67,0,0,0,115,
+    16,0,0,0,124,1,124,0,95,0,124,2,124,0,95,1,
+    100,0,83,0,41,1,78,41,2,114,100,0,0,0,114,35,
+    0,0,0,41,3,114,102,0,0,0,114,100,0,0,0,114,
+    35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,114,186,0,0,0,229,3,0,0,115,4,0,0,
+    0,0,1,6,1,122,28,69,120,116,101,110,115,105,111,110,
+    70,105,108,101,76,111,97,100,101,114,46,95,95,105,110,105,
+    116,95,95,99,2,0,0,0,0,0,0,0,2,0,0,0,
+    2,0,0,0,67,0,0,0,115,24,0,0,0,124,0,106,
+    0,124,1,106,0,107,2,111,22,124,0,106,1,124,1,106,
+    1,107,2,83,0,41,1,78,41,2,114,213,0,0,0,114,
+    113,0,0,0,41,2,114,102,0,0,0,114,214,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
+    215,0,0,0,233,3,0,0,115,4,0,0,0,0,1,12,
+    1,122,26,69,120,116,101,110,115,105,111,110,70,105,108,101,
+    76,111,97,100,101,114,46,95,95,101,113,95,95,99,1,0,
+    0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,0,
+    0,0,115,20,0,0,0,116,0,124,0,106,1,131,1,116,
+    0,124,0,106,2,131,1,65,0,83,0,41,1,78,41,3,
+    114,216,0,0,0,114,100,0,0,0,114,35,0,0,0,41,
+    1,114,102,0,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,114,217,0,0,0,237,3,0,0,115,2,
+    0,0,0,0,1,122,28,69,120,116,101,110,115,105,111,110,
+    70,105,108,101,76,111,97,100,101,114,46,95,95,104,97,115,
+    104,95,95,99,2,0,0,0,0,0,0,0,3,0,0,0,
+    5,0,0,0,67,0,0,0,115,36,0,0,0,116,0,160,
+    1,116,2,106,3,124,1,161,2,125,2,116,0,160,4,100,
+    1,124,1,106,5,124,0,106,6,161,3,1,0,124,2,83,
+    0,41,2,122,38,67,114,101,97,116,101,32,97,110,32,117,
+    110,105,116,105,97,108,105,122,101,100,32,101,120,116,101,110,
+    115,105,111,110,32,109,111,100,117,108,101,122,38,101,120,116,
+    101,110,115,105,111,110,32,109,111,100,117,108,101,32,123,33,
+    114,125,32,108,111,97,100,101,100,32,102,114,111,109,32,123,
+    33,114,125,41,7,114,116,0,0,0,114,189,0,0,0,114,
+    143,0,0,0,90,14,99,114,101,97,116,101,95,100,121,110,
+    97,109,105,99,114,130,0,0,0,114,100,0,0,0,114,35,
+    0,0,0,41,3,114,102,0,0,0,114,166,0,0,0,114,
+    191,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,114,187,0,0,0,240,3,0,0,115,10,0,0,
+    0,0,2,4,1,10,1,6,1,12,1,122,33,69,120,116,
+    101,110,115,105,111,110,70,105,108,101,76,111,97,100,101,114,
+    46,99,114,101,97,116,101,95,109,111,100,117,108,101,99,2,
+    0,0,0,0,0,0,0,2,0,0,0,5,0,0,0,67,
+    0,0,0,115,36,0,0,0,116,0,160,1,116,2,106,3,
+    124,1,161,2,1,0,116,0,160,4,100,1,124,0,106,5,
+    124,0,106,6,161,3,1,0,100,2,83,0,41,3,122,30,
+    73,110,105,116,105,97,108,105,122,101,32,97,110,32,101,120,
+    116,101,110,115,105,111,110,32,109,111,100,117,108,101,122,40,
+    101,120,116,101,110,115,105,111,110,32,109,111,100,117,108,101,
+    32,123,33,114,125,32,101,120,101,99,117,116,101,100,32,102,
+    114,111,109,32,123,33,114,125,78,41,7,114,116,0,0,0,
+    114,189,0,0,0,114,143,0,0,0,90,12,101,120,101,99,
+    95,100,121,110,97,109,105,99,114,130,0,0,0,114,100,0,
+    0,0,114,35,0,0,0,41,2,114,102,0,0,0,114,191,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,114,192,0,0,0,248,3,0,0,115,6,0,0,0,
+    0,2,14,1,6,1,122,31,69,120,116,101,110,115,105,111,
+    110,70,105,108,101,76,111,97,100,101,114,46,101,120,101,99,
+    95,109,111,100,117,108,101,99,2,0,0,0,0,0,0,0,
+    2,0,0,0,4,0,0,0,3,0,0,0,115,36,0,0,
+    0,116,0,124,0,106,1,131,1,100,1,25,0,137,0,116,
+    2,135,0,102,1,100,2,100,3,132,8,116,3,68,0,131,
+    1,131,1,83,0,41,4,122,49,82,101,116,117,114,110,32,
+    84,114,117,101,32,105,102,32,116,104,101,32,101,120,116,101,
+    110,115,105,111,110,32,109,111,100,117,108,101,32,105,115,32,
+    97,32,112,97,99,107,97,103,101,46,114,29,0,0,0,99,
+    1,0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,
+    51,0,0,0,115,26,0,0,0,124,0,93,18,125,1,136,
+    0,100,0,124,1,23,0,107,2,86,0,1,0,113,2,100,
+    1,83,0,41,2,114,186,0,0,0,78,114,2,0,0,0,
+    41,2,114,22,0,0,0,218,6,115,117,102,102,105,120,41,
+    1,218,9,102,105,108,101,95,110,97,109,101,114,2,0,0,
+    0,114,4,0,0,0,250,9,60,103,101,110,101,120,112,114,
+    62,1,4,0,0,115,2,0,0,0,4,1,122,49,69,120,
+    116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,101,
+    114,46,105,115,95,112,97,99,107,97,103,101,46,60,108,111,
+    99,97,108,115,62,46,60,103,101,110,101,120,112,114,62,41,
+    4,114,38,0,0,0,114,35,0,0,0,218,3,97,110,121,
+    218,18,69,88,84,69,78,83,73,79,78,95,83,85,70,70,
+    73,88,69,83,41,2,114,102,0,0,0,114,121,0,0,0,
+    114,2,0,0,0,41,1,114,228,0,0,0,114,4,0,0,
+    0,114,161,0,0,0,254,3,0,0,115,6,0,0,0,0,
+    2,14,1,12,1,122,30,69,120,116,101,110,115,105,111,110,
+    70,105,108,101,76,111,97,100,101,114,46,105,115,95,112,97,
+    99,107,97,103,101,99,2,0,0,0,0,0,0,0,2,0,
+    0,0,1,0,0,0,67,0,0,0,115,4,0,0,0,100,
+    1,83,0,41,2,122,63,82,101,116,117,114,110,32,78,111,
+    110,101,32,97,115,32,97,110,32,101,120,116,101,110,115,105,
+    111,110,32,109,111,100,117,108,101,32,99,97,110,110,111,116,
+    32,99,114,101,97,116,101,32,97,32,99,111,100,101,32,111,
+    98,106,101,99,116,46,78,114,2,0,0,0,41,2,114,102,
+    0,0,0,114,121,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,114,188,0,0,0,4,4,0,0,
+    115,2,0,0,0,0,2,122,28,69,120,116,101,110,115,105,
+    111,110,70,105,108,101,76,111,97,100,101,114,46,103,101,116,
     95,99,111,100,101,99,2,0,0,0,0,0,0,0,2,0,
     0,0,1,0,0,0,67,0,0,0,115,4,0,0,0,100,
-    1,83,0,41,2,122,39,82,101,116,117,114,110,32,78,111,
-    110,101,32,97,115,32,116,104,101,114,101,32,105,115,32,110,
-    111,32,115,111,117,114,99,101,32,99,111,100,101,46,78,114,
-    2,0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    202,0,0,0,212,3,0,0,115,2,0,0,0,0,2,122,
-    31,83,111,117,114,99,101,108,101,115,115,70,105,108,101,76,
-    111,97,100,101,114,46,103,101,116,95,115,111,117,114,99,101,
-    78,41,6,114,107,0,0,0,114,106,0,0,0,114,108,0,
-    0,0,114,109,0,0,0,114,188,0,0,0,114,202,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,225,0,0,0,192,3,0,0,115,4,
-    0,0,0,12,4,8,16,114,225,0,0,0,99,0,0,0,
-    0,0,0,0,0,0,0,0,0,3,0,0,0,64,0,0,
-    0,115,92,0,0,0,101,0,90,1,100,0,90,2,100,1,
-    90,3,100,2,100,3,132,0,90,4,100,4,100,5,132,0,
-    90,5,100,6,100,7,132,0,90,6,100,8,100,9,132,0,
-    90,7,100,10,100,11,132,0,90,8,100,12,100,13,132,0,
-    90,9,100,14,100,15,132,0,90,10,100,16,100,17,132,0,
-    90,11,101,12,100,18,100,19,132,0,131,1,90,13,100,20,
-    83,0,41,21,218,19,69,120,116,101,110,115,105,111,110,70,
-    105,108,101,76,111,97,100,101,114,122,93,76,111,97,100,101,
-    114,32,102,111,114,32,101,120,116,101,110,115,105,111,110,32,
-    109,111,100,117,108,101,115,46,10,10,32,32,32,32,84,104,
-    101,32,99,111,110,115,116,114,117,99,116,111,114,32,105,115,
-    32,100,101,115,105,103,110,101,100,32,116,111,32,119,111,114,
-    107,32,119,105,116,104,32,70,105,108,101,70,105,110,100,101,
-    114,46,10,10,32,32,32,32,99,3,0,0,0,0,0,0,
-    0,3,0,0,0,2,0,0,0,67,0,0,0,115,16,0,
-    0,0,124,1,124,0,95,0,124,2,124,0,95,1,100,0,
-    83,0,41,1,78,41,2,114,100,0,0,0,114,35,0,0,
-    0,41,3,114,102,0,0,0,114,100,0,0,0,114,35,0,
-    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,186,0,0,0,229,3,0,0,115,4,0,0,0,0,
-    1,6,1,122,28,69,120,116,101,110,115,105,111,110,70,105,
-    108,101,76,111,97,100,101,114,46,95,95,105,110,105,116,95,
-    95,99,2,0,0,0,0,0,0,0,2,0,0,0,2,0,
-    0,0,67,0,0,0,115,24,0,0,0,124,0,106,0,124,
-    1,106,0,107,2,111,22,124,0,106,1,124,1,106,1,107,
-    2,83,0,41,1,78,41,2,114,213,0,0,0,114,113,0,
-    0,0,41,2,114,102,0,0,0,114,214,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,215,0,
-    0,0,233,3,0,0,115,4,0,0,0,0,1,12,1,122,
-    26,69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,
-    97,100,101,114,46,95,95,101,113,95,95,99,1,0,0,0,
-    0,0,0,0,1,0,0,0,3,0,0,0,67,0,0,0,
-    115,20,0,0,0,116,0,124,0,106,1,131,1,116,0,124,
-    0,106,2,131,1,65,0,83,0,41,1,78,41,3,114,216,
-    0,0,0,114,100,0,0,0,114,35,0,0,0,41,1,114,
-    102,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,217,0,0,0,237,3,0,0,115,2,0,0,
-    0,0,1,122,28,69,120,116,101,110,115,105,111,110,70,105,
-    108,101,76,111,97,100,101,114,46,95,95,104,97,115,104,95,
-    95,99,2,0,0,0,0,0,0,0,3,0,0,0,5,0,
-    0,0,67,0,0,0,115,36,0,0,0,116,0,160,1,116,
-    2,106,3,124,1,161,2,125,2,116,0,160,4,100,1,124,
-    1,106,5,124,0,106,6,161,3,1,0,124,2,83,0,41,
-    2,122,38,67,114,101,97,116,101,32,97,110,32,117,110,105,
-    116,105,97,108,105,122,101,100,32,101,120,116,101,110,115,105,
-    111,110,32,109,111,100,117,108,101,122,38,101,120,116,101,110,
-    115,105,111,110,32,109,111,100,117,108,101,32,123,33,114,125,
-    32,108,111,97,100,101,100,32,102,114,111,109,32,123,33,114,
-    125,41,7,114,116,0,0,0,114,189,0,0,0,114,143,0,
-    0,0,90,14,99,114,101,97,116,101,95,100,121,110,97,109,
-    105,99,114,130,0,0,0,114,100,0,0,0,114,35,0,0,
-    0,41,3,114,102,0,0,0,114,166,0,0,0,114,191,0,
-    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,187,0,0,0,240,3,0,0,115,10,0,0,0,0,
-    2,4,1,10,1,6,1,12,1,122,33,69,120,116,101,110,
-    115,105,111,110,70,105,108,101,76,111,97,100,101,114,46,99,
-    114,101,97,116,101,95,109,111,100,117,108,101,99,2,0,0,
-    0,0,0,0,0,2,0,0,0,5,0,0,0,67,0,0,
-    0,115,36,0,0,0,116,0,160,1,116,2,106,3,124,1,
-    161,2,1,0,116,0,160,4,100,1,124,0,106,5,124,0,
-    106,6,161,3,1,0,100,2,83,0,41,3,122,30,73,110,
-    105,116,105,97,108,105,122,101,32,97,110,32,101,120,116,101,
-    110,115,105,111,110,32,109,111,100,117,108,101,122,40,101,120,
-    116,101,110,115,105,111,110,32,109,111,100,117,108,101,32,123,
-    33,114,125,32,101,120,101,99,117,116,101,100,32,102,114,111,
-    109,32,123,33,114,125,78,41,7,114,116,0,0,0,114,189,
-    0,0,0,114,143,0,0,0,90,12,101,120,101,99,95,100,
-    121,110,97,109,105,99,114,130,0,0,0,114,100,0,0,0,
-    114,35,0,0,0,41,2,114,102,0,0,0,114,191,0,0,
+    1,83,0,41,2,122,53,82,101,116,117,114,110,32,78,111,
+    110,101,32,97,115,32,101,120,116,101,110,115,105,111,110,32,
+    109,111,100,117,108,101,115,32,104,97,118,101,32,110,111,32,
+    115,111,117,114,99,101,32,99,111,100,101,46,78,114,2,0,
+    0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,114,202,0,
+    0,0,8,4,0,0,115,2,0,0,0,0,2,122,30,69,
+    120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,
+    101,114,46,103,101,116,95,115,111,117,114,99,101,99,2,0,
+    0,0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,
+    0,0,115,6,0,0,0,124,0,106,0,83,0,41,1,122,
+    58,82,101,116,117,114,110,32,116,104,101,32,112,97,116,104,
+    32,116,111,32,116,104,101,32,115,111,117,114,99,101,32,102,
+    105,108,101,32,97,115,32,102,111,117,110,100,32,98,121,32,
+    116,104,101,32,102,105,110,100,101,114,46,41,1,114,35,0,
+    0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,114,159,0,
+    0,0,12,4,0,0,115,2,0,0,0,0,3,122,32,69,
+    120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,
+    101,114,46,103,101,116,95,102,105,108,101,110,97,109,101,78,
+    41,14,114,107,0,0,0,114,106,0,0,0,114,108,0,0,
+    0,114,109,0,0,0,114,186,0,0,0,114,215,0,0,0,
+    114,217,0,0,0,114,187,0,0,0,114,192,0,0,0,114,
+    161,0,0,0,114,188,0,0,0,114,202,0,0,0,114,118,
+    0,0,0,114,159,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,2,0,0,0,114,4,0,0,0,114,226,0,0,
+    0,221,3,0,0,115,18,0,0,0,12,8,8,4,8,4,
+    8,3,8,8,8,6,8,6,8,4,8,4,114,226,0,0,
+    0,99,0,0,0,0,0,0,0,0,0,0,0,0,2,0,
+    0,0,64,0,0,0,115,96,0,0,0,101,0,90,1,100,
+    0,90,2,100,1,90,3,100,2,100,3,132,0,90,4,100,
+    4,100,5,132,0,90,5,100,6,100,7,132,0,90,6,100,
+    8,100,9,132,0,90,7,100,10,100,11,132,0,90,8,100,
+    12,100,13,132,0,90,9,100,14,100,15,132,0,90,10,100,
+    16,100,17,132,0,90,11,100,18,100,19,132,0,90,12,100,
+    20,100,21,132,0,90,13,100,22,83,0,41,23,218,14,95,
+    78,97,109,101,115,112,97,99,101,80,97,116,104,97,38,1,
+    0,0,82,101,112,114,101,115,101,110,116,115,32,97,32,110,
+    97,109,101,115,112,97,99,101,32,112,97,99,107,97,103,101,
+    39,115,32,112,97,116,104,46,32,32,73,116,32,117,115,101,
+    115,32,116,104,101,32,109,111,100,117,108,101,32,110,97,109,
+    101,10,32,32,32,32,116,111,32,102,105,110,100,32,105,116,
+    115,32,112,97,114,101,110,116,32,109,111,100,117,108,101,44,
+    32,97,110,100,32,102,114,111,109,32,116,104,101,114,101,32,
+    105,116,32,108,111,111,107,115,32,117,112,32,116,104,101,32,
+    112,97,114,101,110,116,39,115,10,32,32,32,32,95,95,112,
+    97,116,104,95,95,46,32,32,87,104,101,110,32,116,104,105,
+    115,32,99,104,97,110,103,101,115,44,32,116,104,101,32,109,
+    111,100,117,108,101,39,115,32,111,119,110,32,112,97,116,104,
+    32,105,115,32,114,101,99,111,109,112,117,116,101,100,44,10,
+    32,32,32,32,117,115,105,110,103,32,112,97,116,104,95,102,
+    105,110,100,101,114,46,32,32,70,111,114,32,116,111,112,45,
+    108,101,118,101,108,32,109,111,100,117,108,101,115,44,32,116,
+    104,101,32,112,97,114,101,110,116,32,109,111,100,117,108,101,
+    39,115,32,112,97,116,104,10,32,32,32,32,105,115,32,115,
+    121,115,46,112,97,116,104,46,99,4,0,0,0,0,0,0,
+    0,4,0,0,0,3,0,0,0,67,0,0,0,115,36,0,
+    0,0,124,1,124,0,95,0,124,2,124,0,95,1,116,2,
+    124,0,160,3,161,0,131,1,124,0,95,4,124,3,124,0,
+    95,5,100,0,83,0,41,1,78,41,6,218,5,95,110,97,
+    109,101,218,5,95,112,97,116,104,114,95,0,0,0,218,16,
+    95,103,101,116,95,112,97,114,101,110,116,95,112,97,116,104,
+    218,17,95,108,97,115,116,95,112,97,114,101,110,116,95,112,
+    97,116,104,218,12,95,112,97,116,104,95,102,105,110,100,101,
+    114,41,4,114,102,0,0,0,114,100,0,0,0,114,35,0,
+    0,0,218,11,112,97,116,104,95,102,105,110,100,101,114,114,
+    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,186,
+    0,0,0,25,4,0,0,115,8,0,0,0,0,1,6,1,
+    6,1,14,1,122,23,95,78,97,109,101,115,112,97,99,101,
+    80,97,116,104,46,95,95,105,110,105,116,95,95,99,1,0,
+    0,0,0,0,0,0,4,0,0,0,3,0,0,0,67,0,
+    0,0,115,38,0,0,0,124,0,106,0,160,1,100,1,161,
+    1,92,3,125,1,125,2,125,3,124,2,100,2,107,2,114,
+    30,100,3,83,0,124,1,100,4,102,2,83,0,41,5,122,
+    62,82,101,116,117,114,110,115,32,97,32,116,117,112,108,101,
+    32,111,102,32,40,112,97,114,101,110,116,45,109,111,100,117,
+    108,101,45,110,97,109,101,44,32,112,97,114,101,110,116,45,
+    112,97,116,104,45,97,116,116,114,45,110,97,109,101,41,114,
+    59,0,0,0,114,30,0,0,0,41,2,114,6,0,0,0,
+    114,35,0,0,0,90,8,95,95,112,97,116,104,95,95,41,
+    2,114,233,0,0,0,114,32,0,0,0,41,4,114,102,0,
+    0,0,114,224,0,0,0,218,3,100,111,116,90,2,109,101,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
+    23,95,102,105,110,100,95,112,97,114,101,110,116,95,112,97,
+    116,104,95,110,97,109,101,115,31,4,0,0,115,8,0,0,
+    0,0,2,18,1,8,2,4,3,122,38,95,78,97,109,101,
+    115,112,97,99,101,80,97,116,104,46,95,102,105,110,100,95,
+    112,97,114,101,110,116,95,112,97,116,104,95,110,97,109,101,
+    115,99,1,0,0,0,0,0,0,0,3,0,0,0,3,0,
+    0,0,67,0,0,0,115,28,0,0,0,124,0,160,0,161,
+    0,92,2,125,1,125,2,116,1,116,2,106,3,124,1,25,
+    0,124,2,131,2,83,0,41,1,78,41,4,114,240,0,0,
+    0,114,112,0,0,0,114,6,0,0,0,218,7,109,111,100,
+    117,108,101,115,41,3,114,102,0,0,0,90,18,112,97,114,
+    101,110,116,95,109,111,100,117,108,101,95,110,97,109,101,90,
+    14,112,97,116,104,95,97,116,116,114,95,110,97,109,101,114,
+    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,235,
+    0,0,0,41,4,0,0,115,4,0,0,0,0,1,12,1,
+    122,31,95,78,97,109,101,115,112,97,99,101,80,97,116,104,
+    46,95,103,101,116,95,112,97,114,101,110,116,95,112,97,116,
+    104,99,1,0,0,0,0,0,0,0,3,0,0,0,4,0,
+    0,0,67,0,0,0,115,80,0,0,0,116,0,124,0,160,
+    1,161,0,131,1,125,1,124,1,124,0,106,2,107,3,114,
+    74,124,0,160,3,124,0,106,4,124,1,161,2,125,2,124,
+    2,100,0,107,9,114,68,124,2,106,5,100,0,107,8,114,
+    68,124,2,106,6,114,68,124,2,106,6,124,0,95,7,124,
+    1,124,0,95,2,124,0,106,7,83,0,41,1,78,41,8,
+    114,95,0,0,0,114,235,0,0,0,114,236,0,0,0,114,
+    237,0,0,0,114,233,0,0,0,114,122,0,0,0,114,158,
+    0,0,0,114,234,0,0,0,41,3,114,102,0,0,0,90,
+    11,112,97,114,101,110,116,95,112,97,116,104,114,166,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,192,0,0,0,248,3,0,0,115,6,0,0,0,0,2,
-    14,1,6,1,122,31,69,120,116,101,110,115,105,111,110,70,
-    105,108,101,76,111,97,100,101,114,46,101,120,101,99,95,109,
-    111,100,117,108,101,99,2,0,0,0,0,0,0,0,2,0,
-    0,0,4,0,0,0,3,0,0,0,115,36,0,0,0,116,
-    0,124,0,106,1,131,1,100,1,25,0,137,0,116,2,135,
-    0,102,1,100,2,100,3,132,8,116,3,68,0,131,1,131,
-    1,83,0,41,4,122,49,82,101,116,117,114,110,32,84,114,
-    117,101,32,105,102,32,116,104,101,32,101,120,116,101,110,115,
-    105,111,110,32,109,111,100,117,108,101,32,105,115,32,97,32,
-    112,97,99,107,97,103,101,46,114,29,0,0,0,99,1,0,
-    0,0,0,0,0,0,2,0,0,0,4,0,0,0,51,0,
-    0,0,115,26,0,0,0,124,0,93,18,125,1,136,0,100,
-    0,124,1,23,0,107,2,86,0,1,0,113,2,100,1,83,
-    0,41,2,114,186,0,0,0,78,114,2,0,0,0,41,2,
-    114,22,0,0,0,218,6,115,117,102,102,105,120,41,1,218,
-    9,102,105,108,101,95,110,97,109,101,114,2,0,0,0,114,
-    4,0,0,0,250,9,60,103,101,110,101,120,112,114,62,1,
-    4,0,0,115,2,0,0,0,4,1,122,49,69,120,116,101,
-    110,115,105,111,110,70,105,108,101,76,111,97,100,101,114,46,
-    105,115,95,112,97,99,107,97,103,101,46,60,108,111,99,97,
-    108,115,62,46,60,103,101,110,101,120,112,114,62,41,4,114,
-    38,0,0,0,114,35,0,0,0,218,3,97,110,121,218,18,
-    69,88,84,69,78,83,73,79,78,95,83,85,70,70,73,88,
-    69,83,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
-    0,0,0,41,1,114,228,0,0,0,114,4,0,0,0,114,
-    161,0,0,0,254,3,0,0,115,6,0,0,0,0,2,14,
-    1,12,1,122,30,69,120,116,101,110,115,105,111,110,70,105,
-    108,101,76,111,97,100,101,114,46,105,115,95,112,97,99,107,
-    97,103,101,99,2,0,0,0,0,0,0,0,2,0,0,0,
-    1,0,0,0,67,0,0,0,115,4,0,0,0,100,1,83,
-    0,41,2,122,63,82,101,116,117,114,110,32,78,111,110,101,
-    32,97,115,32,97,110,32,101,120,116,101,110,115,105,111,110,
-    32,109,111,100,117,108,101,32,99,97,110,110,111,116,32,99,
-    114,101,97,116,101,32,97,32,99,111,100,101,32,111,98,106,
-    101,99,116,46,78,114,2,0,0,0,41,2,114,102,0,0,
-    0,114,121,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,188,0,0,0,4,4,0,0,115,2,
-    0,0,0,0,2,122,28,69,120,116,101,110,115,105,111,110,
-    70,105,108,101,76,111,97,100,101,114,46,103,101,116,95,99,
-    111,100,101,99,2,0,0,0,0,0,0,0,2,0,0,0,
-    1,0,0,0,67,0,0,0,115,4,0,0,0,100,1,83,
-    0,41,2,122,53,82,101,116,117,114,110,32,78,111,110,101,
-    32,97,115,32,101,120,116,101,110,115,105,111,110,32,109,111,
-    100,117,108,101,115,32,104,97,118,101,32,110,111,32,115,111,
-    117,114,99,101,32,99,111,100,101,46,78,114,2,0,0,0,
-    41,2,114,102,0,0,0,114,121,0,0,0,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,114,202,0,0,0,
-    8,4,0,0,115,2,0,0,0,0,2,122,30,69,120,116,
-    101,110,115,105,111,110,70,105,108,101,76,111,97,100,101,114,
-    46,103,101,116,95,115,111,117,114,99,101,99,2,0,0,0,
-    0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,
-    115,6,0,0,0,124,0,106,0,83,0,41,1,122,58,82,
-    101,116,117,114,110,32,116,104,101,32,112,97,116,104,32,116,
-    111,32,116,104,101,32,115,111,117,114,99,101,32,102,105,108,
-    101,32,97,115,32,102,111,117,110,100,32,98,121,32,116,104,
-    101,32,102,105,110,100,101,114,46,41,1,114,35,0,0,0,
-    41,2,114,102,0,0,0,114,121,0,0,0,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,114,159,0,0,0,
-    12,4,0,0,115,2,0,0,0,0,3,122,32,69,120,116,
-    101,110,115,105,111,110,70,105,108,101,76,111,97,100,101,114,
-    46,103,101,116,95,102,105,108,101,110,97,109,101,78,41,14,
-    114,107,0,0,0,114,106,0,0,0,114,108,0,0,0,114,
-    109,0,0,0,114,186,0,0,0,114,215,0,0,0,114,217,
-    0,0,0,114,187,0,0,0,114,192,0,0,0,114,161,0,
-    0,0,114,188,0,0,0,114,202,0,0,0,114,118,0,0,
-    0,114,159,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,2,0,0,0,114,4,0,0,0,114,226,0,0,0,221,
-    3,0,0,115,18,0,0,0,12,8,8,4,8,4,8,3,
-    8,8,8,6,8,6,8,4,8,4,114,226,0,0,0,99,
-    0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,
-    64,0,0,0,115,96,0,0,0,101,0,90,1,100,0,90,
-    2,100,1,90,3,100,2,100,3,132,0,90,4,100,4,100,
-    5,132,0,90,5,100,6,100,7,132,0,90,6,100,8,100,
-    9,132,0,90,7,100,10,100,11,132,0,90,8,100,12,100,
-    13,132,0,90,9,100,14,100,15,132,0,90,10,100,16,100,
-    17,132,0,90,11,100,18,100,19,132,0,90,12,100,20,100,
-    21,132,0,90,13,100,22,83,0,41,23,218,14,95,78,97,
-    109,101,115,112,97,99,101,80,97,116,104,97,38,1,0,0,
-    82,101,112,114,101,115,101,110,116,115,32,97,32,110,97,109,
-    101,115,112,97,99,101,32,112,97,99,107,97,103,101,39,115,
-    32,112,97,116,104,46,32,32,73,116,32,117,115,101,115,32,
-    116,104,101,32,109,111,100,117,108,101,32,110,97,109,101,10,
-    32,32,32,32,116,111,32,102,105,110,100,32,105,116,115,32,
-    112,97,114,101,110,116,32,109,111,100,117,108,101,44,32,97,
-    110,100,32,102,114,111,109,32,116,104,101,114,101,32,105,116,
-    32,108,111,111,107,115,32,117,112,32,116,104,101,32,112,97,
-    114,101,110,116,39,115,10,32,32,32,32,95,95,112,97,116,
-    104,95,95,46,32,32,87,104,101,110,32,116,104,105,115,32,
-    99,104,97,110,103,101,115,44,32,116,104,101,32,109,111,100,
-    117,108,101,39,115,32,111,119,110,32,112,97,116,104,32,105,
-    115,32,114,101,99,111,109,112,117,116,101,100,44,10,32,32,
-    32,32,117,115,105,110,103,32,112,97,116,104,95,102,105,110,
-    100,101,114,46,32,32,70,111,114,32,116,111,112,45,108,101,
-    118,101,108,32,109,111,100,117,108,101,115,44,32,116,104,101,
-    32,112,97,114,101,110,116,32,109,111,100,117,108,101,39,115,
-    32,112,97,116,104,10,32,32,32,32,105,115,32,115,121,115,
-    46,112,97,116,104,46,99,4,0,0,0,0,0,0,0,4,
-    0,0,0,3,0,0,0,67,0,0,0,115,36,0,0,0,
-    124,1,124,0,95,0,124,2,124,0,95,1,116,2,124,0,
-    160,3,161,0,131,1,124,0,95,4,124,3,124,0,95,5,
-    100,0,83,0,41,1,78,41,6,218,5,95,110,97,109,101,
-    218,5,95,112,97,116,104,114,95,0,0,0,218,16,95,103,
-    101,116,95,112,97,114,101,110,116,95,112,97,116,104,218,17,
-    95,108,97,115,116,95,112,97,114,101,110,116,95,112,97,116,
-    104,218,12,95,112,97,116,104,95,102,105,110,100,101,114,41,
-    4,114,102,0,0,0,114,100,0,0,0,114,35,0,0,0,
-    218,11,112,97,116,104,95,102,105,110,100,101,114,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,
-    0,25,4,0,0,115,8,0,0,0,0,1,6,1,6,1,
-    14,1,122,23,95,78,97,109,101,115,112,97,99,101,80,97,
-    116,104,46,95,95,105,110,105,116,95,95,99,1,0,0,0,
-    0,0,0,0,4,0,0,0,3,0,0,0,67,0,0,0,
-    115,38,0,0,0,124,0,106,0,160,1,100,1,161,1,92,
-    3,125,1,125,2,125,3,124,2,100,2,107,2,114,30,100,
-    3,83,0,124,1,100,4,102,2,83,0,41,5,122,62,82,
-    101,116,117,114,110,115,32,97,32,116,117,112,108,101,32,111,
-    102,32,40,112,97,114,101,110,116,45,109,111,100,117,108,101,
-    45,110,97,109,101,44,32,112,97,114,101,110,116,45,112,97,
-    116,104,45,97,116,116,114,45,110,97,109,101,41,114,59,0,
-    0,0,114,30,0,0,0,41,2,114,6,0,0,0,114,35,
-    0,0,0,90,8,95,95,112,97,116,104,95,95,41,2,114,
-    233,0,0,0,114,32,0,0,0,41,4,114,102,0,0,0,
-    114,224,0,0,0,218,3,100,111,116,90,2,109,101,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,218,23,95,
-    102,105,110,100,95,112,97,114,101,110,116,95,112,97,116,104,
-    95,110,97,109,101,115,31,4,0,0,115,8,0,0,0,0,
-    2,18,1,8,2,4,3,122,38,95,78,97,109,101,115,112,
-    97,99,101,80,97,116,104,46,95,102,105,110,100,95,112,97,
-    114,101,110,116,95,112,97,116,104,95,110,97,109,101,115,99,
-    1,0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,
-    67,0,0,0,115,28,0,0,0,124,0,160,0,161,0,92,
-    2,125,1,125,2,116,1,116,2,106,3,124,1,25,0,124,
-    2,131,2,83,0,41,1,78,41,4,114,240,0,0,0,114,
-    112,0,0,0,114,6,0,0,0,218,7,109,111,100,117,108,
-    101,115,41,3,114,102,0,0,0,90,18,112,97,114,101,110,
-    116,95,109,111,100,117,108,101,95,110,97,109,101,90,14,112,
-    97,116,104,95,97,116,116,114,95,110,97,109,101,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,114,235,0,0,
-    0,41,4,0,0,115,4,0,0,0,0,1,12,1,122,31,
-    95,78,97,109,101,115,112,97,99,101,80,97,116,104,46,95,
-    103,101,116,95,112,97,114,101,110,116,95,112,97,116,104,99,
-    1,0,0,0,0,0,0,0,3,0,0,0,4,0,0,0,
-    67,0,0,0,115,80,0,0,0,116,0,124,0,160,1,161,
-    0,131,1,125,1,124,1,124,0,106,2,107,3,114,74,124,
-    0,160,3,124,0,106,4,124,1,161,2,125,2,124,2,100,
-    0,107,9,114,68,124,2,106,5,100,0,107,8,114,68,124,
-    2,106,6,114,68,124,2,106,6,124,0,95,7,124,1,124,
-    0,95,2,124,0,106,7,83,0,41,1,78,41,8,114,95,
-    0,0,0,114,235,0,0,0,114,236,0,0,0,114,237,0,
-    0,0,114,233,0,0,0,114,122,0,0,0,114,158,0,0,
-    0,114,234,0,0,0,41,3,114,102,0,0,0,90,11,112,
-    97,114,101,110,116,95,112,97,116,104,114,166,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,12,
-    95,114,101,99,97,108,99,117,108,97,116,101,45,4,0,0,
-    115,16,0,0,0,0,2,12,1,10,1,14,3,18,1,6,
-    1,8,1,6,1,122,27,95,78,97,109,101,115,112,97,99,
-    101,80,97,116,104,46,95,114,101,99,97,108,99,117,108,97,
-    116,101,99,1,0,0,0,0,0,0,0,1,0,0,0,3,
-    0,0,0,67,0,0,0,115,12,0,0,0,116,0,124,0,
-    160,1,161,0,131,1,83,0,41,1,78,41,2,218,4,105,
-    116,101,114,114,242,0,0,0,41,1,114,102,0,0,0,114,
+    218,12,95,114,101,99,97,108,99,117,108,97,116,101,45,4,
+    0,0,115,16,0,0,0,0,2,12,1,10,1,14,3,18,
+    1,6,1,8,1,6,1,122,27,95,78,97,109,101,115,112,
+    97,99,101,80,97,116,104,46,95,114,101,99,97,108,99,117,
+    108,97,116,101,99,1,0,0,0,0,0,0,0,1,0,0,
+    0,3,0,0,0,67,0,0,0,115,12,0,0,0,116,0,
+    124,0,160,1,161,0,131,1,83,0,41,1,78,41,2,218,
+    4,105,116,101,114,114,242,0,0,0,41,1,114,102,0,0,
+    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    218,8,95,95,105,116,101,114,95,95,58,4,0,0,115,2,
+    0,0,0,0,1,122,23,95,78,97,109,101,115,112,97,99,
+    101,80,97,116,104,46,95,95,105,116,101,114,95,95,99,3,
+    0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,67,
+    0,0,0,115,14,0,0,0,124,2,124,0,106,0,124,1,
+    60,0,100,0,83,0,41,1,78,41,1,114,234,0,0,0,
+    41,3,114,102,0,0,0,218,5,105,110,100,101,120,114,35,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,218,11,95,95,115,101,116,105,116,101,109,95,95,61,
+    4,0,0,115,2,0,0,0,0,1,122,26,95,78,97,109,
+    101,115,112,97,99,101,80,97,116,104,46,95,95,115,101,116,
+    105,116,101,109,95,95,99,1,0,0,0,0,0,0,0,1,
+    0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,0,
+    116,0,124,0,160,1,161,0,131,1,83,0,41,1,78,41,
+    2,114,31,0,0,0,114,242,0,0,0,41,1,114,102,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,218,7,95,95,108,101,110,95,95,64,4,0,0,115,2,
+    0,0,0,0,1,122,22,95,78,97,109,101,115,112,97,99,
+    101,80,97,116,104,46,95,95,108,101,110,95,95,99,1,0,
+    0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,0,
+    0,0,115,12,0,0,0,100,1,160,0,124,0,106,1,161,
+    1,83,0,41,2,78,122,20,95,78,97,109,101,115,112,97,
+    99,101,80,97,116,104,40,123,33,114,125,41,41,2,114,48,
+    0,0,0,114,234,0,0,0,41,1,114,102,0,0,0,114,
     2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,8,
-    95,95,105,116,101,114,95,95,58,4,0,0,115,2,0,0,
+    95,95,114,101,112,114,95,95,67,4,0,0,115,2,0,0,
     0,0,1,122,23,95,78,97,109,101,115,112,97,99,101,80,
-    97,116,104,46,95,95,105,116,101,114,95,95,99,3,0,0,
-    0,0,0,0,0,3,0,0,0,3,0,0,0,67,0,0,
-    0,115,14,0,0,0,124,2,124,0,106,0,124,1,60,0,
-    100,0,83,0,41,1,78,41,1,114,234,0,0,0,41,3,
-    114,102,0,0,0,218,5,105,110,100,101,120,114,35,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,11,95,95,115,101,116,105,116,101,109,95,95,61,4,0,
-    0,115,2,0,0,0,0,1,122,26,95,78,97,109,101,115,
-    112,97,99,101,80,97,116,104,46,95,95,115,101,116,105,116,
-    101,109,95,95,99,1,0,0,0,0,0,0,0,1,0,0,
-    0,3,0,0,0,67,0,0,0,115,12,0,0,0,116,0,
-    124,0,160,1,161,0,131,1,83,0,41,1,78,41,2,114,
-    31,0,0,0,114,242,0,0,0,41,1,114,102,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    7,95,95,108,101,110,95,95,64,4,0,0,115,2,0,0,
-    0,0,1,122,22,95,78,97,109,101,115,112,97,99,101,80,
-    97,116,104,46,95,95,108,101,110,95,95,99,1,0,0,0,
-    0,0,0,0,1,0,0,0,3,0,0,0,67,0,0,0,
-    115,12,0,0,0,100,1,160,0,124,0,106,1,161,1,83,
-    0,41,2,78,122,20,95,78,97,109,101,115,112,97,99,101,
-    80,97,116,104,40,123,33,114,125,41,41,2,114,48,0,0,
-    0,114,234,0,0,0,41,1,114,102,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,218,8,95,95,
-    114,101,112,114,95,95,67,4,0,0,115,2,0,0,0,0,
-    1,122,23,95,78,97,109,101,115,112,97,99,101,80,97,116,
-    104,46,95,95,114,101,112,114,95,95,99,2,0,0,0,0,
-    0,0,0,2,0,0,0,3,0,0,0,67,0,0,0,115,
-    12,0,0,0,124,1,124,0,160,0,161,0,107,6,83,0,
-    41,1,78,41,1,114,242,0,0,0,41,2,114,102,0,0,
-    0,218,4,105,116,101,109,114,2,0,0,0,114,2,0,0,
-    0,114,4,0,0,0,218,12,95,95,99,111,110,116,97,105,
-    110,115,95,95,70,4,0,0,115,2,0,0,0,0,1,122,
-    27,95,78,97,109,101,115,112,97,99,101,80,97,116,104,46,
-    95,95,99,111,110,116,97,105,110,115,95,95,99,2,0,0,
+    97,116,104,46,95,95,114,101,112,114,95,95,99,2,0,0,
     0,0,0,0,0,2,0,0,0,3,0,0,0,67,0,0,
-    0,115,16,0,0,0,124,0,106,0,160,1,124,1,161,1,
-    1,0,100,0,83,0,41,1,78,41,2,114,234,0,0,0,
-    114,165,0,0,0,41,2,114,102,0,0,0,114,249,0,0,
+    0,115,12,0,0,0,124,1,124,0,160,0,161,0,107,6,
+    83,0,41,1,78,41,1,114,242,0,0,0,41,2,114,102,
+    0,0,0,218,4,105,116,101,109,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,218,12,95,95,99,111,110,116,
+    97,105,110,115,95,95,70,4,0,0,115,2,0,0,0,0,
+    1,122,27,95,78,97,109,101,115,112,97,99,101,80,97,116,
+    104,46,95,95,99,111,110,116,97,105,110,115,95,95,99,2,
+    0,0,0,0,0,0,0,2,0,0,0,3,0,0,0,67,
+    0,0,0,115,16,0,0,0,124,0,106,0,160,1,124,1,
+    161,1,1,0,100,0,83,0,41,1,78,41,2,114,234,0,
+    0,0,114,165,0,0,0,41,2,114,102,0,0,0,114,249,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,114,165,0,0,0,73,4,0,0,115,2,0,0,0,
+    0,1,122,21,95,78,97,109,101,115,112,97,99,101,80,97,
+    116,104,46,97,112,112,101,110,100,78,41,14,114,107,0,0,
+    0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,0,
+    114,186,0,0,0,114,240,0,0,0,114,235,0,0,0,114,
+    242,0,0,0,114,244,0,0,0,114,246,0,0,0,114,247,
+    0,0,0,114,248,0,0,0,114,250,0,0,0,114,165,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,
+    0,114,4,0,0,0,114,232,0,0,0,18,4,0,0,115,
+    20,0,0,0,12,7,8,6,8,10,8,4,8,13,8,3,
+    8,3,8,3,8,3,8,3,114,232,0,0,0,99,0,0,
+    0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,0,
+    0,0,115,80,0,0,0,101,0,90,1,100,0,90,2,100,
+    1,100,2,132,0,90,3,101,4,100,3,100,4,132,0,131,
+    1,90,5,100,5,100,6,132,0,90,6,100,7,100,8,132,
+    0,90,7,100,9,100,10,132,0,90,8,100,11,100,12,132,
+    0,90,9,100,13,100,14,132,0,90,10,100,15,100,16,132,
+    0,90,11,100,17,83,0,41,18,218,16,95,78,97,109,101,
+    115,112,97,99,101,76,111,97,100,101,114,99,4,0,0,0,
+    0,0,0,0,4,0,0,0,4,0,0,0,67,0,0,0,
+    115,18,0,0,0,116,0,124,1,124,2,124,3,131,3,124,
+    0,95,1,100,0,83,0,41,1,78,41,2,114,232,0,0,
+    0,114,234,0,0,0,41,4,114,102,0,0,0,114,100,0,
+    0,0,114,35,0,0,0,114,238,0,0,0,114,2,0,0,
+    0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,0,
+    79,4,0,0,115,2,0,0,0,0,1,122,25,95,78,97,
+    109,101,115,112,97,99,101,76,111,97,100,101,114,46,95,95,
+    105,110,105,116,95,95,99,2,0,0,0,0,0,0,0,2,
+    0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,0,
+    100,1,160,0,124,1,106,1,161,1,83,0,41,2,122,115,
+    82,101,116,117,114,110,32,114,101,112,114,32,102,111,114,32,
+    116,104,101,32,109,111,100,117,108,101,46,10,10,32,32,32,
+    32,32,32,32,32,84,104,101,32,109,101,116,104,111,100,32,
+    105,115,32,100,101,112,114,101,99,97,116,101,100,46,32,32,
+    84,104,101,32,105,109,112,111,114,116,32,109,97,99,104,105,
+    110,101,114,121,32,100,111,101,115,32,116,104,101,32,106,111,
+    98,32,105,116,115,101,108,102,46,10,10,32,32,32,32,32,
+    32,32,32,122,25,60,109,111,100,117,108,101,32,123,33,114,
+    125,32,40,110,97,109,101,115,112,97,99,101,41,62,41,2,
+    114,48,0,0,0,114,107,0,0,0,41,2,114,172,0,0,
+    0,114,191,0,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,218,11,109,111,100,117,108,101,95,114,101,
+    112,114,82,4,0,0,115,2,0,0,0,0,7,122,28,95,
+    78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,
+    109,111,100,117,108,101,95,114,101,112,114,99,2,0,0,0,
+    0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,
+    115,4,0,0,0,100,1,83,0,41,2,78,84,114,2,0,
+    0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,114,161,0,
+    0,0,91,4,0,0,115,2,0,0,0,0,1,122,27,95,
+    78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,
+    105,115,95,112,97,99,107,97,103,101,99,2,0,0,0,0,
+    0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,115,
+    4,0,0,0,100,1,83,0,41,2,78,114,30,0,0,0,
+    114,2,0,0,0,41,2,114,102,0,0,0,114,121,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,165,0,0,0,73,4,0,0,115,2,0,0,0,0,1,
-    122,21,95,78,97,109,101,115,112,97,99,101,80,97,116,104,
-    46,97,112,112,101,110,100,78,41,14,114,107,0,0,0,114,
-    106,0,0,0,114,108,0,0,0,114,109,0,0,0,114,186,
-    0,0,0,114,240,0,0,0,114,235,0,0,0,114,242,0,
-    0,0,114,244,0,0,0,114,246,0,0,0,114,247,0,0,
-    0,114,248,0,0,0,114,250,0,0,0,114,165,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,232,0,0,0,18,4,0,0,115,20,0,
-    0,0,12,7,8,6,8,10,8,4,8,13,8,3,8,3,
-    8,3,8,3,8,3,114,232,0,0,0,99,0,0,0,0,
-    0,0,0,0,0,0,0,0,3,0,0,0,64,0,0,0,
-    115,80,0,0,0,101,0,90,1,100,0,90,2,100,1,100,
-    2,132,0,90,3,101,4,100,3,100,4,132,0,131,1,90,
-    5,100,5,100,6,132,0,90,6,100,7,100,8,132,0,90,
-    7,100,9,100,10,132,0,90,8,100,11,100,12,132,0,90,
-    9,100,13,100,14,132,0,90,10,100,15,100,16,132,0,90,
-    11,100,17,83,0,41,18,218,16,95,78,97,109,101,115,112,
-    97,99,101,76,111,97,100,101,114,99,4,0,0,0,0,0,
-    0,0,4,0,0,0,4,0,0,0,67,0,0,0,115,18,
-    0,0,0,116,0,124,1,124,2,124,3,131,3,124,0,95,
-    1,100,0,83,0,41,1,78,41,2,114,232,0,0,0,114,
-    234,0,0,0,41,4,114,102,0,0,0,114,100,0,0,0,
-    114,35,0,0,0,114,238,0,0,0,114,2,0,0,0,114,
-    2,0,0,0,114,4,0,0,0,114,186,0,0,0,79,4,
-    0,0,115,2,0,0,0,0,1,122,25,95,78,97,109,101,
-    115,112,97,99,101,76,111,97,100,101,114,46,95,95,105,110,
-    105,116,95,95,99,2,0,0,0,0,0,0,0,2,0,0,
-    0,3,0,0,0,67,0,0,0,115,12,0,0,0,100,1,
-    160,0,124,1,106,1,161,1,83,0,41,2,122,115,82,101,
-    116,117,114,110,32,114,101,112,114,32,102,111,114,32,116,104,
-    101,32,109,111,100,117,108,101,46,10,10,32,32,32,32,32,
-    32,32,32,84,104,101,32,109,101,116,104,111,100,32,105,115,
-    32,100,101,112,114,101,99,97,116,101,100,46,32,32,84,104,
-    101,32,105,109,112,111,114,116,32,109,97,99,104,105,110,101,
-    114,121,32,100,111,101,115,32,116,104,101,32,106,111,98,32,
-    105,116,115,101,108,102,46,10,10,32,32,32,32,32,32,32,
-    32,122,25,60,109,111,100,117,108,101,32,123,33,114,125,32,
-    40,110,97,109,101,115,112,97,99,101,41,62,41,2,114,48,
-    0,0,0,114,107,0,0,0,41,2,114,172,0,0,0,114,
-    191,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,218,11,109,111,100,117,108,101,95,114,101,112,114,
-    82,4,0,0,115,2,0,0,0,0,7,122,28,95,78,97,
-    109,101,115,112,97,99,101,76,111,97,100,101,114,46,109,111,
-    100,117,108,101,95,114,101,112,114,99,2,0,0,0,0,0,
-    0,0,2,0,0,0,1,0,0,0,67,0,0,0,115,4,
-    0,0,0,100,1,83,0,41,2,78,84,114,2,0,0,0,
-    41,2,114,102,0,0,0,114,121,0,0,0,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,114,161,0,0,0,
-    91,4,0,0,115,2,0,0,0,0,1,122,27,95,78,97,
-    109,101,115,112,97,99,101,76,111,97,100,101,114,46,105,115,
-    95,112,97,99,107,97,103,101,99,2,0,0,0,0,0,0,
-    0,2,0,0,0,1,0,0,0,67,0,0,0,115,4,0,
-    0,0,100,1,83,0,41,2,78,114,30,0,0,0,114,2,
-    0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,202,
-    0,0,0,94,4,0,0,115,2,0,0,0,0,1,122,27,
+    114,202,0,0,0,94,4,0,0,115,2,0,0,0,0,1,
+    122,27,95,78,97,109,101,115,112,97,99,101,76,111,97,100,
+    101,114,46,103,101,116,95,115,111,117,114,99,101,99,2,0,
+    0,0,0,0,0,0,2,0,0,0,6,0,0,0,67,0,
+    0,0,115,16,0,0,0,116,0,100,1,100,2,100,3,100,
+    4,100,5,141,4,83,0,41,6,78,114,30,0,0,0,122,
+    8,60,115,116,114,105,110,103,62,114,190,0,0,0,84,41,
+    1,114,204,0,0,0,41,1,114,205,0,0,0,41,2,114,
+    102,0,0,0,114,121,0,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,114,188,0,0,0,97,4,0,
+    0,115,2,0,0,0,0,1,122,25,95,78,97,109,101,115,
+    112,97,99,101,76,111,97,100,101,114,46,103,101,116,95,99,
+    111,100,101,99,2,0,0,0,0,0,0,0,2,0,0,0,
+    1,0,0,0,67,0,0,0,115,4,0,0,0,100,1,83,
+    0,41,2,122,42,85,115,101,32,100,101,102,97,117,108,116,
+    32,115,101,109,97,110,116,105,99,115,32,102,111,114,32,109,
+    111,100,117,108,101,32,99,114,101,97,116,105,111,110,46,78,
+    114,2,0,0,0,41,2,114,102,0,0,0,114,166,0,0,
+    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,187,0,0,0,100,4,0,0,115,0,0,0,0,122,30,
     95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,
-    46,103,101,116,95,115,111,117,114,99,101,99,2,0,0,0,
-    0,0,0,0,2,0,0,0,6,0,0,0,67,0,0,0,
-    115,16,0,0,0,116,0,100,1,100,2,100,3,100,4,100,
-    5,141,4,83,0,41,6,78,114,30,0,0,0,122,8,60,
-    115,116,114,105,110,103,62,114,190,0,0,0,84,41,1,114,
-    204,0,0,0,41,1,114,205,0,0,0,41,2,114,102,0,
-    0,0,114,121,0,0,0,114,2,0,0,0,114,2,0,0,
-    0,114,4,0,0,0,114,188,0,0,0,97,4,0,0,115,
-    2,0,0,0,0,1,122,25,95,78,97,109,101,115,112,97,
-    99,101,76,111,97,100,101,114,46,103,101,116,95,99,111,100,
-    101,99,2,0,0,0,0,0,0,0,2,0,0,0,1,0,
-    0,0,67,0,0,0,115,4,0,0,0,100,1,83,0,41,
-    2,122,42,85,115,101,32,100,101,102,97,117,108,116,32,115,
-    101,109,97,110,116,105,99,115,32,102,111,114,32,109,111,100,
-    117,108,101,32,99,114,101,97,116,105,111,110,46,78,114,2,
-    0,0,0,41,2,114,102,0,0,0,114,166,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,187,
-    0,0,0,100,4,0,0,115,0,0,0,0,122,30,95,78,
-    97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,99,
-    114,101,97,116,101,95,109,111,100,117,108,101,99,2,0,0,
-    0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,
-    0,115,4,0,0,0,100,0,83,0,41,1,78,114,2,0,
-    0,0,41,2,114,102,0,0,0,114,191,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,192,0,
-    0,0,103,4,0,0,115,2,0,0,0,0,1,122,28,95,
-    78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,
-    101,120,101,99,95,109,111,100,117,108,101,99,2,0,0,0,
-    0,0,0,0,2,0,0,0,4,0,0,0,67,0,0,0,
-    115,26,0,0,0,116,0,160,1,100,1,124,0,106,2,161,
-    2,1,0,116,0,160,3,124,0,124,1,161,2,83,0,41,
-    2,122,98,76,111,97,100,32,97,32,110,97,109,101,115,112,
-    97,99,101,32,109,111,100,117,108,101,46,10,10,32,32,32,
-    32,32,32,32,32,84,104,105,115,32,109,101,116,104,111,100,
-    32,105,115,32,100,101,112,114,101,99,97,116,101,100,46,32,
-    32,85,115,101,32,101,120,101,99,95,109,111,100,117,108,101,
-    40,41,32,105,110,115,116,101,97,100,46,10,10,32,32,32,
-    32,32,32,32,32,122,38,110,97,109,101,115,112,97,99,101,
-    32,109,111,100,117,108,101,32,108,111,97,100,101,100,32,119,
-    105,116,104,32,112,97,116,104,32,123,33,114,125,41,4,114,
-    116,0,0,0,114,130,0,0,0,114,234,0,0,0,114,193,
-    0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,194,
-    0,0,0,106,4,0,0,115,6,0,0,0,0,7,6,1,
-    8,1,122,28,95,78,97,109,101,115,112,97,99,101,76,111,
-    97,100,101,114,46,108,111,97,100,95,109,111,100,117,108,101,
-    78,41,12,114,107,0,0,0,114,106,0,0,0,114,108,0,
-    0,0,114,186,0,0,0,114,184,0,0,0,114,252,0,0,
-    0,114,161,0,0,0,114,202,0,0,0,114,188,0,0,0,
-    114,187,0,0,0,114,192,0,0,0,114,194,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,251,0,0,0,78,4,0,0,115,16,0,0,
-    0,8,1,8,3,12,9,8,3,8,3,8,3,8,3,8,
-    3,114,251,0,0,0,99,0,0,0,0,0,0,0,0,0,
-    0,0,0,4,0,0,0,64,0,0,0,115,106,0,0,0,
-    101,0,90,1,100,0,90,2,100,1,90,3,101,4,100,2,
-    100,3,132,0,131,1,90,5,101,4,100,4,100,5,132,0,
-    131,1,90,6,101,4,100,6,100,7,132,0,131,1,90,7,
-    101,4,100,8,100,9,132,0,131,1,90,8,101,4,100,17,
-    100,11,100,12,132,1,131,1,90,9,101,4,100,18,100,13,
-    100,14,132,1,131,1,90,10,101,4,100,19,100,15,100,16,
-    132,1,131,1,90,11,100,10,83,0,41,20,218,10,80,97,
-    116,104,70,105,110,100,101,114,122,62,77,101,116,97,32,112,
-    97,116,104,32,102,105,110,100,101,114,32,102,111,114,32,115,
-    121,115,46,112,97,116,104,32,97,110,100,32,112,97,99,107,
-    97,103,101,32,95,95,112,97,116,104,95,95,32,97,116,116,
-    114,105,98,117,116,101,115,46,99,1,0,0,0,0,0,0,
-    0,2,0,0,0,4,0,0,0,67,0,0,0,115,42,0,
-    0,0,120,36,116,0,106,1,160,2,161,0,68,0,93,22,
-    125,1,116,3,124,1,100,1,131,2,114,12,124,1,160,4,
-    161,0,1,0,113,12,87,0,100,2,83,0,41,3,122,125,
-    67,97,108,108,32,116,104,101,32,105,110,118,97,108,105,100,
-    97,116,101,95,99,97,99,104,101,115,40,41,32,109,101,116,
-    104,111,100,32,111,110,32,97,108,108,32,112,97,116,104,32,
-    101,110,116,114,121,32,102,105,110,100,101,114,115,10,32,32,
-    32,32,32,32,32,32,115,116,111,114,101,100,32,105,110,32,
+    46,99,114,101,97,116,101,95,109,111,100,117,108,101,99,2,
+    0,0,0,0,0,0,0,2,0,0,0,1,0,0,0,67,
+    0,0,0,115,4,0,0,0,100,0,83,0,41,1,78,114,
+    2,0,0,0,41,2,114,102,0,0,0,114,191,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
+    192,0,0,0,103,4,0,0,115,2,0,0,0,0,1,122,
+    28,95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,
+    114,46,101,120,101,99,95,109,111,100,117,108,101,99,2,0,
+    0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,0,
+    0,0,115,26,0,0,0,116,0,160,1,100,1,124,0,106,
+    2,161,2,1,0,116,0,160,3,124,0,124,1,161,2,83,
+    0,41,2,122,98,76,111,97,100,32,97,32,110,97,109,101,
+    115,112,97,99,101,32,109,111,100,117,108,101,46,10,10,32,
+    32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,104,
+    111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,100,
+    46,32,32,85,115,101,32,101,120,101,99,95,109,111,100,117,
+    108,101,40,41,32,105,110,115,116,101,97,100,46,10,10,32,
+    32,32,32,32,32,32,32,122,38,110,97,109,101,115,112,97,
+    99,101,32,109,111,100,117,108,101,32,108,111,97,100,101,100,
+    32,119,105,116,104,32,112,97,116,104,32,123,33,114,125,41,
+    4,114,116,0,0,0,114,130,0,0,0,114,234,0,0,0,
+    114,193,0,0,0,41,2,114,102,0,0,0,114,121,0,0,
+    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,194,0,0,0,106,4,0,0,115,6,0,0,0,0,7,
+    6,1,8,1,122,28,95,78,97,109,101,115,112,97,99,101,
+    76,111,97,100,101,114,46,108,111,97,100,95,109,111,100,117,
+    108,101,78,41,12,114,107,0,0,0,114,106,0,0,0,114,
+    108,0,0,0,114,186,0,0,0,114,184,0,0,0,114,252,
+    0,0,0,114,161,0,0,0,114,202,0,0,0,114,188,0,
+    0,0,114,187,0,0,0,114,192,0,0,0,114,194,0,0,
+    0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,114,251,0,0,0,78,4,0,0,115,16,
+    0,0,0,8,1,8,3,12,9,8,3,8,3,8,3,8,
+    3,8,3,114,251,0,0,0,99,0,0,0,0,0,0,0,
+    0,0,0,0,0,4,0,0,0,64,0,0,0,115,106,0,
+    0,0,101,0,90,1,100,0,90,2,100,1,90,3,101,4,
+    100,2,100,3,132,0,131,1,90,5,101,4,100,4,100,5,
+    132,0,131,1,90,6,101,4,100,6,100,7,132,0,131,1,
+    90,7,101,4,100,8,100,9,132,0,131,1,90,8,101,4,
+    100,17,100,11,100,12,132,1,131,1,90,9,101,4,100,18,
+    100,13,100,14,132,1,131,1,90,10,101,4,100,19,100,15,
+    100,16,132,1,131,1,90,11,100,10,83,0,41,20,218,10,
+    80,97,116,104,70,105,110,100,101,114,122,62,77,101,116,97,
+    32,112,97,116,104,32,102,105,110,100,101,114,32,102,111,114,
+    32,115,121,115,46,112,97,116,104,32,97,110,100,32,112,97,
+    99,107,97,103,101,32,95,95,112,97,116,104,95,95,32,97,
+    116,116,114,105,98,117,116,101,115,46,99,1,0,0,0,0,
+    0,0,0,2,0,0,0,4,0,0,0,67,0,0,0,115,
+    42,0,0,0,120,36,116,0,106,1,160,2,161,0,68,0,
+    93,22,125,1,116,3,124,1,100,1,131,2,114,12,124,1,
+    160,4,161,0,1,0,113,12,87,0,100,2,83,0,41,3,
+    122,125,67,97,108,108,32,116,104,101,32,105,110,118,97,108,
+    105,100,97,116,101,95,99,97,99,104,101,115,40,41,32,109,
+    101,116,104,111,100,32,111,110,32,97,108,108,32,112,97,116,
+    104,32,101,110,116,114,121,32,102,105,110,100,101,114,115,10,
+    32,32,32,32,32,32,32,32,115,116,111,114,101,100,32,105,
+    110,32,115,121,115,46,112,97,116,104,95,105,109,112,111,114,
+    116,101,114,95,99,97,99,104,101,115,32,40,119,104,101,114,
+    101,32,105,109,112,108,101,109,101,110,116,101,100,41,46,218,
+    17,105,110,118,97,108,105,100,97,116,101,95,99,97,99,104,
+    101,115,78,41,5,114,6,0,0,0,218,19,112,97,116,104,
+    95,105,109,112,111,114,116,101,114,95,99,97,99,104,101,218,
+    6,118,97,108,117,101,115,114,110,0,0,0,114,254,0,0,
+    0,41,2,114,172,0,0,0,218,6,102,105,110,100,101,114,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
+    254,0,0,0,124,4,0,0,115,6,0,0,0,0,4,16,
+    1,10,1,122,28,80,97,116,104,70,105,110,100,101,114,46,
+    105,110,118,97,108,105,100,97,116,101,95,99,97,99,104,101,
+    115,99,2,0,0,0,0,0,0,0,3,0,0,0,9,0,
+    0,0,67,0,0,0,115,84,0,0,0,116,0,106,1,100,
+    1,107,9,114,28,116,0,106,1,115,28,116,2,160,3,100,
+    2,116,4,161,2,1,0,120,50,116,0,106,1,68,0,93,
+    36,125,2,121,8,124,2,124,1,131,1,83,0,4,0,116,
+    5,107,10,114,70,1,0,1,0,1,0,119,36,89,0,113,
+    36,88,0,113,36,87,0,100,1,83,0,100,1,83,0,41,
+    3,122,46,83,101,97,114,99,104,32,115,121,115,46,112,97,
+    116,104,95,104,111,111,107,115,32,102,111,114,32,97,32,102,
+    105,110,100,101,114,32,102,111,114,32,39,112,97,116,104,39,
+    46,78,122,23,115,121,115,46,112,97,116,104,95,104,111,111,
+    107,115,32,105,115,32,101,109,112,116,121,41,6,114,6,0,
+    0,0,218,10,112,97,116,104,95,104,111,111,107,115,114,61,
+    0,0,0,114,62,0,0,0,114,120,0,0,0,114,101,0,
+    0,0,41,3,114,172,0,0,0,114,35,0,0,0,90,4,
+    104,111,111,107,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,218,11,95,112,97,116,104,95,104,111,111,107,115,
+    132,4,0,0,115,16,0,0,0,0,3,16,1,12,1,12,
+    1,2,1,8,1,14,1,12,2,122,22,80,97,116,104,70,
+    105,110,100,101,114,46,95,112,97,116,104,95,104,111,111,107,
+    115,99,2,0,0,0,0,0,0,0,3,0,0,0,8,0,
+    0,0,67,0,0,0,115,102,0,0,0,124,1,100,1,107,
+    2,114,42,121,12,116,0,160,1,161,0,125,1,87,0,110,
+    20,4,0,116,2,107,10,114,40,1,0,1,0,1,0,100,
+    2,83,0,88,0,121,14,116,3,106,4,124,1,25,0,125,
+    2,87,0,110,40,4,0,116,5,107,10,114,96,1,0,1,
+    0,1,0,124,0,160,6,124,1,161,1,125,2,124,2,116,
+    3,106,4,124,1,60,0,89,0,110,2,88,0,124,2,83,
+    0,41,3,122,210,71,101,116,32,116,104,101,32,102,105,110,
+    100,101,114,32,102,111,114,32,116,104,101,32,112,97,116,104,
+    32,101,110,116,114,121,32,102,114,111,109,32,115,121,115,46,
+    112,97,116,104,95,105,109,112,111,114,116,101,114,95,99,97,
+    99,104,101,46,10,10,32,32,32,32,32,32,32,32,73,102,
+    32,116,104,101,32,112,97,116,104,32,101,110,116,114,121,32,
+    105,115,32,110,111,116,32,105,110,32,116,104,101,32,99,97,
+    99,104,101,44,32,102,105,110,100,32,116,104,101,32,97,112,
+    112,114,111,112,114,105,97,116,101,32,102,105,110,100,101,114,
+    10,32,32,32,32,32,32,32,32,97,110,100,32,99,97,99,
+    104,101,32,105,116,46,32,73,102,32,110,111,32,102,105,110,
+    100,101,114,32,105,115,32,97,118,97,105,108,97,98,108,101,
+    44,32,115,116,111,114,101,32,78,111,110,101,46,10,10,32,
+    32,32,32,32,32,32,32,114,30,0,0,0,78,41,7,114,
+    1,0,0,0,114,45,0,0,0,218,17,70,105,108,101,78,
+    111,116,70,111,117,110,100,69,114,114,111,114,114,6,0,0,
+    0,114,255,0,0,0,218,8,75,101,121,69,114,114,111,114,
+    114,3,1,0,0,41,3,114,172,0,0,0,114,35,0,0,
+    0,114,1,1,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,218,20,95,112,97,116,104,95,105,109,112,
+    111,114,116,101,114,95,99,97,99,104,101,145,4,0,0,115,
+    22,0,0,0,0,8,8,1,2,1,12,1,14,3,6,1,
+    2,1,14,1,14,1,10,1,16,1,122,31,80,97,116,104,
+    70,105,110,100,101,114,46,95,112,97,116,104,95,105,109,112,
+    111,114,116,101,114,95,99,97,99,104,101,99,3,0,0,0,
+    0,0,0,0,6,0,0,0,4,0,0,0,67,0,0,0,
+    115,82,0,0,0,116,0,124,2,100,1,131,2,114,26,124,
+    2,160,1,124,1,161,1,92,2,125,3,125,4,110,14,124,
+    2,160,2,124,1,161,1,125,3,103,0,125,4,124,3,100,
+    0,107,9,114,60,116,3,160,4,124,1,124,3,161,2,83,
+    0,116,3,160,5,124,1,100,0,161,2,125,5,124,4,124,
+    5,95,6,124,5,83,0,41,2,78,114,119,0,0,0,41,
+    7,114,110,0,0,0,114,119,0,0,0,114,183,0,0,0,
+    114,116,0,0,0,114,180,0,0,0,114,162,0,0,0,114,
+    158,0,0,0,41,6,114,172,0,0,0,114,121,0,0,0,
+    114,1,1,0,0,114,122,0,0,0,114,123,0,0,0,114,
+    166,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,218,16,95,108,101,103,97,99,121,95,103,101,116,
+    95,115,112,101,99,167,4,0,0,115,18,0,0,0,0,4,
+    10,1,16,2,10,1,4,1,8,1,12,1,12,1,6,1,
+    122,27,80,97,116,104,70,105,110,100,101,114,46,95,108,101,
+    103,97,99,121,95,103,101,116,95,115,112,101,99,78,99,4,
+    0,0,0,0,0,0,0,9,0,0,0,5,0,0,0,67,
+    0,0,0,115,170,0,0,0,103,0,125,4,120,160,124,2,
+    68,0,93,130,125,5,116,0,124,5,116,1,116,2,102,2,
+    131,2,115,30,113,10,124,0,160,3,124,5,161,1,125,6,
+    124,6,100,1,107,9,114,10,116,4,124,6,100,2,131,2,
+    114,72,124,6,160,5,124,1,124,3,161,2,125,7,110,12,
+    124,0,160,6,124,1,124,6,161,2,125,7,124,7,100,1,
+    107,8,114,94,113,10,124,7,106,7,100,1,107,9,114,108,
+    124,7,83,0,124,7,106,8,125,8,124,8,100,1,107,8,
+    114,130,116,9,100,3,131,1,130,1,124,4,160,10,124,8,
+    161,1,1,0,113,10,87,0,116,11,160,12,124,1,100,1,
+    161,2,125,7,124,4,124,7,95,8,124,7,83,0,100,1,
+    83,0,41,4,122,63,70,105,110,100,32,116,104,101,32,108,
+    111,97,100,101,114,32,111,114,32,110,97,109,101,115,112,97,
+    99,101,95,112,97,116,104,32,102,111,114,32,116,104,105,115,
+    32,109,111,100,117,108,101,47,112,97,99,107,97,103,101,32,
+    110,97,109,101,46,78,114,182,0,0,0,122,19,115,112,101,
+    99,32,109,105,115,115,105,110,103,32,108,111,97,100,101,114,
+    41,13,114,141,0,0,0,114,71,0,0,0,218,5,98,121,
+    116,101,115,114,6,1,0,0,114,110,0,0,0,114,182,0,
+    0,0,114,7,1,0,0,114,122,0,0,0,114,158,0,0,
+    0,114,101,0,0,0,114,147,0,0,0,114,116,0,0,0,
+    114,162,0,0,0,41,9,114,172,0,0,0,114,121,0,0,
+    0,114,35,0,0,0,114,181,0,0,0,218,14,110,97,109,
+    101,115,112,97,99,101,95,112,97,116,104,90,5,101,110,116,
+    114,121,114,1,1,0,0,114,166,0,0,0,114,123,0,0,
+    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    218,9,95,103,101,116,95,115,112,101,99,182,4,0,0,115,
+    40,0,0,0,0,5,4,1,10,1,14,1,2,1,10,1,
+    8,1,10,1,14,2,12,1,8,1,2,1,10,1,4,1,
+    6,1,8,1,8,5,14,2,12,1,6,1,122,20,80,97,
+    116,104,70,105,110,100,101,114,46,95,103,101,116,95,115,112,
+    101,99,99,4,0,0,0,0,0,0,0,6,0,0,0,5,
+    0,0,0,67,0,0,0,115,100,0,0,0,124,2,100,1,
+    107,8,114,14,116,0,106,1,125,2,124,0,160,2,124,1,
+    124,2,124,3,161,3,125,4,124,4,100,1,107,8,114,40,
+    100,1,83,0,124,4,106,3,100,1,107,8,114,92,124,4,
+    106,4,125,5,124,5,114,86,100,2,124,4,95,5,116,6,
+    124,1,124,5,124,0,106,2,131,3,124,4,95,4,124,4,
+    83,0,100,1,83,0,110,4,124,4,83,0,100,1,83,0,
+    41,3,122,141,84,114,121,32,116,111,32,102,105,110,100,32,
+    97,32,115,112,101,99,32,102,111,114,32,39,102,117,108,108,
+    110,97,109,101,39,32,111,110,32,115,121,115,46,112,97,116,
+    104,32,111,114,32,39,112,97,116,104,39,46,10,10,32,32,
+    32,32,32,32,32,32,84,104,101,32,115,101,97,114,99,104,
+    32,105,115,32,98,97,115,101,100,32,111,110,32,115,121,115,
+    46,112,97,116,104,95,104,111,111,107,115,32,97,110,100,32,
     115,121,115,46,112,97,116,104,95,105,109,112,111,114,116,101,
-    114,95,99,97,99,104,101,115,32,40,119,104,101,114,101,32,
-    105,109,112,108,101,109,101,110,116,101,100,41,46,218,17,105,
-    110,118,97,108,105,100,97,116,101,95,99,97,99,104,101,115,
-    78,41,5,114,6,0,0,0,218,19,112,97,116,104,95,105,
-    109,112,111,114,116,101,114,95,99,97,99,104,101,218,6,118,
-    97,108,117,101,115,114,110,0,0,0,114,254,0,0,0,41,
-    2,114,172,0,0,0,218,6,102,105,110,100,101,114,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,254,0,
-    0,0,124,4,0,0,115,6,0,0,0,0,4,16,1,10,
-    1,122,28,80,97,116,104,70,105,110,100,101,114,46,105,110,
-    118,97,108,105,100,97,116,101,95,99,97,99,104,101,115,99,
-    2,0,0,0,0,0,0,0,3,0,0,0,12,0,0,0,
-    67,0,0,0,115,84,0,0,0,116,0,106,1,100,1,107,
-    9,114,28,116,0,106,1,115,28,116,2,160,3,100,2,116,
-    4,161,2,1,0,120,50,116,0,106,1,68,0,93,36,125,
-    2,121,8,124,2,124,1,131,1,83,0,4,0,116,5,107,
-    10,114,70,1,0,1,0,1,0,119,36,89,0,113,36,88,
-    0,113,36,87,0,100,1,83,0,100,1,83,0,41,3,122,
-    46,83,101,97,114,99,104,32,115,121,115,46,112,97,116,104,
-    95,104,111,111,107,115,32,102,111,114,32,97,32,102,105,110,
-    100,101,114,32,102,111,114,32,39,112,97,116,104,39,46,78,
-    122,23,115,121,115,46,112,97,116,104,95,104,111,111,107,115,
-    32,105,115,32,101,109,112,116,121,41,6,114,6,0,0,0,
-    218,10,112,97,116,104,95,104,111,111,107,115,114,61,0,0,
-    0,114,62,0,0,0,114,120,0,0,0,114,101,0,0,0,
-    41,3,114,172,0,0,0,114,35,0,0,0,90,4,104,111,
-    111,107,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,11,95,112,97,116,104,95,104,111,111,107,115,132,4,
-    0,0,115,16,0,0,0,0,3,16,1,12,1,12,1,2,
-    1,8,1,14,1,12,2,122,22,80,97,116,104,70,105,110,
-    100,101,114,46,95,112,97,116,104,95,104,111,111,107,115,99,
-    2,0,0,0,0,0,0,0,3,0,0,0,19,0,0,0,
-    67,0,0,0,115,102,0,0,0,124,1,100,1,107,2,114,
-    42,121,12,116,0,160,1,161,0,125,1,87,0,110,20,4,
-    0,116,2,107,10,114,40,1,0,1,0,1,0,100,2,83,
-    0,88,0,121,14,116,3,106,4,124,1,25,0,125,2,87,
-    0,110,40,4,0,116,5,107,10,114,96,1,0,1,0,1,
-    0,124,0,160,6,124,1,161,1,125,2,124,2,116,3,106,
-    4,124,1,60,0,89,0,110,2,88,0,124,2,83,0,41,
-    3,122,210,71,101,116,32,116,104,101,32,102,105,110,100,101,
-    114,32,102,111,114,32,116,104,101,32,112,97,116,104,32,101,
-    110,116,114,121,32,102,114,111,109,32,115,121,115,46,112,97,
-    116,104,95,105,109,112,111,114,116,101,114,95,99,97,99,104,
-    101,46,10,10,32,32,32,32,32,32,32,32,73,102,32,116,
-    104,101,32,112,97,116,104,32,101,110,116,114,121,32,105,115,
-    32,110,111,116,32,105,110,32,116,104,101,32,99,97,99,104,
-    101,44,32,102,105,110,100,32,116,104,101,32,97,112,112,114,
-    111,112,114,105,97,116,101,32,102,105,110,100,101,114,10,32,
-    32,32,32,32,32,32,32,97,110,100,32,99,97,99,104,101,
-    32,105,116,46,32,73,102,32,110,111,32,102,105,110,100,101,
-    114,32,105,115,32,97,118,97,105,108,97,98,108,101,44,32,
-    115,116,111,114,101,32,78,111,110,101,46,10,10,32,32,32,
-    32,32,32,32,32,114,30,0,0,0,78,41,7,114,1,0,
-    0,0,114,45,0,0,0,218,17,70,105,108,101,78,111,116,
-    70,111,117,110,100,69,114,114,111,114,114,6,0,0,0,114,
-    255,0,0,0,218,8,75,101,121,69,114,114,111,114,114,3,
-    1,0,0,41,3,114,172,0,0,0,114,35,0,0,0,114,
-    1,1,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,218,20,95,112,97,116,104,95,105,109,112,111,114,
-    116,101,114,95,99,97,99,104,101,145,4,0,0,115,22,0,
-    0,0,0,8,8,1,2,1,12,1,14,3,6,1,2,1,
-    14,1,14,1,10,1,16,1,122,31,80,97,116,104,70,105,
-    110,100,101,114,46,95,112,97,116,104,95,105,109,112,111,114,
-    116,101,114,95,99,97,99,104,101,99,3,0,0,0,0,0,
-    0,0,6,0,0,0,4,0,0,0,67,0,0,0,115,82,
-    0,0,0,116,0,124,2,100,1,131,2,114,26,124,2,160,
-    1,124,1,161,1,92,2,125,3,125,4,110,14,124,2,160,
-    2,124,1,161,1,125,3,103,0,125,4,124,3,100,0,107,
-    9,114,60,116,3,160,4,124,1,124,3,161,2,83,0,116,
-    3,160,5,124,1,100,0,161,2,125,5,124,4,124,5,95,
-    6,124,5,83,0,41,2,78,114,119,0,0,0,41,7,114,
-    110,0,0,0,114,119,0,0,0,114,183,0,0,0,114,116,
-    0,0,0,114,180,0,0,0,114,162,0,0,0,114,158,0,
-    0,0,41,6,114,172,0,0,0,114,121,0,0,0,114,1,
-    1,0,0,114,122,0,0,0,114,123,0,0,0,114,166,0,
+    114,95,99,97,99,104,101,46,10,32,32,32,32,32,32,32,
+    32,78,90,9,110,97,109,101,115,112,97,99,101,41,7,114,
+    6,0,0,0,114,35,0,0,0,114,10,1,0,0,114,122,
+    0,0,0,114,158,0,0,0,114,160,0,0,0,114,232,0,
+    0,0,41,6,114,172,0,0,0,114,121,0,0,0,114,35,
+    0,0,0,114,181,0,0,0,114,166,0,0,0,114,9,1,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,16,95,108,101,103,97,99,121,95,103,101,116,95,115,
-    112,101,99,167,4,0,0,115,18,0,0,0,0,4,10,1,
-    16,2,10,1,4,1,8,1,12,1,12,1,6,1,122,27,
-    80,97,116,104,70,105,110,100,101,114,46,95,108,101,103,97,
-    99,121,95,103,101,116,95,115,112,101,99,78,99,4,0,0,
-    0,0,0,0,0,9,0,0,0,5,0,0,0,67,0,0,
-    0,115,170,0,0,0,103,0,125,4,120,160,124,2,68,0,
-    93,130,125,5,116,0,124,5,116,1,116,2,102,2,131,2,
-    115,30,113,10,124,0,160,3,124,5,161,1,125,6,124,6,
-    100,1,107,9,114,10,116,4,124,6,100,2,131,2,114,72,
-    124,6,160,5,124,1,124,3,161,2,125,7,110,12,124,0,
-    160,6,124,1,124,6,161,2,125,7,124,7,100,1,107,8,
-    114,94,113,10,124,7,106,7,100,1,107,9,114,108,124,7,
-    83,0,124,7,106,8,125,8,124,8,100,1,107,8,114,130,
-    116,9,100,3,131,1,130,1,124,4,160,10,124,8,161,1,
-    1,0,113,10,87,0,116,11,160,12,124,1,100,1,161,2,
-    125,7,124,4,124,7,95,8,124,7,83,0,100,1,83,0,
-    41,4,122,63,70,105,110,100,32,116,104,101,32,108,111,97,
-    100,101,114,32,111,114,32,110,97,109,101,115,112,97,99,101,
-    95,112,97,116,104,32,102,111,114,32,116,104,105,115,32,109,
-    111,100,117,108,101,47,112,97,99,107,97,103,101,32,110,97,
-    109,101,46,78,114,182,0,0,0,122,19,115,112,101,99,32,
-    109,105,115,115,105,110,103,32,108,111,97,100,101,114,41,13,
-    114,141,0,0,0,114,71,0,0,0,218,5,98,121,116,101,
-    115,114,6,1,0,0,114,110,0,0,0,114,182,0,0,0,
-    114,7,1,0,0,114,122,0,0,0,114,158,0,0,0,114,
-    101,0,0,0,114,147,0,0,0,114,116,0,0,0,114,162,
-    0,0,0,41,9,114,172,0,0,0,114,121,0,0,0,114,
-    35,0,0,0,114,181,0,0,0,218,14,110,97,109,101,115,
-    112,97,99,101,95,112,97,116,104,90,5,101,110,116,114,121,
-    114,1,1,0,0,114,166,0,0,0,114,123,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,9,
-    95,103,101,116,95,115,112,101,99,182,4,0,0,115,40,0,
-    0,0,0,5,4,1,10,1,14,1,2,1,10,1,8,1,
-    10,1,14,2,12,1,8,1,2,1,10,1,4,1,6,1,
-    8,1,8,5,14,2,12,1,6,1,122,20,80,97,116,104,
-    70,105,110,100,101,114,46,95,103,101,116,95,115,112,101,99,
-    99,4,0,0,0,0,0,0,0,6,0,0,0,5,0,0,
-    0,67,0,0,0,115,100,0,0,0,124,2,100,1,107,8,
-    114,14,116,0,106,1,125,2,124,0,160,2,124,1,124,2,
-    124,3,161,3,125,4,124,4,100,1,107,8,114,40,100,1,
-    83,0,124,4,106,3,100,1,107,8,114,92,124,4,106,4,
-    125,5,124,5,114,86,100,2,124,4,95,5,116,6,124,1,
-    124,5,124,0,106,2,131,3,124,4,95,4,124,4,83,0,
-    100,1,83,0,110,4,124,4,83,0,100,1,83,0,41,3,
-    122,141,84,114,121,32,116,111,32,102,105,110,100,32,97,32,
-    115,112,101,99,32,102,111,114,32,39,102,117,108,108,110,97,
-    109,101,39,32,111,110,32,115,121,115,46,112,97,116,104,32,
-    111,114,32,39,112,97,116,104,39,46,10,10,32,32,32,32,
-    32,32,32,32,84,104,101,32,115,101,97,114,99,104,32,105,
-    115,32,98,97,115,101,100,32,111,110,32,115,121,115,46,112,
-    97,116,104,95,104,111,111,107,115,32,97,110,100,32,115,121,
-    115,46,112,97,116,104,95,105,109,112,111,114,116,101,114,95,
-    99,97,99,104,101,46,10,32,32,32,32,32,32,32,32,78,
-    90,9,110,97,109,101,115,112,97,99,101,41,7,114,6,0,
-    0,0,114,35,0,0,0,114,10,1,0,0,114,122,0,0,
-    0,114,158,0,0,0,114,160,0,0,0,114,232,0,0,0,
-    41,6,114,172,0,0,0,114,121,0,0,0,114,35,0,0,
-    0,114,181,0,0,0,114,166,0,0,0,114,9,1,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    182,0,0,0,214,4,0,0,115,26,0,0,0,0,6,8,
-    1,6,1,14,1,8,1,4,1,10,1,6,1,4,3,6,
-    1,16,1,4,2,6,2,122,20,80,97,116,104,70,105,110,
-    100,101,114,46,102,105,110,100,95,115,112,101,99,99,3,0,
-    0,0,0,0,0,0,4,0,0,0,4,0,0,0,67,0,
-    0,0,115,30,0,0,0,124,0,160,0,124,1,124,2,161,
-    2,125,3,124,3,100,1,107,8,114,24,100,1,83,0,124,
-    3,106,1,83,0,41,2,122,170,102,105,110,100,32,116,104,
-    101,32,109,111,100,117,108,101,32,111,110,32,115,121,115,46,
-    112,97,116,104,32,111,114,32,39,112,97,116,104,39,32,98,
-    97,115,101,100,32,111,110,32,115,121,115,46,112,97,116,104,
-    95,104,111,111,107,115,32,97,110,100,10,32,32,32,32,32,
-    32,32,32,115,121,115,46,112,97,116,104,95,105,109,112,111,
-    114,116,101,114,95,99,97,99,104,101,46,10,10,32,32,32,
-    32,32,32,32,32,84,104,105,115,32,109,101,116,104,111,100,
-    32,105,115,32,100,101,112,114,101,99,97,116,101,100,46,32,
-    32,85,115,101,32,102,105,110,100,95,115,112,101,99,40,41,
-    32,105,110,115,116,101,97,100,46,10,10,32,32,32,32,32,
-    32,32,32,78,41,2,114,182,0,0,0,114,122,0,0,0,
-    41,4,114,172,0,0,0,114,121,0,0,0,114,35,0,0,
-    0,114,166,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,183,0,0,0,238,4,0,0,115,8,
-    0,0,0,0,8,12,1,8,1,4,1,122,22,80,97,116,
-    104,70,105,110,100,101,114,46,102,105,110,100,95,109,111,100,
-    117,108,101,41,1,78,41,2,78,78,41,1,78,41,12,114,
-    107,0,0,0,114,106,0,0,0,114,108,0,0,0,114,109,
-    0,0,0,114,184,0,0,0,114,254,0,0,0,114,3,1,
-    0,0,114,6,1,0,0,114,7,1,0,0,114,10,1,0,
-    0,114,182,0,0,0,114,183,0,0,0,114,2,0,0,0,
+    0,114,182,0,0,0,214,4,0,0,115,26,0,0,0,0,
+    6,8,1,6,1,14,1,8,1,4,1,10,1,6,1,4,
+    3,6,1,16,1,4,2,6,2,122,20,80,97,116,104,70,
+    105,110,100,101,114,46,102,105,110,100,95,115,112,101,99,99,
+    3,0,0,0,0,0,0,0,4,0,0,0,4,0,0,0,
+    67,0,0,0,115,30,0,0,0,124,0,160,0,124,1,124,
+    2,161,2,125,3,124,3,100,1,107,8,114,24,100,1,83,
+    0,124,3,106,1,83,0,41,2,122,170,102,105,110,100,32,
+    116,104,101,32,109,111,100,117,108,101,32,111,110,32,115,121,
+    115,46,112,97,116,104,32,111,114,32,39,112,97,116,104,39,
+    32,98,97,115,101,100,32,111,110,32,115,121,115,46,112,97,
+    116,104,95,104,111,111,107,115,32,97,110,100,10,32,32,32,
+    32,32,32,32,32,115,121,115,46,112,97,116,104,95,105,109,
+    112,111,114,116,101,114,95,99,97,99,104,101,46,10,10,32,
+    32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,104,
+    111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,100,
+    46,32,32,85,115,101,32,102,105,110,100,95,115,112,101,99,
+    40,41,32,105,110,115,116,101,97,100,46,10,10,32,32,32,
+    32,32,32,32,32,78,41,2,114,182,0,0,0,114,122,0,
+    0,0,41,4,114,172,0,0,0,114,121,0,0,0,114,35,
+    0,0,0,114,166,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,114,183,0,0,0,238,4,0,0,
+    115,8,0,0,0,0,8,12,1,8,1,4,1,122,22,80,
+    97,116,104,70,105,110,100,101,114,46,102,105,110,100,95,109,
+    111,100,117,108,101,41,1,78,41,2,78,78,41,1,78,41,
+    12,114,107,0,0,0,114,106,0,0,0,114,108,0,0,0,
+    114,109,0,0,0,114,184,0,0,0,114,254,0,0,0,114,
+    3,1,0,0,114,6,1,0,0,114,7,1,0,0,114,10,
+    1,0,0,114,182,0,0,0,114,183,0,0,0,114,2,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,253,0,0,0,120,4,0,0,115,20,0,0,0,12,
+    4,12,8,12,13,12,22,12,15,2,1,12,31,2,1,12,
+    23,2,1,114,253,0,0,0,99,0,0,0,0,0,0,0,
+    0,0,0,0,0,3,0,0,0,64,0,0,0,115,90,0,
+    0,0,101,0,90,1,100,0,90,2,100,1,90,3,100,2,
+    100,3,132,0,90,4,100,4,100,5,132,0,90,5,101,6,
+    90,7,100,6,100,7,132,0,90,8,100,8,100,9,132,0,
+    90,9,100,19,100,11,100,12,132,1,90,10,100,13,100,14,
+    132,0,90,11,101,12,100,15,100,16,132,0,131,1,90,13,
+    100,17,100,18,132,0,90,14,100,10,83,0,41,20,218,10,
+    70,105,108,101,70,105,110,100,101,114,122,172,70,105,108,101,
+    45,98,97,115,101,100,32,102,105,110,100,101,114,46,10,10,
+    32,32,32,32,73,110,116,101,114,97,99,116,105,111,110,115,
+    32,119,105,116,104,32,116,104,101,32,102,105,108,101,32,115,
+    121,115,116,101,109,32,97,114,101,32,99,97,99,104,101,100,
+    32,102,111,114,32,112,101,114,102,111,114,109,97,110,99,101,
+    44,32,98,101,105,110,103,10,32,32,32,32,114,101,102,114,
+    101,115,104,101,100,32,119,104,101,110,32,116,104,101,32,100,
+    105,114,101,99,116,111,114,121,32,116,104,101,32,102,105,110,
+    100,101,114,32,105,115,32,104,97,110,100,108,105,110,103,32,
+    104,97,115,32,98,101,101,110,32,109,111,100,105,102,105,101,
+    100,46,10,10,32,32,32,32,99,2,0,0,0,0,0,0,
+    0,5,0,0,0,6,0,0,0,7,0,0,0,115,88,0,
+    0,0,103,0,125,3,120,40,124,2,68,0,93,32,92,2,
+    137,0,125,4,124,3,160,0,135,0,102,1,100,1,100,2,
+    132,8,124,4,68,0,131,1,161,1,1,0,113,10,87,0,
+    124,3,124,0,95,1,124,1,112,58,100,3,124,0,95,2,
+    100,4,124,0,95,3,116,4,131,0,124,0,95,5,116,4,
+    131,0,124,0,95,6,100,5,83,0,41,6,122,154,73,110,
+    105,116,105,97,108,105,122,101,32,119,105,116,104,32,116,104,
+    101,32,112,97,116,104,32,116,111,32,115,101,97,114,99,104,
+    32,111,110,32,97,110,100,32,97,32,118,97,114,105,97,98,
+    108,101,32,110,117,109,98,101,114,32,111,102,10,32,32,32,
+    32,32,32,32,32,50,45,116,117,112,108,101,115,32,99,111,
+    110,116,97,105,110,105,110,103,32,116,104,101,32,108,111,97,
+    100,101,114,32,97,110,100,32,116,104,101,32,102,105,108,101,
+    32,115,117,102,102,105,120,101,115,32,116,104,101,32,108,111,
+    97,100,101,114,10,32,32,32,32,32,32,32,32,114,101,99,
+    111,103,110,105,122,101,115,46,99,1,0,0,0,0,0,0,
+    0,2,0,0,0,3,0,0,0,51,0,0,0,115,22,0,
+    0,0,124,0,93,14,125,1,124,1,136,0,102,2,86,0,
+    1,0,113,2,100,0,83,0,41,1,78,114,2,0,0,0,
+    41,2,114,22,0,0,0,114,227,0,0,0,41,1,114,122,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,114,229,0,
+    0,0,11,5,0,0,115,2,0,0,0,4,0,122,38,70,
+    105,108,101,70,105,110,100,101,114,46,95,95,105,110,105,116,
+    95,95,46,60,108,111,99,97,108,115,62,46,60,103,101,110,
+    101,120,112,114,62,114,59,0,0,0,114,89,0,0,0,78,
+    41,7,114,147,0,0,0,218,8,95,108,111,97,100,101,114,
+    115,114,35,0,0,0,218,11,95,112,97,116,104,95,109,116,
+    105,109,101,218,3,115,101,116,218,11,95,112,97,116,104,95,
+    99,97,99,104,101,218,19,95,114,101,108,97,120,101,100,95,
+    112,97,116,104,95,99,97,99,104,101,41,5,114,102,0,0,
+    0,114,35,0,0,0,218,14,108,111,97,100,101,114,95,100,
+    101,116,97,105,108,115,90,7,108,111,97,100,101,114,115,114,
+    168,0,0,0,114,2,0,0,0,41,1,114,122,0,0,0,
+    114,4,0,0,0,114,186,0,0,0,5,5,0,0,115,16,
+    0,0,0,0,4,4,1,14,1,28,1,6,2,10,1,6,
+    1,8,1,122,19,70,105,108,101,70,105,110,100,101,114,46,
+    95,95,105,110,105,116,95,95,99,1,0,0,0,0,0,0,
+    0,1,0,0,0,2,0,0,0,67,0,0,0,115,10,0,
+    0,0,100,1,124,0,95,0,100,2,83,0,41,3,122,31,
+    73,110,118,97,108,105,100,97,116,101,32,116,104,101,32,100,
+    105,114,101,99,116,111,114,121,32,109,116,105,109,101,46,114,
+    89,0,0,0,78,41,1,114,13,1,0,0,41,1,114,102,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,114,254,0,0,0,19,5,0,0,115,2,0,0,0,
+    0,2,122,28,70,105,108,101,70,105,110,100,101,114,46,105,
+    110,118,97,108,105,100,97,116,101,95,99,97,99,104,101,115,
+    99,2,0,0,0,0,0,0,0,3,0,0,0,3,0,0,
+    0,67,0,0,0,115,42,0,0,0,124,0,160,0,124,1,
+    161,1,125,2,124,2,100,1,107,8,114,26,100,1,103,0,
+    102,2,83,0,124,2,106,1,124,2,106,2,112,38,103,0,
+    102,2,83,0,41,2,122,197,84,114,121,32,116,111,32,102,
+    105,110,100,32,97,32,108,111,97,100,101,114,32,102,111,114,
+    32,116,104,101,32,115,112,101,99,105,102,105,101,100,32,109,
+    111,100,117,108,101,44,32,111,114,32,116,104,101,32,110,97,
+    109,101,115,112,97,99,101,10,32,32,32,32,32,32,32,32,
+    112,97,99,107,97,103,101,32,112,111,114,116,105,111,110,115,
+    46,32,82,101,116,117,114,110,115,32,40,108,111,97,100,101,
+    114,44,32,108,105,115,116,45,111,102,45,112,111,114,116,105,
+    111,110,115,41,46,10,10,32,32,32,32,32,32,32,32,84,
+    104,105,115,32,109,101,116,104,111,100,32,105,115,32,100,101,
+    112,114,101,99,97,116,101,100,46,32,32,85,115,101,32,102,
+    105,110,100,95,115,112,101,99,40,41,32,105,110,115,116,101,
+    97,100,46,10,10,32,32,32,32,32,32,32,32,78,41,3,
+    114,182,0,0,0,114,122,0,0,0,114,158,0,0,0,41,
+    3,114,102,0,0,0,114,121,0,0,0,114,166,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    253,0,0,0,120,4,0,0,115,20,0,0,0,12,4,12,
-    8,12,13,12,22,12,15,2,1,12,31,2,1,12,23,2,
-    1,114,253,0,0,0,99,0,0,0,0,0,0,0,0,0,
-    0,0,0,3,0,0,0,64,0,0,0,115,90,0,0,0,
-    101,0,90,1,100,0,90,2,100,1,90,3,100,2,100,3,
-    132,0,90,4,100,4,100,5,132,0,90,5,101,6,90,7,
-    100,6,100,7,132,0,90,8,100,8,100,9,132,0,90,9,
-    100,19,100,11,100,12,132,1,90,10,100,13,100,14,132,0,
-    90,11,101,12,100,15,100,16,132,0,131,1,90,13,100,17,
-    100,18,132,0,90,14,100,10,83,0,41,20,218,10,70,105,
-    108,101,70,105,110,100,101,114,122,172,70,105,108,101,45,98,
-    97,115,101,100,32,102,105,110,100,101,114,46,10,10,32,32,
-    32,32,73,110,116,101,114,97,99,116,105,111,110,115,32,119,
-    105,116,104,32,116,104,101,32,102,105,108,101,32,115,121,115,
-    116,101,109,32,97,114,101,32,99,97,99,104,101,100,32,102,
-    111,114,32,112,101,114,102,111,114,109,97,110,99,101,44,32,
-    98,101,105,110,103,10,32,32,32,32,114,101,102,114,101,115,
-    104,101,100,32,119,104,101,110,32,116,104,101,32,100,105,114,
-    101,99,116,111,114,121,32,116,104,101,32,102,105,110,100,101,
-    114,32,105,115,32,104,97,110,100,108,105,110,103,32,104,97,
-    115,32,98,101,101,110,32,109,111,100,105,102,105,101,100,46,
-    10,10,32,32,32,32,99,2,0,0,0,0,0,0,0,5,
-    0,0,0,6,0,0,0,7,0,0,0,115,88,0,0,0,
-    103,0,125,3,120,40,124,2,68,0,93,32,92,2,137,0,
-    125,4,124,3,160,0,135,0,102,1,100,1,100,2,132,8,
-    124,4,68,0,131,1,161,1,1,0,113,10,87,0,124,3,
-    124,0,95,1,124,1,112,58,100,3,124,0,95,2,100,4,
-    124,0,95,3,116,4,131,0,124,0,95,5,116,4,131,0,
-    124,0,95,6,100,5,83,0,41,6,122,154,73,110,105,116,
-    105,97,108,105,122,101,32,119,105,116,104,32,116,104,101,32,
-    112,97,116,104,32,116,111,32,115,101,97,114,99,104,32,111,
-    110,32,97,110,100,32,97,32,118,97,114,105,97,98,108,101,
-    32,110,117,109,98,101,114,32,111,102,10,32,32,32,32,32,
-    32,32,32,50,45,116,117,112,108,101,115,32,99,111,110,116,
-    97,105,110,105,110,103,32,116,104,101,32,108,111,97,100,101,
-    114,32,97,110,100,32,116,104,101,32,102,105,108,101,32,115,
-    117,102,102,105,120,101,115,32,116,104,101,32,108,111,97,100,
-    101,114,10,32,32,32,32,32,32,32,32,114,101,99,111,103,
-    110,105,122,101,115,46,99,1,0,0,0,0,0,0,0,2,
-    0,0,0,3,0,0,0,51,0,0,0,115,22,0,0,0,
-    124,0,93,14,125,1,124,1,136,0,102,2,86,0,1,0,
-    113,2,100,0,83,0,41,1,78,114,2,0,0,0,41,2,
-    114,22,0,0,0,114,227,0,0,0,41,1,114,122,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,114,229,0,0,0,
-    11,5,0,0,115,2,0,0,0,4,0,122,38,70,105,108,
-    101,70,105,110,100,101,114,46,95,95,105,110,105,116,95,95,
-    46,60,108,111,99,97,108,115,62,46,60,103,101,110,101,120,
-    112,114,62,114,59,0,0,0,114,89,0,0,0,78,41,7,
-    114,147,0,0,0,218,8,95,108,111,97,100,101,114,115,114,
-    35,0,0,0,218,11,95,112,97,116,104,95,109,116,105,109,
-    101,218,3,115,101,116,218,11,95,112,97,116,104,95,99,97,
-    99,104,101,218,19,95,114,101,108,97,120,101,100,95,112,97,
-    116,104,95,99,97,99,104,101,41,5,114,102,0,0,0,114,
-    35,0,0,0,218,14,108,111,97,100,101,114,95,100,101,116,
-    97,105,108,115,90,7,108,111,97,100,101,114,115,114,168,0,
-    0,0,114,2,0,0,0,41,1,114,122,0,0,0,114,4,
-    0,0,0,114,186,0,0,0,5,5,0,0,115,16,0,0,
-    0,0,4,4,1,14,1,28,1,6,2,10,1,6,1,8,
-    1,122,19,70,105,108,101,70,105,110,100,101,114,46,95,95,
-    105,110,105,116,95,95,99,1,0,0,0,0,0,0,0,1,
-    0,0,0,2,0,0,0,67,0,0,0,115,10,0,0,0,
-    100,1,124,0,95,0,100,2,83,0,41,3,122,31,73,110,
-    118,97,108,105,100,97,116,101,32,116,104,101,32,100,105,114,
-    101,99,116,111,114,121,32,109,116,105,109,101,46,114,89,0,
-    0,0,78,41,1,114,13,1,0,0,41,1,114,102,0,0,
+    119,0,0,0,25,5,0,0,115,8,0,0,0,0,7,10,
+    1,8,1,8,1,122,22,70,105,108,101,70,105,110,100,101,
+    114,46,102,105,110,100,95,108,111,97,100,101,114,99,6,0,
+    0,0,0,0,0,0,7,0,0,0,6,0,0,0,67,0,
+    0,0,115,26,0,0,0,124,1,124,2,124,3,131,2,125,
+    6,116,0,124,2,124,3,124,6,124,4,100,1,141,4,83,
+    0,41,2,78,41,2,114,122,0,0,0,114,158,0,0,0,
+    41,1,114,169,0,0,0,41,7,114,102,0,0,0,114,167,
+    0,0,0,114,121,0,0,0,114,35,0,0,0,90,4,115,
+    109,115,108,114,181,0,0,0,114,122,0,0,0,114,2,0,
+    0,0,114,2,0,0,0,114,4,0,0,0,114,10,1,0,
+    0,37,5,0,0,115,6,0,0,0,0,1,10,1,8,1,
+    122,20,70,105,108,101,70,105,110,100,101,114,46,95,103,101,
+    116,95,115,112,101,99,78,99,3,0,0,0,0,0,0,0,
+    14,0,0,0,8,0,0,0,67,0,0,0,115,98,1,0,
+    0,100,1,125,3,124,1,160,0,100,2,161,1,100,3,25,
+    0,125,4,121,24,116,1,124,0,106,2,112,34,116,3,160,
+    4,161,0,131,1,106,5,125,5,87,0,110,24,4,0,116,
+    6,107,10,114,66,1,0,1,0,1,0,100,4,125,5,89,
+    0,110,2,88,0,124,5,124,0,106,7,107,3,114,92,124,
+    0,160,8,161,0,1,0,124,5,124,0,95,7,116,9,131,
+    0,114,114,124,0,106,10,125,6,124,4,160,11,161,0,125,
+    7,110,10,124,0,106,12,125,6,124,4,125,7,124,7,124,
+    6,107,6,114,218,116,13,124,0,106,2,124,4,131,2,125,
+    8,120,72,124,0,106,14,68,0,93,54,92,2,125,9,125,
+    10,100,5,124,9,23,0,125,11,116,13,124,8,124,11,131,
+    2,125,12,116,15,124,12,131,1,114,152,124,0,160,16,124,
+    10,124,1,124,12,124,8,103,1,124,2,161,5,83,0,113,
+    152,87,0,116,17,124,8,131,1,125,3,120,88,124,0,106,
+    14,68,0,93,78,92,2,125,9,125,10,116,13,124,0,106,
+    2,124,4,124,9,23,0,131,2,125,12,116,18,106,19,100,
+    6,124,12,100,3,100,7,141,3,1,0,124,7,124,9,23,
+    0,124,6,107,6,114,226,116,15,124,12,131,1,114,226,124,
+    0,160,16,124,10,124,1,124,12,100,8,124,2,161,5,83,
+    0,113,226,87,0,124,3,144,1,114,94,116,18,160,19,100,
+    9,124,8,161,2,1,0,116,18,160,20,124,1,100,8,161,
+    2,125,13,124,8,103,1,124,13,95,21,124,13,83,0,100,
+    8,83,0,41,10,122,111,84,114,121,32,116,111,32,102,105,
+    110,100,32,97,32,115,112,101,99,32,102,111,114,32,116,104,
+    101,32,115,112,101,99,105,102,105,101,100,32,109,111,100,117,
+    108,101,46,10,10,32,32,32,32,32,32,32,32,82,101,116,
+    117,114,110,115,32,116,104,101,32,109,97,116,99,104,105,110,
+    103,32,115,112,101,99,44,32,111,114,32,78,111,110,101,32,
+    105,102,32,110,111,116,32,102,111,117,110,100,46,10,32,32,
+    32,32,32,32,32,32,70,114,59,0,0,0,114,57,0,0,
+    0,114,89,0,0,0,114,186,0,0,0,122,9,116,114,121,
+    105,110,103,32,123,125,41,1,90,9,118,101,114,98,111,115,
+    105,116,121,78,122,25,112,111,115,115,105,98,108,101,32,110,
+    97,109,101,115,112,97,99,101,32,102,111,114,32,123,125,41,
+    22,114,32,0,0,0,114,39,0,0,0,114,35,0,0,0,
+    114,1,0,0,0,114,45,0,0,0,114,221,0,0,0,114,
+    40,0,0,0,114,13,1,0,0,218,11,95,102,105,108,108,
+    95,99,97,99,104,101,114,5,0,0,0,114,16,1,0,0,
+    114,90,0,0,0,114,15,1,0,0,114,28,0,0,0,114,
+    12,1,0,0,114,44,0,0,0,114,10,1,0,0,114,46,
+    0,0,0,114,116,0,0,0,114,130,0,0,0,114,162,0,
+    0,0,114,158,0,0,0,41,14,114,102,0,0,0,114,121,
+    0,0,0,114,181,0,0,0,90,12,105,115,95,110,97,109,
+    101,115,112,97,99,101,90,11,116,97,105,108,95,109,111,100,
+    117,108,101,114,149,0,0,0,90,5,99,97,99,104,101,90,
+    12,99,97,99,104,101,95,109,111,100,117,108,101,90,9,98,
+    97,115,101,95,112,97,116,104,114,227,0,0,0,114,167,0,
+    0,0,90,13,105,110,105,116,95,102,105,108,101,110,97,109,
+    101,90,9,102,117,108,108,95,112,97,116,104,114,166,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,254,0,0,0,19,5,0,0,115,2,0,0,0,0,2,
-    122,28,70,105,108,101,70,105,110,100,101,114,46,105,110,118,
-    97,108,105,100,97,116,101,95,99,97,99,104,101,115,99,2,
-    0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,67,
-    0,0,0,115,42,0,0,0,124,0,160,0,124,1,161,1,
-    125,2,124,2,100,1,107,8,114,26,100,1,103,0,102,2,
-    83,0,124,2,106,1,124,2,106,2,112,38,103,0,102,2,
-    83,0,41,2,122,197,84,114,121,32,116,111,32,102,105,110,
-    100,32,97,32,108,111,97,100,101,114,32,102,111,114,32,116,
-    104,101,32,115,112,101,99,105,102,105,101,100,32,109,111,100,
-    117,108,101,44,32,111,114,32,116,104,101,32,110,97,109,101,
-    115,112,97,99,101,10,32,32,32,32,32,32,32,32,112,97,
-    99,107,97,103,101,32,112,111,114,116,105,111,110,115,46,32,
-    82,101,116,117,114,110,115,32,40,108,111,97,100,101,114,44,
-    32,108,105,115,116,45,111,102,45,112,111,114,116,105,111,110,
-    115,41,46,10,10,32,32,32,32,32,32,32,32,84,104,105,
-    115,32,109,101,116,104,111,100,32,105,115,32,100,101,112,114,
-    101,99,97,116,101,100,46,32,32,85,115,101,32,102,105,110,
-    100,95,115,112,101,99,40,41,32,105,110,115,116,101,97,100,
-    46,10,10,32,32,32,32,32,32,32,32,78,41,3,114,182,
-    0,0,0,114,122,0,0,0,114,158,0,0,0,41,3,114,
-    102,0,0,0,114,121,0,0,0,114,166,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,119,0,
-    0,0,25,5,0,0,115,8,0,0,0,0,7,10,1,8,
-    1,8,1,122,22,70,105,108,101,70,105,110,100,101,114,46,
-    102,105,110,100,95,108,111,97,100,101,114,99,6,0,0,0,
-    0,0,0,0,7,0,0,0,6,0,0,0,67,0,0,0,
-    115,26,0,0,0,124,1,124,2,124,3,131,2,125,6,116,
-    0,124,2,124,3,124,6,124,4,100,1,141,4,83,0,41,
-    2,78,41,2,114,122,0,0,0,114,158,0,0,0,41,1,
-    114,169,0,0,0,41,7,114,102,0,0,0,114,167,0,0,
-    0,114,121,0,0,0,114,35,0,0,0,90,4,115,109,115,
-    108,114,181,0,0,0,114,122,0,0,0,114,2,0,0,0,
-    114,2,0,0,0,114,4,0,0,0,114,10,1,0,0,37,
-    5,0,0,115,6,0,0,0,0,1,10,1,8,1,122,20,
-    70,105,108,101,70,105,110,100,101,114,46,95,103,101,116,95,
-    115,112,101,99,78,99,3,0,0,0,0,0,0,0,14,0,
-    0,0,16,0,0,0,67,0,0,0,115,98,1,0,0,100,
-    1,125,3,124,1,160,0,100,2,161,1,100,3,25,0,125,
-    4,121,24,116,1,124,0,106,2,112,34,116,3,160,4,161,
-    0,131,1,106,5,125,5,87,0,110,24,4,0,116,6,107,
-    10,114,66,1,0,1,0,1,0,100,4,125,5,89,0,110,
-    2,88,0,124,5,124,0,106,7,107,3,114,92,124,0,160,
-    8,161,0,1,0,124,5,124,0,95,7,116,9,131,0,114,
-    114,124,0,106,10,125,6,124,4,160,11,161,0,125,7,110,
-    10,124,0,106,12,125,6,124,4,125,7,124,7,124,6,107,
-    6,114,218,116,13,124,0,106,2,124,4,131,2,125,8,120,
-    72,124,0,106,14,68,0,93,54,92,2,125,9,125,10,100,
-    5,124,9,23,0,125,11,116,13,124,8,124,11,131,2,125,
-    12,116,15,124,12,131,1,114,152,124,0,160,16,124,10,124,
-    1,124,12,124,8,103,1,124,2,161,5,83,0,113,152,87,
-    0,116,17,124,8,131,1,125,3,120,88,124,0,106,14,68,
-    0,93,78,92,2,125,9,125,10,116,13,124,0,106,2,124,
-    4,124,9,23,0,131,2,125,12,116,18,106,19,100,6,124,
-    12,100,3,100,7,141,3,1,0,124,7,124,9,23,0,124,
-    6,107,6,114,226,116,15,124,12,131,1,114,226,124,0,160,
-    16,124,10,124,1,124,12,100,8,124,2,161,5,83,0,113,
-    226,87,0,124,3,144,1,114,94,116,18,160,19,100,9,124,
-    8,161,2,1,0,116,18,160,20,124,1,100,8,161,2,125,
-    13,124,8,103,1,124,13,95,21,124,13,83,0,100,8,83,
-    0,41,10,122,111,84,114,121,32,116,111,32,102,105,110,100,
-    32,97,32,115,112,101,99,32,102,111,114,32,116,104,101,32,
-    115,112,101,99,105,102,105,101,100,32,109,111,100,117,108,101,
-    46,10,10,32,32,32,32,32,32,32,32,82,101,116,117,114,
-    110,115,32,116,104,101,32,109,97,116,99,104,105,110,103,32,
-    115,112,101,99,44,32,111,114,32,78,111,110,101,32,105,102,
-    32,110,111,116,32,102,111,117,110,100,46,10,32,32,32,32,
-    32,32,32,32,70,114,59,0,0,0,114,57,0,0,0,114,
-    89,0,0,0,114,186,0,0,0,122,9,116,114,121,105,110,
-    103,32,123,125,41,1,90,9,118,101,114,98,111,115,105,116,
-    121,78,122,25,112,111,115,115,105,98,108,101,32,110,97,109,
-    101,115,112,97,99,101,32,102,111,114,32,123,125,41,22,114,
-    32,0,0,0,114,39,0,0,0,114,35,0,0,0,114,1,
-    0,0,0,114,45,0,0,0,114,221,0,0,0,114,40,0,
-    0,0,114,13,1,0,0,218,11,95,102,105,108,108,95,99,
-    97,99,104,101,114,5,0,0,0,114,16,1,0,0,114,90,
-    0,0,0,114,15,1,0,0,114,28,0,0,0,114,12,1,
-    0,0,114,44,0,0,0,114,10,1,0,0,114,46,0,0,
-    0,114,116,0,0,0,114,130,0,0,0,114,162,0,0,0,
-    114,158,0,0,0,41,14,114,102,0,0,0,114,121,0,0,
-    0,114,181,0,0,0,90,12,105,115,95,110,97,109,101,115,
-    112,97,99,101,90,11,116,97,105,108,95,109,111,100,117,108,
-    101,114,149,0,0,0,90,5,99,97,99,104,101,90,12,99,
-    97,99,104,101,95,109,111,100,117,108,101,90,9,98,97,115,
-    101,95,112,97,116,104,114,227,0,0,0,114,167,0,0,0,
-    90,13,105,110,105,116,95,102,105,108,101,110,97,109,101,90,
-    9,102,117,108,108,95,112,97,116,104,114,166,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,182,
-    0,0,0,42,5,0,0,115,70,0,0,0,0,5,4,1,
-    14,1,2,1,24,1,14,1,10,1,10,1,8,1,6,2,
-    6,1,6,1,10,2,6,1,4,2,8,1,12,1,16,1,
-    8,1,10,1,8,1,24,4,8,2,16,1,16,1,16,1,
-    12,1,8,1,10,1,12,1,6,1,12,1,12,1,8,1,
-    4,1,122,20,70,105,108,101,70,105,110,100,101,114,46,102,
-    105,110,100,95,115,112,101,99,99,1,0,0,0,0,0,0,
-    0,9,0,0,0,14,0,0,0,67,0,0,0,115,194,0,
-    0,0,124,0,106,0,125,1,121,22,116,1,160,2,124,1,
-    112,22,116,1,160,3,161,0,161,1,125,2,87,0,110,30,
-    4,0,116,4,116,5,116,6,102,3,107,10,114,58,1,0,
-    1,0,1,0,103,0,125,2,89,0,110,2,88,0,116,7,
-    106,8,160,9,100,1,161,1,115,84,116,10,124,2,131,1,
-    124,0,95,11,110,78,116,10,131,0,125,3,120,64,124,2,
-    68,0,93,56,125,4,124,4,160,12,100,2,161,1,92,3,
-    125,5,125,6,125,7,124,6,114,138,100,3,160,13,124,5,
-    124,7,160,14,161,0,161,2,125,8,110,4,124,5,125,8,
-    124,3,160,15,124,8,161,1,1,0,113,96,87,0,124,3,
-    124,0,95,11,116,7,106,8,160,9,116,16,161,1,114,190,
-    100,4,100,5,132,0,124,2,68,0,131,1,124,0,95,17,
-    100,6,83,0,41,7,122,68,70,105,108,108,32,116,104,101,
-    32,99,97,99,104,101,32,111,102,32,112,111,116,101,110,116,
-    105,97,108,32,109,111,100,117,108,101,115,32,97,110,100,32,
-    112,97,99,107,97,103,101,115,32,102,111,114,32,116,104,105,
-    115,32,100,105,114,101,99,116,111,114,121,46,114,0,0,0,
-    0,114,59,0,0,0,122,5,123,125,46,123,125,99,1,0,
-    0,0,0,0,0,0,2,0,0,0,4,0,0,0,83,0,
-    0,0,115,20,0,0,0,104,0,124,0,93,12,125,1,124,
-    1,160,0,161,0,146,2,113,4,83,0,114,2,0,0,0,
-    41,1,114,90,0,0,0,41,2,114,22,0,0,0,90,2,
-    102,110,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,250,9,60,115,101,116,99,111,109,112,62,119,5,0,0,
-    115,2,0,0,0,6,0,122,41,70,105,108,101,70,105,110,
-    100,101,114,46,95,102,105,108,108,95,99,97,99,104,101,46,
-    60,108,111,99,97,108,115,62,46,60,115,101,116,99,111,109,
-    112,62,78,41,18,114,35,0,0,0,114,1,0,0,0,90,
-    7,108,105,115,116,100,105,114,114,45,0,0,0,114,4,1,
-    0,0,218,15,80,101,114,109,105,115,115,105,111,110,69,114,
-    114,111,114,218,18,78,111,116,65,68,105,114,101,99,116,111,
-    114,121,69,114,114,111,114,114,6,0,0,0,114,7,0,0,
-    0,114,8,0,0,0,114,14,1,0,0,114,15,1,0,0,
-    114,85,0,0,0,114,48,0,0,0,114,90,0,0,0,218,
-    3,97,100,100,114,9,0,0,0,114,16,1,0,0,41,9,
-    114,102,0,0,0,114,35,0,0,0,90,8,99,111,110,116,
-    101,110,116,115,90,21,108,111,119,101,114,95,115,117,102,102,
-    105,120,95,99,111,110,116,101,110,116,115,114,249,0,0,0,
-    114,100,0,0,0,114,239,0,0,0,114,227,0,0,0,90,
-    8,110,101,119,95,110,97,109,101,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,18,1,0,0,90,5,0,
-    0,115,34,0,0,0,0,2,6,1,2,1,22,1,20,3,
-    10,3,12,1,12,7,6,1,10,1,16,1,4,1,18,2,
-    4,1,14,1,6,1,12,1,122,22,70,105,108,101,70,105,
-    110,100,101,114,46,95,102,105,108,108,95,99,97,99,104,101,
-    99,1,0,0,0,0,0,0,0,3,0,0,0,3,0,0,
-    0,7,0,0,0,115,18,0,0,0,135,0,135,1,102,2,
-    100,1,100,2,132,8,125,2,124,2,83,0,41,3,97,20,
-    1,0,0,65,32,99,108,97,115,115,32,109,101,116,104,111,
-    100,32,119,104,105,99,104,32,114,101,116,117,114,110,115,32,
-    97,32,99,108,111,115,117,114,101,32,116,111,32,117,115,101,
-    32,111,110,32,115,121,115,46,112,97,116,104,95,104,111,111,
-    107,10,32,32,32,32,32,32,32,32,119,104,105,99,104,32,
-    119,105,108,108,32,114,101,116,117,114,110,32,97,110,32,105,
-    110,115,116,97,110,99,101,32,117,115,105,110,103,32,116,104,
-    101,32,115,112,101,99,105,102,105,101,100,32,108,111,97,100,
-    101,114,115,32,97,110,100,32,116,104,101,32,112,97,116,104,
-    10,32,32,32,32,32,32,32,32,99,97,108,108,101,100,32,
-    111,110,32,116,104,101,32,99,108,111,115,117,114,101,46,10,
-    10,32,32,32,32,32,32,32,32,73,102,32,116,104,101,32,
-    112,97,116,104,32,99,97,108,108,101,100,32,111,110,32,116,
-    104,101,32,99,108,111,115,117,114,101,32,105,115,32,110,111,
-    116,32,97,32,100,105,114,101,99,116,111,114,121,44,32,73,
-    109,112,111,114,116,69,114,114,111,114,32,105,115,10,32,32,
-    32,32,32,32,32,32,114,97,105,115,101,100,46,10,10,32,
-    32,32,32,32,32,32,32,99,1,0,0,0,0,0,0,0,
-    1,0,0,0,4,0,0,0,19,0,0,0,115,34,0,0,
-    0,116,0,124,0,131,1,115,20,116,1,100,1,124,0,100,
-    2,141,2,130,1,136,0,124,0,102,1,136,1,158,2,142,
-    0,83,0,41,3,122,45,80,97,116,104,32,104,111,111,107,
-    32,102,111,114,32,105,109,112,111,114,116,108,105,98,46,109,
-    97,99,104,105,110,101,114,121,46,70,105,108,101,70,105,110,
-    100,101,114,46,122,30,111,110,108,121,32,100,105,114,101,99,
-    116,111,114,105,101,115,32,97,114,101,32,115,117,112,112,111,
-    114,116,101,100,41,1,114,35,0,0,0,41,2,114,46,0,
-    0,0,114,101,0,0,0,41,1,114,35,0,0,0,41,2,
-    114,172,0,0,0,114,17,1,0,0,114,2,0,0,0,114,
-    4,0,0,0,218,24,112,97,116,104,95,104,111,111,107,95,
-    102,111,114,95,70,105,108,101,70,105,110,100,101,114,131,5,
-    0,0,115,6,0,0,0,0,2,8,1,12,1,122,54,70,
-    105,108,101,70,105,110,100,101,114,46,112,97,116,104,95,104,
-    111,111,107,46,60,108,111,99,97,108,115,62,46,112,97,116,
-    104,95,104,111,111,107,95,102,111,114,95,70,105,108,101,70,
-    105,110,100,101,114,114,2,0,0,0,41,3,114,172,0,0,
-    0,114,17,1,0,0,114,23,1,0,0,114,2,0,0,0,
-    41,2,114,172,0,0,0,114,17,1,0,0,114,4,0,0,
-    0,218,9,112,97,116,104,95,104,111,111,107,121,5,0,0,
-    115,4,0,0,0,0,10,14,6,122,20,70,105,108,101,70,
-    105,110,100,101,114,46,112,97,116,104,95,104,111,111,107,99,
-    1,0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,
-    67,0,0,0,115,12,0,0,0,100,1,160,0,124,0,106,
-    1,161,1,83,0,41,2,78,122,16,70,105,108,101,70,105,
-    110,100,101,114,40,123,33,114,125,41,41,2,114,48,0,0,
-    0,114,35,0,0,0,41,1,114,102,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,114,248,0,0,
-    0,139,5,0,0,115,2,0,0,0,0,1,122,19,70,105,
-    108,101,70,105,110,100,101,114,46,95,95,114,101,112,114,95,
-    95,41,1,78,41,15,114,107,0,0,0,114,106,0,0,0,
-    114,108,0,0,0,114,109,0,0,0,114,186,0,0,0,114,
-    254,0,0,0,114,125,0,0,0,114,183,0,0,0,114,119,
-    0,0,0,114,10,1,0,0,114,182,0,0,0,114,18,1,
-    0,0,114,184,0,0,0,114,24,1,0,0,114,248,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,11,1,0,0,252,4,0,0,115,18,
-    0,0,0,12,9,8,14,8,4,4,2,8,12,8,5,10,
-    48,8,31,12,18,114,11,1,0,0,99,4,0,0,0,0,
-    0,0,0,6,0,0,0,11,0,0,0,67,0,0,0,115,
-    146,0,0,0,124,0,160,0,100,1,161,1,125,4,124,0,
-    160,0,100,2,161,1,125,5,124,4,115,66,124,5,114,36,
-    124,5,106,1,125,4,110,30,124,2,124,3,107,2,114,56,
-    116,2,124,1,124,2,131,2,125,4,110,10,116,3,124,1,
-    124,2,131,2,125,4,124,5,115,84,116,4,124,1,124,2,
-    124,4,100,3,141,3,125,5,121,36,124,5,124,0,100,2,
-    60,0,124,4,124,0,100,1,60,0,124,2,124,0,100,4,
-    60,0,124,3,124,0,100,5,60,0,87,0,110,20,4,0,
-    116,5,107,10,114,140,1,0,1,0,1,0,89,0,110,2,
-    88,0,100,0,83,0,41,6,78,218,10,95,95,108,111,97,
-    100,101,114,95,95,218,8,95,95,115,112,101,99,95,95,41,
-    1,114,122,0,0,0,90,8,95,95,102,105,108,101,95,95,
-    90,10,95,95,99,97,99,104,101,100,95,95,41,6,218,3,
-    103,101,116,114,122,0,0,0,114,225,0,0,0,114,220,0,
-    0,0,114,169,0,0,0,218,9,69,120,99,101,112,116,105,
-    111,110,41,6,90,2,110,115,114,100,0,0,0,90,8,112,
-    97,116,104,110,97,109,101,90,9,99,112,97,116,104,110,97,
-    109,101,114,122,0,0,0,114,166,0,0,0,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,218,14,95,102,105,
-    120,95,117,112,95,109,111,100,117,108,101,145,5,0,0,115,
-    34,0,0,0,0,2,10,1,10,1,4,1,4,1,8,1,
-    8,1,12,2,10,1,4,1,14,1,2,1,8,1,8,1,
-    8,1,12,1,14,2,114,29,1,0,0,99,0,0,0,0,
-    0,0,0,0,3,0,0,0,3,0,0,0,67,0,0,0,
-    115,38,0,0,0,116,0,116,1,160,2,161,0,102,2,125,
-    0,116,3,116,4,102,2,125,1,116,5,116,6,102,2,125,
-    2,124,0,124,1,124,2,103,3,83,0,41,1,122,95,82,
-    101,116,117,114,110,115,32,97,32,108,105,115,116,32,111,102,
-    32,102,105,108,101,45,98,97,115,101,100,32,109,111,100,117,
-    108,101,32,108,111,97,100,101,114,115,46,10,10,32,32,32,
-    32,69,97,99,104,32,105,116,101,109,32,105,115,32,97,32,
-    116,117,112,108,101,32,40,108,111,97,100,101,114,44,32,115,
-    117,102,102,105,120,101,115,41,46,10,32,32,32,32,41,7,
-    114,226,0,0,0,114,143,0,0,0,218,18,101,120,116,101,
-    110,115,105,111,110,95,115,117,102,102,105,120,101,115,114,220,
-    0,0,0,114,86,0,0,0,114,225,0,0,0,114,76,0,
-    0,0,41,3,90,10,101,120,116,101,110,115,105,111,110,115,
-    90,6,115,111,117,114,99,101,90,8,98,121,116,101,99,111,
-    100,101,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,163,0,0,0,168,5,0,0,115,8,0,0,0,0,
-    5,12,1,8,1,8,1,114,163,0,0,0,99,1,0,0,
-    0,0,0,0,0,12,0,0,0,12,0,0,0,67,0,0,
-    0,115,156,1,0,0,124,0,97,0,116,0,106,1,97,1,
-    116,0,106,2,97,2,116,1,106,3,116,4,25,0,125,1,
-    120,56,100,1,68,0,93,48,125,2,124,2,116,1,106,3,
-    107,7,114,58,116,0,160,5,124,2,161,1,125,3,110,10,
-    116,1,106,3,124,2,25,0,125,3,116,6,124,1,124,2,
-    124,3,131,3,1,0,113,32,87,0,100,2,100,3,103,1,
-    102,2,100,4,100,5,100,3,103,2,102,2,102,2,125,4,
-    120,118,124,4,68,0,93,102,92,2,125,5,125,6,116,7,
-    100,6,100,7,132,0,124,6,68,0,131,1,131,1,115,142,
-    116,8,130,1,124,6,100,8,25,0,125,7,124,5,116,1,
-    106,3,107,6,114,174,116,1,106,3,124,5,25,0,125,8,
-    80,0,113,112,121,16,116,0,160,5,124,5,161,1,125,8,
-    80,0,87,0,113,112,4,0,116,9,107,10,114,212,1,0,
-    1,0,1,0,119,112,89,0,113,112,88,0,113,112,87,0,
-    116,9,100,9,131,1,130,1,116,6,124,1,100,10,124,8,
-    131,3,1,0,116,6,124,1,100,11,124,7,131,3,1,0,
-    116,6,124,1,100,12,100,13,160,10,124,6,161,1,131,3,
-    1,0,116,0,160,5,100,14,161,1,125,9,116,6,124,1,
-    100,14,124,9,131,3,1,0,116,0,160,5,100,15,161,1,
-    125,10,116,6,124,1,100,15,124,10,131,3,1,0,124,5,
-    100,4,107,2,144,1,114,88,116,0,160,5,100,16,161,1,
-    125,11,116,6,124,1,100,17,124,11,131,3,1,0,116,6,
-    124,1,100,18,116,11,131,0,131,3,1,0,116,12,160,13,
-    116,2,160,14,161,0,161,1,1,0,124,5,100,4,107,2,
-    144,1,114,152,116,15,160,16,100,19,161,1,1,0,100,20,
-    116,12,107,6,144,1,114,152,100,21,116,17,95,18,100,22,
-    83,0,41,23,122,205,83,101,116,117,112,32,116,104,101,32,
+    114,182,0,0,0,42,5,0,0,115,70,0,0,0,0,5,
+    4,1,14,1,2,1,24,1,14,1,10,1,10,1,8,1,
+    6,2,6,1,6,1,10,2,6,1,4,2,8,1,12,1,
+    16,1,8,1,10,1,8,1,24,4,8,2,16,1,16,1,
+    16,1,12,1,8,1,10,1,12,1,6,1,12,1,12,1,
+    8,1,4,1,122,20,70,105,108,101,70,105,110,100,101,114,
+    46,102,105,110,100,95,115,112,101,99,99,1,0,0,0,0,
+    0,0,0,9,0,0,0,10,0,0,0,67,0,0,0,115,
+    194,0,0,0,124,0,106,0,125,1,121,22,116,1,160,2,
+    124,1,112,22,116,1,160,3,161,0,161,1,125,2,87,0,
+    110,30,4,0,116,4,116,5,116,6,102,3,107,10,114,58,
+    1,0,1,0,1,0,103,0,125,2,89,0,110,2,88,0,
+    116,7,106,8,160,9,100,1,161,1,115,84,116,10,124,2,
+    131,1,124,0,95,11,110,78,116,10,131,0,125,3,120,64,
+    124,2,68,0,93,56,125,4,124,4,160,12,100,2,161,1,
+    92,3,125,5,125,6,125,7,124,6,114,138,100,3,160,13,
+    124,5,124,7,160,14,161,0,161,2,125,8,110,4,124,5,
+    125,8,124,3,160,15,124,8,161,1,1,0,113,96,87,0,
+    124,3,124,0,95,11,116,7,106,8,160,9,116,16,161,1,
+    114,190,100,4,100,5,132,0,124,2,68,0,131,1,124,0,
+    95,17,100,6,83,0,41,7,122,68,70,105,108,108,32,116,
+    104,101,32,99,97,99,104,101,32,111,102,32,112,111,116,101,
+    110,116,105,97,108,32,109,111,100,117,108,101,115,32,97,110,
+    100,32,112,97,99,107,97,103,101,115,32,102,111,114,32,116,
+    104,105,115,32,100,105,114,101,99,116,111,114,121,46,114,0,
+    0,0,0,114,59,0,0,0,122,5,123,125,46,123,125,99,
+    1,0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,
+    83,0,0,0,115,20,0,0,0,104,0,124,0,93,12,125,
+    1,124,1,160,0,161,0,146,2,113,4,83,0,114,2,0,
+    0,0,41,1,114,90,0,0,0,41,2,114,22,0,0,0,
+    90,2,102,110,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,250,9,60,115,101,116,99,111,109,112,62,119,5,
+    0,0,115,2,0,0,0,6,0,122,41,70,105,108,101,70,
+    105,110,100,101,114,46,95,102,105,108,108,95,99,97,99,104,
+    101,46,60,108,111,99,97,108,115,62,46,60,115,101,116,99,
+    111,109,112,62,78,41,18,114,35,0,0,0,114,1,0,0,
+    0,90,7,108,105,115,116,100,105,114,114,45,0,0,0,114,
+    4,1,0,0,218,15,80,101,114,109,105,115,115,105,111,110,
+    69,114,114,111,114,218,18,78,111,116,65,68,105,114,101,99,
+    116,111,114,121,69,114,114,111,114,114,6,0,0,0,114,7,
+    0,0,0,114,8,0,0,0,114,14,1,0,0,114,15,1,
+    0,0,114,85,0,0,0,114,48,0,0,0,114,90,0,0,
+    0,218,3,97,100,100,114,9,0,0,0,114,16,1,0,0,
+    41,9,114,102,0,0,0,114,35,0,0,0,90,8,99,111,
+    110,116,101,110,116,115,90,21,108,111,119,101,114,95,115,117,
+    102,102,105,120,95,99,111,110,116,101,110,116,115,114,249,0,
+    0,0,114,100,0,0,0,114,239,0,0,0,114,227,0,0,
+    0,90,8,110,101,119,95,110,97,109,101,114,2,0,0,0,
+    114,2,0,0,0,114,4,0,0,0,114,18,1,0,0,90,
+    5,0,0,115,34,0,0,0,0,2,6,1,2,1,22,1,
+    20,3,10,3,12,1,12,7,6,1,10,1,16,1,4,1,
+    18,2,4,1,14,1,6,1,12,1,122,22,70,105,108,101,
+    70,105,110,100,101,114,46,95,102,105,108,108,95,99,97,99,
+    104,101,99,1,0,0,0,0,0,0,0,3,0,0,0,3,
+    0,0,0,7,0,0,0,115,18,0,0,0,135,0,135,1,
+    102,2,100,1,100,2,132,8,125,2,124,2,83,0,41,3,
+    97,20,1,0,0,65,32,99,108,97,115,115,32,109,101,116,
+    104,111,100,32,119,104,105,99,104,32,114,101,116,117,114,110,
+    115,32,97,32,99,108,111,115,117,114,101,32,116,111,32,117,
+    115,101,32,111,110,32,115,121,115,46,112,97,116,104,95,104,
+    111,111,107,10,32,32,32,32,32,32,32,32,119,104,105,99,
+    104,32,119,105,108,108,32,114,101,116,117,114,110,32,97,110,
+    32,105,110,115,116,97,110,99,101,32,117,115,105,110,103,32,
+    116,104,101,32,115,112,101,99,105,102,105,101,100,32,108,111,
+    97,100,101,114,115,32,97,110,100,32,116,104,101,32,112,97,
+    116,104,10,32,32,32,32,32,32,32,32,99,97,108,108,101,
+    100,32,111,110,32,116,104,101,32,99,108,111,115,117,114,101,
+    46,10,10,32,32,32,32,32,32,32,32,73,102,32,116,104,
+    101,32,112,97,116,104,32,99,97,108,108,101,100,32,111,110,
+    32,116,104,101,32,99,108,111,115,117,114,101,32,105,115,32,
+    110,111,116,32,97,32,100,105,114,101,99,116,111,114,121,44,
+    32,73,109,112,111,114,116,69,114,114,111,114,32,105,115,10,
+    32,32,32,32,32,32,32,32,114,97,105,115,101,100,46,10,
+    10,32,32,32,32,32,32,32,32,99,1,0,0,0,0,0,
+    0,0,1,0,0,0,4,0,0,0,19,0,0,0,115,34,
+    0,0,0,116,0,124,0,131,1,115,20,116,1,100,1,124,
+    0,100,2,141,2,130,1,136,0,124,0,102,1,136,1,158,
+    2,142,0,83,0,41,3,122,45,80,97,116,104,32,104,111,
+    111,107,32,102,111,114,32,105,109,112,111,114,116,108,105,98,
+    46,109,97,99,104,105,110,101,114,121,46,70,105,108,101,70,
+    105,110,100,101,114,46,122,30,111,110,108,121,32,100,105,114,
+    101,99,116,111,114,105,101,115,32,97,114,101,32,115,117,112,
+    112,111,114,116,101,100,41,1,114,35,0,0,0,41,2,114,
+    46,0,0,0,114,101,0,0,0,41,1,114,35,0,0,0,
+    41,2,114,172,0,0,0,114,17,1,0,0,114,2,0,0,
+    0,114,4,0,0,0,218,24,112,97,116,104,95,104,111,111,
+    107,95,102,111,114,95,70,105,108,101,70,105,110,100,101,114,
+    131,5,0,0,115,6,0,0,0,0,2,8,1,12,1,122,
+    54,70,105,108,101,70,105,110,100,101,114,46,112,97,116,104,
+    95,104,111,111,107,46,60,108,111,99,97,108,115,62,46,112,
+    97,116,104,95,104,111,111,107,95,102,111,114,95,70,105,108,
+    101,70,105,110,100,101,114,114,2,0,0,0,41,3,114,172,
+    0,0,0,114,17,1,0,0,114,23,1,0,0,114,2,0,
+    0,0,41,2,114,172,0,0,0,114,17,1,0,0,114,4,
+    0,0,0,218,9,112,97,116,104,95,104,111,111,107,121,5,
+    0,0,115,4,0,0,0,0,10,14,6,122,20,70,105,108,
+    101,70,105,110,100,101,114,46,112,97,116,104,95,104,111,111,
+    107,99,1,0,0,0,0,0,0,0,1,0,0,0,3,0,
+    0,0,67,0,0,0,115,12,0,0,0,100,1,160,0,124,
+    0,106,1,161,1,83,0,41,2,78,122,16,70,105,108,101,
+    70,105,110,100,101,114,40,123,33,114,125,41,41,2,114,48,
+    0,0,0,114,35,0,0,0,41,1,114,102,0,0,0,114,
+    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,248,
+    0,0,0,139,5,0,0,115,2,0,0,0,0,1,122,19,
+    70,105,108,101,70,105,110,100,101,114,46,95,95,114,101,112,
+    114,95,95,41,1,78,41,15,114,107,0,0,0,114,106,0,
+    0,0,114,108,0,0,0,114,109,0,0,0,114,186,0,0,
+    0,114,254,0,0,0,114,125,0,0,0,114,183,0,0,0,
+    114,119,0,0,0,114,10,1,0,0,114,182,0,0,0,114,
+    18,1,0,0,114,184,0,0,0,114,24,1,0,0,114,248,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,114,11,1,0,0,252,4,0,0,
+    115,18,0,0,0,12,9,8,14,8,4,4,2,8,12,8,
+    5,10,48,8,31,12,18,114,11,1,0,0,99,4,0,0,
+    0,0,0,0,0,6,0,0,0,8,0,0,0,67,0,0,
+    0,115,146,0,0,0,124,0,160,0,100,1,161,1,125,4,
+    124,0,160,0,100,2,161,1,125,5,124,4,115,66,124,5,
+    114,36,124,5,106,1,125,4,110,30,124,2,124,3,107,2,
+    114,56,116,2,124,1,124,2,131,2,125,4,110,10,116,3,
+    124,1,124,2,131,2,125,4,124,5,115,84,116,4,124,1,
+    124,2,124,4,100,3,141,3,125,5,121,36,124,5,124,0,
+    100,2,60,0,124,4,124,0,100,1,60,0,124,2,124,0,
+    100,4,60,0,124,3,124,0,100,5,60,0,87,0,110,20,
+    4,0,116,5,107,10,114,140,1,0,1,0,1,0,89,0,
+    110,2,88,0,100,0,83,0,41,6,78,218,10,95,95,108,
+    111,97,100,101,114,95,95,218,8,95,95,115,112,101,99,95,
+    95,41,1,114,122,0,0,0,90,8,95,95,102,105,108,101,
+    95,95,90,10,95,95,99,97,99,104,101,100,95,95,41,6,
+    218,3,103,101,116,114,122,0,0,0,114,225,0,0,0,114,
+    220,0,0,0,114,169,0,0,0,218,9,69,120,99,101,112,
+    116,105,111,110,41,6,90,2,110,115,114,100,0,0,0,90,
+    8,112,97,116,104,110,97,109,101,90,9,99,112,97,116,104,
+    110,97,109,101,114,122,0,0,0,114,166,0,0,0,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,218,14,95,
+    102,105,120,95,117,112,95,109,111,100,117,108,101,145,5,0,
+    0,115,34,0,0,0,0,2,10,1,10,1,4,1,4,1,
+    8,1,8,1,12,2,10,1,4,1,14,1,2,1,8,1,
+    8,1,8,1,12,1,14,2,114,29,1,0,0,99,0,0,
+    0,0,0,0,0,0,3,0,0,0,3,0,0,0,67,0,
+    0,0,115,38,0,0,0,116,0,116,1,160,2,161,0,102,
+    2,125,0,116,3,116,4,102,2,125,1,116,5,116,6,102,
+    2,125,2,124,0,124,1,124,2,103,3,83,0,41,1,122,
+    95,82,101,116,117,114,110,115,32,97,32,108,105,115,116,32,
+    111,102,32,102,105,108,101,45,98,97,115,101,100,32,109,111,
+    100,117,108,101,32,108,111,97,100,101,114,115,46,10,10,32,
+    32,32,32,69,97,99,104,32,105,116,101,109,32,105,115,32,
+    97,32,116,117,112,108,101,32,40,108,111,97,100,101,114,44,
+    32,115,117,102,102,105,120,101,115,41,46,10,32,32,32,32,
+    41,7,114,226,0,0,0,114,143,0,0,0,218,18,101,120,
+    116,101,110,115,105,111,110,95,115,117,102,102,105,120,101,115,
+    114,220,0,0,0,114,86,0,0,0,114,225,0,0,0,114,
+    76,0,0,0,41,3,90,10,101,120,116,101,110,115,105,111,
+    110,115,90,6,115,111,117,114,99,101,90,8,98,121,116,101,
+    99,111,100,101,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,114,163,0,0,0,168,5,0,0,115,8,0,0,
+    0,0,5,12,1,8,1,8,1,114,163,0,0,0,99,1,
+    0,0,0,0,0,0,0,12,0,0,0,9,0,0,0,67,
+    0,0,0,115,156,1,0,0,124,0,97,0,116,0,106,1,
+    97,1,116,0,106,2,97,2,116,1,106,3,116,4,25,0,
+    125,1,120,56,100,1,68,0,93,48,125,2,124,2,116,1,
+    106,3,107,7,114,58,116,0,160,5,124,2,161,1,125,3,
+    110,10,116,1,106,3,124,2,25,0,125,3,116,6,124,1,
+    124,2,124,3,131,3,1,0,113,32,87,0,100,2,100,3,
+    103,1,102,2,100,4,100,5,100,3,103,2,102,2,102,2,
+    125,4,120,118,124,4,68,0,93,102,92,2,125,5,125,6,
+    116,7,100,6,100,7,132,0,124,6,68,0,131,1,131,1,
+    115,142,116,8,130,1,124,6,100,8,25,0,125,7,124,5,
+    116,1,106,3,107,6,114,174,116,1,106,3,124,5,25,0,
+    125,8,80,0,113,112,121,16,116,0,160,5,124,5,161,1,
+    125,8,80,0,87,0,113,112,4,0,116,9,107,10,114,212,
+    1,0,1,0,1,0,119,112,89,0,113,112,88,0,113,112,
+    87,0,116,9,100,9,131,1,130,1,116,6,124,1,100,10,
+    124,8,131,3,1,0,116,6,124,1,100,11,124,7,131,3,
+    1,0,116,6,124,1,100,12,100,13,160,10,124,6,161,1,
+    131,3,1,0,116,0,160,5,100,14,161,1,125,9,116,6,
+    124,1,100,14,124,9,131,3,1,0,116,0,160,5,100,15,
+    161,1,125,10,116,6,124,1,100,15,124,10,131,3,1,0,
+    124,5,100,4,107,2,144,1,114,88,116,0,160,5,100,16,
+    161,1,125,11,116,6,124,1,100,17,124,11,131,3,1,0,
+    116,6,124,1,100,18,116,11,131,0,131,3,1,0,116,12,
+    160,13,116,2,160,14,161,0,161,1,1,0,124,5,100,4,
+    107,2,144,1,114,152,116,15,160,16,100,19,161,1,1,0,
+    100,20,116,12,107,6,144,1,114,152,100,21,116,17,95,18,
+    100,22,83,0,41,23,122,205,83,101,116,117,112,32,116,104,
+    101,32,112,97,116,104,45,98,97,115,101,100,32,105,109,112,
+    111,114,116,101,114,115,32,102,111,114,32,105,109,112,111,114,
+    116,108,105,98,32,98,121,32,105,109,112,111,114,116,105,110,
+    103,32,110,101,101,100,101,100,10,32,32,32,32,98,117,105,
+    108,116,45,105,110,32,109,111,100,117,108,101,115,32,97,110,
+    100,32,105,110,106,101,99,116,105,110,103,32,116,104,101,109,
+    32,105,110,116,111,32,116,104,101,32,103,108,111,98,97,108,
+    32,110,97,109,101,115,112,97,99,101,46,10,10,32,32,32,
+    32,79,116,104,101,114,32,99,111,109,112,111,110,101,110,116,
+    115,32,97,114,101,32,101,120,116,114,97,99,116,101,100,32,
+    102,114,111,109,32,116,104,101,32,99,111,114,101,32,98,111,
+    111,116,115,116,114,97,112,32,109,111,100,117,108,101,46,10,
+    10,32,32,32,32,41,4,114,50,0,0,0,114,61,0,0,
+    0,218,8,98,117,105,108,116,105,110,115,114,140,0,0,0,
+    90,5,112,111,115,105,120,250,1,47,90,2,110,116,250,1,
+    92,99,1,0,0,0,0,0,0,0,2,0,0,0,3,0,
+    0,0,115,0,0,0,115,26,0,0,0,124,0,93,18,125,
+    1,116,0,124,1,131,1,100,0,107,2,86,0,1,0,113,
+    2,100,1,83,0,41,2,114,29,0,0,0,78,41,1,114,
+    31,0,0,0,41,2,114,22,0,0,0,114,79,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
+    229,0,0,0,204,5,0,0,115,2,0,0,0,4,0,122,
+    25,95,115,101,116,117,112,46,60,108,111,99,97,108,115,62,
+    46,60,103,101,110,101,120,112,114,62,114,60,0,0,0,122,
+    30,105,109,112,111,114,116,108,105,98,32,114,101,113,117,105,
+    114,101,115,32,112,111,115,105,120,32,111,114,32,110,116,114,
+    1,0,0,0,114,25,0,0,0,114,21,0,0,0,114,30,
+    0,0,0,90,7,95,116,104,114,101,97,100,90,8,95,119,
+    101,97,107,114,101,102,90,6,119,105,110,114,101,103,114,171,
+    0,0,0,114,5,0,0,0,122,4,46,112,121,119,122,6,
+    95,100,46,112,121,100,84,78,41,19,114,116,0,0,0,114,
+    6,0,0,0,114,143,0,0,0,114,241,0,0,0,114,107,
+    0,0,0,90,18,95,98,117,105,108,116,105,110,95,102,114,
+    111,109,95,110,97,109,101,114,111,0,0,0,218,3,97,108,
+    108,114,151,0,0,0,114,101,0,0,0,114,26,0,0,0,
+    114,11,0,0,0,114,231,0,0,0,114,147,0,0,0,114,
+    30,1,0,0,114,86,0,0,0,114,165,0,0,0,114,170,
+    0,0,0,114,174,0,0,0,41,12,218,17,95,98,111,111,
+    116,115,116,114,97,112,95,109,111,100,117,108,101,90,11,115,
+    101,108,102,95,109,111,100,117,108,101,90,12,98,117,105,108,
+    116,105,110,95,110,97,109,101,90,14,98,117,105,108,116,105,
+    110,95,109,111,100,117,108,101,90,10,111,115,95,100,101,116,
+    97,105,108,115,90,10,98,117,105,108,116,105,110,95,111,115,
+    114,21,0,0,0,114,25,0,0,0,90,9,111,115,95,109,
+    111,100,117,108,101,90,13,116,104,114,101,97,100,95,109,111,
+    100,117,108,101,90,14,119,101,97,107,114,101,102,95,109,111,
+    100,117,108,101,90,13,119,105,110,114,101,103,95,109,111,100,
+    117,108,101,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,218,6,95,115,101,116,117,112,179,5,0,0,115,76,
+    0,0,0,0,8,4,1,6,1,6,3,10,1,10,1,10,
+    1,12,2,10,1,16,3,22,1,14,2,22,1,8,1,10,
+    1,10,1,4,2,2,1,10,1,6,1,14,1,12,2,8,
+    1,12,1,12,1,18,3,10,1,12,3,10,1,12,3,10,
+    1,10,1,12,3,14,1,14,1,10,1,10,1,10,1,114,
+    36,1,0,0,99,1,0,0,0,0,0,0,0,2,0,0,
+    0,4,0,0,0,67,0,0,0,115,50,0,0,0,116,0,
+    124,0,131,1,1,0,116,1,131,0,125,1,116,2,106,3,
+    160,4,116,5,106,6,124,1,142,0,103,1,161,1,1,0,
+    116,2,106,7,160,8,116,9,161,1,1,0,100,1,83,0,
+    41,2,122,41,73,110,115,116,97,108,108,32,116,104,101,32,
     112,97,116,104,45,98,97,115,101,100,32,105,109,112,111,114,
-    116,101,114,115,32,102,111,114,32,105,109,112,111,114,116,108,
-    105,98,32,98,121,32,105,109,112,111,114,116,105,110,103,32,
-    110,101,101,100,101,100,10,32,32,32,32,98,117,105,108,116,
-    45,105,110,32,109,111,100,117,108,101,115,32,97,110,100,32,
-    105,110,106,101,99,116,105,110,103,32,116,104,101,109,32,105,
-    110,116,111,32,116,104,101,32,103,108,111,98,97,108,32,110,
-    97,109,101,115,112,97,99,101,46,10,10,32,32,32,32,79,
-    116,104,101,114,32,99,111,109,112,111,110,101,110,116,115,32,
-    97,114,101,32,101,120,116,114,97,99,116,101,100,32,102,114,
-    111,109,32,116,104,101,32,99,111,114,101,32,98,111,111,116,
-    115,116,114,97,112,32,109,111,100,117,108,101,46,10,10,32,
-    32,32,32,41,4,114,50,0,0,0,114,61,0,0,0,218,
-    8,98,117,105,108,116,105,110,115,114,140,0,0,0,90,5,
-    112,111,115,105,120,250,1,47,90,2,110,116,250,1,92,99,
-    1,0,0,0,0,0,0,0,2,0,0,0,3,0,0,0,
-    115,0,0,0,115,26,0,0,0,124,0,93,18,125,1,116,
-    0,124,1,131,1,100,0,107,2,86,0,1,0,113,2,100,
-    1,83,0,41,2,114,29,0,0,0,78,41,1,114,31,0,
-    0,0,41,2,114,22,0,0,0,114,79,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,229,0,
-    0,0,204,5,0,0,115,2,0,0,0,4,0,122,25,95,
-    115,101,116,117,112,46,60,108,111,99,97,108,115,62,46,60,
-    103,101,110,101,120,112,114,62,114,60,0,0,0,122,30,105,
-    109,112,111,114,116,108,105,98,32,114,101,113,117,105,114,101,
-    115,32,112,111,115,105,120,32,111,114,32,110,116,114,1,0,
-    0,0,114,25,0,0,0,114,21,0,0,0,114,30,0,0,
-    0,90,7,95,116,104,114,101,97,100,90,8,95,119,101,97,
-    107,114,101,102,90,6,119,105,110,114,101,103,114,171,0,0,
-    0,114,5,0,0,0,122,4,46,112,121,119,122,6,95,100,
-    46,112,121,100,84,78,41,19,114,116,0,0,0,114,6,0,
-    0,0,114,143,0,0,0,114,241,0,0,0,114,107,0,0,
-    0,90,18,95,98,117,105,108,116,105,110,95,102,114,111,109,
-    95,110,97,109,101,114,111,0,0,0,218,3,97,108,108,114,
-    151,0,0,0,114,101,0,0,0,114,26,0,0,0,114,11,
-    0,0,0,114,231,0,0,0,114,147,0,0,0,114,30,1,
-    0,0,114,86,0,0,0,114,165,0,0,0,114,170,0,0,
-    0,114,174,0,0,0,41,12,218,17,95,98,111,111,116,115,
-    116,114,97,112,95,109,111,100,117,108,101,90,11,115,101,108,
-    102,95,109,111,100,117,108,101,90,12,98,117,105,108,116,105,
-    110,95,110,97,109,101,90,14,98,117,105,108,116,105,110,95,
-    109,111,100,117,108,101,90,10,111,115,95,100,101,116,97,105,
-    108,115,90,10,98,117,105,108,116,105,110,95,111,115,114,21,
-    0,0,0,114,25,0,0,0,90,9,111,115,95,109,111,100,
-    117,108,101,90,13,116,104,114,101,97,100,95,109,111,100,117,
-    108,101,90,14,119,101,97,107,114,101,102,95,109,111,100,117,
-    108,101,90,13,119,105,110,114,101,103,95,109,111,100,117,108,
-    101,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,6,95,115,101,116,117,112,179,5,0,0,115,76,0,0,
-    0,0,8,4,1,6,1,6,3,10,1,10,1,10,1,12,
-    2,10,1,16,3,22,1,14,2,22,1,8,1,10,1,10,
-    1,4,2,2,1,10,1,6,1,14,1,12,2,8,1,12,
-    1,12,1,18,3,10,1,12,3,10,1,12,3,10,1,10,
-    1,12,3,14,1,14,1,10,1,10,1,10,1,114,36,1,
-    0,0,99,1,0,0,0,0,0,0,0,2,0,0,0,4,
-    0,0,0,67,0,0,0,115,50,0,0,0,116,0,124,0,
-    131,1,1,0,116,1,131,0,125,1,116,2,106,3,160,4,
-    116,5,106,6,124,1,142,0,103,1,161,1,1,0,116,2,
-    106,7,160,8,116,9,161,1,1,0,100,1,83,0,41,2,
-    122,41,73,110,115,116,97,108,108,32,116,104,101,32,112,97,
-    116,104,45,98,97,115,101,100,32,105,109,112,111,114,116,32,
-    99,111,109,112,111,110,101,110,116,115,46,78,41,10,114,36,
-    1,0,0,114,163,0,0,0,114,6,0,0,0,114,2,1,
-    0,0,114,147,0,0,0,114,11,1,0,0,114,24,1,0,
-    0,218,9,109,101,116,97,95,112,97,116,104,114,165,0,0,
-    0,114,253,0,0,0,41,2,114,35,1,0,0,90,17,115,
-    117,112,112,111,114,116,101,100,95,108,111,97,100,101,114,115,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    8,95,105,110,115,116,97,108,108,243,5,0,0,115,8,0,
-    0,0,0,2,8,1,6,1,20,1,114,38,1,0,0,41,
-    1,114,47,0,0,0,41,1,78,41,3,78,78,78,41,2,
-    114,60,0,0,0,114,60,0,0,0,41,1,84,41,1,78,
-    41,1,78,41,61,114,109,0,0,0,114,10,0,0,0,90,
-    37,95,67,65,83,69,95,73,78,83,69,78,83,73,84,73,
-    86,69,95,80,76,65,84,70,79,82,77,83,95,66,89,84,
-    69,83,95,75,69,89,114,9,0,0,0,114,11,0,0,0,
-    114,17,0,0,0,114,19,0,0,0,114,28,0,0,0,114,
-    38,0,0,0,114,39,0,0,0,114,43,0,0,0,114,44,
-    0,0,0,114,46,0,0,0,114,56,0,0,0,218,4,116,
-    121,112,101,218,8,95,95,99,111,100,101,95,95,114,142,0,
-    0,0,114,15,0,0,0,114,129,0,0,0,114,14,0,0,
-    0,114,18,0,0,0,114,209,0,0,0,114,75,0,0,0,
-    114,74,0,0,0,114,86,0,0,0,114,76,0,0,0,90,
-    23,68,69,66,85,71,95,66,89,84,69,67,79,68,69,95,
-    83,85,70,70,73,88,69,83,90,27,79,80,84,73,77,73,
-    90,69,68,95,66,89,84,69,67,79,68,69,95,83,85,70,
-    70,73,88,69,83,114,81,0,0,0,114,87,0,0,0,114,
-    93,0,0,0,114,97,0,0,0,114,99,0,0,0,114,118,
-    0,0,0,114,125,0,0,0,114,133,0,0,0,114,137,0,
-    0,0,114,139,0,0,0,114,145,0,0,0,114,150,0,0,
-    0,114,152,0,0,0,114,157,0,0,0,218,6,111,98,106,
-    101,99,116,114,164,0,0,0,114,169,0,0,0,114,170,0,
-    0,0,114,185,0,0,0,114,195,0,0,0,114,212,0,0,
-    0,114,220,0,0,0,114,225,0,0,0,114,231,0,0,0,
-    114,226,0,0,0,114,232,0,0,0,114,251,0,0,0,114,
-    253,0,0,0,114,11,1,0,0,114,29,1,0,0,114,163,
-    0,0,0,114,36,1,0,0,114,38,1,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,8,60,109,111,100,117,108,101,62,24,0,0,0,115,
-    116,0,0,0,4,0,4,1,4,1,2,1,6,3,8,17,
-    8,5,8,5,8,6,8,12,8,10,8,9,8,5,8,7,
-    10,22,10,126,16,1,12,2,4,1,4,2,6,2,6,2,
-    8,2,16,45,8,34,8,19,8,12,8,12,8,28,8,17,
-    8,33,8,28,8,24,10,13,10,10,10,11,8,14,6,3,
-    4,1,14,67,14,64,14,29,16,127,0,17,14,41,18,45,
-    18,26,4,3,18,53,14,60,14,42,14,127,0,5,14,127,
-    0,22,10,23,8,11,8,64,
+    116,32,99,111,109,112,111,110,101,110,116,115,46,78,41,10,
+    114,36,1,0,0,114,163,0,0,0,114,6,0,0,0,114,
+    2,1,0,0,114,147,0,0,0,114,11,1,0,0,114,24,
+    1,0,0,218,9,109,101,116,97,95,112,97,116,104,114,165,
+    0,0,0,114,253,0,0,0,41,2,114,35,1,0,0,90,
+    17,115,117,112,112,111,114,116,101,100,95,108,111,97,100,101,
+    114,115,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,218,8,95,105,110,115,116,97,108,108,243,5,0,0,115,
+    8,0,0,0,0,2,8,1,6,1,20,1,114,38,1,0,
+    0,41,1,114,47,0,0,0,41,1,78,41,3,78,78,78,
+    41,2,114,60,0,0,0,114,60,0,0,0,41,1,84,41,
+    1,78,41,1,78,41,61,114,109,0,0,0,114,10,0,0,
+    0,90,37,95,67,65,83,69,95,73,78,83,69,78,83,73,
+    84,73,86,69,95,80,76,65,84,70,79,82,77,83,95,66,
+    89,84,69,83,95,75,69,89,114,9,0,0,0,114,11,0,
+    0,0,114,17,0,0,0,114,19,0,0,0,114,28,0,0,
+    0,114,38,0,0,0,114,39,0,0,0,114,43,0,0,0,
+    114,44,0,0,0,114,46,0,0,0,114,56,0,0,0,218,
+    4,116,121,112,101,218,8,95,95,99,111,100,101,95,95,114,
+    142,0,0,0,114,15,0,0,0,114,129,0,0,0,114,14,
+    0,0,0,114,18,0,0,0,114,209,0,0,0,114,75,0,
+    0,0,114,74,0,0,0,114,86,0,0,0,114,76,0,0,
+    0,90,23,68,69,66,85,71,95,66,89,84,69,67,79,68,
+    69,95,83,85,70,70,73,88,69,83,90,27,79,80,84,73,
+    77,73,90,69,68,95,66,89,84,69,67,79,68,69,95,83,
+    85,70,70,73,88,69,83,114,81,0,0,0,114,87,0,0,
+    0,114,93,0,0,0,114,97,0,0,0,114,99,0,0,0,
+    114,118,0,0,0,114,125,0,0,0,114,133,0,0,0,114,
+    137,0,0,0,114,139,0,0,0,114,145,0,0,0,114,150,
+    0,0,0,114,152,0,0,0,114,157,0,0,0,218,6,111,
+    98,106,101,99,116,114,164,0,0,0,114,169,0,0,0,114,
+    170,0,0,0,114,185,0,0,0,114,195,0,0,0,114,212,
+    0,0,0,114,220,0,0,0,114,225,0,0,0,114,231,0,
+    0,0,114,226,0,0,0,114,232,0,0,0,114,251,0,0,
+    0,114,253,0,0,0,114,11,1,0,0,114,29,1,0,0,
+    114,163,0,0,0,114,36,1,0,0,114,38,1,0,0,114,
+    2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,218,8,60,109,111,100,117,108,101,62,24,0,0,
+    0,115,116,0,0,0,4,0,4,1,4,1,2,1,6,3,
+    8,17,8,5,8,5,8,6,8,12,8,10,8,9,8,5,
+    8,7,10,22,10,126,16,1,12,2,4,1,4,2,6,2,
+    6,2,8,2,16,45,8,34,8,19,8,12,8,12,8,28,
+    8,17,8,33,8,28,8,24,10,13,10,10,10,11,8,14,
+    6,3,4,1,14,67,14,64,14,29,16,127,0,17,14,41,
+    18,45,18,26,4,3,18,53,14,60,14,42,14,127,0,5,
+    14,127,0,22,10,23,8,11,8,64,
 };


From webhook-mailer at python.org  Tue Jan  9 16:37:29 2018
From: webhook-mailer at python.org (Alexander Belopolsky)
Date: Tue, 09 Jan 2018 21:37:29 -0000
Subject: [Python-checkins] Fix when parsing tz offsets microseconds shorter
 than 6 (#4781)
Message-ID: <mailman.10.1515533850.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/f80c0ca13330112fe4d8018609c085ef556cb5bf
commit: f80c0ca13330112fe4d8018609c085ef556cb5bf
branch: master
author: Mario Corchero <mariocj89 at gmail.com>
committer: Alexander Belopolsky <abalkin at users.noreply.github.com>
date: 2018-01-09T16:37:26-05:00
summary:

Fix when parsing tz offsets microseconds shorter than 6 (#4781)

As the remainder was directly parsed as an int, strings like
.600 were parsed as 600 microseconds rather than milliseconds.

files:
M Lib/_strptime.py
M Lib/test/test_strptime.py

diff --git a/Lib/_strptime.py b/Lib/_strptime.py
index f5195af90c8..1be04850acf 100644
--- a/Lib/_strptime.py
+++ b/Lib/_strptime.py
@@ -470,7 +470,10 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
                 minutes = int(z[3:5])
                 seconds = int(z[5:7] or 0)
                 gmtoff = (hours * 60 * 60) + (minutes * 60) + seconds
-                gmtoff_fraction = int(z[8:] or 0)
+                gmtoff_remainder = z[8:]
+                # Pad to always return microseconds.
+                gmtoff_remainder_padding = "0" * (6 - len(gmtoff_remainder))
+                gmtoff_fraction = int(gmtoff_remainder + gmtoff_remainder_padding)
                 if z.startswith("-"):
                     gmtoff = -gmtoff
                     gmtoff_fraction = -gmtoff_fraction
diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py
index 1251886779d..af71008bec5 100644
--- a/Lib/test/test_strptime.py
+++ b/Lib/test/test_strptime.py
@@ -345,6 +345,9 @@ def test_offset(self):
         (*_, offset), _, offset_fraction = _strptime._strptime("-01:30:30.000001", "%z")
         self.assertEqual(offset, -(one_hour + half_hour + half_minute))
         self.assertEqual(offset_fraction, -1)
+        (*_, offset), _, offset_fraction = _strptime._strptime("+01:30:30.001", "%z")
+        self.assertEqual(offset, one_hour + half_hour + half_minute)
+        self.assertEqual(offset_fraction, 1000)
         (*_, offset), _, offset_fraction = _strptime._strptime("Z", "%z")
         self.assertEqual(offset, 0)
         self.assertEqual(offset_fraction, 0)


From lp_benchmark_robot at intel.com  Tue Jan  9 19:42:54 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Tue, 9 Jan 2018 16:42:54 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-09
Message-ID: <a468a7fc-4e69-4028-b148-e7d90b5af3b2@orsmsx108.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-09 03:04:20-08:00.
  - commit:		d135f20
  - previous commit:	9b99747
  - revision date:	2018-01-09 19:14:46+11:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.914% |  -1.183%   |  +5.529%   |  +9.130%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.067% |  +0.027%   |  +22.147%  |  +13.850%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 0.927% |  +0.375%   |  +24.472%  |  +12.227%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 1.320% |  -0.159%   |  +21.819%  |  +11.290%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.461% |  +0.453%   |  +8.971%   |  +11.352%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.438% |  -0.131%   |  +9.329%   |  +12.896%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 0.749% |  -0.651%   |  +4.812%   |  +13.230%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.745% |  -0.289%   |  -1.214%   |  +8.893%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 2.977% |  -0.184%   |  +9.548%   |  +17.519%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.371% |  -1.907%   |  +10.901%  |  +15.910%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.322% |  -1.070%   |  +2.798%   |  +8.526%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.896% |  -0.146%   |  +4.205%   |  +6.115%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.579% |  -0.038%   |  +2.487%   |  +7.937%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.402% |  -0.828%   |  +11.360%  |  +11.813%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.640% |  -1.357%   |  +6.230%   |  +13.268%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 1.041% |  -0.373%   |  +6.281%   |  +12.161%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.649% |  -0.258%   |  +10.135%  |  +11.331%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.829% |  -3.020%   |  +5.231%   |  +13.573%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 2.161% |  -0.475%   |  +2.768%   |  +9.917%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 1.628% |  -0.022%   |  -0.641%   |  +14.556%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.676% |  -1.569%   |  +6.756%   |  +13.166%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.514% |  +0.304%   |  +46.532%  |  +12.753%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.584% |  -1.028%   |  +7.931%   |  +15.457%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.819% |  +0.189%   |  +17.185%  |  +13.769%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 3.454% |  -0.264%   |  +10.929%  |  +7.796%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.720% |  -0.476%   |  +3.301%   |  +5.754%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.624% |  +0.148%   |  -4.744%   |  +3.504%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 0.731% |  +0.187%   |  +4.383%   |  +8.647%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 2.047% |  -1.480%   |  +1.605%   |  +10.754%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.135% |  -0.155%   |  -2.170%   |  +24.598%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 5.920% |  -0.925%   |  +2.531%   |  +18.500%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.717% |  +0.444%   |  +5.150%   |  +17.155%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 3.703% |  -0.131%   |  +11.903%  |  +10.050%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.097% |  +0.011%   |  -0.096%   |  +10.276%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.149% |  -0.215%   |  +10.401%  |  +6.060%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.114% |  -0.083%   |  +1.828%   |  +5.894%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.000% |  -0.418%   |  +9.997%   |  +13.455%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.636% |  +0.860%   |  +1.810%   |  +13.431%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.608% |  -0.143%   |  -0.718%   |  +10.283%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.036% |  +0.048%   |  -4.788%   |  +5.838%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 2.707% |  +0.754%   |  +6.442%   |  +8.244%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.133% |  +0.009%   |  +8.685%   |  +13.991%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.817% |  -0.125%   |  -2.983%   |  +5.584%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.697% |  +0.341%   |  +26.568%  |  +10.023%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.746% |  +0.556%   |  +5.284%   |  +3.551%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.533% |  +0.105%   |  +14.771%  |  +7.891%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.380% |  -0.028%   |  -6.679%   |  +1.241%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.526% |  -0.006%   |  +5.037%   |  +3.795%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 0.983% |  -0.716%   |  +5.285%   |  +10.339%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 2.917% |  -0.201%   |  +5.623%   |  +6.731%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.570% |  -0.939%   |  +18.312%  |  +9.394%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.331% |  -0.821%   |  +10.896%  |  +9.404%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.299% |  -0.375%   |  +7.723%   |  +8.840%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.509% |  -0.675%   |  +11.025%  |  +10.737%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.135% |  -0.651%   |  +9.729%   |  +13.142%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 5.391% |  +0.630%   |  +18.196%  |  +13.079%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.040% |  -0.828%   |  +5.110%   |  +7.550%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 5.302% |  -1.620%   |  +0.541%   |  +1.353%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 7.358% |  -0.136%   |  +4.257%   |  +24.796%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 2.194% |  -0.248%   |  -2.621%   |  +17.563%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.681% |  -0.392%   |  +7.496%   |  +7.706%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 0.956% |  +0.272%   |  +0.772%   |  +13.186%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 1.954% |  +0.062%   |  +1.988%   |  +9.949%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 3.731% |  -0.097%   |  -9.800%   |  +14.094%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.276% |  -0.191%   |  +2.647%   |  +12.583%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-09

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Wed Jan 10 04:12:27 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 10 Jan 2018 09:12:27 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=4
Message-ID: <20180110091227.1.CACB91D9CABBC6C2@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 7, -7] memory blocks, sum=0
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloglMK3uv', '--timeout', '7200']

From webhook-mailer at python.org  Wed Jan 10 16:46:19 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 10 Jan 2018 21:46:19 -0000
Subject: [Python-checkins] bpo-29240: readline now ignores the UTF-8 Mode
 (#5145)
Message-ID: <mailman.12.1515620780.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/2cba6b85797ba60d67389126f184aad5c9e02ff3
commit: 2cba6b85797ba60d67389126f184aad5c9e02ff3
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-10T22:46:15+01:00
summary:

bpo-29240: readline now ignores the UTF-8 Mode (#5145)

Add new fuctions ignoring the UTF-8 mode:

* _Py_DecodeCurrentLocale()
* _Py_EncodeCurrentLocale()
* _PyUnicode_DecodeCurrentLocaleAndSize()
* _PyUnicode_EncodeCurrentLocale()

Modify the readline module to use these functions.

Re-enable test_readline.test_nonascii().

files:
M Include/fileutils.h
M Include/unicodeobject.h
M Lib/test/test_readline.py
M Modules/readline.c
M Objects/unicodeobject.c
M Python/fileutils.c

diff --git a/Include/fileutils.h b/Include/fileutils.h
index d027e18348f..2527d84669d 100644
--- a/Include/fileutils.h
+++ b/Include/fileutils.h
@@ -24,6 +24,14 @@ PyAPI_FUNC(wchar_t*) _Py_DecodeUTF8_surrogateescape(
     const char *s,
     Py_ssize_t size,
     size_t *p_wlen);
+
+PyAPI_FUNC(wchar_t *) _Py_DecodeCurrentLocale(
+    const char *arg,
+    size_t *size);
+
+PyAPI_FUNC(char*) _Py_EncodeCurrentLocale(
+    const wchar_t *text,
+    size_t *error_pos);
 #endif
 
 #ifndef Py_LIMITED_API
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
index 0274de6733a..576e7ad8510 100644
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1810,6 +1810,16 @@ PyAPI_FUNC(PyObject*) PyUnicode_EncodeLocale(
     PyObject *unicode,
     const char *errors
     );
+
+PyAPI_FUNC(PyObject*) _PyUnicode_DecodeCurrentLocaleAndSize(
+    const char *str,
+    Py_ssize_t len,
+    const char *errors);
+
+PyAPI_FUNC(PyObject*) _PyUnicode_EncodeCurrentLocale(
+    PyObject *unicode,
+    const char *errors
+    );
 #endif
 
 /* --- File system encoding ---------------------------------------------- */
diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py
index 28ea38b747e..b4c25dee9d3 100644
--- a/Lib/test/test_readline.py
+++ b/Lib/test/test_readline.py
@@ -152,8 +152,6 @@ def test_auto_history_disabled(self):
         output = run_pty(self.auto_history_script.format(False))
         self.assertIn(b"History length: 0\r\n", output)
 
-    @unittest.skipIf(True,
-                     "FIXME: test broken by bpo-29240")
     def test_nonascii(self):
         try:
             readline.add_history("\xEB\xEF")
diff --git a/Modules/readline.c b/Modules/readline.c
index 811fca8cd92..8db4cfd0152 100644
--- a/Modules/readline.c
+++ b/Modules/readline.c
@@ -132,13 +132,14 @@ static PyModuleDef readlinemodule;
 static PyObject *
 encode(PyObject *b)
 {
-    return PyUnicode_EncodeLocale(b, "surrogateescape");
+    return _PyUnicode_EncodeCurrentLocale(b, "surrogateescape");
 }
 
 static PyObject *
 decode(const char *s)
 {
-    return PyUnicode_DecodeLocale(s, "surrogateescape");
+    return _PyUnicode_DecodeCurrentLocaleAndSize(s, strlen(s),
+                                                 "surrogateescape");
 }
 
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 92a6ad6b979..1a230e03e63 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3395,8 +3395,8 @@ locale_error_handler(const char *errors, int *surrogateescape)
     }
 }
 
-PyObject *
-PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
+static PyObject *
+unicode_encode_locale(PyObject *unicode, const char *errors, int current_locale)
 {
     Py_ssize_t wlen, wlen2;
     wchar_t *wstr;
@@ -3423,7 +3423,12 @@ PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
         /* "surrogateescape" error handler */
         char *str;
 
-        str = Py_EncodeLocale(wstr, &error_pos);
+        if (current_locale) {
+            str = _Py_EncodeCurrentLocale(wstr, &error_pos);
+        }
+        else {
+            str = Py_EncodeLocale(wstr, &error_pos);
+        }
         if (str == NULL) {
             if (error_pos == (size_t)-1) {
                 PyErr_NoMemory();
@@ -3437,7 +3442,12 @@ PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
         PyMem_Free(wstr);
 
         bytes = PyBytes_FromString(str);
-        PyMem_Free(str);
+        if (current_locale) {
+            PyMem_RawFree(str);
+        }
+        else {
+            PyMem_Free(str);
+        }
     }
     else {
         /* strict mode */
@@ -3502,6 +3512,18 @@ PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
     return NULL;
 }
 
+PyObject *
+PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
+{
+    return unicode_encode_locale(unicode, errors, 0);
+}
+
+PyObject *
+_PyUnicode_EncodeCurrentLocale(PyObject *unicode, const char *errors)
+{
+    return unicode_encode_locale(unicode, errors, 1);
+}
+
 PyObject *
 PyUnicode_EncodeFSDefault(PyObject *unicode)
 {
@@ -3524,7 +3546,8 @@ PyUnicode_EncodeFSDefault(PyObject *unicode)
                                          Py_FileSystemDefaultEncodeErrors);
     }
     else {
-        return PyUnicode_EncodeLocale(unicode, Py_FileSystemDefaultEncodeErrors);
+        return unicode_encode_locale(unicode,
+                                     Py_FileSystemDefaultEncodeErrors, 0);
     }
 #endif
 }
@@ -3695,9 +3718,9 @@ mbstowcs_errorpos(const char *str, size_t len)
     return 0;
 }
 
-PyObject*
-PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
-                              const char *errors)
+static PyObject*
+unicode_decode_locale(const char *str, Py_ssize_t len, const char *errors,
+                      int current_locale)
 {
     wchar_t smallbuf[256];
     size_t smallbuf_len = Py_ARRAY_LENGTH(smallbuf);
@@ -3719,7 +3742,12 @@ PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
 
     if (surrogateescape) {
         /* "surrogateescape" error handler */
-        wstr = Py_DecodeLocale(str, &wlen);
+        if (current_locale) {
+            wstr = _Py_DecodeCurrentLocale(str, &wlen);
+        }
+        else {
+            wstr = Py_DecodeLocale(str, &wlen);
+        }
         if (wstr == NULL) {
             if (wlen == (size_t)-1)
                 PyErr_NoMemory();
@@ -3794,11 +3822,25 @@ PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
     return NULL;
 }
 
+PyObject*
+PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
+                              const char *errors)
+{
+    return unicode_decode_locale(str, len, errors, 0);
+}
+
+PyObject*
+_PyUnicode_DecodeCurrentLocaleAndSize(const char *str, Py_ssize_t len,
+                                      const char *errors)
+{
+    return unicode_decode_locale(str, len, errors, 1);
+}
+
 PyObject*
 PyUnicode_DecodeLocale(const char *str, const char *errors)
 {
     Py_ssize_t size = (Py_ssize_t)strlen(str);
-    return PyUnicode_DecodeLocaleAndSize(str, size, errors);
+    return unicode_decode_locale(str, size, errors, 0);
 }
 
 
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 645a1793664..9275494e864 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -263,7 +263,7 @@ decode_ascii_surrogateescape(const char *arg, size_t *size)
 
 #if !defined(__APPLE__) && !defined(__ANDROID__)
 static wchar_t*
-decode_locale(const char* arg, size_t *size)
+decode_current_locale(const char* arg, size_t *size)
 {
     wchar_t *res;
     size_t argsize;
@@ -380,32 +380,13 @@ decode_locale(const char* arg, size_t *size)
 #endif
 
 
-/* Decode a byte string from the locale encoding with the
-   surrogateescape error handler: undecodable bytes are decoded as characters
-   in range U+DC80..U+DCFF. If a byte sequence can be decoded as a surrogate
-   character, escape the bytes using the surrogateescape error handler instead
-   of decoding them.
-
-   Return a pointer to a newly allocated wide character string, use
-   PyMem_RawFree() to free the memory. If size is not NULL, write the number of
-   wide characters excluding the null character into *size
-
-   Return NULL on decoding error or memory allocation error. If *size* is not
-   NULL, *size is set to (size_t)-1 on memory error or set to (size_t)-2 on
-   decoding error.
-
-   Decoding errors should never happen, unless there is a bug in the C
-   library.
-
-   Use the Py_EncodeLocale() function to encode the character string back to a
-   byte string. */
-wchar_t*
-Py_DecodeLocale(const char* arg, size_t *size)
+static wchar_t*
+decode_locale(const char* arg, size_t *size, int ignore_utf8_mode)
 {
 #if defined(__APPLE__) || defined(__ANDROID__)
     return _Py_DecodeUTF8_surrogateescape(arg, strlen(arg), size);
 #else
-    if (Py_UTF8Mode == 1) {
+    if (!ignore_utf8_mode && Py_UTF8Mode == 1) {
         return _Py_DecodeUTF8_surrogateescape(arg, strlen(arg), size);
     }
 
@@ -426,11 +407,45 @@ Py_DecodeLocale(const char* arg, size_t *size)
     }
 #endif
 
-    return decode_locale(arg, size);
+    return decode_current_locale(arg, size);
 #endif   /* __APPLE__ or __ANDROID__ */
 }
 
 
+/* Decode a byte string from the locale encoding with the
+   surrogateescape error handler: undecodable bytes are decoded as characters
+   in range U+DC80..U+DCFF. If a byte sequence can be decoded as a surrogate
+   character, escape the bytes using the surrogateescape error handler instead
+   of decoding them.
+
+   Return a pointer to a newly allocated wide character string, use
+   PyMem_RawFree() to free the memory. If size is not NULL, write the number of
+   wide characters excluding the null character into *size
+
+   Return NULL on decoding error or memory allocation error. If *size* is not
+   NULL, *size is set to (size_t)-1 on memory error or set to (size_t)-2 on
+   decoding error.
+
+   Decoding errors should never happen, unless there is a bug in the C
+   library.
+
+   Use the Py_EncodeLocale() function to encode the character string back to a
+   byte string. */
+wchar_t*
+Py_DecodeLocale(const char* arg, size_t *size)
+{
+    return decode_locale(arg, size, 0);
+}
+
+
+/* Similar to Py_DecodeLocale() but ignore the UTF-8 mode */
+wchar_t*
+_Py_DecodeCurrentLocale(const char* arg, size_t *size)
+{
+    return decode_locale(arg, size, 1);
+}
+
+
 #if !defined(__APPLE__) && !defined(__ANDROID__)
 static char*
 encode_current_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
@@ -508,12 +523,13 @@ encode_current_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
 #endif
 
 static char*
-encode_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
+encode_locale(const wchar_t *text, size_t *error_pos,
+              int raw_malloc, int ignore_utf8_mode)
 {
 #if defined(__APPLE__) || defined(__ANDROID__)
     return _Py_EncodeUTF8_surrogateescape(text, error_pos, raw_malloc);
 #else   /* __APPLE__ */
-    if (Py_UTF8Mode == 1) {
+    if (!ignore_utf8_mode && Py_UTF8Mode == 1) {
         return _Py_EncodeUTF8_surrogateescape(text, error_pos, raw_malloc);
     }
 
@@ -544,7 +560,7 @@ encode_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
 char*
 Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
 {
-    return encode_locale(text, error_pos, 0);
+    return encode_locale(text, error_pos, 0, 0);
 }
 
 
@@ -553,7 +569,15 @@ Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
 char*
 _Py_EncodeLocaleRaw(const wchar_t *text, size_t *error_pos)
 {
-    return encode_locale(text, error_pos, 1);
+    return encode_locale(text, error_pos, 1, 0);
+}
+
+
+/* Similar to _Py_EncodeLocaleRaw() but ignore the UTF-8 Mode */
+char*
+_Py_EncodeCurrentLocale(const wchar_t *text, size_t *error_pos)
+{
+    return encode_locale(text, error_pos, 1, 1);
 }
 
 


From webhook-mailer at python.org  Wed Jan 10 23:56:45 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Thu, 11 Jan 2018 04:56:45 -0000
Subject: [Python-checkins] Fix obvious typos in docstrings (#5151)
Message-ID: <mailman.13.1515646605.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/d55209d5b1e097cde55fa3f83149d614c8ccaf09
commit: d55209d5b1e097cde55fa3f83149d614c8ccaf09
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-10T20:56:41-08:00
summary:

Fix obvious typos in docstrings (#5151)

files:
M Lib/dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index d80054920ce..db14b0f66c2 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -570,7 +570,7 @@ def _process_class(cls, repr, eq, order, hash, init, frozen):
 
 
 # _cls should never be specified by keyword, so start it with an
-#  underscore. The presense of _cls is used to detect if this
+#  underscore. The presence of _cls is used to detect if this
 #  decorator is being called with parameters or not.
 def dataclass(_cls=None, *, init=True, repr=True, eq=True, order=False,
               hash=None, frozen=False):
@@ -677,7 +677,7 @@ class C:
           y: int
 
     c = C(1, 2)
-    assert asdtuple(c) == (1, 2)
+    assert astuple(c) == (1, 2)
 
     If given, 'tuple_factory' will be used instead of built-in tuple.
     The function applies recursively to field values that are
@@ -714,16 +714,17 @@ def make_dataclass(cls_name, fields, *, bases=(), namespace=None, init=True,
     omitted, use the string 'typing.Any'.  Field objects are created by
     the equivalent of calling 'field(name, type [, Field-info])'.
 
-      C = make_class('C', ['x', ('y', int'), ('z', int, Field(init=False))], bases=[Base])
+      C = make_dataclass('C', ['x', ('y', int), ('z', int, field(init=False))], bases=(Base,))
 
     is equivalent to:
 
       @dataclass
       class C(Base):
-          a: int
-          b: int = field(init=False)
+          x: 'typing.Any'
+          y: int
+          z: int = field(init=False)
 
-    For the bases and namespace paremeters, see the builtin type() function.
+    For the bases and namespace parameters, see the builtin type() function.
 
     The parameters init, repr, eq, order, hash, and frozen are passed to
     dataclass().


From webhook-mailer at python.org  Thu Jan 11 00:45:22 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Thu, 11 Jan 2018 05:45:22 -0000
Subject: [Python-checkins] bpo-32320: Add default value support to
 collections.namedtuple() (#4859)
Message-ID: <mailman.14.1515649523.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/3948207c610e931831828d33aaef258185df31db
commit: 3948207c610e931831828d33aaef258185df31db
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-10T21:45:19-08:00
summary:

bpo-32320: Add default value support to collections.namedtuple() (#4859)

files:
A Misc/NEWS.d/next/Library/2017-12-14-01-36-25.bpo-32320.jwOZlr.rst
M Doc/library/collections.rst
M Lib/collections/__init__.py
M Lib/test/test_collections.py

diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst
index 4b0d8c048ae..18aaba65b25 100644
--- a/Doc/library/collections.rst
+++ b/Doc/library/collections.rst
@@ -782,7 +782,7 @@ Named tuples assign meaning to each position in a tuple and allow for more reada
 self-documenting code.  They can be used wherever regular tuples are used, and
 they add the ability to access fields by name instead of position index.
 
-.. function:: namedtuple(typename, field_names, *, rename=False, module=None)
+.. function:: namedtuple(typename, field_names, *, rename=False, defaults=None, module=None)
 
     Returns a new tuple subclass named *typename*.  The new subclass is used to
     create tuple-like objects that have fields accessible by attribute lookup as
@@ -805,6 +805,13 @@ they add the ability to access fields by name instead of position index.
     converted to ``['abc', '_1', 'ghi', '_3']``, eliminating the keyword
     ``def`` and the duplicate fieldname ``abc``.
 
+    *defaults* can be ``None`` or an :term:`iterable` of default values.
+    Since fields with a default value must come after any fields without a
+    default, the *defaults* are applied to the rightmost parameters.  For
+    example, if the fieldnames are ``['x', 'y', 'z']`` and the defaults are
+    ``(1, 2)``, then ``x`` will be a required argument, ``y`` will default to
+    ``1``, and ``z`` will default to ``2``.
+
     If *module* is defined, the ``__module__`` attribute of the named tuple is
     set to that value.
 
@@ -824,6 +831,10 @@ they add the ability to access fields by name instead of position index.
     .. versionchanged:: 3.7
        Remove the *verbose* parameter and the :attr:`_source` attribute.
 
+    .. versionchanged:: 3.7
+       Added the *defaults* parameter and the :attr:`_field_defaults`
+       attribute.
+
 .. doctest::
     :options: +NORMALIZE_WHITESPACE
 
@@ -911,6 +922,18 @@ field names, the method and attribute names start with an underscore.
         >>> Pixel(11, 22, 128, 255, 0)
         Pixel(x=11, y=22, red=128, green=255, blue=0)
 
+.. attribute:: somenamedtuple._fields_defaults
+
+   Dictionary mapping field names to default values.
+
+   .. doctest::
+
+        >>> Account = namedtuple('Account', ['type', 'balance'], defaults=[0])
+        >>> Account._fields_defaults
+        {'balance': 0}
+        >>> Account('premium')
+        Account(type='premium', balance=0)
+
 To retrieve a field whose name is stored in a string, use the :func:`getattr`
 function:
 
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 50cf8141731..7088b88e04a 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -303,7 +303,7 @@ def __eq__(self, other):
 
 _nt_itemgetters = {}
 
-def namedtuple(typename, field_names, *, rename=False, module=None):
+def namedtuple(typename, field_names, *, rename=False, defaults=None, module=None):
     """Returns a new subclass of tuple with named fields.
 
     >>> Point = namedtuple('Point', ['x', 'y'])
@@ -332,7 +332,8 @@ def namedtuple(typename, field_names, *, rename=False, module=None):
     if isinstance(field_names, str):
         field_names = field_names.replace(',', ' ').split()
     field_names = list(map(str, field_names))
-    typename = str(typename)
+    typename = _sys.intern(str(typename))
+
     if rename:
         seen = set()
         for index, name in enumerate(field_names):
@@ -342,6 +343,7 @@ def namedtuple(typename, field_names, *, rename=False, module=None):
                 or name in seen):
                 field_names[index] = f'_{index}'
             seen.add(name)
+
     for name in [typename] + field_names:
         if type(name) is not str:
             raise TypeError('Type names and field names must be strings')
@@ -351,6 +353,7 @@ def namedtuple(typename, field_names, *, rename=False, module=None):
         if _iskeyword(name):
             raise ValueError('Type names and field names cannot be a '
                              f'keyword: {name!r}')
+
     seen = set()
     for name in field_names:
         if name.startswith('_') and not rename:
@@ -360,6 +363,14 @@ def namedtuple(typename, field_names, *, rename=False, module=None):
             raise ValueError(f'Encountered duplicate field name: {name!r}')
         seen.add(name)
 
+    field_defaults = {}
+    if defaults is not None:
+        defaults = tuple(defaults)
+        if len(defaults) > len(field_names):
+            raise TypeError('Got more default values than field names')
+        field_defaults = dict(reversed(list(zip(reversed(field_names),
+                                                reversed(defaults)))))
+
     # Variables used in the methods and docstrings
     field_names = tuple(map(_sys.intern, field_names))
     num_fields = len(field_names)
@@ -372,10 +383,12 @@ def namedtuple(typename, field_names, *, rename=False, module=None):
 
     s = f'def __new__(_cls, {arg_list}): return _tuple_new(_cls, ({arg_list}))'
     namespace = {'_tuple_new': tuple_new, '__name__': f'namedtuple_{typename}'}
-    # Note: exec() has the side-effect of interning the typename and field names
+    # Note: exec() has the side-effect of interning the field names
     exec(s, namespace)
     __new__ = namespace['__new__']
     __new__.__doc__ = f'Create new instance of {typename}({arg_list})'
+    if defaults is not None:
+        __new__.__defaults__ = defaults
 
     @classmethod
     def _make(cls, iterable):
@@ -420,6 +433,7 @@ def __getnewargs__(self):
         '__doc__': f'{typename}({arg_list})',
         '__slots__': (),
         '_fields': field_names,
+        '_fields_defaults': field_defaults,
         '__new__': __new__,
         '_make': _make,
         '_replace': _replace,
diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
index 6c466f41e68..cb662355bb7 100644
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -216,6 +216,57 @@ def test_factory(self):
         self.assertRaises(TypeError, Point._make, [11])                     # catch too few args
         self.assertRaises(TypeError, Point._make, [11, 22, 33])             # catch too many args
 
+    def test_defaults(self):
+        Point = namedtuple('Point', 'x y', defaults=(10, 20))              # 2 defaults
+        self.assertEqual(Point._fields_defaults, {'x': 10, 'y': 20})
+        self.assertEqual(Point(1, 2), (1, 2))
+        self.assertEqual(Point(1), (1, 20))
+        self.assertEqual(Point(), (10, 20))
+
+        Point = namedtuple('Point', 'x y', defaults=(20,))                 # 1 default
+        self.assertEqual(Point._fields_defaults, {'y': 20})
+        self.assertEqual(Point(1, 2), (1, 2))
+        self.assertEqual(Point(1), (1, 20))
+
+        Point = namedtuple('Point', 'x y', defaults=())                     # 0 defaults
+        self.assertEqual(Point._fields_defaults, {})
+        self.assertEqual(Point(1, 2), (1, 2))
+        with self.assertRaises(TypeError):
+            Point(1)
+
+        with self.assertRaises(TypeError):                                  # catch too few args
+            Point()
+        with self.assertRaises(TypeError):                                  # catch too many args
+            Point(1, 2, 3)
+        with self.assertRaises(TypeError):                                  # too many defaults
+            Point = namedtuple('Point', 'x y', defaults=(10, 20, 30))
+        with self.assertRaises(TypeError):                                  # non-iterable defaults
+            Point = namedtuple('Point', 'x y', defaults=10)
+        with self.assertRaises(TypeError):                                  # another non-iterable default
+            Point = namedtuple('Point', 'x y', defaults=False)
+
+        Point = namedtuple('Point', 'x y', defaults=None)                   # default is None
+        self.assertEqual(Point._fields_defaults, {})
+        self.assertIsNone(Point.__new__.__defaults__, None)
+        self.assertEqual(Point(10, 20), (10, 20))
+        with self.assertRaises(TypeError):                                  # catch too few args
+            Point(10)
+
+        Point = namedtuple('Point', 'x y', defaults=[10, 20])               # allow non-tuple iterable
+        self.assertEqual(Point._fields_defaults, {'x': 10, 'y': 20})
+        self.assertEqual(Point.__new__.__defaults__, (10, 20))
+        self.assertEqual(Point(1, 2), (1, 2))
+        self.assertEqual(Point(1), (1, 20))
+        self.assertEqual(Point(), (10, 20))
+
+        Point = namedtuple('Point', 'x y', defaults=iter([10, 20]))         # allow plain iterator
+        self.assertEqual(Point._fields_defaults, {'x': 10, 'y': 20})
+        self.assertEqual(Point.__new__.__defaults__, (10, 20))
+        self.assertEqual(Point(1, 2), (1, 2))
+        self.assertEqual(Point(1), (1, 20))
+        self.assertEqual(Point(), (10, 20))
+
+
     @unittest.skipIf(sys.flags.optimize >= 2,
                      "Docstrings are omitted with -O2 and above")
     def test_factory_doc_attr(self):
diff --git a/Misc/NEWS.d/next/Library/2017-12-14-01-36-25.bpo-32320.jwOZlr.rst b/Misc/NEWS.d/next/Library/2017-12-14-01-36-25.bpo-32320.jwOZlr.rst
new file mode 100644
index 00000000000..6e4aad8f795
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-14-01-36-25.bpo-32320.jwOZlr.rst
@@ -0,0 +1 @@
+collections.namedtuple() now supports default values.


From lp_benchmark_robot at intel.com  Thu Jan 11 00:47:32 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Wed, 10 Jan 2018 21:47:32 -0800
Subject: [Python-checkins] [1 down, 1 up,
 63 flat] Results for Python (master branch) 2018-01-10
Message-ID: <e276480b-0697-4723-8c1a-941cb1635bb0@orsmsx153.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-10 03:03:22-08:00.
  - commit:		f80c0ca
  - previous commit:	d135f20
  - revision date:	2018-01-09 16:37:26-05:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.915% |  +1.922%   |  +7.345%   |  +7.185%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.317% |  -0.272%   |  +21.936%  |  +9.323%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.379% |  -0.185%   |  +24.332%  |  +6.078%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 1.078% |  +0.432%   |  +22.157%  |  +6.036%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 4.311% |  -1.340%   |  +7.751%   |  +13.493%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 2.409% |  +0.022%   |  +9.349%   |  +13.231%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.384% |  +1.271%   |  +6.022%   |  +10.970%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.529% |  +0.620%   |  -0.586%   |  +8.440%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 2.491% |  +0.374%   |  +9.886%   |  +16.213%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.973% |  +1.275%   |  +12.037%  |  +14.071%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.417% |  +0.867%   |  +3.641%   |  +7.809%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.889% |  +1.073%   |  +5.232%   |  +4.641%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.696% |  -0.145%   |  +2.345%   |  +5.541%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.376% |  +1.100%   |  +12.335%  |  +10.894%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.387% |  +2.424%   |  +8.504%   |  +11.256%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.871% |  +0.246%   |  +6.512%   |  +11.561%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 1.274% |  +0.203%   |  +10.318%  |  +10.355%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 2.776% |  +2.754%   |  +7.841%   |  +9.190%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.483% |  -0.582%   |  +2.202%   |  +11.202%  |
+-----+------------------------+--------+------------+------------+------------+
| :-( |              json_loads| 2.806% |  -7.406%   |  -8.094%   |  +12.714%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.516% |  +1.815%   |  +8.449%   |  +10.977%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.221% |  -0.009%   |  +46.527%  |  +10.120%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.479% |  +1.300%   |  +9.128%   |  +12.440%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.546% |  +1.271%   |  +18.238%  |  +12.191%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.898% |  -0.404%   |  +10.569%  |  +7.532%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.937% |  +0.939%   |  +4.209%   |  +5.089%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.585% |  +1.286%   |  -3.397%   |  +3.600%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.968% |  -0.454%   |  +3.949%   |  +7.294%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.515% |  -0.143%   |  +1.465%   |  +11.704%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.023% |  -0.353%   |  -2.530%   |  +24.046%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.218% |  +0.939%   |  +3.446%   |  +16.428%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.598% |  +0.684%   |  +5.799%   |  +14.929%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 4.206% |  -0.678%   |  +11.306%  |  +10.185%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.197% |  -0.012%   |  -0.108%   |  +10.235%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.126% |  +0.617%   |  +10.954%  |  +5.516%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.092% |  +0.717%   |  +2.532%   |  +5.551%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.147% |  +0.030%   |  +10.024%  |  +12.524%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.511% |  +1.681%   |  +3.461%   |  +11.866%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.294% |  -1.795%   |  -2.525%   |  +12.923%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 1.369% |  +0.470%   |  -4.296%   |  +5.732%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 2.818% |  -1.613%   |  +4.933%   |  +7.685%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.549% |  -1.335%   |  +7.466%   |  +14.351%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 2.440% |  -0.544%   |  -3.543%   |  +6.352%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.999% |  +0.049%   |  +26.604%  |  +8.841%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 1.961% |  -0.792%   |  +4.534%   |  +5.629%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 1.050% |  -0.678%   |  +14.193%  |  +8.403%   |
+-----+------------------------+--------+------------+------------+------------+
| :-) | scimark_sparse_mat_mult| 0.757% |  +5.822%   |  -0.469%   |  -4.227%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.557% |  +0.355%   |  +5.375%   |  +2.279%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.213% |  +1.003%   |  +6.235%   |  +7.325%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.220% |  +0.614%   |  +6.202%   |  +4.428%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.226% |  +0.514%   |  +18.732%  |  +9.293%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.699% |  -0.346%   |  +10.588%  |  +7.985%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.656% |  +0.284%   |  +7.985%   |  +8.542%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.248% |  +0.734%   |  +11.678%  |  +9.446%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.258% |  +0.153%   |  +9.867%   |  +10.177%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 5.178% |  +2.010%   |  +19.840%  |  +12.026%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.143% |  +0.740%   |  +5.812%   |  +6.831%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 4.687% |  +0.895%   |  +1.431%   |  -0.034%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 6.284% |  +0.832%   |  +5.054%   |  +24.542%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 2.751% |  +0.937%   |  -1.660%   |  +16.729%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.635% |  +0.854%   |  +8.286%   |  +7.423%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 0.807% |  -0.042%   |  +0.730%   |  +13.645%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.308% |  +0.901%   |  +2.872%   |  +9.181%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 2.660% |  +1.997%   |  -7.607%   |  +13.452%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 0.941% |  +0.572%   |  +3.203%   |  +12.008%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-down-1-up-63-flat-results-for-python-master-branch-2018-01-10

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Thu Jan 11 04:11:18 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 11 Jan 2018 09:11:18 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=3
Message-ID: <20180111091118.1.5F72EC19E046D5F1@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [-7, 1, 7] memory blocks, sum=1
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, 0, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogVTVCH8', '--timeout', '7200']

From webhook-mailer at python.org  Thu Jan 11 04:38:07 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 11 Jan 2018 09:38:07 -0000
Subject: [Python-checkins] bpo-29240: Ignore UTF-8 Mode in time module
 (#5148)
Message-ID: <mailman.15.1515663489.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/cb3ae5588bd7733e76dc09277bb7626652d9bb64
commit: cb3ae5588bd7733e76dc09277bb7626652d9bb64
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-11T10:37:59+01:00
summary:

bpo-29240: Ignore UTF-8 Mode in time module (#5148)

time.strftime() must use the current LC_CTYPE encoding, not UTF-8
if the UTF-8 mode is enabled.

Add _PyUnicode_DecodeCurrentLocale() function.

files:
M Include/unicodeobject.h
M Modules/readline.c
M Modules/timemodule.c
M Objects/unicodeobject.c

diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
index 576e7ad8510..d263026b562 100644
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1811,6 +1811,10 @@ PyAPI_FUNC(PyObject*) PyUnicode_EncodeLocale(
     const char *errors
     );
 
+PyAPI_FUNC(PyObject*) _PyUnicode_DecodeCurrentLocale(
+    const char *str,
+    const char *errors);
+
 PyAPI_FUNC(PyObject*) _PyUnicode_DecodeCurrentLocaleAndSize(
     const char *str,
     Py_ssize_t len,
diff --git a/Modules/readline.c b/Modules/readline.c
index 8db4cfd0152..caf661c81d4 100644
--- a/Modules/readline.c
+++ b/Modules/readline.c
@@ -138,8 +138,7 @@ encode(PyObject *b)
 static PyObject *
 decode(const char *s)
 {
-    return _PyUnicode_DecodeCurrentLocaleAndSize(s, strlen(s),
-                                                 "surrogateescape");
+    return _PyUnicode_DecodeCurrentLocale(s, "surrogateescape");
 }
 
 
diff --git a/Modules/timemodule.c b/Modules/timemodule.c
index 5cae03dd416..4e7f9d9482e 100644
--- a/Modules/timemodule.c
+++ b/Modules/timemodule.c
@@ -418,11 +418,11 @@ tmtotuple(struct tm *p
     SET(8, p->tm_isdst);
 #ifdef HAVE_STRUCT_TM_TM_ZONE
     PyStructSequence_SET_ITEM(v, 9,
-        PyUnicode_DecodeLocale(p->tm_zone, "surrogateescape"));
+        _PyUnicode_DecodeCurrentLocale(p->tm_zone, "surrogateescape"));
     SET(10, p->tm_gmtoff);
 #else
     PyStructSequence_SET_ITEM(v, 9,
-        PyUnicode_DecodeLocale(zone, "surrogateescape"));
+        _PyUnicode_DecodeCurrentLocale(zone, "surrogateescape"));
     PyStructSequence_SET_ITEM(v, 10, _PyLong_FromTime_t(gmtoff));
 #endif /* HAVE_STRUCT_TM_TM_ZONE */
 #undef SET
@@ -809,8 +809,8 @@ time_strftime(PyObject *self, PyObject *args)
 #ifdef HAVE_WCSFTIME
             ret = PyUnicode_FromWideChar(outbuf, buflen);
 #else
-            ret = PyUnicode_DecodeLocaleAndSize(outbuf, buflen,
-                                                "surrogateescape");
+            ret = _PyUnicode_DecodeCurrentLocaleAndSize(outbuf, buflen,
+                                                        "surrogateescape");
 #endif
             PyMem_Free(outbuf);
             break;
@@ -1541,8 +1541,8 @@ PyInit_timezone(PyObject *m) {
     PyModule_AddIntConstant(m, "altzone", timezone-3600);
 #endif
     PyModule_AddIntConstant(m, "daylight", daylight);
-    otz0 = PyUnicode_DecodeLocale(tzname[0], "surrogateescape");
-    otz1 = PyUnicode_DecodeLocale(tzname[1], "surrogateescape");
+    otz0 = _PyUnicode_DecodeCurrentLocale(tzname[0], "surrogateescape");
+    otz1 = _PyUnicode_DecodeCurrentLocale(tzname[1], "surrogateescape");
     PyModule_AddObject(m, "tzname", Py_BuildValue("(NN)", otz0, otz1));
 #else /* !HAVE_TZNAME || __GLIBC__ || __CYGWIN__*/
     {
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 1a230e03e63..a6e02f478b1 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3836,6 +3836,12 @@ _PyUnicode_DecodeCurrentLocaleAndSize(const char *str, Py_ssize_t len,
     return unicode_decode_locale(str, len, errors, 1);
 }
 
+PyObject*
+_PyUnicode_DecodeCurrentLocale(const char *str, const char *errors)
+{
+    return unicode_decode_locale(str, (Py_ssize_t)strlen(str), errors, 1);
+}
+
 PyObject*
 PyUnicode_DecodeLocale(const char *str, const char *errors)
 {


From webhook-mailer at python.org  Thu Jan 11 06:03:28 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 11 Jan 2018 11:03:28 -0000
Subject: [Python-checkins] bpo-31993: Do not create frames for large bytes
 and str objects (#5114)
Message-ID: <mailman.16.1515668610.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/0a2da50e1867831251fad75377d0f10713eb6301
commit: 0a2da50e1867831251fad75377d0f10713eb6301
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-11T13:03:20+02:00
summary:

bpo-31993: Do not create frames for large bytes and str objects (#5114)

when serialize into memory buffer with C pickle implementations.

This optimization already is performed when serialize into memory
with Python pickle implementations or into a file with both
implementations.

files:
M Lib/test/pickletester.py
M Lib/test/test_pickle.py
M Modules/_pickle.c

diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
index 5d983eb617f..f4e3f81249c 100644
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -2097,20 +2097,21 @@ def test_framing_large_objects(self):
         N = 1024 * 1024
         obj = [b'x' * N, b'y' * N, 'z' * N]
         for proto in range(4, pickle.HIGHEST_PROTOCOL + 1):
-            for fast in [True, False]:
+            for fast in [False, True]:
                 with self.subTest(proto=proto, fast=fast):
-                    if hasattr(self, 'pickler'):
+                    if not fast:
+                        # fast=False by default.
+                        # This covers in-memory pickling with pickle.dumps().
+                        pickled = self.dumps(obj, proto)
+                    else:
+                        # Pickler is required when fast=True.
+                        if not hasattr(self, 'pickler'):
+                            continue
                         buf = io.BytesIO()
                         pickler = self.pickler(buf, protocol=proto)
                         pickler.fast = fast
                         pickler.dump(obj)
                         pickled = buf.getvalue()
-                    elif fast:
-                        continue
-                    else:
-                        # Fallback to self.dumps when fast=False and
-                        # self.pickler is not available.
-                        pickled = self.dumps(obj, proto)
                     unpickled = self.loads(pickled)
                     # More informative error message in case of failure.
                     self.assertEqual([len(x) for x in obj],
diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py
index 895ed48df1d..0051a0118a3 100644
--- a/Lib/test/test_pickle.py
+++ b/Lib/test/test_pickle.py
@@ -71,8 +71,6 @@ def loads(self, buf, **kwds):
 class InMemoryPickleTests(AbstractPickleTests, AbstractUnpickleTests,
                           BigmemPickleTests):
 
-    pickler = pickle._Pickler
-    unpickler = pickle._Unpickler
     bad_stack_errors = (pickle.UnpicklingError, IndexError)
     truncated_errors = (pickle.UnpicklingError, EOFError,
                         AttributeError, ValueError,
@@ -84,6 +82,8 @@ def dumps(self, arg, protocol=None):
     def loads(self, buf, **kwds):
         return pickle.loads(buf, **kwds)
 
+    test_framed_write_sizes_with_delayed_writer = None
+
 
 class PersistentPicklerUnpicklerMixin(object):
 
diff --git a/Modules/_pickle.c b/Modules/_pickle.c
index 5cb1fba6cc2..5b3de4d96dd 100644
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -2142,47 +2142,74 @@ save_float(PicklerObject *self, PyObject *obj)
     return 0;
 }
 
-/* No-copy code-path to write large contiguous data directly into the
-   underlying file object, bypassing the output_buffer of the Pickler. */
-static int
-_Pickler_write_large_bytes(
-    PicklerObject *self, const char *header, Py_ssize_t header_size,
-    PyObject *payload)
-{
-    assert(self->output_buffer != NULL);
-    assert(self->write != NULL);
-    PyObject *result;
+/* Perform direct write of the header and payload of the binary object.
 
-    /* Commit the previous frame. */
-    if (_Pickler_CommitFrame(self)) {
-        return -1;
+   The large contiguous data is written directly into the underlying file
+   object, bypassing the output_buffer of the Pickler.  We intentionally
+   do not insert a protocol 4 frame opcode to make it possible to optimize
+   file.read calls in the loader.
+ */
+static int
+_Pickler_write_bytes(PicklerObject *self,
+                     const char *header, Py_ssize_t header_size,
+                     const char *data, Py_ssize_t data_size,
+                     PyObject *payload)
+{
+    int bypass_buffer = (data_size >= FRAME_SIZE_TARGET);
+    int framing = self->framing;
+
+    if (bypass_buffer) {
+        assert(self->output_buffer != NULL);
+        /* Commit the previous frame. */
+        if (_Pickler_CommitFrame(self)) {
+            return -1;
+        }
+        /* Disable framing temporarily */
+        self->framing = 0;
     }
-    /* Disable frameing temporarily */
-    self->framing = 0;
 
     if (_Pickler_Write(self, header, header_size) < 0) {
         return -1;
     }
-    /* Dump the output buffer to the file. */
-    if (_Pickler_FlushToFile(self) < 0) {
-        return -1;
-    }
 
-    /* Stream write the payload into the file without going through the
-       output buffer. */
-    result = PyObject_CallFunctionObjArgs(self->write, payload, NULL);
-    if (result == NULL) {
-        return -1;
-    }
-    Py_DECREF(result);
+    if (bypass_buffer && self->write != NULL) {
+        /* Bypass the in-memory buffer to directly stream large data
+           into the underlying file object. */
+        PyObject *result, *mem = NULL;
+        /* Dump the output buffer to the file. */
+        if (_Pickler_FlushToFile(self) < 0) {
+            return -1;
+        }
 
-    /* Reinitialize the buffer for subsequent calls to _Pickler_Write. */
-    if (_Pickler_ClearBuffer(self) < 0) {
-        return -1;
+        /* Stream write the payload into the file without going through the
+           output buffer. */
+        if (payload == NULL) {
+            payload = mem = PyMemoryView_FromMemory((char *) data, data_size,
+                                                    PyBUF_READ);
+            if (payload == NULL) {
+                return -1;
+            }
+        }
+        result = PyObject_CallFunctionObjArgs(self->write, payload, NULL);
+        Py_XDECREF(mem);
+        if (result == NULL) {
+            return -1;
+        }
+        Py_DECREF(result);
+
+        /* Reinitialize the buffer for subsequent calls to _Pickler_Write. */
+        if (_Pickler_ClearBuffer(self) < 0) {
+            return -1;
+        }
+    }
+    else {
+        if (_Pickler_Write(self, data, data_size) < 0) {
+            return -1;
+        }
     }
 
     /* Re-enable framing for subsequent calls to _Pickler_Write. */
-    self->framing = 1;
+    self->framing = framing;
 
     return 0;
 }
@@ -2265,20 +2292,10 @@ save_bytes(PicklerObject *self, PyObject *obj)
             return -1;          /* string too large */
         }
 
-        if (size < FRAME_SIZE_TARGET || self->write == NULL) {
-            if (_Pickler_Write(self, header, len) < 0) {
-                return -1;
-            }
-            if (_Pickler_Write(self, PyBytes_AS_STRING(obj), size) < 0) {
-                return -1;
-            }
-        }
-        else {
-            /* Bypass the in-memory buffer to directly stream large data
-               into the underlying file object. */
-            if (_Pickler_write_large_bytes(self, header, len, obj) < 0) {
-                return -1;
-            }
+        if (_Pickler_write_bytes(self, header, len,
+                                 PyBytes_AS_STRING(obj), size, obj) < 0)
+        {
+            return -1;
         }
 
         if (memo_put(self, obj) < 0)
@@ -2360,11 +2377,29 @@ raw_unicode_escape(PyObject *obj)
 }
 
 static int
-write_utf8(PicklerObject *self, const char *data, Py_ssize_t size)
+write_unicode_binary(PicklerObject *self, PyObject *obj)
 {
     char header[9];
     Py_ssize_t len;
-    PyObject *mem;
+    PyObject *encoded = NULL;
+    Py_ssize_t size;
+    const char *data;
+
+    if (PyUnicode_READY(obj))
+        return -1;
+
+    data = PyUnicode_AsUTF8AndSize(obj, &size);
+    if (data == NULL) {
+        /* Issue #8383: for strings with lone surrogates, fallback on the
+           "surrogatepass" error handler. */
+        PyErr_Clear();
+        encoded = PyUnicode_AsEncodedString(obj, "utf-8", "surrogatepass");
+        if (encoded == NULL)
+            return -1;
+
+        data = PyBytes_AS_STRING(encoded);
+        size = PyBytes_GET_SIZE(encoded);
+    }
 
     assert(size >= 0);
     if (size <= 0xff && self->proto >= 4) {
@@ -2388,61 +2423,18 @@ write_utf8(PicklerObject *self, const char *data, Py_ssize_t size)
     else {
         PyErr_SetString(PyExc_OverflowError,
                         "cannot serialize a string larger than 4GiB");
+        Py_XDECREF(encoded);
         return -1;
     }
 
-    if (size < FRAME_SIZE_TARGET || self->write == NULL) {
-        if (_Pickler_Write(self, header, len) < 0) {
-            return -1;
-        }
-        if (_Pickler_Write(self, data, size) < 0) {
-            return -1;
-        }
-    }
-    else {
-        /* Bypass the in-memory buffer to directly stream large data
-           into the underlying file object. */
-        mem = PyMemoryView_FromMemory((char *) data, size, PyBUF_READ);
-        if (mem == NULL) {
-            return -1;
-        }
-        if (_Pickler_write_large_bytes(self, header, len, mem) < 0) {
-            Py_DECREF(mem);
-            return -1;
-        }
-        Py_DECREF(mem);
+    if (_Pickler_write_bytes(self, header, len, data, size, encoded) < 0) {
+        Py_XDECREF(encoded);
+        return -1;
     }
+    Py_XDECREF(encoded);
     return 0;
 }
 
-static int
-write_unicode_binary(PicklerObject *self, PyObject *obj)
-{
-    PyObject *encoded = NULL;
-    Py_ssize_t size;
-    const char *data;
-    int r;
-
-    if (PyUnicode_READY(obj))
-        return -1;
-
-    data = PyUnicode_AsUTF8AndSize(obj, &size);
-    if (data != NULL)
-        return write_utf8(self, data, size);
-
-    /* Issue #8383: for strings with lone surrogates, fallback on the
-       "surrogatepass" error handler. */
-    PyErr_Clear();
-    encoded = PyUnicode_AsEncodedString(obj, "utf-8", "surrogatepass");
-    if (encoded == NULL)
-        return -1;
-
-    r = write_utf8(self, PyBytes_AS_STRING(encoded),
-                   PyBytes_GET_SIZE(encoded));
-    Py_DECREF(encoded);
-    return r;
-}
-
 static int
 save_unicode(PicklerObject *self, PyObject *obj)
 {


From webhook-mailer at python.org  Thu Jan 11 13:20:20 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 11 Jan 2018 18:20:20 -0000
Subject: [Python-checkins] bpo-31113: Get rid of recursion in the compiler
 for normal control flow. (#3015)
Message-ID: <mailman.17.1515694822.2789.python-checkins@python.org>

https://github.com/python/cpython/commit/782d6fe4434381c50e0c7ec94a1ef9c6debbc333
commit: 782d6fe4434381c50e0c7ec94a1ef9c6debbc333
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-11T20:20:13+02:00
summary:

bpo-31113: Get rid of recursion in the compiler for normal control flow. (#3015)

files:
A Misc/NEWS.d/next/Core and Builtins/2017-08-07-16-46-56.bpo-31113.XgNEFg.rst
M Lib/test/test_compile.py
M Python/compile.c

diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py
index 29aa362590f..4617a126abe 100644
--- a/Lib/test/test_compile.py
+++ b/Lib/test/test_compile.py
@@ -671,6 +671,11 @@ def __fspath__(self):
 
         compile("42", PathLike("test_compile_pathlike"), "single")
 
+    def test_stack_overflow(self):
+        # bpo-31113: Stack overflow when compile a long sequence of
+        # complex statements.
+        compile("if a: b\n" * 200000, "<dummy>", "exec")
+
 
 class TestExpressionStackSize(unittest.TestCase):
     # These tests check that the computed stack size for a code object
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-08-07-16-46-56.bpo-31113.XgNEFg.rst b/Misc/NEWS.d/next/Core and Builtins/2017-08-07-16-46-56.bpo-31113.XgNEFg.rst
new file mode 100644
index 00000000000..3ecdf2ff086
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-08-07-16-46-56.bpo-31113.XgNEFg.rst	
@@ -0,0 +1 @@
+Get rid of recursion in the compiler for normal control flow.
diff --git a/Python/compile.c b/Python/compile.c
index 7ba2ff34df2..31efc28da43 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -4920,83 +4920,42 @@ struct assembler {
 };
 
 static void
-dfs(struct compiler *c, basicblock *b, struct assembler *a)
+dfs(struct compiler *c, basicblock *b, struct assembler *a, int end)
 {
-    int i;
-    struct instr *instr = NULL;
-
-    if (b->b_seen)
-        return;
-    b->b_seen = 1;
-    if (b->b_next != NULL)
-        dfs(c, b->b_next, a);
-    for (i = 0; i < b->b_iused; i++) {
-        instr = &b->b_instr[i];
-        if (instr->i_jrel || instr->i_jabs)
-            dfs(c, instr->i_target, a);
+    int i, j;
+
+    /* Get rid of recursion for normal control flow.
+       Since the number of blocks is limited, unused space in a_postorder
+       (from a_nblocks to end) can be used as a stack for still not ordered
+       blocks. */
+    for (j = end; b && !b->b_seen; b = b->b_next) {
+        b->b_seen = 1;
+        assert(a->a_nblocks < j);
+        a->a_postorder[--j] = b;
+    }
+    while (j < end) {
+        b = a->a_postorder[j++];
+        for (i = 0; i < b->b_iused; i++) {
+            struct instr *instr = &b->b_instr[i];
+            if (instr->i_jrel || instr->i_jabs)
+                dfs(c, instr->i_target, a, j);
+        }
+        assert(a->a_nblocks < j);
+        a->a_postorder[a->a_nblocks++] = b;
     }
-    a->a_postorder[a->a_nblocks++] = b;
 }
 
-static int
-stackdepth_walk(struct compiler *c, basicblock *b, int depth, int maxdepth)
+Py_LOCAL_INLINE(void)
+stackdepth_push(basicblock ***sp, basicblock *b, int depth)
 {
-    int i, new_depth, target_depth, effect;
-    struct instr *instr;
-    assert(!b->b_seen || b->b_startdepth == depth);
-    if (b->b_seen || b->b_startdepth >= depth) {
-        return maxdepth;
-    }
-    /* Guard against infinite recursion */
-    b->b_seen = 1;
-    b->b_startdepth = depth;
-    for (i = 0; i < b->b_iused; i++) {
-        instr = &b->b_instr[i];
-        effect = stack_effect(instr->i_opcode, instr->i_oparg, 0);
-        if (effect == PY_INVALID_STACK_EFFECT) {
-            fprintf(stderr, "opcode = %d\n", instr->i_opcode);
-            Py_FatalError("PyCompile_OpcodeStackEffect()");
-        }
-        new_depth = depth + effect;
-        if (new_depth > maxdepth) {
-            maxdepth = new_depth;
-        }
-        assert(new_depth >= 0); /* invalid code or bug in stackdepth() */
-        if (instr->i_jrel || instr->i_jabs) {
-            /* Recursively inspect jump target */
-            effect = stack_effect(instr->i_opcode, instr->i_oparg, 1);
-            assert(effect != PY_INVALID_STACK_EFFECT);
-            target_depth = depth + effect;
-            if (target_depth > maxdepth) {
-                maxdepth = target_depth;
-            }
-            assert(target_depth >= 0); /* invalid code or bug in stackdepth() */
-            if (instr->i_opcode == CONTINUE_LOOP) {
-                /* Pops a variable number of values from the stack,
-                 * but the target should be already proceeding.
-                 */
-                assert(instr->i_target->b_seen);
-                assert(instr->i_target->b_startdepth <= depth);
-                goto out; /* remaining code is dead */
-            }
-            maxdepth = stackdepth_walk(c, instr->i_target,
-                                       target_depth, maxdepth);
-        }
-        depth = new_depth;
-        if (instr->i_opcode == JUMP_ABSOLUTE ||
-            instr->i_opcode == JUMP_FORWARD ||
-            instr->i_opcode == RETURN_VALUE ||
-            instr->i_opcode == RAISE_VARARGS ||
-            instr->i_opcode == BREAK_LOOP)
-        {
-            goto out; /* remaining code is dead */
-        }
+    /* XXX b->b_startdepth > depth only for the target of SETUP_FINALLY,
+     * SETUP_WITH and SETUP_ASYNC_WITH. */
+    assert(b->b_startdepth < 0 || b->b_startdepth >= depth);
+    if (b->b_startdepth < depth) {
+        assert(b->b_startdepth < 0);
+        b->b_startdepth = depth;
+        *(*sp)++ = b;
     }
-    if (b->b_next)
-        maxdepth = stackdepth_walk(c, b->b_next, depth, maxdepth);
-out:
-    b->b_seen = 0;
-    return maxdepth;
 }
 
 /* Find the flow path that needs the largest stack.  We assume that
@@ -5005,16 +4964,79 @@ stackdepth_walk(struct compiler *c, basicblock *b, int depth, int maxdepth)
 static int
 stackdepth(struct compiler *c)
 {
-    basicblock *b, *entryblock;
-    entryblock = NULL;
+    basicblock *b, *entryblock = NULL;
+    basicblock **stack, **sp;
+    int nblocks = 0, maxdepth = 0;
     for (b = c->u->u_blocks; b != NULL; b = b->b_list) {
-        b->b_seen = 0;
         b->b_startdepth = INT_MIN;
         entryblock = b;
+        nblocks++;
     }
     if (!entryblock)
         return 0;
-    return stackdepth_walk(c, entryblock, 0, 0);
+    stack = (basicblock **)PyObject_Malloc(sizeof(basicblock *) * nblocks);
+    if (!stack) {
+        PyErr_NoMemory();
+        return -1;
+    }
+
+    sp = stack;
+    stackdepth_push(&sp, entryblock, 0);
+    while (sp != stack) {
+        b = *--sp;
+        int depth = b->b_startdepth;
+        assert(depth >= 0);
+        basicblock *next = b->b_next;
+        for (int i = 0; i < b->b_iused; i++) {
+            struct instr *instr = &b->b_instr[i];
+            int effect = stack_effect(instr->i_opcode, instr->i_oparg, 0);
+            if (effect == PY_INVALID_STACK_EFFECT) {
+                fprintf(stderr, "opcode = %d\n", instr->i_opcode);
+                Py_FatalError("PyCompile_OpcodeStackEffect()");
+            }
+            int new_depth = depth + effect;
+            if (new_depth > maxdepth) {
+                maxdepth = new_depth;
+            }
+            assert(depth >= 0); /* invalid code or bug in stackdepth() */
+            if (instr->i_jrel || instr->i_jabs) {
+                effect = stack_effect(instr->i_opcode, instr->i_oparg, 1);
+                assert(effect != PY_INVALID_STACK_EFFECT);
+                int target_depth = depth + effect;
+                if (target_depth > maxdepth) {
+                    maxdepth = target_depth;
+                }
+                assert(target_depth >= 0); /* invalid code or bug in stackdepth() */
+                if (instr->i_opcode == CONTINUE_LOOP) {
+                    /* Pops a variable number of values from the stack,
+                     * but the target should be already proceeding.
+                     */
+                    assert(instr->i_target->b_startdepth >= 0);
+                    assert(instr->i_target->b_startdepth <= depth);
+                    /* remaining code is dead */
+                    next = NULL;
+                    break;
+                }
+                stackdepth_push(&sp, instr->i_target, target_depth);
+            }
+            depth = new_depth;
+            if (instr->i_opcode == JUMP_ABSOLUTE ||
+                instr->i_opcode == JUMP_FORWARD ||
+                instr->i_opcode == RETURN_VALUE ||
+                instr->i_opcode == RAISE_VARARGS ||
+                instr->i_opcode == BREAK_LOOP)
+            {
+                /* remaining code is dead */
+                next = NULL;
+                break;
+            }
+        }
+        if (next != NULL) {
+            stackdepth_push(&sp, next, depth);
+        }
+    }
+    PyObject_Free(stack);
+    return maxdepth;
 }
 
 static int
@@ -5320,7 +5342,7 @@ makecode(struct compiler *c, struct assembler *a)
     Py_ssize_t nlocals;
     int nlocals_int;
     int flags;
-    int argcount, kwonlyargcount;
+    int argcount, kwonlyargcount, maxdepth;
 
     tmp = dict_keys_inorder(c->u->u_consts, 0);
     if (!tmp)
@@ -5360,8 +5382,12 @@ makecode(struct compiler *c, struct assembler *a)
 
     argcount = Py_SAFE_DOWNCAST(c->u->u_argcount, Py_ssize_t, int);
     kwonlyargcount = Py_SAFE_DOWNCAST(c->u->u_kwonlyargcount, Py_ssize_t, int);
+    maxdepth = stackdepth(c);
+    if (maxdepth < 0) {
+        goto error;
+    }
     co = PyCode_New(argcount, kwonlyargcount,
-                    nlocals_int, stackdepth(c), flags,
+                    nlocals_int, maxdepth, flags,
                     bytecode, consts, names, varnames,
                     freevars, cellvars,
                     c->c_filename, c->u->u_name,
@@ -5448,7 +5474,7 @@ assemble(struct compiler *c, int addNone)
     }
     if (!assemble_init(&a, nblocks, c->u->u_firstlineno))
         goto error;
-    dfs(c, entryblock, &a);
+    dfs(c, entryblock, &a, nblocks);
 
     /* Can't modify the bytecode after computing jump offsets. */
     assemble_jump_offsets(&a, c);


From lp_benchmark_robot at intel.com  Thu Jan 11 21:55:32 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Thu, 11 Jan 2018 18:55:32 -0800
Subject: [Python-checkins] [1 up,
 64 flat] Results for Python (master branch) 2018-01-11
Message-ID: <1964fb9f-13bf-4633-913e-eab747832208@orsmsx153.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-11 03:03:07-08:00.
  - commit:		0a2da50
  - previous commit:	f80c0ca
  - revision date:	2018-01-11 13:03:20+02:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.905% |  -0.113%   |  +7.240%   |  +7.432%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.081% |  -0.237%   |  +21.751%  |  +9.054%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 0.605% |  -0.040%   |  +24.302%  |  +6.613%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.764% |  -0.005%   |  +22.154%  |  +6.332%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.638% |  +0.952%   |  +8.629%   |  +11.982%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 2.059% |  +1.533%   |  +10.739%  |  +11.507%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.414% |  +0.217%   |  +6.226%   |  +10.131%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.570% |  +0.200%   |  -0.385%   |  +7.797%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 4.175% |  -0.451%   |  +9.480%   |  +17.384%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 1.159% |  +0.491%   |  +12.469%  |  +13.852%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.585% |  +0.643%   |  +4.261%   |  +7.520%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 1.332% |  +0.739%   |  +5.932%   |  +3.095%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.860% |  +0.200%   |  +2.541%   |  +6.191%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.109% |  +1.421%   |  +13.580%  |  +10.560%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.434% |  +0.472%   |  +8.936%   |  +9.624%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.638% |  -0.301%   |  +6.230%   |  +11.782%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.284% |  +0.196%   |  +10.494%  |  +10.759%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.262% |  +0.927%   |  +8.695%   |  +9.560%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.834% |  +0.711%   |  +2.898%   |  +8.637%   |
+-----+------------------------+--------+------------+------------+------------+
| :-) |              json_loads| 1.177% |  +8.023%   |  +0.578%   |  +10.196%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.489% |  +2.006%   |  +10.285%  |  +9.478%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 3.022% |  +0.631%   |  +46.864%  |  +12.005%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.900% |  +1.841%   |  +10.800%  |  +11.839%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.514% |  -0.732%   |  +17.639%  |  +12.186%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.124% |  -0.688%   |  +9.954%   |  +9.014%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 3.579% |  +0.153%   |  +4.356%   |  +5.335%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.192% |  -0.585%   |  -4.003%   |  +4.076%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 2.030% |  +0.205%   |  +4.145%   |  +7.655%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.349% |  +3.009%   |  +4.430%   |  +9.602%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.755% |  -0.548%   |  -3.093%   |  +24.539%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.121% |  +0.099%   |  +3.542%   |  +21.753%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list|10.572% |  -2.403%   |  +3.535%   |  +20.932%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 3.132% |  +1.484%   |  +12.622%  |  +9.163%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.045% |  +0.308%   |  +0.201%   |  +9.921%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.129% |  +0.455%   |  +11.359%  |  +5.124%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.094% |  +0.277%   |  +2.802%   |  +5.339%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.232% |  -0.305%   |  +9.749%   |  +12.553%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.717% |  -0.404%   |  +3.071%   |  +10.756%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.525% |  +1.339%   |  -1.152%   |  +10.477%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.941% |  +0.889%   |  -3.369%   |  +3.919%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 2.394% |  +1.144%   |  +6.021%   |  +7.619%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.731% |  +0.772%   |  +8.180%   |  +14.333%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.743% |  +0.447%   |  -3.080%   |  +3.882%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 3.204% |  -0.574%   |  +26.183%  |  +10.757%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.052% |  -0.155%   |  +4.386%   |  +4.236%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.561% |  +0.316%   |  +14.465%  |  +8.787%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 2.827% |  -0.046%   |  -0.516%   |  -3.029%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.519% |  +0.859%   |  +6.188%   |  -2.575%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.411% |  +1.464%   |  +7.608%   |  +5.410%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.391% |  +0.352%   |  +6.533%   |  +4.894%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 4.474% |  -0.318%   |  +18.473%  |  +10.280%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.693% |  +1.926%   |  +12.309%  |  +5.233%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.098% |  +1.305%   |  +9.186%   |  +6.500%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.550% |  +0.887%   |  +12.461%  |  +7.292%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.224% |  -0.367%   |  +9.537%   |  +10.308%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 3.749% |  -1.575%   |  +18.577%  |  +10.270%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.974% |  +0.583%   |  +6.361%   |  +6.888%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 6.686% |  -0.925%   |  +0.520%   |  +3.699%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 7.643% |  -1.506%   |  +3.624%   |  +24.202%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 1.592% |  +2.595%   |  +0.978%   |  +14.530%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 2.078% |  +0.190%   |  +8.461%   |  +6.562%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.783% |  +1.281%   |  +2.001%   |  +11.913%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.666% |  -0.978%   |  +1.922%   |  +9.564%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 1.220% |  +0.711%   |  -6.842%   |  +13.554%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.332% |  +0.648%   |  +3.830%   |  +12.188%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-up-64-flat-results-for-python-master-branch-2018-01-11

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Fri Jan 12 00:53:52 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Fri, 12 Jan 2018 05:53:52 -0000
Subject: [Python-checkins] bpo-32467: Let collections.abc.ValuesView inherit
 from Collection (#5152)
Message-ID: <mailman.0.1515736433.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/02556fbade5e1e864dd09d5768a8dbbf5b3a0dac
commit: 02556fbade5e1e864dd09d5768a8dbbf5b3a0dac
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-11T21:53:49-08:00
summary:

bpo-32467: Let collections.abc.ValuesView inherit from Collection (#5152)

files:
A Misc/NEWS.d/next/Library/2018-01-11-00-33-42.bpo-32467.YVEOv6.rst
M Doc/library/collections.abc.rst
M Lib/_collections_abc.py
M Lib/test/test_collections.py

diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst
index 60154532094..2a3fb142f72 100644
--- a/Doc/library/collections.abc.rst
+++ b/Doc/library/collections.abc.rst
@@ -87,7 +87,8 @@ ABC                        Inherits from          Abstract Methods        Mixin
                            :class:`Set`                                   ``__iter__``
 :class:`KeysView`          :class:`MappingView`,                          ``__contains__``,
                            :class:`Set`                                   ``__iter__``
-:class:`ValuesView`        :class:`MappingView`                           ``__contains__``, ``__iter__``
+:class:`ValuesView`        :class:`MappingView`,                          ``__contains__``, ``__iter__``
+                           :class:`Collection`
 :class:`Awaitable`                                ``__await__``
 :class:`Coroutine`         :class:`Awaitable`     ``send``, ``throw``     ``close``
 :class:`AsyncIterable`                            ``__aiter__``
diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py
index a5c7bfcda1d..dbe30dff1fe 100644
--- a/Lib/_collections_abc.py
+++ b/Lib/_collections_abc.py
@@ -746,7 +746,7 @@ def __iter__(self):
 ItemsView.register(dict_items)
 
 
-class ValuesView(MappingView):
+class ValuesView(MappingView, Collection):
 
     __slots__ = ()
 
diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
index cb662355bb7..a55239e5730 100644
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -843,13 +843,13 @@ def test_Collection(self):
             self.assertFalse(issubclass(type(x), Collection), repr(type(x)))
         # Check some non-collection iterables
         non_col_iterables = [_test_gen(), iter(b''), iter(bytearray()),
-                             (x for x in []), dict().values()]
+                             (x for x in [])]
         for x in non_col_iterables:
             self.assertNotIsInstance(x, Collection)
             self.assertFalse(issubclass(type(x), Collection), repr(type(x)))
         # Check some collections
         samples = [set(), frozenset(), dict(), bytes(), str(), tuple(),
-                   list(), dict().keys(), dict().items()]
+                   list(), dict().keys(), dict().items(), dict().values()]
         for x in samples:
             self.assertIsInstance(x, Collection)
             self.assertTrue(issubclass(type(x), Collection), repr(type(x)))
diff --git a/Misc/NEWS.d/next/Library/2018-01-11-00-33-42.bpo-32467.YVEOv6.rst b/Misc/NEWS.d/next/Library/2018-01-11-00-33-42.bpo-32467.YVEOv6.rst
new file mode 100644
index 00000000000..58bf066f9f5
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-11-00-33-42.bpo-32467.YVEOv6.rst
@@ -0,0 +1 @@
+collections.abc.ValuesView now inherits from collections.abc.Collection.


From webhook-mailer at python.org  Fri Jan 12 01:06:37 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Fri, 12 Jan 2018 06:06:37 -0000
Subject: [Python-checkins] bpo-31145: Use dataclasses to create a
 prioritization wrapper (#5153)
Message-ID: <mailman.1.1515737199.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/0c3be9651f4f149f4a78bb7043d26db9e75cabc0
commit: 0c3be9651f4f149f4a78bb7043d26db9e75cabc0
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-11T22:06:34-08:00
summary:

bpo-31145: Use dataclasses to create a prioritization wrapper (#5153)

files:
M Doc/library/heapq.rst
M Doc/library/queue.rst

diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst
index e36ca8d7b34..0b1a3c8b00c 100644
--- a/Doc/library/heapq.rst
+++ b/Doc/library/heapq.rst
@@ -187,6 +187,17 @@ a tie-breaker so that two tasks with the same priority are returned in the order
 they were added. And since no two entry counts are the same, the tuple
 comparison will never attempt to directly compare two tasks.
 
+Another solution to the problem of non-comparable tasks is to create a wrapper
+class that ignores the task item and only compares the priority field::
+
+    from dataclasses import dataclass, field
+    from typing import Any
+
+    @dataclass(order=True)
+    class PrioritizedItem:
+        priority: int
+        item: Any=field(compare=False)
+
 The remaining challenges revolve around finding a pending task and making
 changes to its priority or removing it entirely.  Finding a task can be done
 with a dictionary pointing to an entry in the queue.
diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst
index bd0fc2d8f3c..f9a43bbac3b 100644
--- a/Doc/library/queue.rst
+++ b/Doc/library/queue.rst
@@ -56,6 +56,16 @@ The :mod:`queue` module defines the following classes and exceptions:
    one returned by ``sorted(list(entries))[0]``).  A typical pattern for entries
    is a tuple in the form: ``(priority_number, data)``.
 
+   If the *data* elements are not comparable, the data can be wrapped in a class
+   that ignores the data item and only compares the priority number::
+
+        from dataclasses import dataclass, field
+        from typing import Any
+
+        @dataclass(order=True)
+        class PrioritizedItem:
+            priority: int
+            item: Any=field(compare=False)
 
 .. exception:: Empty
 


From webhook-mailer at python.org  Fri Jan 12 01:18:57 2018
From: webhook-mailer at python.org (Mariatta)
Date: Fri, 12 Jan 2018 06:18:57 -0000
Subject: [Python-checkins] Lexical Analysis ref doc: Fix a typo in the
 string concatenation internal link (GH-5157)
Message-ID: <mailman.2.1515737939.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/3764bb075a73074f55568fc7c37adde6054eb59a
commit: 3764bb075a73074f55568fc7c37adde6054eb59a
branch: master
author: ?smail Ar?l?k <arilik.ismail at gmail.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-11T22:18:54-08:00
summary:

Lexical Analysis ref doc: Fix a typo in the string concatenation internal link (GH-5157)

In lexical analysis reference documentation, the internal link to
the string literal concatenation section was written as`.. _string-catenation:`.
Changed that to `.. _string-concatenation:`.

files:
M Doc/reference/lexical_analysis.rst

diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst
index 0f9cb684c68..95b0f53a83e 100644
--- a/Doc/reference/lexical_analysis.rst
+++ b/Doc/reference/lexical_analysis.rst
@@ -571,7 +571,7 @@ that a single backslash followed by a newline is interpreted as those two
 characters as part of the literal, *not* as a line continuation.
 
 
-.. _string-catenation:
+.. _string-concatenation:
 
 String literal concatenation
 ----------------------------


From webhook-mailer at python.org  Fri Jan 12 01:20:35 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Fri, 12 Jan 2018 06:20:35 -0000
Subject: [Python-checkins] OrderedDict import is no longer needed (#4890)
Message-ID: <mailman.3.1515738036.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/05565ed27e4445e99b59268ba42cf4dd598ece77
commit: 05565ed27e4445e99b59268ba42cf4dd598ece77
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-11T22:20:33-08:00
summary:

OrderedDict import is no longer needed (#4890)

files:
M Lib/argparse.py

diff --git a/Lib/argparse.py b/Lib/argparse.py
index d8bbd352fd3..e3da7f0443c 100644
--- a/Lib/argparse.py
+++ b/Lib/argparse.py
@@ -83,7 +83,6 @@
 ]
 
 
-import collections as _collections
 import os as _os
 import re as _re
 import sys as _sys
@@ -1084,7 +1083,7 @@ def __init__(self,
 
         self._prog_prefix = prog
         self._parser_class = parser_class
-        self._name_parser_map = _collections.OrderedDict()
+        self._name_parser_map = {}
         self._choices_actions = []
 
         super(_SubParsersAction, self).__init__(


From webhook-mailer at python.org  Fri Jan 12 01:50:48 2018
From: webhook-mailer at python.org (Mariatta)
Date: Fri, 12 Jan 2018 06:50:48 -0000
Subject: [Python-checkins] Lexical Analysis ref doc: Fix a typo in the
 string concatenation internal link (GH-5157) (GH-5160)
Message-ID: <mailman.4.1515739850.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/22b1050672c21600224afcf8d94e49c49e3affa5
commit: 22b1050672c21600224afcf8d94e49c49e3affa5
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-11T22:50:45-08:00
summary:

Lexical Analysis ref doc: Fix a typo in the string concatenation internal link (GH-5157) (GH-5160)

In lexical analysis reference documentation, the internal link to
the string literal concatenation section was written as`.. _string-catenation:`.
Changed that to `.. _string-concatenation:`.
(cherry picked from commit 3764bb075a73074f55568fc7c37adde6054eb59a)

files:
M Doc/reference/lexical_analysis.rst

diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst
index c0f2ee59b8c..7ad83947e04 100644
--- a/Doc/reference/lexical_analysis.rst
+++ b/Doc/reference/lexical_analysis.rst
@@ -571,7 +571,7 @@ that a single backslash followed by a newline is interpreted as those two
 characters as part of the literal, *not* as a line continuation.
 
 
-.. _string-catenation:
+.. _string-concatenation:
 
 String literal concatenation
 ----------------------------


From solipsis at pitrou.net  Fri Jan 12 04:13:34 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 12 Jan 2018 09:13:34 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=7
Message-ID: <20180112091334.1.ACFCD516E88F42FA@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_asyncio leaked [0, 0, 3] memory blocks, sum=3
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogr4AaZX', '--timeout', '7200']

From webhook-mailer at python.org  Fri Jan 12 04:18:51 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Fri, 12 Jan 2018 09:18:51 -0000
Subject: [Python-checkins] bpo-32473: Improve ABCMeta._dump_registry()
 readability (GH-5091)
Message-ID: <mailman.5.1515748734.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ae12f5d4c98f2095c2aadd58981453e955044697
commit: ae12f5d4c98f2095c2aadd58981453e955044697
branch: master
author: yahya-abou-imran <yahya-abou-imran at protonmail.com>
committer: INADA Naoki <methane at users.noreply.github.com>
date: 2018-01-12T18:18:44+09:00
summary:

bpo-32473: Improve ABCMeta._dump_registry() readability (GH-5091)

files:
A Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst
M Lib/abc.py

diff --git a/Lib/abc.py b/Lib/abc.py
index d13a0de89b4..9bdc36dce65 100644
--- a/Lib/abc.py
+++ b/Lib/abc.py
@@ -170,9 +170,11 @@ def _dump_registry(cls, file=None):
         """Debug helper to print the ABC registry."""
         print("Class: %s.%s" % (cls.__module__, cls.__qualname__), file=file)
         print("Inv.counter: %s" % ABCMeta._abc_invalidation_counter, file=file)
-        for name in sorted(cls.__dict__.keys()):
+        for name in cls.__dict__:
             if name.startswith("_abc_"):
                 value = getattr(cls, name)
+                if isinstance(value, WeakSet):
+                    value = set(value)
                 print("%s: %r" % (name, value), file=file)
 
     def __instancecheck__(cls, instance):
diff --git a/Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst b/Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst
new file mode 100644
index 00000000000..95b9d45e422
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst
@@ -0,0 +1 @@
+Improve ABCMeta._dump_registry() output readability


From webhook-mailer at python.org  Fri Jan 12 05:47:43 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Fri, 12 Jan 2018 10:47:43 -0000
Subject: [Python-checkins] bpo-32473: Improve ABCMeta._dump_registry()
 readability (GH-5091)
Message-ID: <mailman.6.1515754066.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/a91662affeb0aae2515cdc5e8f82269337105bf4
commit: a91662affeb0aae2515cdc5e8f82269337105bf4
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: INADA Naoki <methane at users.noreply.github.com>
date: 2018-01-12T19:47:36+09:00
summary:

bpo-32473: Improve ABCMeta._dump_registry() readability (GH-5091)

(cherry picked from commit ae12f5d4c98f2095c2aadd58981453e955044697)

files:
A Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst
M Lib/abc.py

diff --git a/Lib/abc.py b/Lib/abc.py
index 43a34a0bbde..a092db2618a 100644
--- a/Lib/abc.py
+++ b/Lib/abc.py
@@ -170,9 +170,11 @@ def _dump_registry(cls, file=None):
         """Debug helper to print the ABC registry."""
         print("Class: %s.%s" % (cls.__module__, cls.__qualname__), file=file)
         print("Inv.counter: %s" % ABCMeta._abc_invalidation_counter, file=file)
-        for name in sorted(cls.__dict__.keys()):
+        for name in sorted(cls.__dict__):
             if name.startswith("_abc_"):
                 value = getattr(cls, name)
+                if isinstance(value, WeakSet):
+                    value = set(value)
                 print("%s: %r" % (name, value), file=file)
 
     def __instancecheck__(cls, instance):
diff --git a/Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst b/Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst
new file mode 100644
index 00000000000..95b9d45e422
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-10-20-37-59.bpo-32473.mP_yJG.rst
@@ -0,0 +1 @@
+Improve ABCMeta._dump_registry() output readability


From webhook-mailer at python.org  Fri Jan 12 09:26:35 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Fri, 12 Jan 2018 14:26:35 -0000
Subject: [Python-checkins] bpo-32521: nis libtirpc (#5137)
Message-ID: <mailman.7.1515767196.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/f3031b8a7ad71d3b6ed05da7f3041d9efbe773cf
commit: f3031b8a7ad71d3b6ed05da7f3041d9efbe773cf
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-12T15:26:32+01:00
summary:

bpo-32521: nis libtirpc (#5137)

glibc has removed Sun RPC. Use replacement libtirpc headers and library in
nis module

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
M setup.py

diff --git a/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst b/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
new file mode 100644
index 00000000000..5ca9bcf7689
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
@@ -0,0 +1,2 @@
+glibc has removed Sun RPC. Use replacement libtirpc headers and library in
+nis module.
diff --git a/setup.py b/setup.py
index f1933f78dee..250425ebd31 100644
--- a/setup.py
+++ b/setup.py
@@ -1373,12 +1373,19 @@ class db_found(Exception): pass
             # Sun yellow pages. Some systems have the functions in libc.
             if (host_platform not in ['cygwin', 'qnx6'] and
                 find_file('rpcsvc/yp_prot.h', inc_dirs, []) is not None):
-                if (self.compiler.find_library_file(lib_dirs, 'nsl')):
-                    libs = ['nsl']
-                else:
-                    libs = []
+                nis_libs = []
+                nis_includes = []
+                if self.compiler.find_library_file(lib_dirs, 'nsl'):
+                    nis_libs.append('nsl')
+                if self.compiler.find_library_file(lib_dirs, 'tirpc'):
+                    # Sun RPC has been moved from glibc to libtirpc
+                    # rpcsvc/yp_prot.h is still in /usr/include, but
+                    # rpc/rpc.h has been moved into tirpc/ subdir.
+                    nis_libs.append('tirpc')
+                    nis_includes.append('/usr/include/tirpc')
                 exts.append( Extension('nis', ['nismodule.c'],
-                                       libraries = libs) )
+                                       libraries = nis_libs,
+                                       include_dirs=nis_includes) )
             else:
                 missing.append('nis')
         else:


From webhook-mailer at python.org  Fri Jan 12 10:29:39 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Fri, 12 Jan 2018 15:29:39 -0000
Subject: [Python-checkins] [3.6] bpo-32521: nis libtirpc (GH-5137) (#5165)
Message-ID: <mailman.8.1515770981.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/d55d6825d7bd0fc120c95d579065623ce6392a2f
commit: d55d6825d7bd0fc120c95d579065623ce6392a2f
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Christian Heimes <christian at python.org>
date: 2018-01-12T16:29:35+01:00
summary:

[3.6] bpo-32521: nis libtirpc (GH-5137) (#5165)

glibc has removed Sun RPC. Use replacement libtirpc headers and library in
nis module

Signed-off-by: Christian Heimes <christian at python.org>
(cherry picked from commit f3031b8a7ad71d3b6ed05da7f3041d9efbe773cf)

files:
A Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
M setup.py

diff --git a/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst b/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
new file mode 100644
index 00000000000..5ca9bcf7689
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
@@ -0,0 +1,2 @@
+glibc has removed Sun RPC. Use replacement libtirpc headers and library in
+nis module.
diff --git a/setup.py b/setup.py
index 5e2b81bbb6b..e8891e6b79e 100644
--- a/setup.py
+++ b/setup.py
@@ -1365,12 +1365,19 @@ class db_found(Exception): pass
             # Sun yellow pages. Some systems have the functions in libc.
             if (host_platform not in ['cygwin', 'qnx6'] and
                 find_file('rpcsvc/yp_prot.h', inc_dirs, []) is not None):
-                if (self.compiler.find_library_file(lib_dirs, 'nsl')):
-                    libs = ['nsl']
-                else:
-                    libs = []
+                nis_libs = []
+                nis_includes = []
+                if self.compiler.find_library_file(lib_dirs, 'nsl'):
+                    nis_libs.append('nsl')
+                if self.compiler.find_library_file(lib_dirs, 'tirpc'):
+                    # Sun RPC has been moved from glibc to libtirpc
+                    # rpcsvc/yp_prot.h is still in /usr/include, but
+                    # rpc/rpc.h has been moved into tirpc/ subdir.
+                    nis_libs.append('tirpc')
+                    nis_includes.append('/usr/include/tirpc')
                 exts.append( Extension('nis', ['nismodule.c'],
-                                       libraries = libs) )
+                                       libraries = nis_libs,
+                                       include_dirs=nis_includes) )
             else:
                 missing.append('nis')
         else:


From webhook-mailer at python.org  Fri Jan 12 10:35:28 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Fri, 12 Jan 2018 15:35:28 -0000
Subject: [Python-checkins] bpo-32521: nis libtirpc (GH-5137) (#5166)
Message-ID: <mailman.9.1515771329.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ab95b3074ee43098edf3f23b07fb18ef57ee614d
commit: ab95b3074ee43098edf3f23b07fb18ef57ee614d
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Christian Heimes <christian at python.org>
date: 2018-01-12T16:35:25+01:00
summary:

bpo-32521: nis libtirpc (GH-5137) (#5166)

glibc has removed Sun RPC. Use replacement libtirpc headers and library in
nis module

Signed-off-by: Christian Heimes <christian at python.org>
(cherry picked from commit f3031b8a7ad71d3b6ed05da7f3041d9efbe773cf)

files:
A Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
M setup.py

diff --git a/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst b/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
new file mode 100644
index 00000000000..5ca9bcf7689
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-08-18-02-33.bpo-32521.Kh-KoN.rst
@@ -0,0 +1,2 @@
+glibc has removed Sun RPC. Use replacement libtirpc headers and library in
+nis module.
diff --git a/setup.py b/setup.py
index 5503486ccbf..784a8d21f1f 100644
--- a/setup.py
+++ b/setup.py
@@ -1349,12 +1349,19 @@ class db_found(Exception): pass
             # Sun yellow pages. Some systems have the functions in libc.
             if (host_platform not in ['cygwin', 'atheos', 'qnx6'] and
                 find_file('rpcsvc/yp_prot.h', inc_dirs, []) is not None):
-                if (self.compiler.find_library_file(lib_dirs, 'nsl')):
-                    libs = ['nsl']
-                else:
-                    libs = []
+                nis_libs = []
+                nis_includes = []
+                if self.compiler.find_library_file(lib_dirs, 'nsl'):
+                    nis_libs.append('nsl')
+                if self.compiler.find_library_file(lib_dirs, 'tirpc'):
+                    # Sun RPC has been moved from glibc to libtirpc
+                    # rpcsvc/yp_prot.h is still in /usr/include, but
+                    # rpc/rpc.h has been moved into tirpc/ subdir.
+                    nis_libs.append('tirpc')
+                    nis_includes.append('/usr/include/tirpc')
                 exts.append( Extension('nis', ['nismodule.c'],
-                                       libraries = libs) )
+                                       libraries = nis_libs,
+                                       include_dirs=nis_includes) )
             else:
                 missing.append('nis')
         else:


From webhook-mailer at python.org  Fri Jan 12 17:28:40 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Fri, 12 Jan 2018 22:28:40 -0000
Subject: [Python-checkins] bpo-31993: Do not use memoryview when pickle
 large strings. (#5154)
Message-ID: <mailman.10.1515796121.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/5b76bdba071e7bbd9fda0b9b100d1506d95c04bd
commit: 5b76bdba071e7bbd9fda0b9b100d1506d95c04bd
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-13T00:28:31+02:00
summary:

bpo-31993: Do not use memoryview when pickle large strings. (#5154)

PyMemoryView_FromMemory() created a memoryview referring to
the internal data of the string.  When the string is destroyed
the memoryview become referring to a freed memory.

files:
M Lib/test/pickletester.py
M Misc/NEWS.d/3.7.0a4.rst
M Modules/_pickle.c

diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
index f4e3f81249c..062ec5ae72b 100644
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -2169,67 +2169,67 @@ def remove_frames(pickled, keep_frame=None):
     def test_framed_write_sizes_with_delayed_writer(self):
         class ChunkAccumulator:
             """Accumulate pickler output in a list of raw chunks."""
-
             def __init__(self):
                 self.chunks = []
-
             def write(self, chunk):
                 self.chunks.append(chunk)
-
             def concatenate_chunks(self):
-                # Some chunks can be memoryview instances, we need to convert
-                # them to bytes to be able to call join
-                return b"".join([c.tobytes() if hasattr(c, 'tobytes') else c
-                                 for c in self.chunks])
-
-        small_objects = [(str(i).encode('ascii'), i % 42, {'i': str(i)})
-                         for i in range(int(1e4))]
+                return b"".join(self.chunks)
 
         for proto in range(4, pickle.HIGHEST_PROTOCOL + 1):
+            objects = [(str(i).encode('ascii'), i % 42, {'i': str(i)})
+                       for i in range(int(1e4))]
+            # Add a large unique ASCII string
+            objects.append('0123456789abcdef' *
+                           (self.FRAME_SIZE_TARGET // 16 + 1))
+
             # Protocol 4 packs groups of small objects into frames and issues
             # calls to write only once or twice per frame:
             # The C pickler issues one call to write per-frame (header and
             # contents) while Python pickler issues two calls to write: one for
             # the frame header and one for the frame binary contents.
             writer = ChunkAccumulator()
-            self.pickler(writer, proto).dump(small_objects)
+            self.pickler(writer, proto).dump(objects)
 
             # Actually read the binary content of the chunks after the end
-            # of the call to dump: ant memoryview passed to write should not
+            # of the call to dump: any memoryview passed to write should not
             # be released otherwise this delayed access would not be possible.
             pickled = writer.concatenate_chunks()
             reconstructed = self.loads(pickled)
-            self.assertEqual(reconstructed, small_objects)
+            self.assertEqual(reconstructed, objects)
             self.assertGreater(len(writer.chunks), 1)
 
-            n_frames, remainder = divmod(len(pickled), self.FRAME_SIZE_TARGET)
-            if remainder > 0:
-                n_frames += 1
+            # memoryviews should own the memory.
+            del objects
+            support.gc_collect()
+            self.assertEqual(writer.concatenate_chunks(), pickled)
 
+            n_frames = (len(pickled) - 1) // self.FRAME_SIZE_TARGET + 1
             # There should be at least one call to write per frame
             self.assertGreaterEqual(len(writer.chunks), n_frames)
 
             # but not too many either: there can be one for the proto,
-            # one per-frame header and one per frame for the actual contents.
-            self.assertGreaterEqual(2 * n_frames + 1, len(writer.chunks))
+            # one per-frame header, one per frame for the actual contents,
+            # and two for the header.
+            self.assertLessEqual(len(writer.chunks), 2 * n_frames + 3)
 
-            chunk_sizes = [len(c) for c in writer.chunks[:-1]]
+            chunk_sizes = [len(c) for c in writer.chunks]
             large_sizes = [s for s in chunk_sizes
                            if s >= self.FRAME_SIZE_TARGET]
-            small_sizes = [s for s in chunk_sizes
-                           if s < self.FRAME_SIZE_TARGET]
+            medium_sizes = [s for s in chunk_sizes
+                           if 9 < s < self.FRAME_SIZE_TARGET]
+            small_sizes = [s for s in chunk_sizes if s <= 9]
 
             # Large chunks should not be too large:
             for chunk_size in large_sizes:
-                self.assertGreater(2 * self.FRAME_SIZE_TARGET, chunk_size)
-
-            last_chunk_size = len(writer.chunks[-1])
-            self.assertGreater(2 * self.FRAME_SIZE_TARGET, last_chunk_size)
-
-            # Small chunks (if any) should be very small
-            # (only proto and frame headers)
-            for chunk_size in small_sizes:
-                self.assertGreaterEqual(9, chunk_size)
+                self.assertLess(chunk_size, 2 * self.FRAME_SIZE_TARGET,
+                                chunk_sizes)
+            # There shouldn't bee too many small chunks: the protocol header,
+            # the frame headers and the large string headers are written
+            # in small chunks.
+            self.assertLessEqual(len(small_sizes),
+                                 len(large_sizes) + len(medium_sizes) + 3,
+                                 chunk_sizes)
 
     def test_nested_names(self):
         global Nested
diff --git a/Misc/NEWS.d/3.7.0a4.rst b/Misc/NEWS.d/3.7.0a4.rst
index ca9aa171bfa..a6322ed405b 100644
--- a/Misc/NEWS.d/3.7.0a4.rst
+++ b/Misc/NEWS.d/3.7.0a4.rst
@@ -693,9 +693,9 @@ ctypes._aix.find_library() Patch by: Michael Felt
 .. nonce: -OMNg8
 .. section: Library
 
-The picklers no longer allocate temporary memory when dumping large
-``bytes`` and ``str`` objects into a file object. Instead the data is
-directly streamed into the underlying file object.
+The pickler now uses less memory when serializing large bytes and str
+objects into a file.  Pickles created with protocol 4 will require less
+memory for unpickling large bytes and str objects.
 
 ..
 
diff --git a/Modules/_pickle.c b/Modules/_pickle.c
index 5b3de4d96dd..f8cbea12e80 100644
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -2184,8 +2184,9 @@ _Pickler_write_bytes(PicklerObject *self,
         /* Stream write the payload into the file without going through the
            output buffer. */
         if (payload == NULL) {
-            payload = mem = PyMemoryView_FromMemory((char *) data, data_size,
-                                                    PyBUF_READ);
+            /* TODO: It would be better to use a memoryview with a linked
+               original string if this is possible. */
+            payload = mem = PyBytes_FromStringAndSize(data, data_size);
             if (payload == NULL) {
                 return -1;
             }


From webhook-mailer at python.org  Fri Jan 12 18:09:02 2018
From: webhook-mailer at python.org (Brett Cannon)
Date: Fri, 12 Jan 2018 23:09:02 -0000
Subject: [Python-checkins] bpo-32248: Introduce the concept of
 Loader.get_resource_reader() (GH-5108)
Message-ID: <mailman.11.1515798543.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/bca42186b69e2e615d29d0d4fdb493c9fe71c48b
commit: bca42186b69e2e615d29d0d4fdb493c9fe71c48b
branch: master
author: Brett Cannon <brettcannon at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-12T15:08:59-08:00
summary:

bpo-32248: Introduce the concept of Loader.get_resource_reader() (GH-5108)

files:
A Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
M Doc/library/importlib.rst
M Doc/whatsnew/3.7.rst
M Lib/importlib/abc.py

diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
index e99c6067a3d..5fa1d7d869d 100644
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -233,7 +233,6 @@ ABC hierarchy::
      |    +-- MetaPathFinder
      |    +-- PathEntryFinder
      +-- Loader
-          +-- ResourceReader
           +-- ResourceLoader --------+
           +-- InspectLoader          |
                +-- ExecutionLoader --+
@@ -370,6 +369,13 @@ ABC hierarchy::
     An abstract base class for a :term:`loader`.
     See :pep:`302` for the exact definition for a loader.
 
+    For loaders that wish to support resource reading, they should
+    implement a ``get_resource_reader(fullname)`` method as specified
+    by :class:`importlib.abc.ResourceReader`.
+
+    .. versionchanged:: 3.7
+       Introduced the optional ``get_resource_reader()`` method.
+
     .. method:: create_module(spec)
 
        A method that returns the module object to use when
@@ -471,8 +477,7 @@ ABC hierarchy::
 
 .. class:: ResourceReader
 
-    An :term:`abstract base class` for :term:`package`
-    :term:`loaders <loader>` to provide the ability to read
+    An :term:`abstract base class` to provide the ability to read
     *resources*.
 
     From the perspective of this ABC, a *resource* is a binary
@@ -487,13 +492,20 @@ ABC hierarchy::
     expected to be a :term:`path-like object` which represents
     conceptually just a file name. This means that no subdirectory
     paths should be included in the *resource* argument. This is
-    because the location of the package that the loader is for acts
-    as the "directory". Hence the metaphor for directories and file
+    because the location of the package the reader is for, acts as the
+    "directory". Hence the metaphor for directories and file
     names is packages and resources, respectively. This is also why
     instances of this class are expected to directly correlate to
     a specific package (instead of potentially representing multiple
     packages or a module).
 
+    Loaders that wish to support resource reading are expected to
+    provide a method called ``get_resource_loader(fullname)`` which
+    returns an object implementing this ABC's interface. If the module
+    specified by fullname is not a package, this method should return
+    :const:`None`. An object compatible with this ABC should only be
+    returned when the specified module is a package.
+
     .. versionadded:: 3.7
 
     .. abstractmethod:: open_resource(resource)
@@ -529,9 +541,10 @@ ABC hierarchy::
         are known a priori and the non-resource names would be useful.
         For instance, returning subdirectory names is allowed so that
         when it is known that the package and resources are stored on
-        the file system then those subdirectory names can be used.
+        the file system then those subdirectory names can be used
+        directly.
 
-        The abstract method returns an empty iterator.
+        The abstract method returns an iterator of no items.
 
 
 .. class:: ResourceLoader
@@ -540,6 +553,10 @@ ABC hierarchy::
     :pep:`302` protocol for loading arbitrary resources from the storage
     back-end.
 
+    .. deprecated:: 3.7
+       This ABC is deprecated in favour of supporting resource loading
+       through :class:`importlib.abc.ResourceReader`.
+
     .. abstractmethod:: get_data(path)
 
         An abstract method to return the bytes for the data located at *path*.
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 992d9ba6e58..1041d31f302 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -328,8 +328,8 @@ importlib.resources
 This module provides several new APIs and one new ABC for access to, opening,
 and reading *resources* inside packages.  Resources are roughly akin to files
 inside of packages, but they needn't be actual files on the physical file
-system.  Module loaders can implement the
-:class:`importlib.abc.ResourceReader` ABC to support this new module's API.
+system.  Module loaders can provide :class:`importlib.abc.ResourceReader`
+implementations to support this new module's API.
 
 
 Improved Modules
@@ -429,6 +429,12 @@ and the ``--directory`` to the command line of the module :mod:`~http.server`.
 With this parameter, the server serves the specified directory, by default it uses the current working directory.
 (Contributed by St?phane Wirtel and Julien Palard in :issue:`28707`.)
 
+importlib
+---------
+
+The :class:`importlib.abc.ResourceReader` ABC was introduced to
+support the loading of resource from packages.
+
 locale
 ------
 
@@ -761,6 +767,9 @@ Deprecated
 
 - The :mod:`macpath` is now deprecated and will be removed in Python 3.8.
 
+- The :class:`importlib.abc.ResourceLoader` ABC has been deprecated in
+  favour of :class:`importlib.abc.ResourceReader`.
+
 
 Changes in the C API
 --------------------
@@ -785,8 +794,8 @@ Windows Only
   been used. If the specified version is not available py.exe will error exit.
   (Contributed by Steve Barnes in :issue:`30291`.)
 
-- The launcher can be run as "py -0" to produce a list of the installed pythons,
-  *with default marked with an asterix*. Running "py -0p" will include the paths.
+- The launcher can be run as ``py -0`` to produce a list of the installed pythons,
+  *with default marked with an asterisk*. Running ``py -0p`` will include the paths.
   If py is run with a version specifier that cannot be matched it will also print
   the *short form* list of available specifiers.
   (Contributed by Steve Barnes in :issue:`30362`.)
diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py
index b772db3758c..bbff7af5885 100644
--- a/Lib/importlib/abc.py
+++ b/Lib/importlib/abc.py
@@ -342,9 +342,14 @@ def set_data(self, path, data):
 _register(SourceLoader, machinery.SourceFileLoader)
 
 
-class ResourceReader(Loader):
+class ResourceReader:
 
-    """Abstract base class for loaders to provide resource reading support."""
+    """Abstract base class to provide resource-reading support.
+
+    Loaders that support resource reading are expected to implement
+    the ``get_resource_reader(fullname)`` method and have it either return None
+    or an object compatible with this ABC.
+    """
 
     @abc.abstractmethod
     def open_resource(self, resource):
diff --git a/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst b/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
new file mode 100644
index 00000000000..a41fde94b3d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
@@ -0,0 +1,13 @@
+Add :class:`importlib.abc.ResourceReader` as an ABC to provide a
+unified API for reading resources contained within packages. Loaders
+wishing to support resource reading are expected to implement the
+``get_resource_reader(fullname)`` method.
+
+Also add :mod:`importlib.resources` as the stdlib port of the
+``importlib_resources`` PyPI package. The modules provides a high-level
+API for end-users to read resources in a nicer fashion than having to
+directly interact with low-level details such as loaders.
+
+Thanks to this work, :class:`importlib.abc.ResourceLoader` has now
+been documented as deprecated due to its under-specified nature and
+lack of features as provided by :class:`importlib.abc.ResourceReader`.


From webhook-mailer at python.org  Fri Jan 12 22:36:00 2018
From: webhook-mailer at python.org (Mariatta)
Date: Sat, 13 Jan 2018 03:36:00 -0000
Subject: [Python-checkins] Improve enum.Flag code example (GH-5167)
Message-ID: <mailman.12.1515814561.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/0f31c74fcfdec8f9e6157de2c366f2273de81677
commit: 0f31c74fcfdec8f9e6157de2c366f2273de81677
branch: master
author: Julian Kahnert <mail at juliankahnert.de>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-12T19:35:57-08:00
summary:

Improve enum.Flag code example (GH-5167)

The code example that demonstrate how to use enum.Flag was missing
the import of enum.auto.

files:
M Doc/library/enum.rst

diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst
index 6548adf789d..5c1b226efc7 100644
--- a/Doc/library/enum.rst
+++ b/Doc/library/enum.rst
@@ -654,7 +654,7 @@ value and let :class:`Flag` select an appropriate value.
 Like :class:`IntFlag`, if a combination of :class:`Flag` members results in no
 flags being set, the boolean evaluation is :data:`False`::
 
-    >>> from enum import Flag
+    >>> from enum import Flag, auto
     >>> class Color(Flag):
     ...     RED = auto()
     ...     BLUE = auto()


From webhook-mailer at python.org  Fri Jan 12 23:23:24 2018
From: webhook-mailer at python.org (Mariatta)
Date: Sat, 13 Jan 2018 04:23:24 -0000
Subject: [Python-checkins] Improve enum.Flag code example (GH-5167) (GH-5171)
Message-ID: <mailman.13.1515817405.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/29b1aff71805df4c5b6f55c02bda94d07c044c81
commit: 29b1aff71805df4c5b6f55c02bda94d07c044c81
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-12T20:23:21-08:00
summary:

Improve enum.Flag code example (GH-5167) (GH-5171)

The code example that demonstrate how to use enum.Flag was missing
the import of enum.auto.
(cherry picked from commit 0f31c74fcfdec8f9e6157de2c366f2273de81677)

files:
M Doc/library/enum.rst

diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst
index 6548adf789d..5c1b226efc7 100644
--- a/Doc/library/enum.rst
+++ b/Doc/library/enum.rst
@@ -654,7 +654,7 @@ value and let :class:`Flag` select an appropriate value.
 Like :class:`IntFlag`, if a combination of :class:`Flag` members results in no
 flags being set, the boolean evaluation is :data:`False`::
 
-    >>> from enum import Flag
+    >>> from enum import Flag, auto
     >>> class Color(Flag):
     ...     RED = auto()
     ...     BLUE = auto()


From solipsis at pitrou.net  Sat Jan 13 04:13:18 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 13 Jan 2018 09:13:18 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=82
Message-ID: <20180113091318.1.CE7923A34BAD3960@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_asyncio leaked [3, 0, 0] memory blocks, sum=3
test_collections leaked [0, 0, 7] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [44, 0, 0] references, sum=44
test_multiprocessing_fork leaked [21, 1, 0] memory blocks, sum=22
test_multiprocessing_fork leaked [2, 0, 0] file descriptors, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflog6SxiaT', '--timeout', '7200']

From webhook-mailer at python.org  Sat Jan 13 13:35:42 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sat, 13 Jan 2018 18:35:42 -0000
Subject: [Python-checkins] Add itertools recipe for directly finding the
 n-th combination (#5161)
Message-ID: <mailman.14.1515868543.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/d37258dd2e189141906bd234385096cd8e885d8d
commit: d37258dd2e189141906bd234385096cd8e885d8d
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-13T10:35:40-08:00
summary:

Add itertools recipe for directly finding the n-th combination (#5161)

files:
M Doc/library/itertools.rst
M Lib/test/test_itertools.py

diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst
index 5af5422eefe..0b3829f19fa 100644
--- a/Doc/library/itertools.rst
+++ b/Doc/library/itertools.rst
@@ -859,6 +859,29 @@ which incur interpreter overhead.
        indices = sorted(random.randrange(n) for i in range(r))
        return tuple(pool[i] for i in indices)
 
+   def nth_combination(iterable, r, index):
+       'Equivalent to list(combinations(iterable, r))[index]'
+       pool = tuple(iterable)
+       n = len(pool)
+       if r < 0 or r > n:
+           raise ValueError
+       c = 1
+       k = min(r, n-r)
+       for i in range(1, k+1):
+           c = c * (n - k + i) // i
+       if index < 0:
+           index += c
+       if index < 0 or index >= c:
+           raise IndexError
+       result = []
+       while r:
+           c, n, r = c*r//n, n-1, r-1
+           while index >= c:
+               index -= c
+               c, n = c*(n-r)//n, n-1
+           result.append(pool[-1-n])
+       return tuple(result)
+
 Note, many of the above recipes can be optimized by replacing global lookups
 with local variables defined as default values.  For example, the
 *dotproduct* recipe can be written as::
diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
index a359905e2e4..4fcc02acbf4 100644
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -2262,6 +2262,30 @@ def test_permutations_sizeof(self):
 ...     # first_true([a,b], x, f) --> a if f(a) else b if f(b) else x
 ...     return next(filter(pred, iterable), default)
 
+>>> def nth_combination(iterable, r, index):
+...     'Equivalent to list(combinations(iterable, r))[index]'
+...     pool = tuple(iterable)
+...     n = len(pool)
+...     if r < 0 or r > n:
+...         raise ValueError
+...     c = 1
+...     k = min(r, n-r)
+...     for i in range(1, k+1):
+...         c = c * (n - k + i) // i
+...     if index < 0:
+...         index += c
+...     if index < 0 or index >= c:
+...         raise IndexError
+...     result = []
+...     while r:
+...         c, n, r = c*r//n, n-1, r-1
+...         while index >= c:
+...             index -= c
+...             c, n = c*(n-r)//n, n-1
+...         result.append(pool[-1-n])
+...     return tuple(result)
+
+
 This is not part of the examples but it tests to make sure the definitions
 perform as purported.
 
@@ -2345,6 +2369,15 @@ def test_permutations_sizeof(self):
 >>> first_true('ABC0DEF1', '9', str.isdigit)
 '0'
 
+>>> population = 'ABCDEFGH'
+>>> for r in range(len(population) + 1):
+...     seq = list(combinations(population, r))
+...     for i in range(len(seq)):
+...         assert nth_combination(population, r, i) == seq[i]
+...     for i in range(-len(seq), 0):
+...         assert nth_combination(population, r, i) == seq[i]
+
+
 """
 
 __test__ = {'libreftest' : libreftest}


From webhook-mailer at python.org  Sat Jan 13 14:21:18 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sat, 13 Jan 2018 19:21:18 -0000
Subject: [Python-checkins] Add itertools recipe for directly finding the
 n-th combination (GH-5161) (#5174)
Message-ID: <mailman.15.1515871279.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/cf4cd4bccbf7c8931a3c6209457c6f7add4c7b5c
commit: cf4cd4bccbf7c8931a3c6209457c6f7add4c7b5c
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-13T11:21:15-08:00
summary:

Add itertools recipe for directly finding the n-th combination (GH-5161) (#5174)

(cherry picked from commit d37258dd2e189141906bd234385096cd8e885d8d)

files:
M Doc/library/itertools.rst
M Lib/test/test_itertools.py

diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst
index d01ce8f3757..700a13a07fe 100644
--- a/Doc/library/itertools.rst
+++ b/Doc/library/itertools.rst
@@ -859,6 +859,29 @@ which incur interpreter overhead.
        indices = sorted(random.randrange(n) for i in range(r))
        return tuple(pool[i] for i in indices)
 
+   def nth_combination(iterable, r, index):
+       'Equivalent to list(combinations(iterable, r))[index]'
+       pool = tuple(iterable)
+       n = len(pool)
+       if r < 0 or r > n:
+           raise ValueError
+       c = 1
+       k = min(r, n-r)
+       for i in range(1, k+1):
+           c = c * (n - k + i) // i
+       if index < 0:
+           index += c
+       if index < 0 or index >= c:
+           raise IndexError
+       result = []
+       while r:
+           c, n, r = c*r//n, n-1, r-1
+           while index >= c:
+               index -= c
+               c, n = c*(n-r)//n, n-1
+           result.append(pool[-1-n])
+       return tuple(result)
+
 Note, many of the above recipes can be optimized by replacing global lookups
 with local variables defined as default values.  For example, the
 *dotproduct* recipe can be written as::
diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
index a978134e88e..1ad37ae35be 100644
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -2229,6 +2229,30 @@ def test_permutations_sizeof(self):
 ...     # first_true([a,b], x, f) --> a if f(a) else b if f(b) else x
 ...     return next(filter(pred, iterable), default)
 
+>>> def nth_combination(iterable, r, index):
+...     'Equivalent to list(combinations(iterable, r))[index]'
+...     pool = tuple(iterable)
+...     n = len(pool)
+...     if r < 0 or r > n:
+...         raise ValueError
+...     c = 1
+...     k = min(r, n-r)
+...     for i in range(1, k+1):
+...         c = c * (n - k + i) // i
+...     if index < 0:
+...         index += c
+...     if index < 0 or index >= c:
+...         raise IndexError
+...     result = []
+...     while r:
+...         c, n, r = c*r//n, n-1, r-1
+...         while index >= c:
+...             index -= c
+...             c, n = c*(n-r)//n, n-1
+...         result.append(pool[-1-n])
+...     return tuple(result)
+
+
 This is not part of the examples but it tests to make sure the definitions
 perform as purported.
 
@@ -2312,6 +2336,15 @@ def test_permutations_sizeof(self):
 >>> first_true('ABC0DEF1', '9', str.isdigit)
 '0'
 
+>>> population = 'ABCDEFGH'
+>>> for r in range(len(population) + 1):
+...     seq = list(combinations(population, r))
+...     for i in range(len(seq)):
+...         assert nth_combination(population, r, i) == seq[i]
+...     for i in range(-len(seq), 0):
+...         assert nth_combination(population, r, i) == seq[i]
+
+
 """
 
 __test__ = {'libreftest' : libreftest}


From solipsis at pitrou.net  Sun Jan 14 04:15:07 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 14 Jan 2018 09:15:07 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=8
Message-ID: <20180114091507.1.E6FFC2D50BB940B4@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_asyncio leaked [0, 3, 0] memory blocks, sum=3
test_collections leaked [-7, 8, 0] memory blocks, sum=1
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloggIVUgU', '--timeout', '7200']

From webhook-mailer at python.org  Sun Jan 14 13:20:16 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 14 Jan 2018 18:20:16 -0000
Subject: [Python-checkins] bpo-29476: Simplify set_add_entry() (#5175)
Message-ID: <mailman.17.1515954017.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/3329992e31bd0494a7d7312853f7ffd054737e27
commit: 3329992e31bd0494a7d7312853f7ffd054737e27
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-14T10:20:13-08:00
summary:

bpo-29476: Simplify set_add_entry() (#5175)

files:
M Objects/setobject.c

diff --git a/Objects/setobject.c b/Objects/setobject.c
index cd5d2dd83c0..c920fb98534 100644
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -187,7 +187,7 @@ set_add_entry(PySetObject *so, PyObject *key, Py_hash_t hash)
                 goto restart;
             mask = so->mask;                 /* help avoid a register spill */
         }
-        else if (entry->hash == -1 && freeslot == NULL)
+        else if (entry->hash == -1)
             freeslot = entry;
 
         if (i + LINEAR_PROBES <= mask) {
@@ -216,7 +216,7 @@ set_add_entry(PySetObject *so, PyObject *key, Py_hash_t hash)
                         goto restart;
                     mask = so->mask;
                 }
-                else if (entry->hash == -1 && freeslot == NULL)
+                else if (entry->hash == -1)
                     freeslot = entry;
             }
         }


From webhook-mailer at python.org  Sun Jan 14 23:57:54 2018
From: webhook-mailer at python.org (Mariatta)
Date: Mon, 15 Jan 2018 04:57:54 -0000
Subject: [Python-checkins] bpo-15221: Update os.path.is*() documentation
 (GH-5185)
Message-ID: <mailman.18.1515992275.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/b3dd18d4035803b50c65a434955966d1b3b1f363
commit: b3dd18d4035803b50c65a434955966d1b3b1f363
branch: master
author: Cheryl Sabella <cheryl.sabella at gmail.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-14T20:57:51-08:00
summary:

bpo-15221: Update os.path.is*() documentation (GH-5185)

`os.path.is*()` can return False if the file can't be accessed.
The behaviour is documented in details in `os.path.exists()`.
Link to `os.path.exists()` from `os.path.is*()`.

files:
M Doc/library/os.path.rst

diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst
index 06d4ecedd86..36bb21c222e 100644
--- a/Doc/library/os.path.rst
+++ b/Doc/library/os.path.rst
@@ -240,8 +240,9 @@ the :mod:`glob` module.)
 
 .. function:: isfile(path)
 
-   Return ``True`` if *path* is an existing regular file.  This follows symbolic
-   links, so both :func:`islink` and :func:`isfile` can be true for the same path.
+   Return ``True`` if *path* is an :func:`existing <exists>` regular file.
+   This follows symbolic links, so both :func:`islink` and :func:`isfile` can
+   be true for the same path.
 
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
@@ -249,8 +250,9 @@ the :mod:`glob` module.)
 
 .. function:: isdir(path)
 
-   Return ``True`` if *path* is an existing directory.  This follows symbolic
-   links, so both :func:`islink` and :func:`isdir` can be true for the same path.
+   Return ``True`` if *path* is an :func:`existing <exists>` directory.  This
+   follows symbolic links, so both :func:`islink` and :func:`isdir` can be true
+   for the same path.
 
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
@@ -258,8 +260,9 @@ the :mod:`glob` module.)
 
 .. function:: islink(path)
 
-   Return ``True`` if *path* refers to a directory entry that is a symbolic link.
-   Always ``False`` if symbolic links are not supported by the Python runtime.
+   Return ``True`` if *path* refers to an :func:`existing <exists>` directory
+   entry that is a symbolic link.  Always ``False`` if symbolic links are not
+   supported by the Python runtime.
 
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.


From webhook-mailer at python.org  Mon Jan 15 00:08:39 2018
From: webhook-mailer at python.org (Mariatta)
Date: Mon, 15 Jan 2018 05:08:39 -0000
Subject: [Python-checkins] bpo-26330: Update shutil.disk_usage()
 documentation (GH-5184)
Message-ID: <mailman.19.1515992920.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ee3b83547c6b0cac1da2cb44aaaea533a1d1bbc8
commit: ee3b83547c6b0cac1da2cb44aaaea533a1d1bbc8
branch: master
author: Cheryl Sabella <cheryl.sabella at gmail.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-14T21:08:37-08:00
summary:

bpo-26330: Update shutil.disk_usage() documentation (GH-5184)

Clarify that on Windows, path must be a directory.
On Unix, path can be a file or a directory.

files:
M Doc/library/shutil.rst

diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
index 2b84fa29372..1527deb167f 100644
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -318,7 +318,8 @@ Directory and files operations
 
    Return disk usage statistics about the given path as a :term:`named tuple`
    with the attributes *total*, *used* and *free*, which are the amount of
-   total, used and free space, in bytes.
+   total, used and free space, in bytes.  On Windows, *path* must be a
+   directory; on Unix, it can be a file or directory.
 
    .. versionadded:: 3.3
 


From webhook-mailer at python.org  Mon Jan 15 00:44:52 2018
From: webhook-mailer at python.org (Mariatta)
Date: Mon, 15 Jan 2018 05:44:52 -0000
Subject: [Python-checkins] bpo-15221: Update os.path.is*() documentation
 (GH-5185) (GH-5186)
Message-ID: <mailman.20.1515995093.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/bb80645d06c16461b0b58d8983cd7fa6962ba3c6
commit: bb80645d06c16461b0b58d8983cd7fa6962ba3c6
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-14T21:44:49-08:00
summary:

bpo-15221: Update os.path.is*() documentation (GH-5185) (GH-5186)

`os.path.is*()` can return False if the file can't be accessed.
The behaviour is documented in details in `os.path.exists()`.
Link to `os.path.exists()` from `os.path.is*()`.
(cherry picked from commit b3dd18d4035803b50c65a434955966d1b3b1f363)

files:
M Doc/library/os.path.rst

diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst
index 406054e5d7a..06493f9505d 100644
--- a/Doc/library/os.path.rst
+++ b/Doc/library/os.path.rst
@@ -246,8 +246,9 @@ the :mod:`glob` module.)
 
 .. function:: isfile(path)
 
-   Return ``True`` if *path* is an existing regular file.  This follows symbolic
-   links, so both :func:`islink` and :func:`isfile` can be true for the same path.
+   Return ``True`` if *path* is an :func:`existing <exists>` regular file.
+   This follows symbolic links, so both :func:`islink` and :func:`isfile` can
+   be true for the same path.
 
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
@@ -255,8 +256,9 @@ the :mod:`glob` module.)
 
 .. function:: isdir(path)
 
-   Return ``True`` if *path* is an existing directory.  This follows symbolic
-   links, so both :func:`islink` and :func:`isdir` can be true for the same path.
+   Return ``True`` if *path* is an :func:`existing <exists>` directory.  This
+   follows symbolic links, so both :func:`islink` and :func:`isdir` can be true
+   for the same path.
 
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
@@ -264,8 +266,9 @@ the :mod:`glob` module.)
 
 .. function:: islink(path)
 
-   Return ``True`` if *path* refers to a directory entry that is a symbolic link.
-   Always ``False`` if symbolic links are not supported by the Python runtime.
+   Return ``True`` if *path* refers to an :func:`existing <exists>` directory
+   entry that is a symbolic link.  Always ``False`` if symbolic links are not
+   supported by the Python runtime.
 
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.


From solipsis at pitrou.net  Mon Jan 15 04:12:07 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 15 Jan 2018 09:12:07 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-5
Message-ID: <20180115091207.1.2CC135AB8FA8363D@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [-7, 1, 0] memory blocks, sum=-6
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, -2, 1] memory blocks, sum=-1
test_multiprocessing_forkserver leaked [0, 0, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogLvrb56', '--timeout', '7200']

From webhook-mailer at python.org  Mon Jan 15 04:45:57 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 15 Jan 2018 09:45:57 -0000
Subject: [Python-checkins] bpo-29240: Fix locale encodings in UTF-8 Mode
 (#5170)
Message-ID: <mailman.21.1516009559.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/7ed7aead9503102d2ed316175f198104e0cd674c
commit: 7ed7aead9503102d2ed316175f198104e0cd674c
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-15T10:45:49+01:00
summary:

bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)

Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
  encoding error, they return the position of the error and an error
  message which are used to raise Unicode errors in
  PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
* Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
* PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
  cases, especially for the strict error handler.
* Add _Py_DecodeUTF8Ex(): return more information on decoding error
  and supports the strict error handler.
* Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
* Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
* Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
  and time zone name.
* Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
  and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
  the "current" locale.
* Remove _PyUnicode_DecodeCurrentLocale(),
  _PyUnicode_DecodeCurrentLocaleAndSize() and
  _PyUnicode_EncodeCurrentLocale().

files:
M Doc/c-api/sys.rst
M Doc/c-api/unicode.rst
M Include/fileutils.h
M Include/unicodeobject.h
M Modules/_datetimemodule.c
M Modules/_localemodule.c
M Modules/getpath.c
M Modules/readline.c
M Modules/timemodule.c
M Objects/unicodeobject.c
M Python/fileutils.c
M Python/pathconfig.c

diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 20bc7bd3df3..e4da96c493c 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -106,6 +106,16 @@ Operating System Utilities
    surrogate character, escape the bytes using the surrogateescape error
    handler instead of decoding them.
 
+   Encoding, highest priority to lowest priority:
+
+   * ``UTF-8`` on macOS and Android;
+   * ``UTF-8`` if the Python UTF-8 mode is enabled;
+   * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
+     ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
+     and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
+     ``ISO-8859-1`` encoding.
+   * the current locale encoding.
+
    Return a pointer to a newly allocated wide character string, use
    :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write
    the number of wide characters excluding the null character into ``*size``
@@ -137,6 +147,18 @@ Operating System Utilities
    :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters
    in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
 
+   Encoding, highest priority to lowest priority:
+
+   * ``UTF-8`` on macOS and Android;
+   * ``UTF-8`` if the Python UTF-8 mode is enabled;
+   * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
+     ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
+     and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
+     ``ISO-8859-1`` encoding.
+   * the current locale encoding.
+
+   The function uses the UTF-8 encoding in the Python UTF-8 mode.
+
    Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free`
    to free the memory. Return ``NULL`` on encoding error or memory allocation
    error
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 45aff1b7e3c..3f6c0559907 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -770,12 +770,20 @@ system.
    :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
    Python startup).
 
+   This function ignores the Python UTF-8 mode.
+
    .. seealso::
 
       The :c:func:`Py_DecodeLocale` function.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.7
+      The function now also uses the current locale encoding for the
+      ``surrogateescape`` error handler. Previously, :c:func:`Py_DecodeLocale`
+      was used for the ``surrogateescape``, and the current locale encoding was
+      used for ``strict``.
+
 
 .. c:function:: PyObject* PyUnicode_DecodeLocale(const char *str, const char *errors)
 
@@ -797,12 +805,20 @@ system.
    :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
    Python startup).
 
+   This function ignores the Python UTF-8 mode.
+
    .. seealso::
 
       The :c:func:`Py_EncodeLocale` function.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.7
+      The function now also uses the current locale encoding for the
+      ``surrogateescape`` error handler. Previously, :c:func:`Py_EncodeLocale`
+      was used for the ``surrogateescape``, and the current locale encoding was
+      used for ``strict``.
+
 
 File System Encoding
 """"""""""""""""""""
diff --git a/Include/fileutils.h b/Include/fileutils.h
index 2527d84669d..b4f8b11a635 100644
--- a/Include/fileutils.h
+++ b/Include/fileutils.h
@@ -20,18 +20,41 @@ PyAPI_FUNC(char*) _Py_EncodeLocaleRaw(
 #endif
 
 #ifdef Py_BUILD_CORE
+PyAPI_FUNC(int) _Py_DecodeUTF8Ex(
+    const char *arg,
+    Py_ssize_t arglen,
+    wchar_t **wstr,
+    size_t *wlen,
+    const char **reason,
+    int surrogateescape);
+
+PyAPI_FUNC(int) _Py_EncodeUTF8Ex(
+    const wchar_t *text,
+    char **str,
+    size_t *error_pos,
+    const char **reason,
+    int raw_malloc,
+    int surrogateescape);
+
 PyAPI_FUNC(wchar_t*) _Py_DecodeUTF8_surrogateescape(
-    const char *s,
-    Py_ssize_t size,
-    size_t *p_wlen);
+    const char *arg,
+    Py_ssize_t arglen);
 
-PyAPI_FUNC(wchar_t *) _Py_DecodeCurrentLocale(
+PyAPI_FUNC(int) _Py_DecodeLocaleEx(
     const char *arg,
-    size_t *size);
+    wchar_t **wstr,
+    size_t *wlen,
+    const char **reason,
+    int current_locale,
+    int surrogateescape);
 
-PyAPI_FUNC(char*) _Py_EncodeCurrentLocale(
+PyAPI_FUNC(int) _Py_EncodeLocaleEx(
     const wchar_t *text,
-    size_t *error_pos);
+    char **str,
+    size_t *error_pos,
+    const char **reason,
+    int current_locale,
+    int surrogateescape);
 #endif
 
 #ifndef Py_LIMITED_API
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
index d263026b562..0274de6733a 100644
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1810,20 +1810,6 @@ PyAPI_FUNC(PyObject*) PyUnicode_EncodeLocale(
     PyObject *unicode,
     const char *errors
     );
-
-PyAPI_FUNC(PyObject*) _PyUnicode_DecodeCurrentLocale(
-    const char *str,
-    const char *errors);
-
-PyAPI_FUNC(PyObject*) _PyUnicode_DecodeCurrentLocaleAndSize(
-    const char *str,
-    Py_ssize_t len,
-    const char *errors);
-
-PyAPI_FUNC(PyObject*) _PyUnicode_EncodeCurrentLocale(
-    PyObject *unicode,
-    const char *errors
-    );
 #endif
 
 /* --- File system encoding ---------------------------------------------- */
diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
index 624196702b6..e68c7c0a1c5 100644
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -696,7 +696,7 @@ static int parse_isoformat_date(const char *dtstr,
     if (NULL == p) {
         return -1;
     }
-    
+
     if (*(p++) != '-') {
         return -2;
     }
diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
index e364668d3dd..324b694b830 100644
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -572,8 +572,9 @@ PyIntl_bind_textdomain_codeset(PyObject* self,PyObject*args)
     if (!PyArg_ParseTuple(args, "sz", &domain, &codeset))
         return NULL;
     codeset = bind_textdomain_codeset(domain, codeset);
-    if (codeset)
+    if (codeset) {
         return PyUnicode_DecodeLocale(codeset, NULL);
+    }
     Py_RETURN_NONE;
 }
 #endif
diff --git a/Modules/getpath.c b/Modules/getpath.c
index 85e737b61d0..e6a3e8e78cd 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -449,8 +449,8 @@ search_for_exec_prefix(const _PyCoreConfig *core_config,
             n = fread(buf, 1, MAXPATHLEN, f);
             buf[n] = '\0';
             fclose(f);
-            rel_builddir_path = _Py_DecodeUTF8_surrogateescape(buf, n, NULL);
-            if (rel_builddir_path != NULL) {
+            rel_builddir_path = _Py_DecodeUTF8_surrogateescape(buf, n);
+            if (rel_builddir_path) {
                 wcsncpy(exec_prefix, calculate->argv0_path, MAXPATHLEN);
                 exec_prefix[MAXPATHLEN] = L'\0';
                 joinpath(exec_prefix, rel_builddir_path);
diff --git a/Modules/readline.c b/Modules/readline.c
index caf661c81d4..811fca8cd92 100644
--- a/Modules/readline.c
+++ b/Modules/readline.c
@@ -132,13 +132,13 @@ static PyModuleDef readlinemodule;
 static PyObject *
 encode(PyObject *b)
 {
-    return _PyUnicode_EncodeCurrentLocale(b, "surrogateescape");
+    return PyUnicode_EncodeLocale(b, "surrogateescape");
 }
 
 static PyObject *
 decode(const char *s)
 {
-    return _PyUnicode_DecodeCurrentLocale(s, "surrogateescape");
+    return PyUnicode_DecodeLocale(s, "surrogateescape");
 }
 
 
diff --git a/Modules/timemodule.c b/Modules/timemodule.c
index 4e7f9d9482e..b17ab5ae824 100644
--- a/Modules/timemodule.c
+++ b/Modules/timemodule.c
@@ -418,11 +418,11 @@ tmtotuple(struct tm *p
     SET(8, p->tm_isdst);
 #ifdef HAVE_STRUCT_TM_TM_ZONE
     PyStructSequence_SET_ITEM(v, 9,
-        _PyUnicode_DecodeCurrentLocale(p->tm_zone, "surrogateescape"));
+        PyUnicode_DecodeLocale(p->tm_zone, "surrogateescape"));
     SET(10, p->tm_gmtoff);
 #else
     PyStructSequence_SET_ITEM(v, 9,
-        _PyUnicode_DecodeCurrentLocale(zone, "surrogateescape"));
+        PyUnicode_DecodeLocale(zone, "surrogateescape"));
     PyStructSequence_SET_ITEM(v, 10, _PyLong_FromTime_t(gmtoff));
 #endif /* HAVE_STRUCT_TM_TM_ZONE */
 #undef SET
@@ -809,8 +809,7 @@ time_strftime(PyObject *self, PyObject *args)
 #ifdef HAVE_WCSFTIME
             ret = PyUnicode_FromWideChar(outbuf, buflen);
 #else
-            ret = _PyUnicode_DecodeCurrentLocaleAndSize(outbuf, buflen,
-                                                        "surrogateescape");
+            ret = PyUnicode_DecodeLocaleAndSize(outbuf, buflen, "surrogateescape");
 #endif
             PyMem_Free(outbuf);
             break;
@@ -1541,8 +1540,8 @@ PyInit_timezone(PyObject *m) {
     PyModule_AddIntConstant(m, "altzone", timezone-3600);
 #endif
     PyModule_AddIntConstant(m, "daylight", daylight);
-    otz0 = _PyUnicode_DecodeCurrentLocale(tzname[0], "surrogateescape");
-    otz1 = _PyUnicode_DecodeCurrentLocale(tzname[1], "surrogateescape");
+    otz0 = PyUnicode_DecodeLocale(tzname[0], "surrogateescape");
+    otz1 = PyUnicode_DecodeLocale(tzname[1], "surrogateescape");
     PyModule_AddObject(m, "tzname", Py_BuildValue("(NN)", otz0, otz1));
 #else /* !HAVE_TZNAME || __GLIBC__ || __CYGWIN__*/
     {
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index a6e02f478b1..07330119dc3 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3327,53 +3327,6 @@ PyUnicode_AsEncodedObject(PyObject *unicode,
     return NULL;
 }
 
-static size_t
-wcstombs_errorpos(const wchar_t *wstr)
-{
-    size_t len;
-#if SIZEOF_WCHAR_T == 2
-    wchar_t buf[3];
-#else
-    wchar_t buf[2];
-#endif
-    char outbuf[MB_LEN_MAX];
-    const wchar_t *start, *previous;
-
-#if SIZEOF_WCHAR_T == 2
-    buf[2] = 0;
-#else
-    buf[1] = 0;
-#endif
-    start = wstr;
-    while (*wstr != L'\0')
-    {
-        previous = wstr;
-#if SIZEOF_WCHAR_T == 2
-        if (Py_UNICODE_IS_HIGH_SURROGATE(wstr[0])
-            && Py_UNICODE_IS_LOW_SURROGATE(wstr[1]))
-        {
-            buf[0] = wstr[0];
-            buf[1] = wstr[1];
-            wstr += 2;
-        }
-        else {
-            buf[0] = *wstr;
-            buf[1] = 0;
-            wstr++;
-        }
-#else
-        buf[0] = *wstr;
-        wstr++;
-#endif
-        len = wcstombs(outbuf, buf, sizeof(outbuf));
-        if (len == (size_t)-1)
-            return previous - start;
-    }
-
-    /* failed to find the unencodable character */
-    return 0;
-}
-
 static int
 locale_error_handler(const char *errors, int *surrogateescape)
 {
@@ -3396,130 +3349,60 @@ locale_error_handler(const char *errors, int *surrogateescape)
 }
 
 static PyObject *
-unicode_encode_locale(PyObject *unicode, const char *errors, int current_locale)
+unicode_encode_locale(PyObject *unicode, const char *errors,
+                      int current_locale)
 {
-    Py_ssize_t wlen, wlen2;
-    wchar_t *wstr;
-    char *errmsg;
-    PyObject *bytes, *reason, *exc;
-    size_t error_pos, errlen;
     int surrogateescape;
-
     if (locale_error_handler(errors, &surrogateescape) < 0)
         return NULL;
 
-    wstr = PyUnicode_AsWideCharString(unicode, &wlen);
-    if (wstr == NULL)
+    Py_ssize_t wlen;
+    wchar_t *wstr = PyUnicode_AsWideCharString(unicode, &wlen);
+    if (wstr == NULL) {
         return NULL;
+    }
 
-    wlen2 = wcslen(wstr);
+    Py_ssize_t wlen2 = wcslen(wstr);
     if (wlen2 != wlen) {
         PyMem_Free(wstr);
         PyErr_SetString(PyExc_ValueError, "embedded null character");
         return NULL;
     }
 
-    if (surrogateescape) {
-        /* "surrogateescape" error handler */
-        char *str;
-
-        if (current_locale) {
-            str = _Py_EncodeCurrentLocale(wstr, &error_pos);
-        }
-        else {
-            str = Py_EncodeLocale(wstr, &error_pos);
-        }
-        if (str == NULL) {
-            if (error_pos == (size_t)-1) {
-                PyErr_NoMemory();
-                PyMem_Free(wstr);
-                return NULL;
-            }
-            else {
-                goto encode_error;
+    char *str;
+    size_t error_pos;
+    const char *reason;
+    int res = _Py_EncodeLocaleEx(wstr, &str, &error_pos, &reason,
+                                 current_locale, surrogateescape);
+    if (res != 0) {
+        if (res == -2) {
+            PyObject *exc;
+            exc = PyObject_CallFunction(PyExc_UnicodeEncodeError, "sOnns",
+                    "locale", unicode,
+                    (Py_ssize_t)error_pos,
+                    (Py_ssize_t)(error_pos+1),
+                    reason);
+            if (exc != NULL) {
+                PyCodec_StrictErrors(exc);
+                Py_DECREF(exc);
             }
-        }
-        PyMem_Free(wstr);
-
-        bytes = PyBytes_FromString(str);
-        if (current_locale) {
-            PyMem_RawFree(str);
+            return NULL;
         }
         else {
-            PyMem_Free(str);
-        }
-    }
-    else {
-        /* strict mode */
-        size_t len, len2;
-
-        len = wcstombs(NULL, wstr, 0);
-        if (len == (size_t)-1) {
-            error_pos = (size_t)-1;
-            goto encode_error;
-        }
-
-        bytes = PyBytes_FromStringAndSize(NULL, len);
-        if (bytes == NULL) {
+            PyErr_NoMemory();
             PyMem_Free(wstr);
             return NULL;
         }
-
-        len2 = wcstombs(PyBytes_AS_STRING(bytes), wstr, len+1);
-        if (len2 == (size_t)-1 || len2 > len) {
-            Py_DECREF(bytes);
-            error_pos = (size_t)-1;
-            goto encode_error;
-        }
-        PyMem_Free(wstr);
     }
-    return bytes;
-
-encode_error:
-    errmsg = strerror(errno);
-    assert(errmsg != NULL);
-
-    if (error_pos == (size_t)-1)
-        error_pos = wcstombs_errorpos(wstr);
-
     PyMem_Free(wstr);
 
-    wstr = Py_DecodeLocale(errmsg, &errlen);
-    if (wstr != NULL) {
-        reason = PyUnicode_FromWideChar(wstr, errlen);
-        PyMem_RawFree(wstr);
-    } else {
-        errmsg = NULL;
-    }
-
-    if (errmsg == NULL)
-        reason = PyUnicode_FromString(
-            "wcstombs() encountered an unencodable "
-            "wide character");
-    if (reason == NULL)
-        return NULL;
-
-    exc = PyObject_CallFunction(PyExc_UnicodeEncodeError, "sOnnO",
-                                "locale", unicode,
-                                (Py_ssize_t)error_pos,
-                                (Py_ssize_t)(error_pos+1),
-                                reason);
-    Py_DECREF(reason);
-    if (exc != NULL) {
-        PyCodec_StrictErrors(exc);
-        Py_DECREF(exc);
-    }
-    return NULL;
+    PyObject *bytes = PyBytes_FromString(str);
+    PyMem_RawFree(str);
+    return bytes;
 }
 
 PyObject *
 PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
-{
-    return unicode_encode_locale(unicode, errors, 0);
-}
-
-PyObject *
-_PyUnicode_EncodeCurrentLocale(PyObject *unicode, const char *errors)
 {
     return unicode_encode_locale(unicode, errors, 1);
 }
@@ -3687,51 +3570,11 @@ PyUnicode_AsEncodedUnicode(PyObject *unicode,
     return NULL;
 }
 
-static size_t
-mbstowcs_errorpos(const char *str, size_t len)
-{
-#ifdef HAVE_MBRTOWC
-    const char *start = str;
-    mbstate_t mbs;
-    size_t converted;
-    wchar_t ch;
-
-    memset(&mbs, 0, sizeof mbs);
-    while (len)
-    {
-        converted = mbrtowc(&ch, str, len, &mbs);
-        if (converted == 0)
-            /* Reached end of string */
-            break;
-        if (converted == (size_t)-1 || converted == (size_t)-2) {
-            /* Conversion error or incomplete character */
-            return str - start;
-        }
-        else {
-            str += converted;
-            len -= converted;
-        }
-    }
-    /* failed to find the undecodable byte sequence */
-    return 0;
-#endif
-    return 0;
-}
-
 static PyObject*
 unicode_decode_locale(const char *str, Py_ssize_t len, const char *errors,
                       int current_locale)
 {
-    wchar_t smallbuf[256];
-    size_t smallbuf_len = Py_ARRAY_LENGTH(smallbuf);
-    wchar_t *wstr;
-    size_t wlen, wlen2;
-    PyObject *unicode;
     int surrogateescape;
-    size_t error_pos, errlen;
-    char *errmsg;
-    PyObject *exc, *reason = NULL;   /* initialize to prevent gcc warning */
-
     if (locale_error_handler(errors, &surrogateescape) < 0)
         return NULL;
 
@@ -3740,113 +3583,47 @@ unicode_decode_locale(const char *str, Py_ssize_t len, const char *errors,
         return NULL;
     }
 
-    if (surrogateescape) {
-        /* "surrogateescape" error handler */
-        if (current_locale) {
-            wstr = _Py_DecodeCurrentLocale(str, &wlen);
+    wchar_t *wstr;
+    size_t wlen;
+    const char *reason;
+    int res = _Py_DecodeLocaleEx(str, &wstr, &wlen, &reason,
+                                 current_locale, surrogateescape);
+    if (res != 0) {
+        if (res == -2) {
+            PyObject *exc;
+            exc = PyObject_CallFunction(PyExc_UnicodeDecodeError, "sy#nns",
+                                        "locale", str, len,
+                                        (Py_ssize_t)wlen,
+                                        (Py_ssize_t)(wlen + 1),
+                                        reason);
+            if (exc != NULL) {
+                PyCodec_StrictErrors(exc);
+                Py_DECREF(exc);
+            }
         }
         else {
-            wstr = Py_DecodeLocale(str, &wlen);
-        }
-        if (wstr == NULL) {
-            if (wlen == (size_t)-1)
-                PyErr_NoMemory();
-            else
-                PyErr_SetFromErrno(PyExc_OSError);
-            return NULL;
+            PyErr_NoMemory();
         }
-
-        unicode = PyUnicode_FromWideChar(wstr, wlen);
-        PyMem_RawFree(wstr);
+        return NULL;
     }
-    else {
-        /* strict mode */
-#ifndef HAVE_BROKEN_MBSTOWCS
-        wlen = mbstowcs(NULL, str, 0);
-#else
-        wlen = len;
-#endif
-        if (wlen == (size_t)-1)
-            goto decode_error;
-        if (wlen+1 <= smallbuf_len) {
-            wstr = smallbuf;
-        }
-        else {
-            wstr = PyMem_New(wchar_t, wlen+1);
-            if (!wstr)
-                return PyErr_NoMemory();
-        }
 
-        wlen2 = mbstowcs(wstr, str, wlen+1);
-        if (wlen2 == (size_t)-1) {
-            if (wstr != smallbuf)
-                PyMem_Free(wstr);
-            goto decode_error;
-        }
-#ifdef HAVE_BROKEN_MBSTOWCS
-        assert(wlen2 == wlen);
-#endif
-        unicode = PyUnicode_FromWideChar(wstr, wlen2);
-        if (wstr != smallbuf)
-            PyMem_Free(wstr);
-    }
+    PyObject *unicode = PyUnicode_FromWideChar(wstr, wlen);
+    PyMem_RawFree(wstr);
     return unicode;
-
-decode_error:
-    errmsg = strerror(errno);
-    assert(errmsg != NULL);
-
-    error_pos = mbstowcs_errorpos(str, len);
-    wstr = Py_DecodeLocale(errmsg, &errlen);
-    if (wstr != NULL) {
-        reason = PyUnicode_FromWideChar(wstr, errlen);
-        PyMem_RawFree(wstr);
-    }
-
-    if (reason == NULL)
-        reason = PyUnicode_FromString(
-            "mbstowcs() encountered an invalid multibyte sequence");
-    if (reason == NULL)
-        return NULL;
-
-    exc = PyObject_CallFunction(PyExc_UnicodeDecodeError, "sy#nnO",
-                                "locale", str, len,
-                                (Py_ssize_t)error_pos,
-                                (Py_ssize_t)(error_pos+1),
-                                reason);
-    Py_DECREF(reason);
-    if (exc != NULL) {
-        PyCodec_StrictErrors(exc);
-        Py_DECREF(exc);
-    }
-    return NULL;
 }
 
 PyObject*
 PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
                               const char *errors)
-{
-    return unicode_decode_locale(str, len, errors, 0);
-}
-
-PyObject*
-_PyUnicode_DecodeCurrentLocaleAndSize(const char *str, Py_ssize_t len,
-                                      const char *errors)
 {
     return unicode_decode_locale(str, len, errors, 1);
 }
 
-PyObject*
-_PyUnicode_DecodeCurrentLocale(const char *str, const char *errors)
-{
-    return unicode_decode_locale(str, (Py_ssize_t)strlen(str), errors, 1);
-}
-
 PyObject*
 PyUnicode_DecodeLocale(const char *str, const char *errors)
 {
     Py_ssize_t size = (Py_ssize_t)strlen(str);
-    return unicode_decode_locale(str, size, errors, 0);
+    return unicode_decode_locale(str, size, errors, 1);
 }
 
 
@@ -3878,7 +3655,8 @@ PyUnicode_DecodeFSDefaultAndSize(const char *s, Py_ssize_t size)
                                 Py_FileSystemDefaultEncodeErrors);
     }
     else {
-        return PyUnicode_DecodeLocaleAndSize(s, size, Py_FileSystemDefaultEncodeErrors);
+        return unicode_decode_locale(s, size,
+                                     Py_FileSystemDefaultEncodeErrors, 0);
     }
 #endif
 }
@@ -5128,17 +4906,23 @@ PyUnicode_DecodeUTF8Stateful(const char *s,
 }
 
 
-/* UTF-8 decoder using the surrogateescape error handler .
+/* UTF-8 decoder: use surrogateescape error handler if 'surrogateescape' is
+   non-zero, use strict error handler otherwise.
 
-   On success, return a pointer to a newly allocated wide character string (use
-   PyMem_RawFree() to free the memory) and write the output length (in number
-   of wchar_t units) into *p_wlen (if p_wlen is set).
+   On success, write a pointer to a newly allocated wide character string into
+   *wstr (use PyMem_RawFree() to free the memory) and write the output length
+   (in number of wchar_t units) into *wlen (if wlen is set).
 
-   On memory allocation failure, return -1 and write (size_t)-1 into *p_wlen
-   (if p_wlen is set). */
-wchar_t*
-_Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size, size_t *p_wlen)
+   On memory allocation failure, return -1.
+
+   On decoding error (if surrogateescape is zero), return -2. If wlen is
+   non-NULL, write the start of the illegal byte sequence into *wlen. If reason
+   is not NULL, write the decoding error message into *reason. */
+int
+_Py_DecodeUTF8Ex(const char *s, Py_ssize_t size, wchar_t **wstr, size_t *wlen,
+                 const char **reason, int surrogateescape)
 {
+    const char *orig_s = s;
     const char *e;
     wchar_t *unicode;
     Py_ssize_t outpos;
@@ -5146,18 +4930,12 @@ _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size, size_t *p_wlen)
     /* Note: size will always be longer than the resulting Unicode
        character count */
     if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) < (size + 1)) {
-        if (p_wlen) {
-            *p_wlen = (size_t)-1;
-        }
-        return NULL;
+        return -1;
     }
 
     unicode = PyMem_RawMalloc((size + 1) * sizeof(wchar_t));
     if (!unicode) {
-        if (p_wlen) {
-            *p_wlen = (size_t)-1;
-        }
-        return NULL;
+        return -1;
     }
 
     /* Unpack UTF-8 encoded data */
@@ -5175,7 +4953,7 @@ _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size, size_t *p_wlen)
             Py_UNREACHABLE();
 #else
             assert(ch > 0xFFFF && ch <= MAX_UNICODE);
-            /*  compute and append the two surrogates: */
+            /* write a surrogate pair */
             unicode[outpos++] = (wchar_t)Py_UNICODE_HIGH_SURROGATE(ch);
             unicode[outpos++] = (wchar_t)Py_UNICODE_LOW_SURROGATE(ch);
 #endif
@@ -5183,60 +4961,88 @@ _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size, size_t *p_wlen)
         else {
             if (!ch && s == e)
                 break;
+            if (!surrogateescape) {
+                PyMem_RawFree(unicode );
+                if (reason != NULL) {
+                    switch (ch) {
+                    case 0:
+                        *reason = "unexpected end of data";
+                        break;
+                    case 1:
+                        *reason = "invalid start byte";
+                        break;
+                    /* 2, 3, 4 */
+                    default:
+                        *reason = "invalid continuation byte";
+                        break;
+                    }
+                }
+                if (wlen != NULL) {
+                    *wlen = s - orig_s;
+                }
+                return -2;
+            }
             /* surrogateescape */
             unicode[outpos++] = 0xDC00 + (unsigned char)*s++;
         }
     }
     unicode[outpos] = L'\0';
-    if (p_wlen) {
-        *p_wlen = outpos;
+    if (wlen) {
+        *wlen = outpos;
     }
-    return unicode;
+    *wstr = unicode;
+    return 0;
+}
+
+wchar_t*
+_Py_DecodeUTF8_surrogateescape(const char *arg, Py_ssize_t arglen)
+{
+    wchar_t *wstr;
+    int res = _Py_DecodeUTF8Ex(arg, arglen, &wstr, NULL, NULL, 1);
+    if (res != 0) {
+        return NULL;
+    }
+    return wstr;
 }
 
 
 /* UTF-8 encoder using the surrogateescape error handler .
 
-   On success, return a pointer to a newly allocated character string (use
-   PyMem_Free() to free the memory).
+   On success, return 0 and write the newly allocated character string (use
+   PyMem_Free() to free the memory) into *str.
 
-   On encoding failure, return NULL and write the position of the invalid
-   surrogate character into *error_pos (if error_pos is set).
+   On encoding failure, return -2 and write the position of the invalid
+   surrogate character into *error_pos (if error_pos is set) and the decoding
+   error message into *reason (if reason is set).
 
-   On memory allocation failure, return NULL and write (size_t)-1 into
-   *error_pos (if error_pos is set). */
-char*
-_Py_EncodeUTF8_surrogateescape(const wchar_t *text, size_t *error_pos,
-                               int raw_malloc)
+   On memory allocation failure, return -1. */
+int
+_Py_EncodeUTF8Ex(const wchar_t *text, char **str, size_t *error_pos,
+                 const char **reason, int raw_malloc, int surrogateescape)
 {
     const Py_ssize_t max_char_size = 4;
     Py_ssize_t len = wcslen(text);
 
     assert(len >= 0);
 
+    if (len > PY_SSIZE_T_MAX / max_char_size - 1) {
+        return -1;
+    }
     char *bytes;
-    if (len <= PY_SSIZE_T_MAX / max_char_size - 1) {
-        if (raw_malloc) {
-            bytes = PyMem_RawMalloc((len + 1) * max_char_size);
-        }
-        else {
-            bytes = PyMem_Malloc((len + 1) * max_char_size);
-        }
+    if (raw_malloc) {
+        bytes = PyMem_RawMalloc((len + 1) * max_char_size);
     }
     else {
-        bytes = NULL;
+        bytes = PyMem_Malloc((len + 1) * max_char_size);
     }
     if (bytes == NULL) {
-        if (error_pos != NULL) {
-            *error_pos = (size_t)-1;
-        }
-        return NULL;
+        return -1;
     }
 
     char *p = bytes;
     Py_ssize_t i;
-    for (i = 0; i < len;) {
-        Py_UCS4 ch = text[i++];
+    for (i = 0; i < len; i++) {
+        Py_UCS4 ch = text[i];
 
         if (ch < 0x80) {
             /* Encode ASCII */
@@ -5250,11 +5056,20 @@ _Py_EncodeUTF8_surrogateescape(const wchar_t *text, size_t *error_pos,
         }
         else if (Py_UNICODE_IS_SURROGATE(ch)) {
             /* surrogateescape error handler */
-            if (!(0xDC80 <= ch && ch <= 0xDCFF)) {
+            if (!surrogateescape || !(0xDC80 <= ch && ch <= 0xDCFF)) {
                 if (error_pos != NULL) {
-                    *error_pos = (size_t)i - 1;
+                    *error_pos = (size_t)i;
                 }
-                goto error;
+                if (reason != NULL) {
+                    *reason = "encoding error";
+                }
+                if (raw_malloc) {
+                    PyMem_RawFree(bytes);
+                }
+                else {
+                    PyMem_Free(bytes);
+                }
+                return -2;
             }
             *p++ = (char)(ch & 0xff);
         }
@@ -5286,18 +5101,16 @@ _Py_EncodeUTF8_surrogateescape(const wchar_t *text, size_t *error_pos,
         if (error_pos != NULL) {
             *error_pos = (size_t)-1;
         }
-        goto error;
-    }
-    return bytes2;
-
- error:
-    if (raw_malloc) {
-        PyMem_RawFree(bytes);
-    }
-    else {
-        PyMem_Free(bytes);
+        if (raw_malloc) {
+            PyMem_RawFree(bytes);
+        }
+        else {
+            PyMem_Free(bytes);
+        }
+        return -1;
     }
-    return NULL;
+    *str = bytes2;
+    return 0;
 }
 
 
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 9275494e864..a50075eced0 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -20,9 +20,6 @@ extern int winerror_to_errno(int);
 #include <fcntl.h>
 #endif /* HAVE_FCNTL_H */
 
-extern char* _Py_EncodeUTF8_surrogateescape(const wchar_t *text,
-                                            size_t *error_pos, int raw_malloc);
-
 #ifdef O_CLOEXEC
 /* Does open() support the O_CLOEXEC flag? Possible values:
 
@@ -69,7 +66,10 @@ _Py_device_encoding(int fd)
     Py_RETURN_NONE;
 }
 
-#if !defined(__APPLE__) && !defined(MS_WINDOWS)
+#if !defined(__APPLE__) && !defined(__ANDROID__) && !defined(MS_WINDOWS)
+
+#define USE_FORCE_ASCII
+
 extern int _Py_normalize_encoding(const char *, char *, size_t);
 
 /* Workaround FreeBSD and OpenIndiana locale encoding issue with the C locale.
@@ -90,7 +90,7 @@ extern int _Py_normalize_encoding(const char *, char *, size_t);
 
        1: the workaround is used: Py_EncodeLocale() uses
           encode_ascii_surrogateescape() and Py_DecodeLocale() uses
-          decode_ascii_surrogateescape()
+          decode_ascii()
        0: the workaround is not used: Py_EncodeLocale() uses wcstombs() and
           Py_DecodeLocale() uses mbstowcs()
       -1: unknown, need to call check_force_ascii() to get the value
@@ -180,16 +180,15 @@ check_force_ascii(void)
     return 1;
 }
 
-static char*
-encode_ascii_surrogateescape(const wchar_t *text, size_t *error_pos, int raw_malloc)
+static int
+encode_ascii(const wchar_t *text, char **str,
+             size_t *error_pos, const char **reason,
+             int raw_malloc, int surrogateescape)
 {
     char *result = NULL, *out;
     size_t len, i;
     wchar_t ch;
 
-    if (error_pos != NULL)
-        *error_pos = (size_t)-1;
-
     len = wcslen(text);
 
     /* +1 for NULL byte */
@@ -199,8 +198,9 @@ encode_ascii_surrogateescape(const wchar_t *text, size_t *error_pos, int raw_mal
     else {
         result = PyMem_Malloc(len + 1);
     }
-    if (result == NULL)
-        return NULL;
+    if (result == NULL) {
+        return -1;
+    }
 
     out = result;
     for (i=0; i<len; i++) {
@@ -210,60 +210,84 @@ encode_ascii_surrogateescape(const wchar_t *text, size_t *error_pos, int raw_mal
             /* ASCII character */
             *out++ = (char)ch;
         }
-        else if (0xdc80 <= ch && ch <= 0xdcff) {
+        else if (surrogateescape && 0xdc80 <= ch && ch <= 0xdcff) {
             /* UTF-8b surrogate */
             *out++ = (char)(ch - 0xdc00);
         }
         else {
-            if (error_pos != NULL) {
-                *error_pos = i;
-            }
             if (raw_malloc) {
                 PyMem_RawFree(result);
             }
             else {
                 PyMem_Free(result);
             }
-            return NULL;
+            if (error_pos != NULL) {
+                *error_pos = i;
+            }
+            if (reason) {
+                *reason = "encoding error";
+            }
+            return -2;
         }
     }
     *out = '\0';
-    return result;
+    *str = result;
+    return 0;
 }
-#endif   /* !defined(__APPLE__) && !defined(MS_WINDOWS) */
+#endif   /* !defined(__APPLE__) && !defined(__ANDROID__) && !defined(MS_WINDOWS) */
+
 
-#if !defined(__APPLE__) && (!defined(MS_WINDOWS) || !defined(HAVE_MBRTOWC))
-static wchar_t*
-decode_ascii_surrogateescape(const char *arg, size_t *size)
+#if !defined(HAVE_MBRTOWC) || defined(USE_FORCE_ASCII)
+static int
+decode_ascii(const char *arg, wchar_t **wstr, size_t *wlen,
+             const char **reason, int surrogateescape)
 {
     wchar_t *res;
     unsigned char *in;
     wchar_t *out;
     size_t argsize = strlen(arg) + 1;
 
-    if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t))
-        return NULL;
-    res = PyMem_RawMalloc(argsize*sizeof(wchar_t));
-    if (!res)
-        return NULL;
+    if (argsize > PY_SSIZE_T_MAX / sizeof(wchar_t)) {
+        return -1;
+    }
+    res = PyMem_RawMalloc(argsize * sizeof(wchar_t));
+    if (!res) {
+        return -1;
+    }
 
-    in = (unsigned char*)arg;
     out = res;
-    while(*in)
-        if(*in < 128)
-            *out++ = *in++;
-        else
-            *out++ = 0xdc00 + *in++;
+    for (in = (unsigned char*)arg; *in; in++) {
+        unsigned char ch = *in;
+        if (ch < 128) {
+            *out++ = ch;
+        }
+        else {
+            if (!surrogateescape) {
+                PyMem_RawFree(res);
+                if (wlen) {
+                    *wlen = in - (unsigned char*)arg;
+                }
+                if (reason) {
+                    *reason = "decoding error";
+                }
+                return -2;
+            }
+            *out++ = 0xdc00 + ch;
+        }
+    }
     *out = 0;
-    if (size != NULL)
-        *size = out - res;
-    return res;
+
+    if (wlen != NULL) {
+        *wlen = out - res;
+    }
+    *wstr = res;
+    return 0;
 }
-#endif
+#endif   /* !HAVE_MBRTOWC */
 
-#if !defined(__APPLE__) && !defined(__ANDROID__)
-static wchar_t*
-decode_current_locale(const char* arg, size_t *size)
+static int
+decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen,
+                      const char **reason, int surrogateescape)
 {
     wchar_t *res;
     size_t argsize;
@@ -284,15 +308,15 @@ decode_current_locale(const char* arg, size_t *size)
     argsize = mbstowcs(NULL, arg, 0);
 #endif
     if (argsize != (size_t)-1) {
-        if (argsize == PY_SSIZE_T_MAX)
-            goto oom;
-        argsize += 1;
-        if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t))
-            goto oom;
-        res = (wchar_t *)PyMem_RawMalloc(argsize*sizeof(wchar_t));
-        if (!res)
-            goto oom;
-        count = mbstowcs(res, arg, argsize);
+        if (argsize > PY_SSIZE_T_MAX / sizeof(wchar_t) - 1) {
+            return -1;
+        }
+        res = (wchar_t *)PyMem_RawMalloc((argsize + 1) * sizeof(wchar_t));
+        if (!res) {
+            return -1;
+        }
+
+        count = mbstowcs(res, arg, argsize + 1);
         if (count != (size_t)-1) {
             wchar_t *tmp;
             /* Only use the result if it contains no
@@ -301,13 +325,16 @@ decode_current_locale(const char* arg, size_t *size)
                          !Py_UNICODE_IS_SURROGATE(*tmp); tmp++)
                 ;
             if (*tmp == 0) {
-                if (size != NULL)
-                    *size = count;
-                return res;
+                if (wlen != NULL) {
+                    *wlen = count;
+                }
+                *wstr = res;
+                return 0;
             }
         }
         PyMem_RawFree(res);
     }
+
     /* Conversion failed. Fall back to escaping with surrogateescape. */
 #ifdef HAVE_MBRTOWC
     /* Try conversion with mbrtwoc (C99), and escape non-decodable bytes. */
@@ -315,30 +342,37 @@ decode_current_locale(const char* arg, size_t *size)
     /* Overallocate; as multi-byte characters are in the argument, the
        actual output could use less memory. */
     argsize = strlen(arg) + 1;
-    if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t))
-        goto oom;
-    res = (wchar_t*)PyMem_RawMalloc(argsize*sizeof(wchar_t));
-    if (!res)
-        goto oom;
+    if (argsize > PY_SSIZE_T_MAX / sizeof(wchar_t)) {
+        return -1;
+    }
+    res = (wchar_t*)PyMem_RawMalloc(argsize * sizeof(wchar_t));
+    if (!res) {
+        return -1;
+    }
+
     in = (unsigned char*)arg;
     out = res;
     memset(&mbs, 0, sizeof mbs);
     while (argsize) {
         size_t converted = mbrtowc(out, (char*)in, argsize, &mbs);
-        if (converted == 0)
+        if (converted == 0) {
             /* Reached end of string; null char stored. */
             break;
+        }
+
         if (converted == (size_t)-2) {
             /* Incomplete character. This should never happen,
                since we provide everything that we have -
                unless there is a bug in the C library, or I
                misunderstood how mbrtowc works. */
-            PyMem_RawFree(res);
-            if (size != NULL)
-                *size = (size_t)-2;
-            return NULL;
+            goto decode_error;
         }
+
         if (converted == (size_t)-1) {
+            if (!surrogateescape) {
+                goto decode_error;
+            }
+
             /* Conversion error. Escape as UTF-8b, and start over
                in the initial shift state. */
             *out++ = 0xdc00 + *in++;
@@ -346,12 +380,18 @@ decode_current_locale(const char* arg, size_t *size)
             memset(&mbs, 0, sizeof mbs);
             continue;
         }
+
         if (Py_UNICODE_IS_SURROGATE(*out)) {
+            if (!surrogateescape) {
+                goto decode_error;
+            }
+
             /* Surrogate character.  Escape the original
                byte sequence with surrogateescape. */
             argsize -= converted;
-            while (converted--)
+            while (converted--) {
                 *out++ = 0xdc00 + *in++;
+            }
             continue;
         }
         /* successfully converted some bytes */
@@ -359,55 +399,80 @@ decode_current_locale(const char* arg, size_t *size)
         argsize -= converted;
         out++;
     }
-    if (size != NULL)
-        *size = out - res;
+    if (wlen != NULL) {
+        *wlen = out - res;
+    }
+    *wstr = res;
+    return 0;
+
+decode_error:
+    PyMem_RawFree(res);
+    if (wlen) {
+        *wlen = in - (unsigned char*)arg;
+    }
+    if (reason) {
+        *reason = "decoding error";
+    }
+    return -2;
 #else   /* HAVE_MBRTOWC */
     /* Cannot use C locale for escaping; manually escape as if charset
        is ASCII (i.e. escape all bytes > 128. This will still roundtrip
        correctly in the locale's charset, which must be an ASCII superset. */
-    res = decode_ascii_surrogateescape(arg, size);
-    if (res == NULL)
-        goto oom;
+    return decode_ascii(arg, wstr, wlen, reason, surrogateescape);
 #endif   /* HAVE_MBRTOWC */
-    return res;
-
-oom:
-    if (size != NULL) {
-        *size = (size_t)-1;
-    }
-    return NULL;
 }
-#endif
 
 
-static wchar_t*
-decode_locale(const char* arg, size_t *size, int ignore_utf8_mode)
+/* Decode a byte string from the locale encoding.
+
+   Use the strict error handler if 'surrogateescape' is zero.  Use the
+   surrogateescape error handler if 'surrogateescape' is non-zero: undecodable
+   bytes are decoded as characters in range U+DC80..U+DCFF. If a byte sequence
+   can be decoded as a surrogate character, escape the bytes using the
+   surrogateescape error handler instead of decoding them.
+
+   On sucess, return 0 and write the newly allocated wide character string into
+   *wstr (use PyMem_RawFree() to free the memory). If wlen is not NULL, write
+   the number of wide characters excluding the null character into *wlen.
+
+   On memory allocation failure, return -1.
+
+   On decoding error, return -2. If wlen is not NULL, write the start of
+   invalid byte sequence in the input string into *wlen. If reason is not NULL,
+   write the decoding error message into *reason.
+
+   Use the Py_EncodeLocaleEx() function to encode the character string back to
+   a byte string. */
+int
+_Py_DecodeLocaleEx(const char* arg, wchar_t **wstr, size_t *wlen,
+                   const char **reason,
+                   int current_locale, int surrogateescape)
 {
+    if (current_locale) {
+        return decode_current_locale(arg, wstr, wlen, reason, surrogateescape);
+    }
+
 #if defined(__APPLE__) || defined(__ANDROID__)
-    return _Py_DecodeUTF8_surrogateescape(arg, strlen(arg), size);
+    return _Py_DecodeUTF8Ex(arg, strlen(arg), wstr, wlen, reason,
+                            surrogateescape);
 #else
-    if (!ignore_utf8_mode && Py_UTF8Mode == 1) {
-        return _Py_DecodeUTF8_surrogateescape(arg, strlen(arg), size);
+    if (Py_UTF8Mode == 1) {
+        return _Py_DecodeUTF8Ex(arg, strlen(arg), wstr, wlen, reason,
+                                surrogateescape);
     }
 
-#ifndef MS_WINDOWS
-    if (force_ascii == -1)
+#ifdef USE_FORCE_ASCII
+    if (force_ascii == -1) {
         force_ascii = check_force_ascii();
+    }
 
     if (force_ascii) {
         /* force ASCII encoding to workaround mbstowcs() issue */
-        wchar_t *wstr = decode_ascii_surrogateescape(arg, size);
-        if (wstr == NULL) {
-            if (size != NULL) {
-                *size = (size_t)-1;
-            }
-            return NULL;
-        }
-        return wstr;
+        return decode_ascii(arg, wstr, wlen, reason, surrogateescape);
     }
 #endif
 
-    return decode_current_locale(arg, size);
+    return decode_current_locale(arg, wstr, wlen, reason, surrogateescape);
 #endif   /* __APPLE__ or __ANDROID__ */
 }
 
@@ -432,23 +497,24 @@ decode_locale(const char* arg, size_t *size, int ignore_utf8_mode)
    Use the Py_EncodeLocale() function to encode the character string back to a
    byte string. */
 wchar_t*
-Py_DecodeLocale(const char* arg, size_t *size)
+Py_DecodeLocale(const char* arg, size_t *wlen)
 {
-    return decode_locale(arg, size, 0);
-}
-
-
-/* Similar to Py_DecodeLocale() but ignore the UTF-8 mode */
-wchar_t*
-_Py_DecodeCurrentLocale(const char* arg, size_t *size)
-{
-    return decode_locale(arg, size, 1);
+    wchar_t *wstr;
+    int res = _Py_DecodeLocaleEx(arg, &wstr, wlen, NULL, 0, 1);
+    if (res != 0) {
+        if (wlen != NULL) {
+            *wlen = (size_t)res;
+        }
+        return NULL;
+    }
+    return wstr;
 }
 
 
-#if !defined(__APPLE__) && !defined(__ANDROID__)
-static char*
-encode_current_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
+static int
+encode_current_locale(const wchar_t *text, char **str,
+                      size_t *error_pos, const char **reason,
+                      int raw_malloc, int surrogateescape)
 {
     const size_t len = wcslen(text);
     char *result = NULL, *bytes = NULL;
@@ -464,38 +530,37 @@ encode_current_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
         for (i=0; i < len; i++) {
             c = text[i];
             if (c >= 0xdc80 && c <= 0xdcff) {
+                if (!surrogateescape) {
+                    goto encode_error;
+                }
                 /* UTF-8b surrogate */
                 if (bytes != NULL) {
                     *bytes++ = c - 0xdc00;
                     size--;
                 }
-                else
+                else {
                     size++;
+                }
                 continue;
             }
             else {
                 buf[0] = c;
-                if (bytes != NULL)
+                if (bytes != NULL) {
                     converted = wcstombs(bytes, buf, size);
-                else
+                }
+                else {
                     converted = wcstombs(NULL, buf, 0);
+                }
                 if (converted == (size_t)-1) {
-                    if (raw_malloc) {
-                        PyMem_RawFree(result);
-                    }
-                    else {
-                        PyMem_Free(result);
-                    }
-                    if (error_pos != NULL)
-                        *error_pos = i;
-                    return NULL;
+                    goto encode_error;
                 }
                 if (bytes != NULL) {
                     bytes += converted;
                     size -= converted;
                 }
-                else
+                else {
                     size += converted;
+                }
             }
         }
         if (result != NULL) {
@@ -511,40 +576,80 @@ encode_current_locale(const wchar_t *text, size_t *error_pos, int raw_malloc)
             result = PyMem_Malloc(size);
         }
         if (result == NULL) {
-            if (error_pos != NULL) {
-                *error_pos = (size_t)-1;
-            }
-            return NULL;
+            return -1;
         }
         bytes = result;
     }
-    return result;
+    *str = result;
+    return 0;
+
+encode_error:
+    if (raw_malloc) {
+        PyMem_RawFree(result);
+    }
+    else {
+        PyMem_Free(result);
+    }
+    if (error_pos != NULL) {
+        *error_pos = i;
+    }
+    if (reason) {
+        *reason = "encoding error";
+    }
+    return -2;
 }
-#endif
 
-static char*
-encode_locale(const wchar_t *text, size_t *error_pos,
-              int raw_malloc, int ignore_utf8_mode)
+static int
+encode_locale_ex(const wchar_t *text, char **str, size_t *error_pos,
+                 const char **reason,
+                 int raw_malloc, int current_locale, int surrogateescape)
 {
+    if (current_locale) {
+        return encode_current_locale(text, str, error_pos, reason,
+                                     raw_malloc, surrogateescape);
+    }
+
 #if defined(__APPLE__) || defined(__ANDROID__)
-    return _Py_EncodeUTF8_surrogateescape(text, error_pos, raw_malloc);
+    return _Py_EncodeUTF8Ex(text, str, error_pos, reason,
+                            raw_malloc, surrogateescape);
 #else   /* __APPLE__ */
-    if (!ignore_utf8_mode && Py_UTF8Mode == 1) {
-        return _Py_EncodeUTF8_surrogateescape(text, error_pos, raw_malloc);
+    if (Py_UTF8Mode == 1) {
+        return _Py_EncodeUTF8Ex(text, str, error_pos, reason,
+                                raw_malloc, surrogateescape);
     }
 
-#ifndef MS_WINDOWS
-    if (force_ascii == -1)
+#ifdef USE_FORCE_ASCII
+    if (force_ascii == -1) {
         force_ascii = check_force_ascii();
+    }
 
-    if (force_ascii)
-        return encode_ascii_surrogateescape(text, error_pos, raw_malloc);
+    if (force_ascii) {
+        return encode_ascii(text, str, error_pos, reason,
+                            raw_malloc, surrogateescape);
+    }
 #endif
 
-    return encode_current_locale(text, error_pos, raw_malloc);
+    return encode_current_locale(text, str, error_pos, reason,
+                                 raw_malloc, surrogateescape);
 #endif   /* __APPLE__ or __ANDROID__ */
 }
 
+static char*
+encode_locale(const wchar_t *text, size_t *error_pos,
+              int raw_malloc, int current_locale)
+{
+    char *str;
+    int res = encode_locale_ex(text, &str, error_pos, NULL,
+                               raw_malloc, current_locale, 1);
+    if (res != -2 && error_pos) {
+        *error_pos = (size_t)-1;
+    }
+    if (res != 0) {
+        return NULL;
+    }
+    return str;
+}
+
 /* Encode a wide character string to the locale encoding with the
    surrogateescape error handler: surrogate characters in the range
    U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
@@ -573,11 +678,13 @@ _Py_EncodeLocaleRaw(const wchar_t *text, size_t *error_pos)
 }
 
 
-/* Similar to _Py_EncodeLocaleRaw() but ignore the UTF-8 Mode */
-char*
-_Py_EncodeCurrentLocale(const wchar_t *text, size_t *error_pos)
+int
+_Py_EncodeLocaleEx(const wchar_t *text, char **str,
+                   size_t *error_pos, const char **reason,
+                   int current_locale, int surrogateescape)
 {
-    return encode_locale(text, error_pos, 1, 1);
+    return encode_locale_ex(text, str, error_pos, reason, 1,
+                            current_locale, surrogateescape);
 }
 
 
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index 9591fcc4911..7ebd69bf622 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -382,8 +382,8 @@ _Py_FindEnvConfigValue(FILE *env_file, const wchar_t *key,
             /* Comment - skip */
             continue;
         }
-        tmpbuffer = _Py_DecodeUTF8_surrogateescape(buffer, n, NULL);
-        if (tmpbuffer != NULL) {
+        tmpbuffer = _Py_DecodeUTF8_surrogateescape(buffer, n);
+        if (tmpbuffer) {
             wchar_t * state;
             wchar_t * tok = wcstok(tmpbuffer, L" \t\r\n", &state);
             if ((tok != NULL) && !wcscmp(tok, key)) {


From webhook-mailer at python.org  Mon Jan 15 09:32:19 2018
From: webhook-mailer at python.org (Mariatta)
Date: Mon, 15 Jan 2018 14:32:19 -0000
Subject: [Python-checkins] bpo-26330: Update shutil.disk_usage()
 documentation (GH-5184) (GH-5188)
Message-ID: <mailman.22.1516026741.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/fb8569e36f2629654d5bc9c7ba05978edce408f4
commit: fb8569e36f2629654d5bc9c7ba05978edce408f4
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-15T06:32:11-08:00
summary:

bpo-26330: Update shutil.disk_usage() documentation (GH-5184) (GH-5188)

Clarify that on Windows, path must be a directory.
On Unix, path can be a file or a directory.
(cherry picked from commit ee3b83547c6b0cac1da2cb44aaaea533a1d1bbc8)

files:
M Doc/library/shutil.rst

diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
index 41e5bafa53d..3d8cb0301e7 100644
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -318,7 +318,8 @@ Directory and files operations
 
    Return disk usage statistics about the given path as a :term:`named tuple`
    with the attributes *total*, *used* and *free*, which are the amount of
-   total, used and free space, in bytes.
+   total, used and free space, in bytes.  On Windows, *path* must be a
+   directory; on Unix, it can be a file or directory.
 
    .. versionadded:: 3.3
 


From webhook-mailer at python.org  Mon Jan 15 09:58:04 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 15 Jan 2018 14:58:04 -0000
Subject: [Python-checkins] bpo-31900: Fix localeconv() encoding for
 LC_NUMERIC (#4174)
Message-ID: <mailman.23.1516028286.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/cb064fc2321ce8673fe365e9ef60445a27657f54
commit: cb064fc2321ce8673fe365e9ef60445a27657f54
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-15T15:58:02+01:00
summary:

bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174)

* Add _Py_GetLocaleconvNumeric() function: decode decimal_point and
  thousands_sep fields of localeconv() from the LC_NUMERIC encoding,
  rather than decoding from the LC_CTYPE encoding.
* Modify locale.localeconv() and "n" formatter of str.format() (for
  int, float and complex to use _Py_GetLocaleconvNumeric()
  internally.

files:
A Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst
M Doc/library/locale.rst
M Doc/library/stdtypes.rst
M Doc/whatsnew/3.7.rst
M Include/fileutils.h
M Modules/_localemodule.c
M Python/fileutils.c
M Python/formatter_unicode.c

diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst
index 7da94a23964..2fd44fe8e90 100644
--- a/Doc/library/locale.rst
+++ b/Doc/library/locale.rst
@@ -147,6 +147,16 @@ The :mod:`locale` module defines the following exception and functions:
    | ``CHAR_MAX`` | Nothing is specified in this locale.    |
    +--------------+-----------------------------------------+
 
+   The function sets temporarily the ``LC_CTYPE`` locale to the ``LC_NUMERIC``
+   locale to decode ``decimal_point`` and ``thousands_sep`` byte strings if
+   they are non-ASCII or longer than 1 byte, and the ``LC_NUMERIC`` locale is
+   different than the ``LC_CTYPE`` locale. This temporary change affects other
+   threads.
+
+   .. versionchanged:: 3.7
+      The function now sets temporarily the ``LC_CTYPE`` locale to the
+      ``LC_NUMERIC`` locale in some cases.
+
 
 .. function:: nl_langinfo(option)
 
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index de2fb27c2d7..120b0d3399c 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1599,6 +1599,20 @@ expression support in the :mod:`re` module).
    See :ref:`formatstrings` for a description of the various formatting options
    that can be specified in format strings.
 
+   .. note::
+      When formatting a number (:class:`int`, :class:`float`, :class:`float`
+      and subclasses) with the ``n`` type (ex: ``'{:n}'.format(1234)``), the
+      function sets temporarily the ``LC_CTYPE`` locale to the ``LC_NUMERIC``
+      locale to decode ``decimal_point`` and ``thousands_sep`` fields of
+      :c:func:`localeconv` if they are non-ASCII or longer than 1 byte, and the
+      ``LC_NUMERIC`` locale is different than the ``LC_CTYPE`` locale. This
+      temporary change affects other threads.
+
+   .. versionchanged:: 3.7
+      When formatting a number with the ``n`` type, the function sets
+      temporarily the ``LC_CTYPE`` locale to the ``LC_NUMERIC`` locale in some
+      cases.
+
 
 .. method:: str.format_map(mapping)
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 1041d31f302..009df38d8ec 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -866,6 +866,9 @@ Changes in Python behavior
 Changes in the Python API
 -------------------------
 
+* The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE``
+  locale to the ``LC_NUMERIC`` locale in some cases.
+
 * The ``asyncio.windows_utils.socketpair()`` function has been
   removed: use directly :func:`socket.socketpair` which is available on all
   platforms since Python 3.5 (before, it wasn't available on Windows).
diff --git a/Include/fileutils.h b/Include/fileutils.h
index b4f8b11a635..21eefdef87a 100644
--- a/Include/fileutils.h
+++ b/Include/fileutils.h
@@ -160,6 +160,11 @@ PyAPI_FUNC(int) _Py_get_blocking(int fd);
 PyAPI_FUNC(int) _Py_set_blocking(int fd, int blocking);
 #endif   /* !MS_WINDOWS */
 
+PyAPI_FUNC(int) _Py_GetLocaleconvNumeric(
+    PyObject **decimal_point,
+    PyObject **thousands_sep,
+    const char **grouping);
+
 #endif   /* Py_LIMITED_API */
 
 #ifdef __cplusplus
diff --git a/Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst b/Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst
new file mode 100644
index 00000000000..2d8e3ce6002
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst
@@ -0,0 +1,9 @@
+The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE``
+locale to the ``LC_NUMERIC`` locale to decode ``decimal_point`` and
+``thousands_sep`` byte strings if they are non-ASCII or longer than 1 byte, and
+the ``LC_NUMERIC`` locale is different than the ``LC_CTYPE`` locale.  This
+temporary change affects other threads.
+
+Same change for the :meth:`str.format` method when formatting a number
+(:class:`int`, :class:`float`, :class:`float` and subclasses) with the ``n``
+type (ex: ``'{:n}'.format(1234)``).
diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
index 324b694b830..f9eeeb72dd9 100644
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -139,8 +139,9 @@ PyLocale_localeconv(PyObject* self)
     PyObject *x;
 
     result = PyDict_New();
-    if (!result)
+    if (!result) {
         return NULL;
+    }
 
     /* if LC_NUMERIC is different in the C library, use saved value */
     l = localeconv();
@@ -171,12 +172,6 @@ PyLocale_localeconv(PyObject* self)
         RESULT(#i, x); \
     } while (0)
 
-    /* Numeric information */
-    RESULT_STRING(decimal_point);
-    RESULT_STRING(thousands_sep);
-    x = copy_grouping(l->grouping);
-    RESULT("grouping", x);
-
     /* Monetary information */
     RESULT_STRING(int_curr_symbol);
     RESULT_STRING(currency_symbol);
@@ -195,10 +190,36 @@ PyLocale_localeconv(PyObject* self)
     RESULT_INT(n_sep_by_space);
     RESULT_INT(p_sign_posn);
     RESULT_INT(n_sign_posn);
+
+    /* Numeric information */
+    PyObject *decimal_point, *thousands_sep;
+    const char *grouping;
+    if (_Py_GetLocaleconvNumeric(&decimal_point,
+                                 &thousands_sep,
+                                 &grouping) < 0) {
+        goto failed;
+    }
+
+    if (PyDict_SetItemString(result, "decimal_point", decimal_point) < 0) {
+        Py_DECREF(decimal_point);
+        Py_DECREF(thousands_sep);
+        goto failed;
+    }
+    Py_DECREF(decimal_point);
+
+    if (PyDict_SetItemString(result, "thousands_sep", thousands_sep) < 0) {
+        Py_DECREF(thousands_sep);
+        goto failed;
+    }
+    Py_DECREF(thousands_sep);
+
+    x = copy_grouping(grouping);
+    RESULT("grouping", x);
+
     return result;
 
   failed:
-    Py_XDECREF(result);
+    Py_DECREF(result);
     return NULL;
 }
 
diff --git a/Python/fileutils.c b/Python/fileutils.c
index a50075eced0..9a1435cfb32 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -1746,3 +1746,80 @@ _Py_set_blocking(int fd, int blocking)
     return -1;
 }
 #endif
+
+
+int
+_Py_GetLocaleconvNumeric(PyObject **decimal_point, PyObject **thousands_sep,
+                         const char **grouping)
+{
+    int res = -1;
+
+    struct lconv *lc = localeconv();
+
+    int change_locale = 0;
+    if (decimal_point != NULL &&
+        (strlen(lc->decimal_point) > 1 || ((unsigned char)lc->decimal_point[0]) > 127))
+    {
+        change_locale = 1;
+    }
+    if (thousands_sep != NULL &&
+        (strlen(lc->thousands_sep) > 1 || ((unsigned char)lc->thousands_sep[0]) > 127))
+    {
+        change_locale = 1;
+    }
+
+    /* Keep a copy of the LC_CTYPE locale */
+    char *oldloc = NULL, *loc = NULL;
+    if (change_locale) {
+        oldloc = setlocale(LC_CTYPE, NULL);
+        if (!oldloc) {
+            PyErr_SetString(PyExc_RuntimeWarning, "faild to get LC_CTYPE locale");
+            return -1;
+        }
+
+        oldloc = _PyMem_Strdup(oldloc);
+        if (!oldloc) {
+            PyErr_NoMemory();
+            return -1;
+        }
+
+        loc = setlocale(LC_NUMERIC, NULL);
+        if (loc != NULL && strcmp(loc, oldloc) == 0) {
+            loc = NULL;
+        }
+
+        if (loc != NULL) {
+            /* Only set the locale temporarilty the LC_CTYPE locale
+               if LC_NUMERIC locale is different than LC_CTYPE locale and
+               decimal_point and/or thousands_sep are non-ASCII or longer than
+               1 byte */
+            setlocale(LC_CTYPE, loc);
+        }
+    }
+
+    if (decimal_point != NULL) {
+        *decimal_point = PyUnicode_DecodeLocale(lc->decimal_point, NULL);
+        if (*decimal_point == NULL) {
+            goto error;
+        }
+    }
+    if (thousands_sep != NULL) {
+        *thousands_sep = PyUnicode_DecodeLocale(lc->thousands_sep, NULL);
+        if (*thousands_sep == NULL) {
+            goto error;
+        }
+    }
+
+    if (grouping != NULL) {
+        *grouping = lc->grouping;
+    }
+
+    res = 0;
+
+error:
+    if (loc != NULL) {
+        setlocale(LC_CTYPE, oldloc);
+    }
+    PyMem_Free(oldloc);
+    return res;
+}
diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
index 397ae7faafc..71e673d9f83 100644
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -704,18 +704,11 @@ get_locale_info(enum LocaleType type, LocaleInfo *locale_info)
 {
     switch (type) {
     case LT_CURRENT_LOCALE: {
-        struct lconv *locale_data = localeconv();
-        locale_info->decimal_point = PyUnicode_DecodeLocale(
-                                         locale_data->decimal_point,
-                                         NULL);
-        if (locale_info->decimal_point == NULL)
+        if (_Py_GetLocaleconvNumeric(&locale_info->decimal_point,
+                                     &locale_info->thousands_sep,
+                                     &locale_info->grouping) < 0) {
             return -1;
-        locale_info->thousands_sep = PyUnicode_DecodeLocale(
-                                         locale_data->thousands_sep,
-                                         NULL);
-        if (locale_info->thousands_sep == NULL)
-            return -1;
-        locale_info->grouping = locale_data->grouping;
+        }
         break;
     }
     case LT_DEFAULT_LOCALE:


From webhook-mailer at python.org  Mon Jan 15 16:39:07 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 15 Jan 2018 21:39:07 -0000
Subject: [Python-checkins] bpo-32539: Fix OSError for os.listdir() for
 extended-length paths on Windows (#5169)
Message-ID: <mailman.24.1516052349.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/27f32e938ff51fd5d90a29abbbabc6b81d156f33
commit: 27f32e938ff51fd5d90a29abbbabc6b81d156f33
branch: 2.7
author: Anthony Sottile <asottile at umich.edu>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2018-01-15T23:39:04+02:00
summary:

bpo-32539: Fix OSError for os.listdir() for extended-length paths on Windows (#5169)

See https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx?f=255&MSPPError=-2147217396#maxpath

Paths that begin with `\\?\` are "extended-length paths".

files:
A Misc/NEWS.d/next/Library/2018-01-12-09-20-22.bpo-32539.D7AbdE.rst
M Lib/test/test_os.py
M Modules/posixmodule.c

diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
index aca03fdabb1..84e20e0c013 100644
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -900,6 +900,56 @@ def test_CTRL_BREAK_EVENT(self):
         self._kill_with_event(signal.CTRL_BREAK_EVENT, "CTRL_BREAK_EVENT")
 
 
+ at unittest.skipUnless(sys.platform == "win32", "Win32 specific tests")
+class Win32ListdirTests(unittest.TestCase):
+    """Test listdir on Windows."""
+
+    def setUp(self):
+        self.created_paths = []
+        for i in range(2):
+            dir_name = 'SUB%d' % i
+            dir_path = os.path.join(support.TESTFN, dir_name)
+            file_name = 'FILE%d' % i
+            file_path = os.path.join(support.TESTFN, file_name)
+            os.makedirs(dir_path)
+            with open(file_path, 'w') as f:
+                f.write("I'm %s and proud of it. Blame test_os.\n" % file_path)
+            self.created_paths.extend([dir_name, file_name])
+        self.created_paths.sort()
+
+    def tearDown(self):
+        shutil.rmtree(support.TESTFN)
+
+    def test_listdir_no_extended_path(self):
+        """Test when the path is not an "extended" path."""
+        # unicode
+        fs_encoding = sys.getfilesystemencoding()
+        self.assertEqual(
+                sorted(os.listdir(support.TESTFN.decode(fs_encoding))),
+                [path.decode(fs_encoding) for path in self.created_paths])
+
+        # bytes
+        self.assertEqual(
+                sorted(os.listdir(os.fsencode(support.TESTFN))),
+                self.created_paths)
+
+    def test_listdir_extended_path(self):
+        """Test when the path starts with '\\\\?\\'."""
+        # See: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath
+        # unicode
+        fs_encoding = sys.getfilesystemencoding()
+        path = u'\\\\?\\' + os.path.abspath(support.TESTFN.decode(fs_encoding))
+        self.assertEqual(
+                sorted(os.listdir(path)),
+                [path.decode(fs_encoding) for path in self.created_paths])
+
+        # bytes
+        path = b'\\\\?\\' + os.path.abspath(support.TESTFN)
+        self.assertEqual(
+                sorted(os.listdir(path)),
+                self.created_paths)
+
+
 class SpawnTests(unittest.TestCase):
     def _test_invalid_env(self, spawn):
         args = [sys.executable, '-c', 'pass']
diff --git a/Misc/NEWS.d/next/Library/2018-01-12-09-20-22.bpo-32539.D7AbdE.rst b/Misc/NEWS.d/next/Library/2018-01-12-09-20-22.bpo-32539.D7AbdE.rst
new file mode 100644
index 00000000000..cc24a6b953d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-12-09-20-22.bpo-32539.D7AbdE.rst
@@ -0,0 +1,2 @@
+Fix ``OSError`` for ``os.listdir`` with deep paths (starting with ``\\?\``) on
+windows.  Patch by Anthony Sottile.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index c9886d514b7..cecbb45fb4b 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -2385,7 +2385,7 @@ posix_listdir(PyObject *self, PyObject *args)
     if (len > 0) {
         char ch = namebuf[len-1];
         if (ch != SEP && ch != ALTSEP && ch != ':')
-            namebuf[len++] = '/';
+            namebuf[len++] = SEP;
         strcpy(namebuf + len, "*.*");
     }
 


From webhook-mailer at python.org  Mon Jan 15 17:23:50 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 15 Jan 2018 22:23:50 -0000
Subject: [Python-checkins] [3.6] bpo-31900: Fix localeconv() encoding for
 LC_NUMERIC (#4174) (#5192)
Message-ID: <mailman.25.1516055031.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/5f959c4f9eca404b8bc4bc6348fed27c4b907b89
commit: 5f959c4f9eca404b8bc4bc6348fed27c4b907b89
branch: 3.6
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-15T23:23:47+01:00
summary:

[3.6] bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174) (#5192)

* Add _Py_GetLocaleconvNumeric() function: decode decimal_point and
  thousands_sep fields of localeconv() from the LC_NUMERIC encoding,
  rather than decoding from the LC_CTYPE encoding.
* Modify locale.localeconv() and "n" formatter of str.format() (for
  int, float and complex to use _Py_GetLocaleconvNumeric()
  internally.

(cherry picked from commit cb064fc2321ce8673fe365e9ef60445a27657f54)

files:
A Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst
M Doc/library/locale.rst
M Doc/library/stdtypes.rst
M Doc/whatsnew/3.6.rst
M Include/fileutils.h
M Modules/_localemodule.c
M Python/fileutils.c
M Python/formatter_unicode.c

diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst
index b04442bc162..9a0c570533a 100644
--- a/Doc/library/locale.rst
+++ b/Doc/library/locale.rst
@@ -147,6 +147,16 @@ The :mod:`locale` module defines the following exception and functions:
    | ``CHAR_MAX`` | Nothing is specified in this locale.    |
    +--------------+-----------------------------------------+
 
+   The function sets temporarily the ``LC_CTYPE`` locale to the ``LC_NUMERIC``
+   locale to decode ``decimal_point`` and ``thousands_sep`` byte strings if
+   they are non-ASCII or longer than 1 byte, and the ``LC_NUMERIC`` locale is
+   different than the ``LC_CTYPE`` locale. This temporary change affects other
+   threads.
+
+   .. versionchanged:: 3.6.5
+      The function now sets temporarily the ``LC_CTYPE`` locale to the
+      ``LC_NUMERIC`` locale in some cases.
+
 
 .. function:: nl_langinfo(option)
 
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index 196a4c00056..d8a1647e8b5 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1599,6 +1599,20 @@ expression support in the :mod:`re` module).
    See :ref:`formatstrings` for a description of the various formatting options
    that can be specified in format strings.
 
+   .. note::
+      When formatting a number (:class:`int`, :class:`float`, :class:`float`
+      and subclasses) with the ``n`` type (ex: ``'{:n}'.format(1234)``), the
+      function sets temporarily the ``LC_CTYPE`` locale to the ``LC_NUMERIC``
+      locale to decode ``decimal_point`` and ``thousands_sep`` fields of
+      :c:func:`localeconv` if they are non-ASCII or longer than 1 byte, and the
+      ``LC_NUMERIC`` locale is different than the ``LC_CTYPE`` locale. This
+      temporary change affects other threads.
+
+   .. versionchanged:: 3.6.5
+      When formatting a number with the ``n`` type, the function sets
+      temporarily the ``LC_CTYPE`` locale to the ``LC_NUMERIC`` locale in some
+      cases.
+
 
 .. method:: str.format_map(mapping)
 
diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst
index 847b50140a6..f83508c9250 100644
--- a/Doc/whatsnew/3.6.rst
+++ b/Doc/whatsnew/3.6.rst
@@ -2346,3 +2346,11 @@ It has been replaced by the new ``make regen-all`` target.
 (Contributed by Victor Stinner in :issue:`23404`.)
 
 .. versionchanged:: 3.6.2
+
+
+Notable changes in Python 3.6.5
+===============================
+
+The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE``
+locale to the ``LC_NUMERIC`` locale in some cases.
+(Contributed by Victor Stinner in :issue:`31900`.)
diff --git a/Include/fileutils.h b/Include/fileutils.h
index 900c70faad7..875715df97a 100644
--- a/Include/fileutils.h
+++ b/Include/fileutils.h
@@ -119,6 +119,11 @@ PyAPI_FUNC(int) _Py_get_blocking(int fd);
 PyAPI_FUNC(int) _Py_set_blocking(int fd, int blocking);
 #endif   /* !MS_WINDOWS */
 
+PyAPI_FUNC(int) _Py_GetLocaleconvNumeric(
+    PyObject **decimal_point,
+    PyObject **thousands_sep,
+    const char **grouping);
+
 #endif   /* Py_LIMITED_API */
 
 #ifdef __cplusplus
diff --git a/Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst b/Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst
new file mode 100644
index 00000000000..2d8e3ce6002
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-30-15-55-32.bpo-31900.-S9xc4.rst
@@ -0,0 +1,9 @@
+The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE``
+locale to the ``LC_NUMERIC`` locale to decode ``decimal_point`` and
+``thousands_sep`` byte strings if they are non-ASCII or longer than 1 byte, and
+the ``LC_NUMERIC`` locale is different than the ``LC_CTYPE`` locale.  This
+temporary change affects other threads.
+
+Same change for the :meth:`str.format` method when formatting a number
+(:class:`int`, :class:`float`, :class:`float` and subclasses) with the ``n``
+type (ex: ``'{:n}'.format(1234)``).
diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
index 71c9146ccb8..95b370b1ad0 100644
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -171,12 +171,6 @@ PyLocale_localeconv(PyObject* self)
         RESULT(#i, x); \
     } while (0)
 
-    /* Numeric information */
-    RESULT_STRING(decimal_point);
-    RESULT_STRING(thousands_sep);
-    x = copy_grouping(l->grouping);
-    RESULT("grouping", x);
-
     /* Monetary information */
     RESULT_STRING(int_curr_symbol);
     RESULT_STRING(currency_symbol);
@@ -195,6 +189,32 @@ PyLocale_localeconv(PyObject* self)
     RESULT_INT(n_sep_by_space);
     RESULT_INT(p_sign_posn);
     RESULT_INT(n_sign_posn);
+
+    /* Numeric information */
+    PyObject *decimal_point, *thousands_sep;
+    const char *grouping;
+    if (_Py_GetLocaleconvNumeric(&decimal_point,
+                                 &thousands_sep,
+                                 &grouping) < 0) {
+        goto failed;
+    }
+
+    if (PyDict_SetItemString(result, "decimal_point", decimal_point) < 0) {
+        Py_DECREF(decimal_point);
+        Py_DECREF(thousands_sep);
+        goto failed;
+    }
+    Py_DECREF(decimal_point);
+
+    if (PyDict_SetItemString(result, "thousands_sep", thousands_sep) < 0) {
+        Py_DECREF(thousands_sep);
+        goto failed;
+    }
+    Py_DECREF(thousands_sep);
+
+    x = copy_grouping(grouping);
+    RESULT("grouping", x);
+
     return result;
 
   failed:
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 97505e5bc6d..14dd81b03f0 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -1597,3 +1597,80 @@ _Py_set_blocking(int fd, int blocking)
     return -1;
 }
 #endif
+
+
+int
+_Py_GetLocaleconvNumeric(PyObject **decimal_point, PyObject **thousands_sep,
+                         const char **grouping)
+{
+    int res = -1;
+
+    struct lconv *lc = localeconv();
+
+    int change_locale = 0;
+    if (decimal_point != NULL &&
+        (strlen(lc->decimal_point) > 1 || ((unsigned char)lc->decimal_point[0]) > 127))
+    {
+        change_locale = 1;
+    }
+    if (thousands_sep != NULL &&
+        (strlen(lc->thousands_sep) > 1 || ((unsigned char)lc->thousands_sep[0]) > 127))
+    {
+        change_locale = 1;
+    }
+
+    /* Keep a copy of the LC_CTYPE locale */
+    char *oldloc = NULL, *loc = NULL;
+    if (change_locale) {
+        oldloc = setlocale(LC_CTYPE, NULL);
+        if (!oldloc) {
+            PyErr_SetString(PyExc_RuntimeWarning, "faild to get LC_CTYPE locale");
+            return -1;
+        }
+
+        oldloc = _PyMem_Strdup(oldloc);
+        if (!oldloc) {
+            PyErr_NoMemory();
+            return -1;
+        }
+
+        loc = setlocale(LC_NUMERIC, NULL);
+        if (loc != NULL && strcmp(loc, oldloc) == 0) {
+            loc = NULL;
+        }
+
+        if (loc != NULL) {
+            /* Only set the locale temporarilty the LC_CTYPE locale
+               if LC_NUMERIC locale is different than LC_CTYPE locale and
+               decimal_point and/or thousands_sep are non-ASCII or longer than
+               1 byte */
+            setlocale(LC_CTYPE, loc);
+        }
+    }
+
+    if (decimal_point != NULL) {
+        *decimal_point = PyUnicode_DecodeLocale(lc->decimal_point, NULL);
+        if (*decimal_point == NULL) {
+            goto error;
+        }
+    }
+    if (thousands_sep != NULL) {
+        *thousands_sep = PyUnicode_DecodeLocale(lc->thousands_sep, NULL);
+        if (*thousands_sep == NULL) {
+            goto error;
+        }
+    }
+
+    if (grouping != NULL) {
+        *grouping = lc->grouping;
+    }
+
+    res = 0;
+
+error:
+    if (loc != NULL) {
+        setlocale(LC_CTYPE, oldloc);
+    }
+    PyMem_Free(oldloc);
+    return res;
+}
diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
index d2be76f1e1a..d3ef650e6ce 100644
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -707,18 +707,11 @@ get_locale_info(enum LocaleType type, LocaleInfo *locale_info)
 {
     switch (type) {
     case LT_CURRENT_LOCALE: {
-        struct lconv *locale_data = localeconv();
-        locale_info->decimal_point = PyUnicode_DecodeLocale(
-                                         locale_data->decimal_point,
-                                         NULL);
-        if (locale_info->decimal_point == NULL)
+        if (_Py_GetLocaleconvNumeric(&locale_info->decimal_point,
+                                     &locale_info->thousands_sep,
+                                     &locale_info->grouping) < 0) {
             return -1;
-        locale_info->thousands_sep = PyUnicode_DecodeLocale(
-                                         locale_data->thousands_sep,
-                                         NULL);
-        if (locale_info->thousands_sep == NULL)
-            return -1;
-        locale_info->grouping = locale_data->grouping;
+        }
         break;
     }
     case LT_DEFAULT_LOCALE:


From webhook-mailer at python.org  Mon Jan 15 17:43:28 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 15 Jan 2018 22:43:28 -0000
Subject: [Python-checkins] [3.6] bpo-32555: Fix locale encodings (#5193)
Message-ID: <mailman.26.1516056210.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/b92c159efada05b3a5ff9d0dbce3fcb2334631f6
commit: b92c159efada05b3a5ff9d0dbce3fcb2334631f6
branch: 3.6
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-15T23:43:24+01:00
summary:

[3.6] bpo-32555: Fix locale encodings (#5193)

On FreeBSD and Solaris, os.strerror() now always decode the byte
string from the current locale encoding, rather than using
ASCII/surrogateescape in some cases.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() which has an
  additional current_locale parameter.
* PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() and
* PyUnicode_EncodeLocale() now always use the current locale
* encoding, instead of using Py_DecodeLocale()/Py_EncodeLocale().
* Document encoding in Py_DecodeLocale() and Py_EncodeLocale()
  documentations.
* Add USE_FORCE_ASCII define to not define
  decode_ascii_surrogateescape() on Android.

files:
A Misc/NEWS.d/next/Library/2018-01-15-17-52-47.bpo-32555.CMq2zF.rst
M Doc/c-api/sys.rst
M Doc/c-api/unicode.rst
M Include/fileutils.h
M Objects/unicodeobject.c
M Python/fileutils.c

diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 035cdc16824..48e2b2bb5e9 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -66,9 +66,18 @@ Operating System Utilities
    surrogate character, escape the bytes using the surrogateescape error
    handler instead of decoding them.
 
+   Encoding, highest priority to lowest priority:
+
+   * ``UTF-8`` on macOS and Android;
+   * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
+     ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
+     and :c:func:`mbstowcs` and :c:func:`wcstombs` functions use the
+     ``ISO-8859-1`` encoding.
+   * the current locale encoding (``LC_CTYPE`` locale).
+
    Return a pointer to a newly allocated wide character string, use
    :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write
-   the number of wide characters excluding the null character into ``*size``
+   the number of wide characters excluding the null character into ``*size``.
 
    Return ``NULL`` on decoding error or memory allocation error. If *size* is
    not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to
@@ -94,6 +103,15 @@ Operating System Utilities
    :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters
    in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
 
+   Encoding, highest priority to lowest priority:
+
+   * ``UTF-8`` on macOS and Android;
+   * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
+     ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
+     and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
+     ``ISO-8859-1`` encoding.
+   * the current locale encoding.
+
    Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free`
    to free the memory. Return ``NULL`` on encoding error or memory allocation
    error
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 6e91576ee8f..b9acaec949b 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -773,6 +773,12 @@ system.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.6.5
+      The function now also uses the current locale encoding for the
+      ``surrogateescape`` error handler. Previously, :c:func:`Py_DecodeLocale`
+      was used for the ``surrogateescape``, and the current locale encoding was
+      used for ``strict``.
+
 
 .. c:function:: PyObject* PyUnicode_DecodeLocale(const char *str, const char *errors)
 
@@ -800,6 +806,12 @@ system.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.6.5
+      The function now also uses the current locale encoding for the
+      ``surrogateescape`` error handler. Previously, :c:func:`Py_EncodeLocale`
+      was used for the ``surrogateescape``, and the current locale encoding was
+      used for ``strict``.
+
 
 File System Encoding
 """"""""""""""""""""
diff --git a/Include/fileutils.h b/Include/fileutils.h
index 875715df97a..8fa70baa21c 100644
--- a/Include/fileutils.h
+++ b/Include/fileutils.h
@@ -17,6 +17,16 @@ PyAPI_FUNC(char*) Py_EncodeLocale(
 
 #ifndef Py_LIMITED_API
 
+PyAPI_FUNC(wchar_t *) _Py_DecodeLocaleEx(
+    const char *arg,
+    size_t *size,
+    int current_locale);
+
+PyAPI_FUNC(char*) _Py_EncodeLocaleEx(
+    const wchar_t *text,
+    size_t *error_pos,
+    int current_locale);
+
 PyAPI_FUNC(PyObject *) _Py_device_encoding(int);
 
 #ifdef MS_WINDOWS
diff --git a/Misc/NEWS.d/next/Library/2018-01-15-17-52-47.bpo-32555.CMq2zF.rst b/Misc/NEWS.d/next/Library/2018-01-15-17-52-47.bpo-32555.CMq2zF.rst
new file mode 100644
index 00000000000..054f8cac1f6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-15-17-52-47.bpo-32555.CMq2zF.rst
@@ -0,0 +1,3 @@
+On FreeBSD and Solaris, os.strerror() now always decode the byte string from
+the current locale encoding, rather than using ASCII/surrogateescape in some
+cases.
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 64905e84b14..86cac96cd62 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3439,8 +3439,9 @@ locale_error_handler(const char *errors, int *surrogateescape)
     }
 }
 
-PyObject *
-PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
+static PyObject *
+unicode_encode_locale(PyObject *unicode, const char *errors,
+                      int current_locale)
 {
     Py_ssize_t wlen, wlen2;
     wchar_t *wstr;
@@ -3469,7 +3470,7 @@ PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
         /* "surrogateescape" error handler */
         char *str;
 
-        str = Py_EncodeLocale(wstr, &error_pos);
+        str = _Py_EncodeLocaleEx(wstr, &error_pos, current_locale);
         if (str == NULL) {
             if (error_pos == (size_t)-1) {
                 PyErr_NoMemory();
@@ -3549,6 +3550,12 @@ PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
     return NULL;
 }
 
+PyObject *
+PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
+{
+    return unicode_encode_locale(unicode, errors, 1);
+}
+
 PyObject *
 PyUnicode_EncodeFSDefault(PyObject *unicode)
 {
@@ -3571,7 +3578,8 @@ PyUnicode_EncodeFSDefault(PyObject *unicode)
                                          Py_FileSystemDefaultEncodeErrors);
     }
     else {
-        return PyUnicode_EncodeLocale(unicode, Py_FileSystemDefaultEncodeErrors);
+        return unicode_encode_locale(unicode,
+                                     Py_FileSystemDefaultEncodeErrors, 0);
     }
 #endif
 }
@@ -3741,9 +3749,9 @@ mbstowcs_errorpos(const char *str, size_t len)
     return 0;
 }
 
-PyObject*
-PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
-                              const char *errors)
+static PyObject*
+unicode_decode_locale(const char *str, Py_ssize_t len,
+                      const char *errors, int current_locale)
 {
     wchar_t smallbuf[256];
     size_t smallbuf_len = Py_ARRAY_LENGTH(smallbuf);
@@ -3766,7 +3774,7 @@ PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
 
     if (surrogateescape) {
         /* "surrogateescape" error handler */
-        wstr = Py_DecodeLocale(str, &wlen);
+        wstr = _Py_DecodeLocaleEx(str, &wlen, current_locale);
         if (wstr == NULL) {
             if (wlen == (size_t)-1)
                 PyErr_NoMemory();
@@ -3844,11 +3852,18 @@ PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len,
     return NULL;
 }
 
+PyObject*
+PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t size,
+                              const char *errors)
+{
+    return unicode_decode_locale(str, size, errors, 1);
+}
+
 PyObject*
 PyUnicode_DecodeLocale(const char *str, const char *errors)
 {
     Py_ssize_t size = (Py_ssize_t)strlen(str);
-    return PyUnicode_DecodeLocaleAndSize(str, size, errors);
+    return unicode_decode_locale(str, size, errors, 1);
 }
 
 
@@ -3880,7 +3895,8 @@ PyUnicode_DecodeFSDefaultAndSize(const char *s, Py_ssize_t size)
                                 Py_FileSystemDefaultEncodeErrors);
     }
     else {
-        return PyUnicode_DecodeLocaleAndSize(s, size, Py_FileSystemDefaultEncodeErrors);
+        return unicode_decode_locale(s, size,
+                                     Py_FileSystemDefaultEncodeErrors, 0);
     }
 #endif
 }
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 14dd81b03f0..7b87b72f73c 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -70,7 +70,10 @@ _Py_device_encoding(int fd)
     Py_RETURN_NONE;
 }
 
-#if !defined(__APPLE__) && !defined(MS_WINDOWS)
+#if !defined(__APPLE__) && !defined(__ANDROID__) && !defined(MS_WINDOWS)
+
+#define USE_FORCE_ASCII
+
 extern int _Py_normalize_encoding(const char *, char *, size_t);
 
 /* Workaround FreeBSD and OpenIndiana locale encoding issue with the C locale.
@@ -221,7 +224,7 @@ encode_ascii_surrogateescape(const wchar_t *text, size_t *error_pos)
 }
 #endif   /* !defined(__APPLE__) && !defined(MS_WINDOWS) */
 
-#if !defined(__APPLE__) && (!defined(MS_WINDOWS) || !defined(HAVE_MBRTOWC))
+#if !defined(HAVE_MBRTOWC) || defined(USE_FORCE_ASCII)
 static wchar_t*
 decode_ascii_surrogateescape(const char *arg, size_t *size)
 {
@@ -251,39 +254,9 @@ decode_ascii_surrogateescape(const char *arg, size_t *size)
 #endif
 
 
-/* Decode a byte string from the locale encoding with the
-   surrogateescape error handler: undecodable bytes are decoded as characters
-   in range U+DC80..U+DCFF. If a byte sequence can be decoded as a surrogate
-   character, escape the bytes using the surrogateescape error handler instead
-   of decoding them.
-
-   Return a pointer to a newly allocated wide character string, use
-   PyMem_RawFree() to free the memory. If size is not NULL, write the number of
-   wide characters excluding the null character into *size
-
-   Return NULL on decoding error or memory allocation error. If *size* is not
-   NULL, *size is set to (size_t)-1 on memory error or set to (size_t)-2 on
-   decoding error.
-
-   Decoding errors should never happen, unless there is a bug in the C
-   library.
-
-   Use the Py_EncodeLocale() function to encode the character string back to a
-   byte string. */
-wchar_t*
-Py_DecodeLocale(const char* arg, size_t *size)
+static wchar_t*
+decode_current_locale(const char* arg, size_t *size)
 {
-#if defined(__APPLE__) || defined(__ANDROID__)
-    wchar_t *wstr;
-    wstr = _Py_DecodeUTF8_surrogateescape(arg, strlen(arg));
-    if (size != NULL) {
-        if (wstr != NULL)
-            *size = wcslen(wstr);
-        else
-            *size = (size_t)-1;
-    }
-    return wstr;
-#else
     wchar_t *res;
     size_t argsize;
     size_t count;
@@ -293,19 +266,6 @@ Py_DecodeLocale(const char* arg, size_t *size)
     mbstate_t mbs;
 #endif
 
-#ifndef MS_WINDOWS
-    if (force_ascii == -1)
-        force_ascii = check_force_ascii();
-
-    if (force_ascii) {
-        /* force ASCII encoding to workaround mbstowcs() issue */
-        res = decode_ascii_surrogateescape(arg, size);
-        if (res == NULL)
-            goto oom;
-        return res;
-    }
-#endif
-
 #ifdef HAVE_BROKEN_MBSTOWCS
     /* Some platforms have a broken implementation of
      * mbstowcs which does not count the characters that
@@ -402,72 +362,96 @@ Py_DecodeLocale(const char* arg, size_t *size)
         goto oom;
 #endif   /* HAVE_MBRTOWC */
     return res;
+
 oom:
     if (size != NULL)
         *size = (size_t)-1;
     return NULL;
+}
+
+
+static wchar_t*
+decode_locale(const char* arg, size_t *size, int current_locale)
+{
+    if (current_locale) {
+        return decode_current_locale(arg, size);
+    }
+
+#if defined(__APPLE__) || defined(__ANDROID__)
+    wchar_t *wstr;
+    wstr = _Py_DecodeUTF8_surrogateescape(arg, strlen(arg));
+    if (size != NULL) {
+        if (wstr != NULL)
+            *size = wcslen(wstr);
+        else
+            *size = (size_t)-1;
+    }
+    return wstr;
+#else
+
+#ifdef USE_FORCE_ASCII
+    if (force_ascii == -1) {
+        force_ascii = check_force_ascii();
+    }
+
+    if (force_ascii) {
+        /* force ASCII encoding to workaround mbstowcs() issue */
+        wchar_t *res = decode_ascii_surrogateescape(arg, size);
+        if (res == NULL) {
+            if (size != NULL)
+                *size = (size_t)-1;
+            return NULL;
+        }
+        return res;
+    }
+#endif
+
+    return decode_current_locale(arg, size);
 #endif   /* __APPLE__ or __ANDROID__ */
 }
 
-/* Encode a wide character string to the locale encoding with the
-   surrogateescape error handler: surrogate characters in the range
-   U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
 
-   Return a pointer to a newly allocated byte string, use PyMem_Free() to free
-   the memory. Return NULL on encoding or memory allocation error.
+/* Decode a byte string from the locale encoding with the
+   surrogateescape error handler: undecodable bytes are decoded as characters
+   in range U+DC80..U+DCFF. If a byte sequence can be decoded as a surrogate
+   character, escape the bytes using the surrogateescape error handler instead
+   of decoding them.
 
-   If error_pos is not NULL, *error_pos is set to the index of the invalid
-   character on encoding error, or set to (size_t)-1 otherwise.
+   Return a pointer to a newly allocated wide character string, use
+   PyMem_RawFree() to free the memory. If size is not NULL, write the number of
+   wide characters excluding the null character into *size
 
-   Use the Py_DecodeLocale() function to decode the bytes string back to a wide
-   character string. */
-char*
-Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
+   Return NULL on decoding error or memory allocation error. If *size* is not
+   NULL, *size is set to (size_t)-1 on memory error or set to (size_t)-2 on
+   decoding error.
+
+   Decoding errors should never happen, unless there is a bug in the C
+   library.
+
+   Use the Py_EncodeLocale() function to encode the character string back to a
+   byte string. */
+wchar_t*
+Py_DecodeLocale(const char* arg, size_t *size)
 {
-#if defined(__APPLE__) || defined(__ANDROID__)
-    Py_ssize_t len;
-    PyObject *unicode, *bytes = NULL;
-    char *cpath;
+    return decode_locale(arg, size, 0);
+}
 
-    unicode = PyUnicode_FromWideChar(text, wcslen(text));
-    if (unicode == NULL)
-        return NULL;
 
-    bytes = _PyUnicode_AsUTF8String(unicode, "surrogateescape");
-    Py_DECREF(unicode);
-    if (bytes == NULL) {
-        PyErr_Clear();
-        if (error_pos != NULL)
-            *error_pos = (size_t)-1;
-        return NULL;
-    }
+wchar_t*
+_Py_DecodeLocaleEx(const char* arg, size_t *size, int current_locale)
+{
+    return decode_locale(arg, size, current_locale);
+}
 
-    len = PyBytes_GET_SIZE(bytes);
-    cpath = PyMem_Malloc(len+1);
-    if (cpath == NULL) {
-        PyErr_Clear();
-        Py_DECREF(bytes);
-        if (error_pos != NULL)
-            *error_pos = (size_t)-1;
-        return NULL;
-    }
-    memcpy(cpath, PyBytes_AsString(bytes), len + 1);
-    Py_DECREF(bytes);
-    return cpath;
-#else   /* __APPLE__ */
+
+static char*
+encode_current_locale(const wchar_t *text, size_t *error_pos)
+{
     const size_t len = wcslen(text);
     char *result = NULL, *bytes = NULL;
     size_t i, size, converted;
     wchar_t c, buf[2];
 
-#ifndef MS_WINDOWS
-    if (force_ascii == -1)
-        force_ascii = check_force_ascii();
-
-    if (force_ascii)
-        return encode_ascii_surrogateescape(text, error_pos);
-#endif
-
     /* The function works in two steps:
        1. compute the length of the output buffer in bytes (size)
        2. outputs the bytes */
@@ -522,10 +506,89 @@ Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
         bytes = result;
     }
     return result;
+}
+
+
+static char*
+encode_locale(const wchar_t *text, size_t *error_pos, int current_locale)
+{
+    if (current_locale) {
+        return encode_current_locale(text, error_pos);
+    }
+
+#if defined(__APPLE__) || defined(__ANDROID__)
+    Py_ssize_t len;
+    PyObject *unicode, *bytes = NULL;
+    char *cpath;
+
+    unicode = PyUnicode_FromWideChar(text, wcslen(text));
+    if (unicode == NULL)
+        return NULL;
+
+    bytes = _PyUnicode_AsUTF8String(unicode, "surrogateescape");
+    Py_DECREF(unicode);
+    if (bytes == NULL) {
+        PyErr_Clear();
+        if (error_pos != NULL)
+            *error_pos = (size_t)-1;
+        return NULL;
+    }
+
+    len = PyBytes_GET_SIZE(bytes);
+    cpath = PyMem_Malloc(len+1);
+    if (cpath == NULL) {
+        PyErr_Clear();
+        Py_DECREF(bytes);
+        if (error_pos != NULL)
+            *error_pos = (size_t)-1;
+        return NULL;
+    }
+    memcpy(cpath, PyBytes_AsString(bytes), len + 1);
+    Py_DECREF(bytes);
+    return cpath;
+#else   /* __APPLE__ */
+
+#ifdef USE_FORCE_ASCII
+    if (force_ascii == -1) {
+        force_ascii = check_force_ascii();
+    }
+
+    if (force_ascii) {
+        return encode_ascii_surrogateescape(text, error_pos);
+    }
+#endif
+
+    return encode_current_locale(text, error_pos);
 #endif   /* __APPLE__ or __ANDROID__ */
 }
 
 
+/* Encode a wide character string to the locale encoding with the
+   surrogateescape error handler: surrogate characters in the range
+   U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
+
+   Return a pointer to a newly allocated byte string, use PyMem_Free() to free
+   the memory. Return NULL on encoding or memory allocation error.
+
+   If error_pos is not NULL, *error_pos is set to the index of the invalid
+   character on encoding error, or set to (size_t)-1 otherwise.
+
+   Use the Py_DecodeLocale() function to decode the bytes string back to a wide
+   character string. */
+char*
+Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
+{
+    return encode_locale(text, error_pos, 0);
+}
+
+
+char*
+_Py_EncodeLocaleEx(const wchar_t *text, size_t *error_pos, int current_locale)
+{
+    return encode_locale(text, error_pos, current_locale);
+}
+
+
 #ifdef MS_WINDOWS
 static __int64 secs_between_epochs = 11644473600; /* Seconds between 1.1.1601 and 1.1.1970 */
 


From webhook-mailer at python.org  Mon Jan 15 17:56:35 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 15 Jan 2018 22:56:35 -0000
Subject: [Python-checkins] Make asyncio tests run when there's no SSL module
 (#5196)
Message-ID: <mailman.27.1516056995.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/21102f0dc20cc347677191817c1b66e20ef7bf21
commit: 21102f0dc20cc347677191817c1b66e20ef7bf21
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-15T17:56:32-05:00
summary:

Make asyncio tests run when there's no SSL module (#5196)

files:
M Lib/test/test_asyncio/functional.py

diff --git a/Lib/test/test_asyncio/functional.py b/Lib/test/test_asyncio/functional.py
index 5fd174b6f43..fbec462c1db 100644
--- a/Lib/test/test_asyncio/functional.py
+++ b/Lib/test/test_asyncio/functional.py
@@ -5,7 +5,6 @@
 import pprint
 import select
 import socket
-import ssl
 import tempfile
 import threading
 
@@ -146,8 +145,6 @@ def start_tls(self, ssl_context, *,
                   server_side=False,
                   server_hostname=None):
 
-        assert isinstance(ssl_context, ssl.SSLContext)
-
         ssl_sock = ssl_context.wrap_socket(
             self.__sock, server_side=server_side,
             server_hostname=server_hostname,


From webhook-mailer at python.org  Mon Jan 15 18:07:14 2018
From: webhook-mailer at python.org (Barry Warsaw)
Date: Mon, 15 Jan 2018 23:07:14 -0000
Subject: [Python-checkins] Implement the get_resource_reader() API for file
 system imports (#5168)
Message-ID: <mailman.28.1516057635.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/5ec0feeeecc1617987ec6cdc6d62b916e718a5cf
commit: 5ec0feeeecc1617987ec6cdc6d62b916e718a5cf
branch: master
author: Barry Warsaw <barry at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-15T15:07:11-08:00
summary:

Implement the get_resource_reader() API for file system imports (#5168)

files:
M Lib/importlib/_bootstrap_external.py
M Lib/importlib/abc.py
M Lib/importlib/resources.py
M Lib/test/test_importlib/test_resource.py
M Lib/test/test_importlib/util.py
M Python/importlib_external.h

diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py
index e808507d3bb..cf75719c86f 100644
--- a/Lib/importlib/_bootstrap_external.py
+++ b/Lib/importlib/_bootstrap_external.py
@@ -6,12 +6,11 @@
 work. One should use importlib as the public-facing version of this module.
 
 """
-#
-# IMPORTANT: Whenever making changes to this module, be sure to run
-# a top-level make in order to get the frozen version of the module
-# updated. Not doing so will result in the Makefile to fail for
-# all others who don't have a ./python around to freeze the module
-# in the early stages of compilation.
+# IMPORTANT: Whenever making changes to this module, be sure to run a top-level
+# `make regen-importlib` followed by `make` in order to get the frozen version
+# of the module updated. Not doing so will result in the Makefile to fail for
+# all others who don't have a ./python around to freeze the module in the early
+# stages of compilation.
 #
 
 # See importlib._setup() for what is injected into the global namespace.
@@ -911,6 +910,33 @@ def get_data(self, path):
         with _io.FileIO(path, 'r') as file:
             return file.read()
 
+    # ResourceReader ABC API.
+
+    @_check_name
+    def get_resource_reader(self, module):
+        if self.is_package(module):
+            return self
+        return None
+
+    def open_resource(self, resource):
+        path = _path_join(_path_split(self.path)[0], resource)
+        return _io.FileIO(path, 'r')
+
+    def resource_path(self, resource):
+        if not self.is_resource(resource):
+            raise FileNotFoundError
+        path = _path_join(_path_split(self.path)[0], resource)
+        return path
+
+    def is_resource(self, name):
+        if path_sep in name:
+            return False
+        path = _path_join(_path_split(self.path)[0], name)
+        return _path_isfile(path)
+
+    def contents(self):
+        return iter(_os.listdir(_path_split(self.path)[0]))
+
 
 class SourceFileLoader(FileLoader, SourceLoader):
 
diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py
index bbff7af5885..d2f45093a45 100644
--- a/Lib/importlib/abc.py
+++ b/Lib/importlib/abc.py
@@ -342,7 +342,7 @@ def set_data(self, path, data):
 _register(SourceLoader, machinery.SourceFileLoader)
 
 
-class ResourceReader:
+class ResourceReader(metaclass=abc.ABCMeta):
 
     """Abstract base class to provide resource-reading support.
 
@@ -383,3 +383,6 @@ def is_resource(self, name):
     def contents(self):
         """Return an iterator of strings over the contents of the package."""
         return iter([])
+
+
+_register(ResourceReader, machinery.SourceFileLoader)
diff --git a/Lib/importlib/resources.py b/Lib/importlib/resources.py
index 8511f24d8e7..20888dfa8ea 100644
--- a/Lib/importlib/resources.py
+++ b/Lib/importlib/resources.py
@@ -59,8 +59,10 @@ def _get_resource_reader(
     # hook wants to create a weak reference to the object, but
     # zipimport.zipimporter does not support weak references, resulting in a
     # TypeError.  That seems terrible.
-    if hasattr(package.__spec__.loader, 'open_resource'):
-        return cast(resources_abc.ResourceReader, package.__spec__.loader)
+    spec = package.__spec__
+    if hasattr(spec.loader, 'get_resource_reader'):
+        return cast(resources_abc.ResourceReader,
+                    spec.loader.get_resource_reader(spec.name))
     return None
 
 
diff --git a/Lib/test/test_importlib/test_resource.py b/Lib/test/test_importlib/test_resource.py
index c35f7897485..c38ad0358a0 100644
--- a/Lib/test/test_importlib/test_resource.py
+++ b/Lib/test/test_importlib/test_resource.py
@@ -1,10 +1,10 @@
 import sys
 import unittest
 
-from importlib import resources
 from . import data01
 from . import zipdata02
 from . import util
+from importlib import resources
 
 
 class ResourceTests:
diff --git a/Lib/test/test_importlib/util.py b/Lib/test/test_importlib/util.py
index bfb7cad6e59..7b07b046a58 100644
--- a/Lib/test/test_importlib/util.py
+++ b/Lib/test/test_importlib/util.py
@@ -397,6 +397,9 @@ def caseok_env_changed(self, *, should_exist):
 
 def create_package(file, path, is_package=True, contents=()):
     class Reader(ResourceReader):
+        def get_resource_reader(self, package):
+            return self
+
         def open_resource(self, path):
             self._path = path
             if isinstance(file, Exception):
diff --git a/Python/importlib_external.h b/Python/importlib_external.h
index 902bef45e32..e298bd0c20c 100644
--- a/Python/importlib_external.h
+++ b/Python/importlib_external.h
@@ -77,7 +77,7 @@ const unsigned char _Py_M__importlib_external[] = {
     111,122,101,110,32,105,109,112,111,114,116,108,105,98,46,95,
     98,111,111,116,115,116,114,97,112,95,101,120,116,101,114,110,
     97,108,62,218,11,95,114,101,108,97,120,95,99,97,115,101,
-    37,0,0,0,115,2,0,0,0,0,2,122,37,95,109,97,
+    36,0,0,0,115,2,0,0,0,0,2,122,37,95,109,97,
     107,101,95,114,101,108,97,120,95,99,97,115,101,46,60,108,
     111,99,97,108,115,62,46,95,114,101,108,97,120,95,99,97,
     115,101,99,0,0,0,0,0,0,0,0,0,0,0,0,1,
@@ -87,7 +87,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,99,107,101,100,32,99,97,115,101,45,105,110,115,101,110,
     115,105,116,105,118,101,108,121,46,70,114,2,0,0,0,114,
     2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,5,0,0,0,41,0,0,0,115,2,0,0,
+    0,0,0,114,5,0,0,0,40,0,0,0,115,2,0,0,
     0,0,2,41,5,218,3,115,121,115,218,8,112,108,97,116,
     102,111,114,109,218,10,115,116,97,114,116,115,119,105,116,104,
     218,27,95,67,65,83,69,95,73,78,83,69,78,83,73,84,
@@ -96,7 +96,7 @@ const unsigned char _Py_M__importlib_external[] = {
     95,80,76,65,84,70,79,82,77,83,95,83,84,82,95,75,
     69,89,41,1,114,5,0,0,0,114,2,0,0,0,41,1,
     114,3,0,0,0,114,4,0,0,0,218,16,95,109,97,107,
-    101,95,114,101,108,97,120,95,99,97,115,101,30,0,0,0,
+    101,95,114,101,108,97,120,95,99,97,115,101,29,0,0,0,
     115,14,0,0,0,0,1,12,1,12,1,6,2,4,2,14,
     4,8,3,114,11,0,0,0,99,1,0,0,0,0,0,0,
     0,1,0,0,0,4,0,0,0,67,0,0,0,115,20,0,
@@ -108,7 +108,7 @@ const unsigned char _Py_M__importlib_external[] = {
     4,0,0,0,218,6,108,105,116,116,108,101,41,2,218,3,
     105,110,116,218,8,116,111,95,98,121,116,101,115,41,1,218,
     1,120,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,7,95,119,95,108,111,110,103,47,0,0,0,115,2,
+    0,218,7,95,119,95,108,111,110,103,46,0,0,0,115,2,
     0,0,0,0,2,114,17,0,0,0,99,1,0,0,0,0,
     0,0,0,1,0,0,0,4,0,0,0,67,0,0,0,115,
     12,0,0,0,116,0,160,1,124,0,100,1,161,2,83,0,
@@ -119,7 +119,7 @@ const unsigned char _Py_M__importlib_external[] = {
     10,102,114,111,109,95,98,121,116,101,115,41,1,90,9,105,
     110,116,95,98,121,116,101,115,114,2,0,0,0,114,2,0,
     0,0,114,4,0,0,0,218,7,95,114,95,108,111,110,103,
-    52,0,0,0,115,2,0,0,0,0,2,114,19,0,0,0,
+    51,0,0,0,115,2,0,0,0,0,2,114,19,0,0,0,
     99,0,0,0,0,0,0,0,0,1,0,0,0,4,0,0,
     0,71,0,0,0,115,20,0,0,0,116,0,160,1,100,1,
     100,2,132,0,124,0,68,0,131,1,161,1,83,0,41,3,
@@ -132,14 +132,14 @@ const unsigned char _Py_M__importlib_external[] = {
     116,114,105,112,218,15,112,97,116,104,95,115,101,112,97,114,
     97,116,111,114,115,41,2,218,2,46,48,218,4,112,97,114,
     116,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    250,10,60,108,105,115,116,99,111,109,112,62,59,0,0,0,
+    250,10,60,108,105,115,116,99,111,109,112,62,58,0,0,0,
     115,2,0,0,0,6,1,122,30,95,112,97,116,104,95,106,
     111,105,110,46,60,108,111,99,97,108,115,62,46,60,108,105,
     115,116,99,111,109,112,62,41,2,218,8,112,97,116,104,95,
     115,101,112,218,4,106,111,105,110,41,1,218,10,112,97,116,
     104,95,112,97,114,116,115,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,10,95,112,97,116,104,95,106,111,
-    105,110,57,0,0,0,115,4,0,0,0,0,2,10,1,114,
+    105,110,56,0,0,0,115,4,0,0,0,0,2,10,1,114,
     28,0,0,0,99,1,0,0,0,0,0,0,0,5,0,0,
     0,5,0,0,0,67,0,0,0,115,96,0,0,0,116,0,
     116,1,131,1,100,1,107,2,114,36,124,0,160,2,116,3,
@@ -157,7 +157,7 @@ const unsigned char _Py_M__importlib_external[] = {
     112,108,105,116,41,5,218,4,112,97,116,104,90,5,102,114,
     111,110,116,218,1,95,218,4,116,97,105,108,114,16,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,11,95,112,97,116,104,95,115,112,108,105,116,63,0,0,
+    218,11,95,112,97,116,104,95,115,112,108,105,116,62,0,0,
     0,115,16,0,0,0,0,2,12,1,16,1,8,1,14,1,
     8,1,18,1,12,1,114,38,0,0,0,99,1,0,0,0,
     0,0,0,0,1,0,0,0,3,0,0,0,67,0,0,0,
@@ -173,7 +173,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,41,2,114,1,0,0,0,90,4,115,116,97,116,41,1,
     114,35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
     4,0,0,0,218,10,95,112,97,116,104,95,115,116,97,116,
-    75,0,0,0,115,2,0,0,0,0,7,114,39,0,0,0,
+    74,0,0,0,115,2,0,0,0,0,7,114,39,0,0,0,
     99,2,0,0,0,0,0,0,0,3,0,0,0,8,0,0,
     0,67,0,0,0,115,48,0,0,0,121,12,116,0,124,0,
     131,1,125,2,87,0,110,20,4,0,116,1,107,10,114,32,
@@ -187,7 +187,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,35,0,0,0,218,4,109,111,100,101,90,9,115,116,97,
     116,95,105,110,102,111,114,2,0,0,0,114,2,0,0,0,
     114,4,0,0,0,218,18,95,112,97,116,104,95,105,115,95,
-    109,111,100,101,95,116,121,112,101,85,0,0,0,115,10,0,
+    109,111,100,101,95,116,121,112,101,84,0,0,0,115,10,0,
     0,0,0,2,2,1,12,1,14,1,6,1,114,43,0,0,
     0,99,1,0,0,0,0,0,0,0,1,0,0,0,3,0,
     0,0,67,0,0,0,115,10,0,0,0,116,0,124,0,100,
@@ -196,7 +196,7 @@ const unsigned char _Py_M__importlib_external[] = {
     46,105,115,102,105,108,101,46,105,0,128,0,0,41,1,114,
     43,0,0,0,41,1,114,35,0,0,0,114,2,0,0,0,
     114,2,0,0,0,114,4,0,0,0,218,12,95,112,97,116,
-    104,95,105,115,102,105,108,101,94,0,0,0,115,2,0,0,
+    104,95,105,115,102,105,108,101,93,0,0,0,115,2,0,0,
     0,0,2,114,44,0,0,0,99,1,0,0,0,0,0,0,
     0,1,0,0,0,3,0,0,0,67,0,0,0,115,22,0,
     0,0,124,0,115,12,116,0,160,1,161,0,125,0,116,2,
@@ -206,7 +206,7 @@ const unsigned char _Py_M__importlib_external[] = {
     3,114,1,0,0,0,218,6,103,101,116,99,119,100,114,43,
     0,0,0,41,1,114,35,0,0,0,114,2,0,0,0,114,
     2,0,0,0,114,4,0,0,0,218,11,95,112,97,116,104,
-    95,105,115,100,105,114,99,0,0,0,115,6,0,0,0,0,
+    95,105,115,100,105,114,98,0,0,0,115,6,0,0,0,0,
     2,4,1,8,1,114,46,0,0,0,233,182,1,0,0,99,
     3,0,0,0,0,0,0,0,6,0,0,0,11,0,0,0,
     67,0,0,0,115,162,0,0,0,100,1,160,0,124,0,116,
@@ -241,7 +241,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,90,8,112,97,116,104,95,116,109,112,90,2,102,100,
     218,4,102,105,108,101,114,2,0,0,0,114,2,0,0,0,
     114,4,0,0,0,218,13,95,119,114,105,116,101,95,97,116,
-    111,109,105,99,106,0,0,0,115,26,0,0,0,0,5,16,
+    111,109,105,99,105,0,0,0,115,26,0,0,0,0,5,16,
     1,6,1,26,1,2,3,14,1,20,1,16,1,14,1,2,
     1,14,1,14,1,6,1,114,56,0,0,0,105,64,13,0,
     0,233,2,0,0,0,114,13,0,0,0,115,2,0,0,0,
@@ -348,7 +348,7 @@ const unsigned char _Py_M__importlib_external[] = {
     108,109,111,115,116,95,102,105,108,101,110,97,109,101,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,17,99,
     97,99,104,101,95,102,114,111,109,95,115,111,117,114,99,101,
-    10,1,0,0,115,48,0,0,0,0,18,8,1,6,1,6,
+    9,1,0,0,115,48,0,0,0,0,18,8,1,6,1,6,
     1,8,1,4,1,8,1,12,1,10,1,12,1,16,1,8,
     1,8,1,8,1,24,1,8,1,12,1,6,2,8,1,8,
     1,8,1,8,1,14,1,14,1,114,81,0,0,0,99,1,
@@ -422,7 +422,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,118,101,108,90,13,98,97,115,101,95,102,105,108,101,110,
     97,109,101,114,2,0,0,0,114,2,0,0,0,114,4,0,
     0,0,218,17,115,111,117,114,99,101,95,102,114,111,109,95,
-    99,97,99,104,101,55,1,0,0,115,46,0,0,0,0,9,
+    99,97,99,104,101,54,1,0,0,115,46,0,0,0,0,9,
     12,1,8,1,10,1,12,1,12,1,8,1,6,1,10,1,
     10,1,8,1,6,1,10,1,8,1,16,1,10,1,6,1,
     8,1,16,1,8,1,6,1,8,1,14,1,114,87,0,0,
@@ -456,7 +456,7 @@ const unsigned char _Py_M__importlib_external[] = {
     36,0,0,0,90,9,101,120,116,101,110,115,105,111,110,218,
     11,115,111,117,114,99,101,95,112,97,116,104,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,218,15,95,103,101,
-    116,95,115,111,117,114,99,101,102,105,108,101,89,1,0,0,
+    116,95,115,111,117,114,99,101,102,105,108,101,88,1,0,0,
     115,20,0,0,0,0,7,12,1,4,1,16,1,24,1,4,
     1,2,1,12,1,18,1,18,1,114,93,0,0,0,99,1,
     0,0,0,0,0,0,0,1,0,0,0,8,0,0,0,67,
@@ -470,7 +470,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,68,0,0,0,114,76,0,0,0,41,1,218,8,102,105,
     108,101,110,97,109,101,114,2,0,0,0,114,2,0,0,0,
     114,4,0,0,0,218,11,95,103,101,116,95,99,97,99,104,
-    101,100,108,1,0,0,115,16,0,0,0,0,1,14,1,2,
+    101,100,107,1,0,0,115,16,0,0,0,0,1,14,1,2,
     1,8,1,14,1,8,1,14,1,4,2,114,97,0,0,0,
     99,1,0,0,0,0,0,0,0,2,0,0,0,8,0,0,
     0,67,0,0,0,115,52,0,0,0,121,14,116,0,124,0,
@@ -484,7 +484,7 @@ const unsigned char _Py_M__importlib_external[] = {
     3,114,39,0,0,0,114,41,0,0,0,114,40,0,0,0,
     41,2,114,35,0,0,0,114,42,0,0,0,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,218,10,95,99,97,
-    108,99,95,109,111,100,101,120,1,0,0,115,12,0,0,0,
+    108,99,95,109,111,100,101,119,1,0,0,115,12,0,0,0,
     0,2,2,1,14,1,14,1,10,3,8,1,114,99,0,0,
     0,99,1,0,0,0,0,0,0,0,3,0,0,0,8,0,
     0,0,3,0,0,0,115,68,0,0,0,100,6,135,0,102,
@@ -521,7 +521,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,108,102,114,100,0,0,0,218,4,97,114,103,115,90,6,
     107,119,97,114,103,115,41,1,218,6,109,101,116,104,111,100,
     114,2,0,0,0,114,4,0,0,0,218,19,95,99,104,101,
-    99,107,95,110,97,109,101,95,119,114,97,112,112,101,114,140,
+    99,107,95,110,97,109,101,95,119,114,97,112,112,101,114,139,
     1,0,0,115,12,0,0,0,0,1,8,1,8,1,10,1,
     4,1,18,1,122,40,95,99,104,101,99,107,95,110,97,109,
     101,46,60,108,111,99,97,108,115,62,46,95,99,104,101,99,
@@ -539,7 +539,7 @@ const unsigned char _Py_M__importlib_external[] = {
     116,116,114,218,8,95,95,100,105,99,116,95,95,218,6,117,
     112,100,97,116,101,41,3,90,3,110,101,119,90,3,111,108,
     100,114,53,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,218,5,95,119,114,97,112,151,1,0,0,
+    114,4,0,0,0,218,5,95,119,114,97,112,150,1,0,0,
     115,8,0,0,0,0,1,10,1,10,1,22,1,122,26,95,
     99,104,101,99,107,95,110,97,109,101,46,60,108,111,99,97,
     108,115,62,46,95,119,114,97,112,41,1,78,41,3,218,10,
@@ -547,7 +547,7 @@ const unsigned char _Py_M__importlib_external[] = {
     9,78,97,109,101,69,114,114,111,114,41,3,114,104,0,0,
     0,114,105,0,0,0,114,115,0,0,0,114,2,0,0,0,
     41,1,114,104,0,0,0,114,4,0,0,0,218,11,95,99,
-    104,101,99,107,95,110,97,109,101,132,1,0,0,115,14,0,
+    104,101,99,107,95,110,97,109,101,131,1,0,0,115,14,0,
     0,0,0,8,14,7,2,1,10,1,14,2,14,5,10,1,
     114,118,0,0,0,99,2,0,0,0,0,0,0,0,5,0,
     0,0,6,0,0,0,67,0,0,0,115,60,0,0,0,124,
@@ -575,7 +575,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,218,6,108,111,97,100,101,114,218,8,112,111,114,116,105,
     111,110,115,218,3,109,115,103,114,2,0,0,0,114,2,0,
     0,0,114,4,0,0,0,218,17,95,102,105,110,100,95,109,
-    111,100,117,108,101,95,115,104,105,109,160,1,0,0,115,10,
+    111,100,117,108,101,95,115,104,105,109,159,1,0,0,115,10,
     0,0,0,0,10,14,1,16,1,4,1,22,1,114,125,0,
     0,0,99,3,0,0,0,0,0,0,0,6,0,0,0,4,
     0,0,0,67,0,0,0,115,158,0,0,0,124,0,100,1,
@@ -642,7 +642,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,90,5,109,97,103,105,99,114,77,0,0,0,114,69,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
     0,218,13,95,99,108,97,115,115,105,102,121,95,112,121,99,
-    177,1,0,0,115,28,0,0,0,0,16,12,1,8,1,16,
+    176,1,0,0,115,28,0,0,0,0,16,12,1,8,1,16,
     1,12,1,12,1,12,1,10,1,12,1,8,1,16,2,8,
     1,16,1,12,1,114,133,0,0,0,99,5,0,0,0,0,
     0,0,0,6,0,0,0,4,0,0,0,67,0,0,0,115,
@@ -696,7 +696,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,114,100,0,0,0,114,132,0,0,0,114,77,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
     23,95,118,97,108,105,100,97,116,101,95,116,105,109,101,115,
-    116,97,109,112,95,112,121,99,210,1,0,0,115,14,0,0,
+    116,97,109,112,95,112,121,99,209,1,0,0,115,14,0,0,
     0,0,19,24,1,10,1,12,1,12,1,8,1,24,1,114,
     137,0,0,0,99,4,0,0,0,0,0,0,0,4,0,0,
     0,3,0,0,0,67,0,0,0,115,38,0,0,0,124,0,
@@ -742,7 +742,7 @@ const unsigned char _Py_M__importlib_external[] = {
     104,97,115,104,114,100,0,0,0,114,132,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,18,95,
     118,97,108,105,100,97,116,101,95,104,97,115,104,95,112,121,
-    99,238,1,0,0,115,8,0,0,0,0,17,16,1,2,1,
+    99,237,1,0,0,115,8,0,0,0,0,17,16,1,2,1,
     10,1,114,139,0,0,0,99,4,0,0,0,0,0,0,0,
     5,0,0,0,5,0,0,0,67,0,0,0,115,80,0,0,
     0,116,0,160,1,124,0,161,1,125,4,116,2,124,4,116,
@@ -765,7 +765,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,100,0,0,0,114,91,0,0,0,114,92,0,0,
     0,218,4,99,111,100,101,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,17,95,99,111,109,112,105,108,101,
-    95,98,121,116,101,99,111,100,101,6,2,0,0,115,16,0,
+    95,98,121,116,101,99,111,100,101,5,2,0,0,115,16,0,
     0,0,0,2,10,1,10,1,12,1,8,1,12,1,4,2,
     10,1,114,145,0,0,0,114,60,0,0,0,99,3,0,0,
     0,0,0,0,0,4,0,0,0,5,0,0,0,67,0,0,
@@ -783,7 +783,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,218,5,109,116,105,109,101,114,136,0,0,0,114,54,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
     0,0,218,22,95,99,111,100,101,95,116,111,95,116,105,109,
-    101,115,116,97,109,112,95,112,121,99,19,2,0,0,115,12,
+    101,115,116,97,109,112,95,112,121,99,18,2,0,0,115,12,
     0,0,0,0,2,8,1,14,1,14,1,14,1,16,1,114,
     150,0,0,0,84,99,3,0,0,0,0,0,0,0,5,0,
     0,0,5,0,0,0,67,0,0,0,115,80,0,0,0,116,
@@ -802,7 +802,7 @@ const unsigned char _Py_M__importlib_external[] = {
     138,0,0,0,90,7,99,104,101,99,107,101,100,114,54,0,
     0,0,114,69,0,0,0,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,17,95,99,111,100,101,95,116,111,
-    95,104,97,115,104,95,112,121,99,29,2,0,0,115,14,0,
+    95,104,97,115,104,95,112,121,99,28,2,0,0,115,14,0,
     0,0,0,2,8,1,12,1,14,1,16,1,10,1,16,1,
     114,152,0,0,0,99,1,0,0,0,0,0,0,0,5,0,
     0,0,6,0,0,0,67,0,0,0,115,62,0,0,0,100,
@@ -829,7 +829,7 @@ const unsigned char _Py_M__importlib_external[] = {
     100,108,105,110,101,218,8,101,110,99,111,100,105,110,103,90,
     15,110,101,119,108,105,110,101,95,100,101,99,111,100,101,114,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    13,100,101,99,111,100,101,95,115,111,117,114,99,101,40,2,
+    13,100,101,99,111,100,101,95,115,111,117,114,99,101,39,2,
     0,0,115,10,0,0,0,0,5,8,1,12,1,10,1,12,
     1,114,157,0,0,0,41,2,114,122,0,0,0,218,26,115,
     117,98,109,111,100,117,108,101,95,115,101,97,114,99,104,95,
@@ -891,7 +891,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,114,161,0,0,0,90,7,100,105,114,110,97,109,101,114,
     2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,23,
     115,112,101,99,95,102,114,111,109,95,102,105,108,101,95,108,
-    111,99,97,116,105,111,110,57,2,0,0,115,62,0,0,0,
+    111,99,97,116,105,111,110,56,2,0,0,115,62,0,0,0,
     0,12,8,4,4,1,10,2,2,1,14,1,14,1,8,2,
     10,8,16,1,6,3,8,1,16,1,14,1,10,1,6,1,
     6,2,4,3,8,2,10,1,2,1,14,1,14,1,6,2,
@@ -928,7 +928,7 @@ const unsigned char _Py_M__importlib_external[] = {
     65,67,72,73,78,69,41,2,218,3,99,108,115,114,3,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
     0,218,14,95,111,112,101,110,95,114,101,103,105,115,116,114,
-    121,137,2,0,0,115,8,0,0,0,0,2,2,1,14,1,
+    121,136,2,0,0,115,8,0,0,0,0,2,2,1,14,1,
     14,1,122,36,87,105,110,100,111,119,115,82,101,103,105,115,
     116,114,121,70,105,110,100,101,114,46,95,111,112,101,110,95,
     114,101,103,105,115,116,114,121,99,2,0,0,0,0,0,0,
@@ -953,7 +953,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,116,114,121,95,107,101,121,114,3,0,0,0,90,4,104,
     107,101,121,218,8,102,105,108,101,112,97,116,104,114,2,0,
     0,0,114,2,0,0,0,114,4,0,0,0,218,16,95,115,
-    101,97,114,99,104,95,114,101,103,105,115,116,114,121,144,2,
+    101,97,114,99,104,95,114,101,103,105,115,116,114,121,143,2,
     0,0,115,22,0,0,0,0,2,6,1,8,2,6,1,6,
     1,22,1,2,1,12,1,26,1,14,1,6,1,122,38,87,
     105,110,100,111,119,115,82,101,103,105,115,116,114,121,70,105,
@@ -976,7 +976,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,116,114,178,0,0,0,114,122,0,0,0,114,168,0,0,
     0,114,166,0,0,0,114,2,0,0,0,114,2,0,0,0,
     114,4,0,0,0,218,9,102,105,110,100,95,115,112,101,99,
-    159,2,0,0,115,26,0,0,0,0,2,10,1,8,1,4,
+    158,2,0,0,115,26,0,0,0,0,2,10,1,8,1,4,
     1,2,1,12,1,14,1,6,1,16,1,14,1,6,1,8,
     1,8,1,122,31,87,105,110,100,111,119,115,82,101,103,105,
     115,116,114,121,70,105,110,100,101,114,46,102,105,110,100,95,
@@ -994,7 +994,7 @@ const unsigned char _Py_M__importlib_external[] = {
     78,41,2,114,182,0,0,0,114,122,0,0,0,41,4,114,
     172,0,0,0,114,121,0,0,0,114,35,0,0,0,114,166,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,218,11,102,105,110,100,95,109,111,100,117,108,101,175,
+    0,0,218,11,102,105,110,100,95,109,111,100,117,108,101,174,
     2,0,0,115,8,0,0,0,0,7,12,1,8,1,6,2,
     122,33,87,105,110,100,111,119,115,82,101,103,105,115,116,114,
     121,70,105,110,100,101,114,46,102,105,110,100,95,109,111,100,
@@ -1004,7 +1004,7 @@ const unsigned char _Py_M__importlib_external[] = {
     11,99,108,97,115,115,109,101,116,104,111,100,114,173,0,0,
     0,114,179,0,0,0,114,182,0,0,0,114,183,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,170,0,0,0,125,2,0,0,115,18,0,
+    4,0,0,0,114,170,0,0,0,124,2,0,0,115,18,0,
     0,0,12,5,4,3,4,2,4,2,12,7,12,15,2,1,
     12,15,2,1,114,170,0,0,0,99,0,0,0,0,0,0,
     0,0,0,0,0,0,2,0,0,0,64,0,0,0,115,48,
@@ -1039,7 +1039,7 @@ const unsigned char _Py_M__importlib_external[] = {
     121,0,0,0,114,96,0,0,0,90,13,102,105,108,101,110,
     97,109,101,95,98,97,115,101,90,9,116,97,105,108,95,110,
     97,109,101,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,161,0,0,0,194,2,0,0,115,8,0,0,0,
+    0,0,114,161,0,0,0,193,2,0,0,115,8,0,0,0,
     0,3,18,1,16,1,14,1,122,24,95,76,111,97,100,101,
     114,66,97,115,105,99,115,46,105,115,95,112,97,99,107,97,
     103,101,99,2,0,0,0,0,0,0,0,2,0,0,0,1,
@@ -1049,7 +1049,7 @@ const unsigned char _Py_M__importlib_external[] = {
     100,117,108,101,32,99,114,101,97,116,105,111,110,46,78,114,
     2,0,0,0,41,2,114,102,0,0,0,114,166,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    13,99,114,101,97,116,101,95,109,111,100,117,108,101,202,2,
+    13,99,114,101,97,116,101,95,109,111,100,117,108,101,201,2,
     0,0,115,0,0,0,0,122,27,95,76,111,97,100,101,114,
     66,97,115,105,99,115,46,99,114,101,97,116,101,95,109,111,
     100,117,108,101,99,2,0,0,0,0,0,0,0,3,0,0,
@@ -1069,7 +1069,7 @@ const unsigned char _Py_M__importlib_external[] = {
     4,101,120,101,99,114,113,0,0,0,41,3,114,102,0,0,
     0,218,6,109,111,100,117,108,101,114,144,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,11,101,
-    120,101,99,95,109,111,100,117,108,101,205,2,0,0,115,10,
+    120,101,99,95,109,111,100,117,108,101,204,2,0,0,115,10,
     0,0,0,0,2,12,1,8,1,6,1,10,1,122,25,95,
     76,111,97,100,101,114,66,97,115,105,99,115,46,101,120,101,
     99,95,109,111,100,117,108,101,99,2,0,0,0,0,0,0,
@@ -1080,14 +1080,14 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,218,17,95,108,111,97,100,95,109,111,100,117,108,
     101,95,115,104,105,109,41,2,114,102,0,0,0,114,121,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,11,108,111,97,100,95,109,111,100,117,108,101,213,2,
+    0,218,11,108,111,97,100,95,109,111,100,117,108,101,212,2,
     0,0,115,2,0,0,0,0,2,122,25,95,76,111,97,100,
     101,114,66,97,115,105,99,115,46,108,111,97,100,95,109,111,
     100,117,108,101,78,41,8,114,107,0,0,0,114,106,0,0,
     0,114,108,0,0,0,114,109,0,0,0,114,161,0,0,0,
     114,187,0,0,0,114,192,0,0,0,114,194,0,0,0,114,
     2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,185,0,0,0,189,2,0,0,115,8,0,0,
+    0,0,0,114,185,0,0,0,188,2,0,0,115,8,0,0,
     0,12,5,8,8,8,3,8,8,114,185,0,0,0,99,0,
     0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,
     0,0,0,115,74,0,0,0,101,0,90,1,100,0,90,2,
@@ -1112,7 +1112,7 @@ const unsigned char _Py_M__importlib_external[] = {
     46,10,32,32,32,32,32,32,32,32,78,41,1,114,40,0,
     0,0,41,2,114,102,0,0,0,114,35,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,10,112,
-    97,116,104,95,109,116,105,109,101,220,2,0,0,115,2,0,
+    97,116,104,95,109,116,105,109,101,219,2,0,0,115,2,0,
     0,0,0,6,122,23,83,111,117,114,99,101,76,111,97,100,
     101,114,46,112,97,116,104,95,109,116,105,109,101,99,2,0,
     0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,0,
@@ -1147,7 +1147,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,32,32,32,32,32,114,149,0,0,0,41,1,114,196,0,
     0,0,41,2,114,102,0,0,0,114,35,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,10,112,
-    97,116,104,95,115,116,97,116,115,228,2,0,0,115,2,0,
+    97,116,104,95,115,116,97,116,115,227,2,0,0,115,2,0,
     0,0,0,11,122,23,83,111,117,114,99,101,76,111,97,100,
     101,114,46,112,97,116,104,95,115,116,97,116,115,99,4,0,
     0,0,0,0,0,0,4,0,0,0,4,0,0,0,67,0,
@@ -1171,7 +1171,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,90,10,99,97,99,104,101,95,112,97,116,104,114,
     54,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
     0,0,0,218,15,95,99,97,99,104,101,95,98,121,116,101,
-    99,111,100,101,241,2,0,0,115,2,0,0,0,0,8,122,
+    99,111,100,101,240,2,0,0,115,2,0,0,0,0,8,122,
     28,83,111,117,114,99,101,76,111,97,100,101,114,46,95,99,
     97,99,104,101,95,98,121,116,101,99,111,100,101,99,3,0,
     0,0,0,0,0,0,3,0,0,0,1,0,0,0,67,0,
@@ -1188,7 +1188,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,32,32,32,32,78,114,2,0,0,0,41,3,114,102,0,
     0,0,114,35,0,0,0,114,54,0,0,0,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,114,198,0,0,0,
-    251,2,0,0,115,0,0,0,0,122,21,83,111,117,114,99,
+    250,2,0,0,115,0,0,0,0,122,21,83,111,117,114,99,
     101,76,111,97,100,101,114,46,115,101,116,95,100,97,116,97,
     99,2,0,0,0,0,0,0,0,5,0,0,0,10,0,0,
     0,67,0,0,0,115,82,0,0,0,124,0,160,0,124,1,
@@ -1208,7 +1208,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,157,0,0,0,41,5,114,102,0,0,0,114,121,0,0,
     0,114,35,0,0,0,114,155,0,0,0,218,3,101,120,99,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    10,103,101,116,95,115,111,117,114,99,101,2,3,0,0,115,
+    10,103,101,116,95,115,111,117,114,99,101,1,3,0,0,115,
     14,0,0,0,0,2,10,1,2,1,14,1,16,1,4,1,
     28,1,122,23,83,111,117,114,99,101,76,111,97,100,101,114,
     46,103,101,116,95,115,111,117,114,99,101,114,89,0,0,0,
@@ -1230,7 +1230,7 @@ const unsigned char _Py_M__importlib_external[] = {
     105,108,101,41,4,114,102,0,0,0,114,54,0,0,0,114,
     35,0,0,0,114,203,0,0,0,114,2,0,0,0,114,2,
     0,0,0,114,4,0,0,0,218,14,115,111,117,114,99,101,
-    95,116,111,95,99,111,100,101,12,3,0,0,115,4,0,0,
+    95,116,111,95,99,111,100,101,11,3,0,0,115,4,0,0,
     0,0,5,12,1,122,27,83,111,117,114,99,101,76,111,97,
     100,101,114,46,115,111,117,114,99,101,95,116,111,95,99,111,
     100,101,99,2,0,0,0,0,0,0,0,15,0,0,0,9,
@@ -1309,7 +1309,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,132,0,0,0,114,69,0,0,0,90,10,98,121,
     116,101,115,95,100,97,116,97,90,11,99,111,100,101,95,111,
     98,106,101,99,116,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,188,0,0,0,20,3,0,0,115,134,0,
+    4,0,0,0,114,188,0,0,0,19,3,0,0,115,134,0,
     0,0,0,7,10,1,4,1,4,1,4,1,4,1,4,1,
     2,1,12,1,14,1,12,2,2,1,14,1,14,1,8,2,
     12,1,2,1,14,1,14,1,6,3,2,1,8,2,2,1,
@@ -1324,1229 +1324,1288 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,196,0,0,0,114,197,0,0,0,114,199,0,0,0,
     114,198,0,0,0,114,202,0,0,0,114,206,0,0,0,114,
     188,0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,195,0,0,0,218,2,0,
+    0,0,0,114,4,0,0,0,114,195,0,0,0,217,2,0,
     0,115,14,0,0,0,8,2,8,8,8,13,8,10,8,7,
     8,10,14,8,114,195,0,0,0,99,0,0,0,0,0,0,
-    0,0,0,0,0,0,4,0,0,0,0,0,0,0,115,80,
+    0,0,0,0,0,0,4,0,0,0,0,0,0,0,115,124,
     0,0,0,101,0,90,1,100,0,90,2,100,1,90,3,100,
     2,100,3,132,0,90,4,100,4,100,5,132,0,90,5,100,
     6,100,7,132,0,90,6,101,7,135,0,102,1,100,8,100,
     9,132,8,131,1,90,8,101,7,100,10,100,11,132,0,131,
-    1,90,9,100,12,100,13,132,0,90,10,135,0,4,0,90,
-    11,83,0,41,14,218,10,70,105,108,101,76,111,97,100,101,
-    114,122,103,66,97,115,101,32,102,105,108,101,32,108,111,97,
-    100,101,114,32,99,108,97,115,115,32,119,104,105,99,104,32,
-    105,109,112,108,101,109,101,110,116,115,32,116,104,101,32,108,
-    111,97,100,101,114,32,112,114,111,116,111,99,111,108,32,109,
-    101,116,104,111,100,115,32,116,104,97,116,10,32,32,32,32,
-    114,101,113,117,105,114,101,32,102,105,108,101,32,115,121,115,
-    116,101,109,32,117,115,97,103,101,46,99,3,0,0,0,0,
-    0,0,0,3,0,0,0,2,0,0,0,67,0,0,0,115,
-    16,0,0,0,124,1,124,0,95,0,124,2,124,0,95,1,
-    100,1,83,0,41,2,122,75,67,97,99,104,101,32,116,104,
-    101,32,109,111,100,117,108,101,32,110,97,109,101,32,97,110,
-    100,32,116,104,101,32,112,97,116,104,32,116,111,32,116,104,
-    101,32,102,105,108,101,32,102,111,117,110,100,32,98,121,32,
-    116,104,101,10,32,32,32,32,32,32,32,32,102,105,110,100,
-    101,114,46,78,41,2,114,100,0,0,0,114,35,0,0,0,
-    41,3,114,102,0,0,0,114,121,0,0,0,114,35,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,186,0,0,0,111,3,0,0,115,4,0,0,0,0,3,
-    6,1,122,19,70,105,108,101,76,111,97,100,101,114,46,95,
-    95,105,110,105,116,95,95,99,2,0,0,0,0,0,0,0,
-    2,0,0,0,2,0,0,0,67,0,0,0,115,24,0,0,
-    0,124,0,106,0,124,1,106,0,107,2,111,22,124,0,106,
-    1,124,1,106,1,107,2,83,0,41,1,78,41,2,218,9,
-    95,95,99,108,97,115,115,95,95,114,113,0,0,0,41,2,
-    114,102,0,0,0,218,5,111,116,104,101,114,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,218,6,95,95,101,
-    113,95,95,117,3,0,0,115,4,0,0,0,0,1,12,1,
-    122,17,70,105,108,101,76,111,97,100,101,114,46,95,95,101,
-    113,95,95,99,1,0,0,0,0,0,0,0,1,0,0,0,
-    3,0,0,0,67,0,0,0,115,20,0,0,0,116,0,124,
-    0,106,1,131,1,116,0,124,0,106,2,131,1,65,0,83,
-    0,41,1,78,41,3,218,4,104,97,115,104,114,100,0,0,
-    0,114,35,0,0,0,41,1,114,102,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,218,8,95,95,
-    104,97,115,104,95,95,121,3,0,0,115,2,0,0,0,0,
-    1,122,19,70,105,108,101,76,111,97,100,101,114,46,95,95,
-    104,97,115,104,95,95,99,2,0,0,0,0,0,0,0,2,
-    0,0,0,3,0,0,0,3,0,0,0,115,16,0,0,0,
-    116,0,116,1,124,0,131,2,160,2,124,1,161,1,83,0,
-    41,1,122,100,76,111,97,100,32,97,32,109,111,100,117,108,
-    101,32,102,114,111,109,32,97,32,102,105,108,101,46,10,10,
-    32,32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,
-    104,111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,
-    100,46,32,32,85,115,101,32,101,120,101,99,95,109,111,100,
-    117,108,101,40,41,32,105,110,115,116,101,97,100,46,10,10,
-    32,32,32,32,32,32,32,32,41,3,218,5,115,117,112,101,
-    114,114,212,0,0,0,114,194,0,0,0,41,2,114,102,0,
-    0,0,114,121,0,0,0,41,1,114,213,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,194,0,0,0,124,3,0,
-    0,115,2,0,0,0,0,10,122,22,70,105,108,101,76,111,
-    97,100,101,114,46,108,111,97,100,95,109,111,100,117,108,101,
-    99,2,0,0,0,0,0,0,0,2,0,0,0,1,0,0,
-    0,67,0,0,0,115,6,0,0,0,124,0,106,0,83,0,
-    41,1,122,58,82,101,116,117,114,110,32,116,104,101,32,112,
-    97,116,104,32,116,111,32,116,104,101,32,115,111,117,114,99,
-    101,32,102,105,108,101,32,97,115,32,102,111,117,110,100,32,
-    98,121,32,116,104,101,32,102,105,110,100,101,114,46,41,1,
-    114,35,0,0,0,41,2,114,102,0,0,0,114,121,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,159,0,0,0,136,3,0,0,115,2,0,0,0,0,3,
-    122,23,70,105,108,101,76,111,97,100,101,114,46,103,101,116,
-    95,102,105,108,101,110,97,109,101,99,2,0,0,0,0,0,
-    0,0,3,0,0,0,9,0,0,0,67,0,0,0,115,32,
-    0,0,0,116,0,160,1,124,1,100,1,161,2,143,10,125,
-    2,124,2,160,2,161,0,83,0,81,0,82,0,88,0,100,
-    2,83,0,41,3,122,39,82,101,116,117,114,110,32,116,104,
-    101,32,100,97,116,97,32,102,114,111,109,32,112,97,116,104,
-    32,97,115,32,114,97,119,32,98,121,116,101,115,46,218,1,
-    114,78,41,3,114,50,0,0,0,114,51,0,0,0,90,4,
-    114,101,97,100,41,3,114,102,0,0,0,114,35,0,0,0,
-    114,55,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,200,0,0,0,141,3,0,0,115,4,0,
-    0,0,0,2,14,1,122,19,70,105,108,101,76,111,97,100,
-    101,114,46,103,101,116,95,100,97,116,97,41,12,114,107,0,
-    0,0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,
-    0,114,186,0,0,0,114,215,0,0,0,114,217,0,0,0,
-    114,118,0,0,0,114,194,0,0,0,114,159,0,0,0,114,
-    200,0,0,0,90,13,95,95,99,108,97,115,115,99,101,108,
-    108,95,95,114,2,0,0,0,114,2,0,0,0,41,1,114,
-    213,0,0,0,114,4,0,0,0,114,212,0,0,0,106,3,
-    0,0,115,12,0,0,0,12,5,8,6,8,4,8,3,16,
-    12,12,5,114,212,0,0,0,99,0,0,0,0,0,0,0,
-    0,0,0,0,0,3,0,0,0,64,0,0,0,115,46,0,
-    0,0,101,0,90,1,100,0,90,2,100,1,90,3,100,2,
-    100,3,132,0,90,4,100,4,100,5,132,0,90,5,100,6,
-    100,7,156,1,100,8,100,9,132,2,90,6,100,10,83,0,
-    41,11,218,16,83,111,117,114,99,101,70,105,108,101,76,111,
-    97,100,101,114,122,62,67,111,110,99,114,101,116,101,32,105,
-    109,112,108,101,109,101,110,116,97,116,105,111,110,32,111,102,
-    32,83,111,117,114,99,101,76,111,97,100,101,114,32,117,115,
-    105,110,103,32,116,104,101,32,102,105,108,101,32,115,121,115,
-    116,101,109,46,99,2,0,0,0,0,0,0,0,3,0,0,
-    0,3,0,0,0,67,0,0,0,115,22,0,0,0,116,0,
-    124,1,131,1,125,2,124,2,106,1,124,2,106,2,100,1,
-    156,2,83,0,41,2,122,33,82,101,116,117,114,110,32,116,
-    104,101,32,109,101,116,97,100,97,116,97,32,102,111,114,32,
-    116,104,101,32,112,97,116,104,46,41,2,114,149,0,0,0,
-    114,207,0,0,0,41,3,114,39,0,0,0,218,8,115,116,
-    95,109,116,105,109,101,90,7,115,116,95,115,105,122,101,41,
-    3,114,102,0,0,0,114,35,0,0,0,114,211,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    197,0,0,0,151,3,0,0,115,4,0,0,0,0,2,8,
-    1,122,27,83,111,117,114,99,101,70,105,108,101,76,111,97,
-    100,101,114,46,112,97,116,104,95,115,116,97,116,115,99,4,
-    0,0,0,0,0,0,0,5,0,0,0,5,0,0,0,67,
-    0,0,0,115,24,0,0,0,116,0,124,1,131,1,125,4,
-    124,0,106,1,124,2,124,3,124,4,100,1,141,3,83,0,
-    41,2,78,41,1,218,5,95,109,111,100,101,41,2,114,99,
-    0,0,0,114,198,0,0,0,41,5,114,102,0,0,0,114,
-    92,0,0,0,114,91,0,0,0,114,54,0,0,0,114,42,
-    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,199,0,0,0,156,3,0,0,115,4,0,0,0,
-    0,2,8,1,122,32,83,111,117,114,99,101,70,105,108,101,
-    76,111,97,100,101,114,46,95,99,97,99,104,101,95,98,121,
-    116,101,99,111,100,101,105,182,1,0,0,41,1,114,222,0,
-    0,0,99,3,0,0,0,1,0,0,0,9,0,0,0,11,
-    0,0,0,67,0,0,0,115,250,0,0,0,116,0,124,1,
-    131,1,92,2,125,4,125,5,103,0,125,6,120,38,124,4,
-    114,54,116,1,124,4,131,1,115,54,116,0,124,4,131,1,
-    92,2,125,4,125,7,124,6,160,2,124,7,161,1,1,0,
-    113,18,87,0,120,110,116,3,124,6,131,1,68,0,93,98,
-    125,7,116,4,124,4,124,7,131,2,125,4,121,14,116,5,
-    160,6,124,4,161,1,1,0,87,0,113,66,4,0,116,7,
-    107,10,114,116,1,0,1,0,1,0,119,66,89,0,113,66,
-    4,0,116,8,107,10,114,162,1,0,125,8,1,0,122,18,
-    116,9,160,10,100,1,124,4,124,8,161,3,1,0,100,2,
-    83,0,100,2,125,8,126,8,88,0,89,0,113,66,88,0,
-    113,66,87,0,121,28,116,11,124,1,124,2,124,3,131,3,
-    1,0,116,9,160,10,100,3,124,1,161,2,1,0,87,0,
-    110,48,4,0,116,8,107,10,114,244,1,0,125,8,1,0,
-    122,18,116,9,160,10,100,1,124,1,124,8,161,3,1,0,
-    87,0,100,2,100,2,125,8,126,8,88,0,89,0,110,2,
-    88,0,100,2,83,0,41,4,122,27,87,114,105,116,101,32,
-    98,121,116,101,115,32,100,97,116,97,32,116,111,32,97,32,
-    102,105,108,101,46,122,27,99,111,117,108,100,32,110,111,116,
-    32,99,114,101,97,116,101,32,123,33,114,125,58,32,123,33,
-    114,125,78,122,12,99,114,101,97,116,101,100,32,123,33,114,
-    125,41,12,114,38,0,0,0,114,46,0,0,0,114,165,0,
-    0,0,114,33,0,0,0,114,28,0,0,0,114,1,0,0,
-    0,90,5,109,107,100,105,114,218,15,70,105,108,101,69,120,
-    105,115,116,115,69,114,114,111,114,114,40,0,0,0,114,116,
-    0,0,0,114,130,0,0,0,114,56,0,0,0,41,9,114,
-    102,0,0,0,114,35,0,0,0,114,54,0,0,0,114,222,
-    0,0,0,218,6,112,97,114,101,110,116,114,96,0,0,0,
-    114,27,0,0,0,114,23,0,0,0,114,201,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,198,
-    0,0,0,161,3,0,0,115,42,0,0,0,0,2,12,1,
-    4,2,14,1,12,1,14,2,14,1,10,1,2,1,14,1,
-    14,2,6,1,16,3,6,1,8,1,22,1,2,1,12,1,
-    16,1,16,2,8,1,122,25,83,111,117,114,99,101,70,105,
-    108,101,76,111,97,100,101,114,46,115,101,116,95,100,97,116,
-    97,78,41,7,114,107,0,0,0,114,106,0,0,0,114,108,
-    0,0,0,114,109,0,0,0,114,197,0,0,0,114,199,0,
-    0,0,114,198,0,0,0,114,2,0,0,0,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,114,220,0,0,0,
-    147,3,0,0,115,6,0,0,0,12,4,8,5,8,5,114,
-    220,0,0,0,99,0,0,0,0,0,0,0,0,0,0,0,
-    0,2,0,0,0,64,0,0,0,115,32,0,0,0,101,0,
-    90,1,100,0,90,2,100,1,90,3,100,2,100,3,132,0,
-    90,4,100,4,100,5,132,0,90,5,100,6,83,0,41,7,
-    218,20,83,111,117,114,99,101,108,101,115,115,70,105,108,101,
-    76,111,97,100,101,114,122,45,76,111,97,100,101,114,32,119,
-    104,105,99,104,32,104,97,110,100,108,101,115,32,115,111,117,
-    114,99,101,108,101,115,115,32,102,105,108,101,32,105,109,112,
-    111,114,116,115,46,99,2,0,0,0,0,0,0,0,5,0,
-    0,0,5,0,0,0,67,0,0,0,115,68,0,0,0,124,
-    0,160,0,124,1,161,1,125,2,124,0,160,1,124,2,161,
-    1,125,3,124,1,124,2,100,1,156,2,125,4,116,2,124,
-    3,124,1,124,4,131,3,1,0,116,3,116,4,124,3,131,
-    1,100,2,100,0,133,2,25,0,124,1,124,2,100,3,141,
-    3,83,0,41,4,78,41,2,114,100,0,0,0,114,35,0,
-    0,0,114,126,0,0,0,41,2,114,100,0,0,0,114,91,
-    0,0,0,41,5,114,159,0,0,0,114,200,0,0,0,114,
-    133,0,0,0,114,145,0,0,0,114,208,0,0,0,41,5,
-    114,102,0,0,0,114,121,0,0,0,114,35,0,0,0,114,
-    54,0,0,0,114,132,0,0,0,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,188,0,0,0,196,3,0,
-    0,115,18,0,0,0,0,1,10,1,10,4,2,1,8,2,
-    12,1,2,1,14,1,2,1,122,29,83,111,117,114,99,101,
-    108,101,115,115,70,105,108,101,76,111,97,100,101,114,46,103,
-    101,116,95,99,111,100,101,99,2,0,0,0,0,0,0,0,
-    2,0,0,0,1,0,0,0,67,0,0,0,115,4,0,0,
-    0,100,1,83,0,41,2,122,39,82,101,116,117,114,110,32,
-    78,111,110,101,32,97,115,32,116,104,101,114,101,32,105,115,
-    32,110,111,32,115,111,117,114,99,101,32,99,111,100,101,46,
-    78,114,2,0,0,0,41,2,114,102,0,0,0,114,121,0,
-    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,202,0,0,0,212,3,0,0,115,2,0,0,0,0,
-    2,122,31,83,111,117,114,99,101,108,101,115,115,70,105,108,
-    101,76,111,97,100,101,114,46,103,101,116,95,115,111,117,114,
-    99,101,78,41,6,114,107,0,0,0,114,106,0,0,0,114,
-    108,0,0,0,114,109,0,0,0,114,188,0,0,0,114,202,
-    0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,114,225,0,0,0,192,3,0,0,
-    115,4,0,0,0,12,4,8,16,114,225,0,0,0,99,0,
-    0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,
-    0,0,0,115,92,0,0,0,101,0,90,1,100,0,90,2,
-    100,1,90,3,100,2,100,3,132,0,90,4,100,4,100,5,
-    132,0,90,5,100,6,100,7,132,0,90,6,100,8,100,9,
-    132,0,90,7,100,10,100,11,132,0,90,8,100,12,100,13,
-    132,0,90,9,100,14,100,15,132,0,90,10,100,16,100,17,
-    132,0,90,11,101,12,100,18,100,19,132,0,131,1,90,13,
-    100,20,83,0,41,21,218,19,69,120,116,101,110,115,105,111,
-    110,70,105,108,101,76,111,97,100,101,114,122,93,76,111,97,
-    100,101,114,32,102,111,114,32,101,120,116,101,110,115,105,111,
-    110,32,109,111,100,117,108,101,115,46,10,10,32,32,32,32,
-    84,104,101,32,99,111,110,115,116,114,117,99,116,111,114,32,
-    105,115,32,100,101,115,105,103,110,101,100,32,116,111,32,119,
-    111,114,107,32,119,105,116,104,32,70,105,108,101,70,105,110,
-    100,101,114,46,10,10,32,32,32,32,99,3,0,0,0,0,
-    0,0,0,3,0,0,0,2,0,0,0,67,0,0,0,115,
-    16,0,0,0,124,1,124,0,95,0,124,2,124,0,95,1,
-    100,0,83,0,41,1,78,41,2,114,100,0,0,0,114,35,
-    0,0,0,41,3,114,102,0,0,0,114,100,0,0,0,114,
-    35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,186,0,0,0,229,3,0,0,115,4,0,0,
-    0,0,1,6,1,122,28,69,120,116,101,110,115,105,111,110,
+    1,90,9,100,12,100,13,132,0,90,10,101,7,100,14,100,
+    15,132,0,131,1,90,11,100,16,100,17,132,0,90,12,100,
+    18,100,19,132,0,90,13,100,20,100,21,132,0,90,14,100,
+    22,100,23,132,0,90,15,135,0,4,0,90,16,83,0,41,
+    24,218,10,70,105,108,101,76,111,97,100,101,114,122,103,66,
+    97,115,101,32,102,105,108,101,32,108,111,97,100,101,114,32,
+    99,108,97,115,115,32,119,104,105,99,104,32,105,109,112,108,
+    101,109,101,110,116,115,32,116,104,101,32,108,111,97,100,101,
+    114,32,112,114,111,116,111,99,111,108,32,109,101,116,104,111,
+    100,115,32,116,104,97,116,10,32,32,32,32,114,101,113,117,
+    105,114,101,32,102,105,108,101,32,115,121,115,116,101,109,32,
+    117,115,97,103,101,46,99,3,0,0,0,0,0,0,0,3,
+    0,0,0,2,0,0,0,67,0,0,0,115,16,0,0,0,
+    124,1,124,0,95,0,124,2,124,0,95,1,100,1,83,0,
+    41,2,122,75,67,97,99,104,101,32,116,104,101,32,109,111,
+    100,117,108,101,32,110,97,109,101,32,97,110,100,32,116,104,
+    101,32,112,97,116,104,32,116,111,32,116,104,101,32,102,105,
+    108,101,32,102,111,117,110,100,32,98,121,32,116,104,101,10,
+    32,32,32,32,32,32,32,32,102,105,110,100,101,114,46,78,
+    41,2,114,100,0,0,0,114,35,0,0,0,41,3,114,102,
+    0,0,0,114,121,0,0,0,114,35,0,0,0,114,2,0,
+    0,0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,
+    0,110,3,0,0,115,4,0,0,0,0,3,6,1,122,19,
     70,105,108,101,76,111,97,100,101,114,46,95,95,105,110,105,
     116,95,95,99,2,0,0,0,0,0,0,0,2,0,0,0,
     2,0,0,0,67,0,0,0,115,24,0,0,0,124,0,106,
     0,124,1,106,0,107,2,111,22,124,0,106,1,124,1,106,
-    1,107,2,83,0,41,1,78,41,2,114,213,0,0,0,114,
-    113,0,0,0,41,2,114,102,0,0,0,114,214,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    215,0,0,0,233,3,0,0,115,4,0,0,0,0,1,12,
-    1,122,26,69,120,116,101,110,115,105,111,110,70,105,108,101,
-    76,111,97,100,101,114,46,95,95,101,113,95,95,99,1,0,
-    0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,0,
-    0,0,115,20,0,0,0,116,0,124,0,106,1,131,1,116,
-    0,124,0,106,2,131,1,65,0,83,0,41,1,78,41,3,
-    114,216,0,0,0,114,100,0,0,0,114,35,0,0,0,41,
-    1,114,102,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,217,0,0,0,237,3,0,0,115,2,
-    0,0,0,0,1,122,28,69,120,116,101,110,115,105,111,110,
-    70,105,108,101,76,111,97,100,101,114,46,95,95,104,97,115,
-    104,95,95,99,2,0,0,0,0,0,0,0,3,0,0,0,
-    5,0,0,0,67,0,0,0,115,36,0,0,0,116,0,160,
-    1,116,2,106,3,124,1,161,2,125,2,116,0,160,4,100,
-    1,124,1,106,5,124,0,106,6,161,3,1,0,124,2,83,
-    0,41,2,122,38,67,114,101,97,116,101,32,97,110,32,117,
-    110,105,116,105,97,108,105,122,101,100,32,101,120,116,101,110,
-    115,105,111,110,32,109,111,100,117,108,101,122,38,101,120,116,
-    101,110,115,105,111,110,32,109,111,100,117,108,101,32,123,33,
-    114,125,32,108,111,97,100,101,100,32,102,114,111,109,32,123,
-    33,114,125,41,7,114,116,0,0,0,114,189,0,0,0,114,
-    143,0,0,0,90,14,99,114,101,97,116,101,95,100,121,110,
-    97,109,105,99,114,130,0,0,0,114,100,0,0,0,114,35,
-    0,0,0,41,3,114,102,0,0,0,114,166,0,0,0,114,
-    191,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,187,0,0,0,240,3,0,0,115,10,0,0,
-    0,0,2,4,1,10,1,6,1,12,1,122,33,69,120,116,
-    101,110,115,105,111,110,70,105,108,101,76,111,97,100,101,114,
-    46,99,114,101,97,116,101,95,109,111,100,117,108,101,99,2,
-    0,0,0,0,0,0,0,2,0,0,0,5,0,0,0,67,
-    0,0,0,115,36,0,0,0,116,0,160,1,116,2,106,3,
-    124,1,161,2,1,0,116,0,160,4,100,1,124,0,106,5,
-    124,0,106,6,161,3,1,0,100,2,83,0,41,3,122,30,
-    73,110,105,116,105,97,108,105,122,101,32,97,110,32,101,120,
-    116,101,110,115,105,111,110,32,109,111,100,117,108,101,122,40,
-    101,120,116,101,110,115,105,111,110,32,109,111,100,117,108,101,
-    32,123,33,114,125,32,101,120,101,99,117,116,101,100,32,102,
-    114,111,109,32,123,33,114,125,78,41,7,114,116,0,0,0,
-    114,189,0,0,0,114,143,0,0,0,90,12,101,120,101,99,
-    95,100,121,110,97,109,105,99,114,130,0,0,0,114,100,0,
-    0,0,114,35,0,0,0,41,2,114,102,0,0,0,114,191,
+    1,107,2,83,0,41,1,78,41,2,218,9,95,95,99,108,
+    97,115,115,95,95,114,113,0,0,0,41,2,114,102,0,0,
+    0,218,5,111,116,104,101,114,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,218,6,95,95,101,113,95,95,116,
+    3,0,0,115,4,0,0,0,0,1,12,1,122,17,70,105,
+    108,101,76,111,97,100,101,114,46,95,95,101,113,95,95,99,
+    1,0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,
+    67,0,0,0,115,20,0,0,0,116,0,124,0,106,1,131,
+    1,116,0,124,0,106,2,131,1,65,0,83,0,41,1,78,
+    41,3,218,4,104,97,115,104,114,100,0,0,0,114,35,0,
+    0,0,41,1,114,102,0,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,218,8,95,95,104,97,115,104,
+    95,95,120,3,0,0,115,2,0,0,0,0,1,122,19,70,
+    105,108,101,76,111,97,100,101,114,46,95,95,104,97,115,104,
+    95,95,99,2,0,0,0,0,0,0,0,2,0,0,0,3,
+    0,0,0,3,0,0,0,115,16,0,0,0,116,0,116,1,
+    124,0,131,2,160,2,124,1,161,1,83,0,41,1,122,100,
+    76,111,97,100,32,97,32,109,111,100,117,108,101,32,102,114,
+    111,109,32,97,32,102,105,108,101,46,10,10,32,32,32,32,
+    32,32,32,32,84,104,105,115,32,109,101,116,104,111,100,32,
+    105,115,32,100,101,112,114,101,99,97,116,101,100,46,32,32,
+    85,115,101,32,101,120,101,99,95,109,111,100,117,108,101,40,
+    41,32,105,110,115,116,101,97,100,46,10,10,32,32,32,32,
+    32,32,32,32,41,3,218,5,115,117,112,101,114,114,212,0,
+    0,0,114,194,0,0,0,41,2,114,102,0,0,0,114,121,
+    0,0,0,41,1,114,213,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,114,194,0,0,0,123,3,0,0,115,2,0,
+    0,0,0,10,122,22,70,105,108,101,76,111,97,100,101,114,
+    46,108,111,97,100,95,109,111,100,117,108,101,99,2,0,0,
+    0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,
+    0,115,6,0,0,0,124,0,106,0,83,0,41,1,122,58,
+    82,101,116,117,114,110,32,116,104,101,32,112,97,116,104,32,
+    116,111,32,116,104,101,32,115,111,117,114,99,101,32,102,105,
+    108,101,32,97,115,32,102,111,117,110,100,32,98,121,32,116,
+    104,101,32,102,105,110,100,101,114,46,41,1,114,35,0,0,
+    0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,0,
+    0,0,114,2,0,0,0,114,4,0,0,0,114,159,0,0,
+    0,135,3,0,0,115,2,0,0,0,0,3,122,23,70,105,
+    108,101,76,111,97,100,101,114,46,103,101,116,95,102,105,108,
+    101,110,97,109,101,99,2,0,0,0,0,0,0,0,3,0,
+    0,0,9,0,0,0,67,0,0,0,115,32,0,0,0,116,
+    0,160,1,124,1,100,1,161,2,143,10,125,2,124,2,160,
+    2,161,0,83,0,81,0,82,0,88,0,100,2,83,0,41,
+    3,122,39,82,101,116,117,114,110,32,116,104,101,32,100,97,
+    116,97,32,102,114,111,109,32,112,97,116,104,32,97,115,32,
+    114,97,119,32,98,121,116,101,115,46,218,1,114,78,41,3,
+    114,50,0,0,0,114,51,0,0,0,90,4,114,101,97,100,
+    41,3,114,102,0,0,0,114,35,0,0,0,114,55,0,0,
+    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,200,0,0,0,140,3,0,0,115,4,0,0,0,0,2,
+    14,1,122,19,70,105,108,101,76,111,97,100,101,114,46,103,
+    101,116,95,100,97,116,97,99,2,0,0,0,0,0,0,0,
+    2,0,0,0,3,0,0,0,67,0,0,0,115,18,0,0,
+    0,124,0,160,0,124,1,161,1,114,14,124,0,83,0,100,
+    0,83,0,41,1,78,41,1,114,161,0,0,0,41,2,114,
+    102,0,0,0,114,191,0,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,218,19,103,101,116,95,114,101,
+    115,111,117,114,99,101,95,114,101,97,100,101,114,147,3,0,
+    0,115,6,0,0,0,0,2,10,1,4,1,122,30,70,105,
+    108,101,76,111,97,100,101,114,46,103,101,116,95,114,101,115,
+    111,117,114,99,101,95,114,101,97,100,101,114,99,2,0,0,
+    0,0,0,0,0,3,0,0,0,4,0,0,0,67,0,0,
+    0,115,32,0,0,0,116,0,116,1,124,0,106,2,131,1,
+    100,1,25,0,124,1,131,2,125,2,116,3,160,4,124,2,
+    100,2,161,2,83,0,41,3,78,114,60,0,0,0,114,219,
+    0,0,0,41,5,114,28,0,0,0,114,38,0,0,0,114,
+    35,0,0,0,114,50,0,0,0,114,51,0,0,0,41,3,
+    114,102,0,0,0,218,8,114,101,115,111,117,114,99,101,114,
+    35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,218,13,111,112,101,110,95,114,101,115,111,117,114,
+    99,101,153,3,0,0,115,4,0,0,0,0,1,20,1,122,
+    24,70,105,108,101,76,111,97,100,101,114,46,111,112,101,110,
+    95,114,101,115,111,117,114,99,101,99,2,0,0,0,0,0,
+    0,0,3,0,0,0,3,0,0,0,67,0,0,0,115,38,
+    0,0,0,124,0,160,0,124,1,161,1,115,14,116,1,130,
+    1,116,2,116,3,124,0,106,4,131,1,100,1,25,0,124,
+    1,131,2,125,2,124,2,83,0,41,2,78,114,60,0,0,
+    0,41,5,218,11,105,115,95,114,101,115,111,117,114,99,101,
+    218,17,70,105,108,101,78,111,116,70,111,117,110,100,69,114,
+    114,111,114,114,28,0,0,0,114,38,0,0,0,114,35,0,
+    0,0,41,3,114,102,0,0,0,114,221,0,0,0,114,35,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,192,0,0,0,248,3,0,0,115,6,0,0,0,
-    0,2,14,1,6,1,122,31,69,120,116,101,110,115,105,111,
-    110,70,105,108,101,76,111,97,100,101,114,46,101,120,101,99,
-    95,109,111,100,117,108,101,99,2,0,0,0,0,0,0,0,
-    2,0,0,0,4,0,0,0,3,0,0,0,115,36,0,0,
-    0,116,0,124,0,106,1,131,1,100,1,25,0,137,0,116,
-    2,135,0,102,1,100,2,100,3,132,8,116,3,68,0,131,
-    1,131,1,83,0,41,4,122,49,82,101,116,117,114,110,32,
-    84,114,117,101,32,105,102,32,116,104,101,32,101,120,116,101,
-    110,115,105,111,110,32,109,111,100,117,108,101,32,105,115,32,
-    97,32,112,97,99,107,97,103,101,46,114,29,0,0,0,99,
-    1,0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,
-    51,0,0,0,115,26,0,0,0,124,0,93,18,125,1,136,
-    0,100,0,124,1,23,0,107,2,86,0,1,0,113,2,100,
-    1,83,0,41,2,114,186,0,0,0,78,114,2,0,0,0,
-    41,2,114,22,0,0,0,218,6,115,117,102,102,105,120,41,
-    1,218,9,102,105,108,101,95,110,97,109,101,114,2,0,0,
-    0,114,4,0,0,0,250,9,60,103,101,110,101,120,112,114,
-    62,1,4,0,0,115,2,0,0,0,4,1,122,49,69,120,
-    116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,101,
-    114,46,105,115,95,112,97,99,107,97,103,101,46,60,108,111,
-    99,97,108,115,62,46,60,103,101,110,101,120,112,114,62,41,
-    4,114,38,0,0,0,114,35,0,0,0,218,3,97,110,121,
-    218,18,69,88,84,69,78,83,73,79,78,95,83,85,70,70,
-    73,88,69,83,41,2,114,102,0,0,0,114,121,0,0,0,
-    114,2,0,0,0,41,1,114,228,0,0,0,114,4,0,0,
-    0,114,161,0,0,0,254,3,0,0,115,6,0,0,0,0,
-    2,14,1,12,1,122,30,69,120,116,101,110,115,105,111,110,
-    70,105,108,101,76,111,97,100,101,114,46,105,115,95,112,97,
-    99,107,97,103,101,99,2,0,0,0,0,0,0,0,2,0,
-    0,0,1,0,0,0,67,0,0,0,115,4,0,0,0,100,
-    1,83,0,41,2,122,63,82,101,116,117,114,110,32,78,111,
-    110,101,32,97,115,32,97,110,32,101,120,116,101,110,115,105,
-    111,110,32,109,111,100,117,108,101,32,99,97,110,110,111,116,
-    32,99,114,101,97,116,101,32,97,32,99,111,100,101,32,111,
-    98,106,101,99,116,46,78,114,2,0,0,0,41,2,114,102,
-    0,0,0,114,121,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,114,188,0,0,0,4,4,0,0,
-    115,2,0,0,0,0,2,122,28,69,120,116,101,110,115,105,
-    111,110,70,105,108,101,76,111,97,100,101,114,46,103,101,116,
-    95,99,111,100,101,99,2,0,0,0,0,0,0,0,2,0,
-    0,0,1,0,0,0,67,0,0,0,115,4,0,0,0,100,
-    1,83,0,41,2,122,53,82,101,116,117,114,110,32,78,111,
-    110,101,32,97,115,32,101,120,116,101,110,115,105,111,110,32,
-    109,111,100,117,108,101,115,32,104,97,118,101,32,110,111,32,
-    115,111,117,114,99,101,32,99,111,100,101,46,78,114,2,0,
-    0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,202,0,
-    0,0,8,4,0,0,115,2,0,0,0,0,2,122,30,69,
-    120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,
-    101,114,46,103,101,116,95,115,111,117,114,99,101,99,2,0,
-    0,0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,
-    0,0,115,6,0,0,0,124,0,106,0,83,0,41,1,122,
-    58,82,101,116,117,114,110,32,116,104,101,32,112,97,116,104,
-    32,116,111,32,116,104,101,32,115,111,117,114,99,101,32,102,
-    105,108,101,32,97,115,32,102,111,117,110,100,32,98,121,32,
-    116,104,101,32,102,105,110,100,101,114,46,41,1,114,35,0,
-    0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,159,0,
-    0,0,12,4,0,0,115,2,0,0,0,0,3,122,32,69,
-    120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,
-    101,114,46,103,101,116,95,102,105,108,101,110,97,109,101,78,
-    41,14,114,107,0,0,0,114,106,0,0,0,114,108,0,0,
-    0,114,109,0,0,0,114,186,0,0,0,114,215,0,0,0,
-    114,217,0,0,0,114,187,0,0,0,114,192,0,0,0,114,
-    161,0,0,0,114,188,0,0,0,114,202,0,0,0,114,118,
-    0,0,0,114,159,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,114,226,0,0,
-    0,221,3,0,0,115,18,0,0,0,12,8,8,4,8,4,
-    8,3,8,8,8,6,8,6,8,4,8,4,114,226,0,0,
-    0,99,0,0,0,0,0,0,0,0,0,0,0,0,2,0,
-    0,0,64,0,0,0,115,96,0,0,0,101,0,90,1,100,
-    0,90,2,100,1,90,3,100,2,100,3,132,0,90,4,100,
-    4,100,5,132,0,90,5,100,6,100,7,132,0,90,6,100,
-    8,100,9,132,0,90,7,100,10,100,11,132,0,90,8,100,
-    12,100,13,132,0,90,9,100,14,100,15,132,0,90,10,100,
-    16,100,17,132,0,90,11,100,18,100,19,132,0,90,12,100,
-    20,100,21,132,0,90,13,100,22,83,0,41,23,218,14,95,
-    78,97,109,101,115,112,97,99,101,80,97,116,104,97,38,1,
-    0,0,82,101,112,114,101,115,101,110,116,115,32,97,32,110,
-    97,109,101,115,112,97,99,101,32,112,97,99,107,97,103,101,
-    39,115,32,112,97,116,104,46,32,32,73,116,32,117,115,101,
-    115,32,116,104,101,32,109,111,100,117,108,101,32,110,97,109,
-    101,10,32,32,32,32,116,111,32,102,105,110,100,32,105,116,
-    115,32,112,97,114,101,110,116,32,109,111,100,117,108,101,44,
-    32,97,110,100,32,102,114,111,109,32,116,104,101,114,101,32,
-    105,116,32,108,111,111,107,115,32,117,112,32,116,104,101,32,
-    112,97,114,101,110,116,39,115,10,32,32,32,32,95,95,112,
-    97,116,104,95,95,46,32,32,87,104,101,110,32,116,104,105,
-    115,32,99,104,97,110,103,101,115,44,32,116,104,101,32,109,
-    111,100,117,108,101,39,115,32,111,119,110,32,112,97,116,104,
-    32,105,115,32,114,101,99,111,109,112,117,116,101,100,44,10,
-    32,32,32,32,117,115,105,110,103,32,112,97,116,104,95,102,
-    105,110,100,101,114,46,32,32,70,111,114,32,116,111,112,45,
-    108,101,118,101,108,32,109,111,100,117,108,101,115,44,32,116,
-    104,101,32,112,97,114,101,110,116,32,109,111,100,117,108,101,
-    39,115,32,112,97,116,104,10,32,32,32,32,105,115,32,115,
-    121,115,46,112,97,116,104,46,99,4,0,0,0,0,0,0,
-    0,4,0,0,0,3,0,0,0,67,0,0,0,115,36,0,
-    0,0,124,1,124,0,95,0,124,2,124,0,95,1,116,2,
-    124,0,160,3,161,0,131,1,124,0,95,4,124,3,124,0,
-    95,5,100,0,83,0,41,1,78,41,6,218,5,95,110,97,
-    109,101,218,5,95,112,97,116,104,114,95,0,0,0,218,16,
-    95,103,101,116,95,112,97,114,101,110,116,95,112,97,116,104,
-    218,17,95,108,97,115,116,95,112,97,114,101,110,116,95,112,
-    97,116,104,218,12,95,112,97,116,104,95,102,105,110,100,101,
-    114,41,4,114,102,0,0,0,114,100,0,0,0,114,35,0,
-    0,0,218,11,112,97,116,104,95,102,105,110,100,101,114,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,186,
-    0,0,0,25,4,0,0,115,8,0,0,0,0,1,6,1,
-    6,1,14,1,122,23,95,78,97,109,101,115,112,97,99,101,
-    80,97,116,104,46,95,95,105,110,105,116,95,95,99,1,0,
-    0,0,0,0,0,0,4,0,0,0,3,0,0,0,67,0,
-    0,0,115,38,0,0,0,124,0,106,0,160,1,100,1,161,
-    1,92,3,125,1,125,2,125,3,124,2,100,2,107,2,114,
-    30,100,3,83,0,124,1,100,4,102,2,83,0,41,5,122,
-    62,82,101,116,117,114,110,115,32,97,32,116,117,112,108,101,
-    32,111,102,32,40,112,97,114,101,110,116,45,109,111,100,117,
-    108,101,45,110,97,109,101,44,32,112,97,114,101,110,116,45,
-    112,97,116,104,45,97,116,116,114,45,110,97,109,101,41,114,
-    59,0,0,0,114,30,0,0,0,41,2,114,6,0,0,0,
-    114,35,0,0,0,90,8,95,95,112,97,116,104,95,95,41,
-    2,114,233,0,0,0,114,32,0,0,0,41,4,114,102,0,
-    0,0,114,224,0,0,0,218,3,100,111,116,90,2,109,101,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    23,95,102,105,110,100,95,112,97,114,101,110,116,95,112,97,
-    116,104,95,110,97,109,101,115,31,4,0,0,115,8,0,0,
-    0,0,2,18,1,8,2,4,3,122,38,95,78,97,109,101,
-    115,112,97,99,101,80,97,116,104,46,95,102,105,110,100,95,
-    112,97,114,101,110,116,95,112,97,116,104,95,110,97,109,101,
-    115,99,1,0,0,0,0,0,0,0,3,0,0,0,3,0,
-    0,0,67,0,0,0,115,28,0,0,0,124,0,160,0,161,
-    0,92,2,125,1,125,2,116,1,116,2,106,3,124,1,25,
-    0,124,2,131,2,83,0,41,1,78,41,4,114,240,0,0,
-    0,114,112,0,0,0,114,6,0,0,0,218,7,109,111,100,
-    117,108,101,115,41,3,114,102,0,0,0,90,18,112,97,114,
-    101,110,116,95,109,111,100,117,108,101,95,110,97,109,101,90,
-    14,112,97,116,104,95,97,116,116,114,95,110,97,109,101,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,235,
-    0,0,0,41,4,0,0,115,4,0,0,0,0,1,12,1,
-    122,31,95,78,97,109,101,115,112,97,99,101,80,97,116,104,
-    46,95,103,101,116,95,112,97,114,101,110,116,95,112,97,116,
-    104,99,1,0,0,0,0,0,0,0,3,0,0,0,4,0,
-    0,0,67,0,0,0,115,80,0,0,0,116,0,124,0,160,
-    1,161,0,131,1,125,1,124,1,124,0,106,2,107,3,114,
-    74,124,0,160,3,124,0,106,4,124,1,161,2,125,2,124,
-    2,100,0,107,9,114,68,124,2,106,5,100,0,107,8,114,
-    68,124,2,106,6,114,68,124,2,106,6,124,0,95,7,124,
-    1,124,0,95,2,124,0,106,7,83,0,41,1,78,41,8,
-    114,95,0,0,0,114,235,0,0,0,114,236,0,0,0,114,
-    237,0,0,0,114,233,0,0,0,114,122,0,0,0,114,158,
-    0,0,0,114,234,0,0,0,41,3,114,102,0,0,0,90,
-    11,112,97,114,101,110,116,95,112,97,116,104,114,166,0,0,
+    0,0,218,13,114,101,115,111,117,114,99,101,95,112,97,116,
+    104,157,3,0,0,115,8,0,0,0,0,1,10,1,4,1,
+    20,1,122,24,70,105,108,101,76,111,97,100,101,114,46,114,
+    101,115,111,117,114,99,101,95,112,97,116,104,99,2,0,0,
+    0,0,0,0,0,3,0,0,0,3,0,0,0,67,0,0,
+    0,115,40,0,0,0,116,0,124,1,107,6,114,12,100,1,
+    83,0,116,1,116,2,124,0,106,3,131,1,100,2,25,0,
+    124,1,131,2,125,2,116,4,124,2,131,1,83,0,41,3,
+    78,70,114,60,0,0,0,41,5,114,25,0,0,0,114,28,
+    0,0,0,114,38,0,0,0,114,35,0,0,0,114,44,0,
+    0,0,41,3,114,102,0,0,0,114,100,0,0,0,114,35,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,114,223,0,0,0,163,3,0,0,115,8,0,0,0,
+    0,1,8,1,4,1,20,1,122,22,70,105,108,101,76,111,
+    97,100,101,114,46,105,115,95,114,101,115,111,117,114,99,101,
+    99,1,0,0,0,0,0,0,0,1,0,0,0,5,0,0,
+    0,67,0,0,0,115,24,0,0,0,116,0,116,1,160,2,
+    116,3,124,0,106,4,131,1,100,1,25,0,161,1,131,1,
+    83,0,41,2,78,114,60,0,0,0,41,5,218,4,105,116,
+    101,114,114,1,0,0,0,218,7,108,105,115,116,100,105,114,
+    114,38,0,0,0,114,35,0,0,0,41,1,114,102,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,12,95,114,101,99,97,108,99,117,108,97,116,101,45,4,
-    0,0,115,16,0,0,0,0,2,12,1,10,1,14,3,18,
-    1,6,1,8,1,6,1,122,27,95,78,97,109,101,115,112,
-    97,99,101,80,97,116,104,46,95,114,101,99,97,108,99,117,
-    108,97,116,101,99,1,0,0,0,0,0,0,0,1,0,0,
-    0,3,0,0,0,67,0,0,0,115,12,0,0,0,116,0,
-    124,0,160,1,161,0,131,1,83,0,41,1,78,41,2,218,
-    4,105,116,101,114,114,242,0,0,0,41,1,114,102,0,0,
+    218,8,99,111,110,116,101,110,116,115,169,3,0,0,115,2,
+    0,0,0,0,1,122,19,70,105,108,101,76,111,97,100,101,
+    114,46,99,111,110,116,101,110,116,115,41,17,114,107,0,0,
+    0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,0,
+    114,186,0,0,0,114,215,0,0,0,114,217,0,0,0,114,
+    118,0,0,0,114,194,0,0,0,114,159,0,0,0,114,200,
+    0,0,0,114,220,0,0,0,114,222,0,0,0,114,225,0,
+    0,0,114,223,0,0,0,114,228,0,0,0,90,13,95,95,
+    99,108,97,115,115,99,101,108,108,95,95,114,2,0,0,0,
+    114,2,0,0,0,41,1,114,213,0,0,0,114,4,0,0,
+    0,114,212,0,0,0,105,3,0,0,115,22,0,0,0,12,
+    5,8,6,8,4,8,3,16,12,12,5,8,7,12,6,8,
+    4,8,6,8,6,114,212,0,0,0,99,0,0,0,0,0,
+    0,0,0,0,0,0,0,3,0,0,0,64,0,0,0,115,
+    46,0,0,0,101,0,90,1,100,0,90,2,100,1,90,3,
+    100,2,100,3,132,0,90,4,100,4,100,5,132,0,90,5,
+    100,6,100,7,156,1,100,8,100,9,132,2,90,6,100,10,
+    83,0,41,11,218,16,83,111,117,114,99,101,70,105,108,101,
+    76,111,97,100,101,114,122,62,67,111,110,99,114,101,116,101,
+    32,105,109,112,108,101,109,101,110,116,97,116,105,111,110,32,
+    111,102,32,83,111,117,114,99,101,76,111,97,100,101,114,32,
+    117,115,105,110,103,32,116,104,101,32,102,105,108,101,32,115,
+    121,115,116,101,109,46,99,2,0,0,0,0,0,0,0,3,
+    0,0,0,3,0,0,0,67,0,0,0,115,22,0,0,0,
+    116,0,124,1,131,1,125,2,124,2,106,1,124,2,106,2,
+    100,1,156,2,83,0,41,2,122,33,82,101,116,117,114,110,
+    32,116,104,101,32,109,101,116,97,100,97,116,97,32,102,111,
+    114,32,116,104,101,32,112,97,116,104,46,41,2,114,149,0,
+    0,0,114,207,0,0,0,41,3,114,39,0,0,0,218,8,
+    115,116,95,109,116,105,109,101,90,7,115,116,95,115,105,122,
+    101,41,3,114,102,0,0,0,114,35,0,0,0,114,211,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,197,0,0,0,177,3,0,0,115,4,0,0,0,0,
+    2,8,1,122,27,83,111,117,114,99,101,70,105,108,101,76,
+    111,97,100,101,114,46,112,97,116,104,95,115,116,97,116,115,
+    99,4,0,0,0,0,0,0,0,5,0,0,0,5,0,0,
+    0,67,0,0,0,115,24,0,0,0,116,0,124,1,131,1,
+    125,4,124,0,106,1,124,2,124,3,124,4,100,1,141,3,
+    83,0,41,2,78,41,1,218,5,95,109,111,100,101,41,2,
+    114,99,0,0,0,114,198,0,0,0,41,5,114,102,0,0,
+    0,114,92,0,0,0,114,91,0,0,0,114,54,0,0,0,
+    114,42,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,114,199,0,0,0,182,3,0,0,115,4,0,
+    0,0,0,2,8,1,122,32,83,111,117,114,99,101,70,105,
+    108,101,76,111,97,100,101,114,46,95,99,97,99,104,101,95,
+    98,121,116,101,99,111,100,101,105,182,1,0,0,41,1,114,
+    231,0,0,0,99,3,0,0,0,1,0,0,0,9,0,0,
+    0,11,0,0,0,67,0,0,0,115,250,0,0,0,116,0,
+    124,1,131,1,92,2,125,4,125,5,103,0,125,6,120,38,
+    124,4,114,54,116,1,124,4,131,1,115,54,116,0,124,4,
+    131,1,92,2,125,4,125,7,124,6,160,2,124,7,161,1,
+    1,0,113,18,87,0,120,110,116,3,124,6,131,1,68,0,
+    93,98,125,7,116,4,124,4,124,7,131,2,125,4,121,14,
+    116,5,160,6,124,4,161,1,1,0,87,0,113,66,4,0,
+    116,7,107,10,114,116,1,0,1,0,1,0,119,66,89,0,
+    113,66,4,0,116,8,107,10,114,162,1,0,125,8,1,0,
+    122,18,116,9,160,10,100,1,124,4,124,8,161,3,1,0,
+    100,2,83,0,100,2,125,8,126,8,88,0,89,0,113,66,
+    88,0,113,66,87,0,121,28,116,11,124,1,124,2,124,3,
+    131,3,1,0,116,9,160,10,100,3,124,1,161,2,1,0,
+    87,0,110,48,4,0,116,8,107,10,114,244,1,0,125,8,
+    1,0,122,18,116,9,160,10,100,1,124,1,124,8,161,3,
+    1,0,87,0,100,2,100,2,125,8,126,8,88,0,89,0,
+    110,2,88,0,100,2,83,0,41,4,122,27,87,114,105,116,
+    101,32,98,121,116,101,115,32,100,97,116,97,32,116,111,32,
+    97,32,102,105,108,101,46,122,27,99,111,117,108,100,32,110,
+    111,116,32,99,114,101,97,116,101,32,123,33,114,125,58,32,
+    123,33,114,125,78,122,12,99,114,101,97,116,101,100,32,123,
+    33,114,125,41,12,114,38,0,0,0,114,46,0,0,0,114,
+    165,0,0,0,114,33,0,0,0,114,28,0,0,0,114,1,
+    0,0,0,90,5,109,107,100,105,114,218,15,70,105,108,101,
+    69,120,105,115,116,115,69,114,114,111,114,114,40,0,0,0,
+    114,116,0,0,0,114,130,0,0,0,114,56,0,0,0,41,
+    9,114,102,0,0,0,114,35,0,0,0,114,54,0,0,0,
+    114,231,0,0,0,218,6,112,97,114,101,110,116,114,96,0,
+    0,0,114,27,0,0,0,114,23,0,0,0,114,201,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,8,95,95,105,116,101,114,95,95,58,4,0,0,115,2,
-    0,0,0,0,1,122,23,95,78,97,109,101,115,112,97,99,
-    101,80,97,116,104,46,95,95,105,116,101,114,95,95,99,3,
-    0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,67,
-    0,0,0,115,14,0,0,0,124,2,124,0,106,0,124,1,
-    60,0,100,0,83,0,41,1,78,41,1,114,234,0,0,0,
-    41,3,114,102,0,0,0,218,5,105,110,100,101,120,114,35,
+    114,198,0,0,0,187,3,0,0,115,42,0,0,0,0,2,
+    12,1,4,2,14,1,12,1,14,2,14,1,10,1,2,1,
+    14,1,14,2,6,1,16,3,6,1,8,1,22,1,2,1,
+    12,1,16,1,16,2,8,1,122,25,83,111,117,114,99,101,
+    70,105,108,101,76,111,97,100,101,114,46,115,101,116,95,100,
+    97,116,97,78,41,7,114,107,0,0,0,114,106,0,0,0,
+    114,108,0,0,0,114,109,0,0,0,114,197,0,0,0,114,
+    199,0,0,0,114,198,0,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,114,229,0,
+    0,0,173,3,0,0,115,6,0,0,0,12,4,8,5,8,
+    5,114,229,0,0,0,99,0,0,0,0,0,0,0,0,0,
+    0,0,0,2,0,0,0,64,0,0,0,115,32,0,0,0,
+    101,0,90,1,100,0,90,2,100,1,90,3,100,2,100,3,
+    132,0,90,4,100,4,100,5,132,0,90,5,100,6,83,0,
+    41,7,218,20,83,111,117,114,99,101,108,101,115,115,70,105,
+    108,101,76,111,97,100,101,114,122,45,76,111,97,100,101,114,
+    32,119,104,105,99,104,32,104,97,110,100,108,101,115,32,115,
+    111,117,114,99,101,108,101,115,115,32,102,105,108,101,32,105,
+    109,112,111,114,116,115,46,99,2,0,0,0,0,0,0,0,
+    5,0,0,0,5,0,0,0,67,0,0,0,115,68,0,0,
+    0,124,0,160,0,124,1,161,1,125,2,124,0,160,1,124,
+    2,161,1,125,3,124,1,124,2,100,1,156,2,125,4,116,
+    2,124,3,124,1,124,4,131,3,1,0,116,3,116,4,124,
+    3,131,1,100,2,100,0,133,2,25,0,124,1,124,2,100,
+    3,141,3,83,0,41,4,78,41,2,114,100,0,0,0,114,
+    35,0,0,0,114,126,0,0,0,41,2,114,100,0,0,0,
+    114,91,0,0,0,41,5,114,159,0,0,0,114,200,0,0,
+    0,114,133,0,0,0,114,145,0,0,0,114,208,0,0,0,
+    41,5,114,102,0,0,0,114,121,0,0,0,114,35,0,0,
+    0,114,54,0,0,0,114,132,0,0,0,114,2,0,0,0,
+    114,2,0,0,0,114,4,0,0,0,114,188,0,0,0,222,
+    3,0,0,115,18,0,0,0,0,1,10,1,10,4,2,1,
+    8,2,12,1,2,1,14,1,2,1,122,29,83,111,117,114,
+    99,101,108,101,115,115,70,105,108,101,76,111,97,100,101,114,
+    46,103,101,116,95,99,111,100,101,99,2,0,0,0,0,0,
+    0,0,2,0,0,0,1,0,0,0,67,0,0,0,115,4,
+    0,0,0,100,1,83,0,41,2,122,39,82,101,116,117,114,
+    110,32,78,111,110,101,32,97,115,32,116,104,101,114,101,32,
+    105,115,32,110,111,32,115,111,117,114,99,101,32,99,111,100,
+    101,46,78,114,2,0,0,0,41,2,114,102,0,0,0,114,
+    121,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,114,202,0,0,0,238,3,0,0,115,2,0,0,
+    0,0,2,122,31,83,111,117,114,99,101,108,101,115,115,70,
+    105,108,101,76,111,97,100,101,114,46,103,101,116,95,115,111,
+    117,114,99,101,78,41,6,114,107,0,0,0,114,106,0,0,
+    0,114,108,0,0,0,114,109,0,0,0,114,188,0,0,0,
+    114,202,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
+    2,0,0,0,114,4,0,0,0,114,234,0,0,0,218,3,
+    0,0,115,4,0,0,0,12,4,8,16,114,234,0,0,0,
+    99,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,
+    0,64,0,0,0,115,92,0,0,0,101,0,90,1,100,0,
+    90,2,100,1,90,3,100,2,100,3,132,0,90,4,100,4,
+    100,5,132,0,90,5,100,6,100,7,132,0,90,6,100,8,
+    100,9,132,0,90,7,100,10,100,11,132,0,90,8,100,12,
+    100,13,132,0,90,9,100,14,100,15,132,0,90,10,100,16,
+    100,17,132,0,90,11,101,12,100,18,100,19,132,0,131,1,
+    90,13,100,20,83,0,41,21,218,19,69,120,116,101,110,115,
+    105,111,110,70,105,108,101,76,111,97,100,101,114,122,93,76,
+    111,97,100,101,114,32,102,111,114,32,101,120,116,101,110,115,
+    105,111,110,32,109,111,100,117,108,101,115,46,10,10,32,32,
+    32,32,84,104,101,32,99,111,110,115,116,114,117,99,116,111,
+    114,32,105,115,32,100,101,115,105,103,110,101,100,32,116,111,
+    32,119,111,114,107,32,119,105,116,104,32,70,105,108,101,70,
+    105,110,100,101,114,46,10,10,32,32,32,32,99,3,0,0,
+    0,0,0,0,0,3,0,0,0,2,0,0,0,67,0,0,
+    0,115,16,0,0,0,124,1,124,0,95,0,124,2,124,0,
+    95,1,100,0,83,0,41,1,78,41,2,114,100,0,0,0,
+    114,35,0,0,0,41,3,114,102,0,0,0,114,100,0,0,
+    0,114,35,0,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,114,186,0,0,0,255,3,0,0,115,4,
+    0,0,0,0,1,6,1,122,28,69,120,116,101,110,115,105,
+    111,110,70,105,108,101,76,111,97,100,101,114,46,95,95,105,
+    110,105,116,95,95,99,2,0,0,0,0,0,0,0,2,0,
+    0,0,2,0,0,0,67,0,0,0,115,24,0,0,0,124,
+    0,106,0,124,1,106,0,107,2,111,22,124,0,106,1,124,
+    1,106,1,107,2,83,0,41,1,78,41,2,114,213,0,0,
+    0,114,113,0,0,0,41,2,114,102,0,0,0,114,214,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,215,0,0,0,3,4,0,0,115,4,0,0,0,0,
+    1,12,1,122,26,69,120,116,101,110,115,105,111,110,70,105,
+    108,101,76,111,97,100,101,114,46,95,95,101,113,95,95,99,
+    1,0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,
+    67,0,0,0,115,20,0,0,0,116,0,124,0,106,1,131,
+    1,116,0,124,0,106,2,131,1,65,0,83,0,41,1,78,
+    41,3,114,216,0,0,0,114,100,0,0,0,114,35,0,0,
+    0,41,1,114,102,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,114,217,0,0,0,7,4,0,0,
+    115,2,0,0,0,0,1,122,28,69,120,116,101,110,115,105,
+    111,110,70,105,108,101,76,111,97,100,101,114,46,95,95,104,
+    97,115,104,95,95,99,2,0,0,0,0,0,0,0,3,0,
+    0,0,5,0,0,0,67,0,0,0,115,36,0,0,0,116,
+    0,160,1,116,2,106,3,124,1,161,2,125,2,116,0,160,
+    4,100,1,124,1,106,5,124,0,106,6,161,3,1,0,124,
+    2,83,0,41,2,122,38,67,114,101,97,116,101,32,97,110,
+    32,117,110,105,116,105,97,108,105,122,101,100,32,101,120,116,
+    101,110,115,105,111,110,32,109,111,100,117,108,101,122,38,101,
+    120,116,101,110,115,105,111,110,32,109,111,100,117,108,101,32,
+    123,33,114,125,32,108,111,97,100,101,100,32,102,114,111,109,
+    32,123,33,114,125,41,7,114,116,0,0,0,114,189,0,0,
+    0,114,143,0,0,0,90,14,99,114,101,97,116,101,95,100,
+    121,110,97,109,105,99,114,130,0,0,0,114,100,0,0,0,
+    114,35,0,0,0,41,3,114,102,0,0,0,114,166,0,0,
+    0,114,191,0,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,114,187,0,0,0,10,4,0,0,115,10,
+    0,0,0,0,2,4,1,10,1,6,1,12,1,122,33,69,
+    120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,
+    101,114,46,99,114,101,97,116,101,95,109,111,100,117,108,101,
+    99,2,0,0,0,0,0,0,0,2,0,0,0,5,0,0,
+    0,67,0,0,0,115,36,0,0,0,116,0,160,1,116,2,
+    106,3,124,1,161,2,1,0,116,0,160,4,100,1,124,0,
+    106,5,124,0,106,6,161,3,1,0,100,2,83,0,41,3,
+    122,30,73,110,105,116,105,97,108,105,122,101,32,97,110,32,
+    101,120,116,101,110,115,105,111,110,32,109,111,100,117,108,101,
+    122,40,101,120,116,101,110,115,105,111,110,32,109,111,100,117,
+    108,101,32,123,33,114,125,32,101,120,101,99,117,116,101,100,
+    32,102,114,111,109,32,123,33,114,125,78,41,7,114,116,0,
+    0,0,114,189,0,0,0,114,143,0,0,0,90,12,101,120,
+    101,99,95,100,121,110,97,109,105,99,114,130,0,0,0,114,
+    100,0,0,0,114,35,0,0,0,41,2,114,102,0,0,0,
+    114,191,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,114,192,0,0,0,18,4,0,0,115,6,0,
+    0,0,0,2,14,1,6,1,122,31,69,120,116,101,110,115,
+    105,111,110,70,105,108,101,76,111,97,100,101,114,46,101,120,
+    101,99,95,109,111,100,117,108,101,99,2,0,0,0,0,0,
+    0,0,2,0,0,0,4,0,0,0,3,0,0,0,115,36,
+    0,0,0,116,0,124,0,106,1,131,1,100,1,25,0,137,
+    0,116,2,135,0,102,1,100,2,100,3,132,8,116,3,68,
+    0,131,1,131,1,83,0,41,4,122,49,82,101,116,117,114,
+    110,32,84,114,117,101,32,105,102,32,116,104,101,32,101,120,
+    116,101,110,115,105,111,110,32,109,111,100,117,108,101,32,105,
+    115,32,97,32,112,97,99,107,97,103,101,46,114,29,0,0,
+    0,99,1,0,0,0,0,0,0,0,2,0,0,0,4,0,
+    0,0,51,0,0,0,115,26,0,0,0,124,0,93,18,125,
+    1,136,0,100,0,124,1,23,0,107,2,86,0,1,0,113,
+    2,100,1,83,0,41,2,114,186,0,0,0,78,114,2,0,
+    0,0,41,2,114,22,0,0,0,218,6,115,117,102,102,105,
+    120,41,1,218,9,102,105,108,101,95,110,97,109,101,114,2,
+    0,0,0,114,4,0,0,0,250,9,60,103,101,110,101,120,
+    112,114,62,27,4,0,0,115,2,0,0,0,4,1,122,49,
+    69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,
+    100,101,114,46,105,115,95,112,97,99,107,97,103,101,46,60,
+    108,111,99,97,108,115,62,46,60,103,101,110,101,120,112,114,
+    62,41,4,114,38,0,0,0,114,35,0,0,0,218,3,97,
+    110,121,218,18,69,88,84,69,78,83,73,79,78,95,83,85,
+    70,70,73,88,69,83,41,2,114,102,0,0,0,114,121,0,
+    0,0,114,2,0,0,0,41,1,114,237,0,0,0,114,4,
+    0,0,0,114,161,0,0,0,24,4,0,0,115,6,0,0,
+    0,0,2,14,1,12,1,122,30,69,120,116,101,110,115,105,
+    111,110,70,105,108,101,76,111,97,100,101,114,46,105,115,95,
+    112,97,99,107,97,103,101,99,2,0,0,0,0,0,0,0,
+    2,0,0,0,1,0,0,0,67,0,0,0,115,4,0,0,
+    0,100,1,83,0,41,2,122,63,82,101,116,117,114,110,32,
+    78,111,110,101,32,97,115,32,97,110,32,101,120,116,101,110,
+    115,105,111,110,32,109,111,100,117,108,101,32,99,97,110,110,
+    111,116,32,99,114,101,97,116,101,32,97,32,99,111,100,101,
+    32,111,98,106,101,99,116,46,78,114,2,0,0,0,41,2,
+    114,102,0,0,0,114,121,0,0,0,114,2,0,0,0,114,
+    2,0,0,0,114,4,0,0,0,114,188,0,0,0,30,4,
+    0,0,115,2,0,0,0,0,2,122,28,69,120,116,101,110,
+    115,105,111,110,70,105,108,101,76,111,97,100,101,114,46,103,
+    101,116,95,99,111,100,101,99,2,0,0,0,0,0,0,0,
+    2,0,0,0,1,0,0,0,67,0,0,0,115,4,0,0,
+    0,100,1,83,0,41,2,122,53,82,101,116,117,114,110,32,
+    78,111,110,101,32,97,115,32,101,120,116,101,110,115,105,111,
+    110,32,109,111,100,117,108,101,115,32,104,97,118,101,32,110,
+    111,32,115,111,117,114,99,101,32,99,111,100,101,46,78,114,
+    2,0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
+    202,0,0,0,34,4,0,0,115,2,0,0,0,0,2,122,
+    30,69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,
+    97,100,101,114,46,103,101,116,95,115,111,117,114,99,101,99,
+    2,0,0,0,0,0,0,0,2,0,0,0,1,0,0,0,
+    67,0,0,0,115,6,0,0,0,124,0,106,0,83,0,41,
+    1,122,58,82,101,116,117,114,110,32,116,104,101,32,112,97,
+    116,104,32,116,111,32,116,104,101,32,115,111,117,114,99,101,
+    32,102,105,108,101,32,97,115,32,102,111,117,110,100,32,98,
+    121,32,116,104,101,32,102,105,110,100,101,114,46,41,1,114,
+    35,0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
+    159,0,0,0,38,4,0,0,115,2,0,0,0,0,3,122,
+    32,69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,
+    97,100,101,114,46,103,101,116,95,102,105,108,101,110,97,109,
+    101,78,41,14,114,107,0,0,0,114,106,0,0,0,114,108,
+    0,0,0,114,109,0,0,0,114,186,0,0,0,114,215,0,
+    0,0,114,217,0,0,0,114,187,0,0,0,114,192,0,0,
+    0,114,161,0,0,0,114,188,0,0,0,114,202,0,0,0,
+    114,118,0,0,0,114,159,0,0,0,114,2,0,0,0,114,
+    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,235,
+    0,0,0,247,3,0,0,115,18,0,0,0,12,8,8,4,
+    8,4,8,3,8,8,8,6,8,6,8,4,8,4,114,235,
+    0,0,0,99,0,0,0,0,0,0,0,0,0,0,0,0,
+    2,0,0,0,64,0,0,0,115,96,0,0,0,101,0,90,
+    1,100,0,90,2,100,1,90,3,100,2,100,3,132,0,90,
+    4,100,4,100,5,132,0,90,5,100,6,100,7,132,0,90,
+    6,100,8,100,9,132,0,90,7,100,10,100,11,132,0,90,
+    8,100,12,100,13,132,0,90,9,100,14,100,15,132,0,90,
+    10,100,16,100,17,132,0,90,11,100,18,100,19,132,0,90,
+    12,100,20,100,21,132,0,90,13,100,22,83,0,41,23,218,
+    14,95,78,97,109,101,115,112,97,99,101,80,97,116,104,97,
+    38,1,0,0,82,101,112,114,101,115,101,110,116,115,32,97,
+    32,110,97,109,101,115,112,97,99,101,32,112,97,99,107,97,
+    103,101,39,115,32,112,97,116,104,46,32,32,73,116,32,117,
+    115,101,115,32,116,104,101,32,109,111,100,117,108,101,32,110,
+    97,109,101,10,32,32,32,32,116,111,32,102,105,110,100,32,
+    105,116,115,32,112,97,114,101,110,116,32,109,111,100,117,108,
+    101,44,32,97,110,100,32,102,114,111,109,32,116,104,101,114,
+    101,32,105,116,32,108,111,111,107,115,32,117,112,32,116,104,
+    101,32,112,97,114,101,110,116,39,115,10,32,32,32,32,95,
+    95,112,97,116,104,95,95,46,32,32,87,104,101,110,32,116,
+    104,105,115,32,99,104,97,110,103,101,115,44,32,116,104,101,
+    32,109,111,100,117,108,101,39,115,32,111,119,110,32,112,97,
+    116,104,32,105,115,32,114,101,99,111,109,112,117,116,101,100,
+    44,10,32,32,32,32,117,115,105,110,103,32,112,97,116,104,
+    95,102,105,110,100,101,114,46,32,32,70,111,114,32,116,111,
+    112,45,108,101,118,101,108,32,109,111,100,117,108,101,115,44,
+    32,116,104,101,32,112,97,114,101,110,116,32,109,111,100,117,
+    108,101,39,115,32,112,97,116,104,10,32,32,32,32,105,115,
+    32,115,121,115,46,112,97,116,104,46,99,4,0,0,0,0,
+    0,0,0,4,0,0,0,3,0,0,0,67,0,0,0,115,
+    36,0,0,0,124,1,124,0,95,0,124,2,124,0,95,1,
+    116,2,124,0,160,3,161,0,131,1,124,0,95,4,124,3,
+    124,0,95,5,100,0,83,0,41,1,78,41,6,218,5,95,
+    110,97,109,101,218,5,95,112,97,116,104,114,95,0,0,0,
+    218,16,95,103,101,116,95,112,97,114,101,110,116,95,112,97,
+    116,104,218,17,95,108,97,115,116,95,112,97,114,101,110,116,
+    95,112,97,116,104,218,12,95,112,97,116,104,95,102,105,110,
+    100,101,114,41,4,114,102,0,0,0,114,100,0,0,0,114,
+    35,0,0,0,218,11,112,97,116,104,95,102,105,110,100,101,
+    114,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,186,0,0,0,51,4,0,0,115,8,0,0,0,0,1,
+    6,1,6,1,14,1,122,23,95,78,97,109,101,115,112,97,
+    99,101,80,97,116,104,46,95,95,105,110,105,116,95,95,99,
+    1,0,0,0,0,0,0,0,4,0,0,0,3,0,0,0,
+    67,0,0,0,115,38,0,0,0,124,0,106,0,160,1,100,
+    1,161,1,92,3,125,1,125,2,125,3,124,2,100,2,107,
+    2,114,30,100,3,83,0,124,1,100,4,102,2,83,0,41,
+    5,122,62,82,101,116,117,114,110,115,32,97,32,116,117,112,
+    108,101,32,111,102,32,40,112,97,114,101,110,116,45,109,111,
+    100,117,108,101,45,110,97,109,101,44,32,112,97,114,101,110,
+    116,45,112,97,116,104,45,97,116,116,114,45,110,97,109,101,
+    41,114,59,0,0,0,114,30,0,0,0,41,2,114,6,0,
+    0,0,114,35,0,0,0,90,8,95,95,112,97,116,104,95,
+    95,41,2,114,242,0,0,0,114,32,0,0,0,41,4,114,
+    102,0,0,0,114,233,0,0,0,218,3,100,111,116,90,2,
+    109,101,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,218,23,95,102,105,110,100,95,112,97,114,101,110,116,95,
+    112,97,116,104,95,110,97,109,101,115,57,4,0,0,115,8,
+    0,0,0,0,2,18,1,8,2,4,3,122,38,95,78,97,
+    109,101,115,112,97,99,101,80,97,116,104,46,95,102,105,110,
+    100,95,112,97,114,101,110,116,95,112,97,116,104,95,110,97,
+    109,101,115,99,1,0,0,0,0,0,0,0,3,0,0,0,
+    3,0,0,0,67,0,0,0,115,28,0,0,0,124,0,160,
+    0,161,0,92,2,125,1,125,2,116,1,116,2,106,3,124,
+    1,25,0,124,2,131,2,83,0,41,1,78,41,4,114,249,
+    0,0,0,114,112,0,0,0,114,6,0,0,0,218,7,109,
+    111,100,117,108,101,115,41,3,114,102,0,0,0,90,18,112,
+    97,114,101,110,116,95,109,111,100,117,108,101,95,110,97,109,
+    101,90,14,112,97,116,104,95,97,116,116,114,95,110,97,109,
+    101,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,244,0,0,0,67,4,0,0,115,4,0,0,0,0,1,
+    12,1,122,31,95,78,97,109,101,115,112,97,99,101,80,97,
+    116,104,46,95,103,101,116,95,112,97,114,101,110,116,95,112,
+    97,116,104,99,1,0,0,0,0,0,0,0,3,0,0,0,
+    4,0,0,0,67,0,0,0,115,80,0,0,0,116,0,124,
+    0,160,1,161,0,131,1,125,1,124,1,124,0,106,2,107,
+    3,114,74,124,0,160,3,124,0,106,4,124,1,161,2,125,
+    2,124,2,100,0,107,9,114,68,124,2,106,5,100,0,107,
+    8,114,68,124,2,106,6,114,68,124,2,106,6,124,0,95,
+    7,124,1,124,0,95,2,124,0,106,7,83,0,41,1,78,
+    41,8,114,95,0,0,0,114,244,0,0,0,114,245,0,0,
+    0,114,246,0,0,0,114,242,0,0,0,114,122,0,0,0,
+    114,158,0,0,0,114,243,0,0,0,41,3,114,102,0,0,
+    0,90,11,112,97,114,101,110,116,95,112,97,116,104,114,166,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,218,11,95,95,115,101,116,105,116,101,109,95,95,61,
-    4,0,0,115,2,0,0,0,0,1,122,26,95,78,97,109,
-    101,115,112,97,99,101,80,97,116,104,46,95,95,115,101,116,
-    105,116,101,109,95,95,99,1,0,0,0,0,0,0,0,1,
+    0,0,218,12,95,114,101,99,97,108,99,117,108,97,116,101,
+    71,4,0,0,115,16,0,0,0,0,2,12,1,10,1,14,
+    3,18,1,6,1,8,1,6,1,122,27,95,78,97,109,101,
+    115,112,97,99,101,80,97,116,104,46,95,114,101,99,97,108,
+    99,117,108,97,116,101,99,1,0,0,0,0,0,0,0,1,
     0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,0,
     116,0,124,0,160,1,161,0,131,1,83,0,41,1,78,41,
-    2,114,31,0,0,0,114,242,0,0,0,41,1,114,102,0,
+    2,114,226,0,0,0,114,251,0,0,0,41,1,114,102,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,7,95,95,108,101,110,95,95,64,4,0,0,115,2,
-    0,0,0,0,1,122,22,95,78,97,109,101,115,112,97,99,
-    101,80,97,116,104,46,95,95,108,101,110,95,95,99,1,0,
-    0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,0,
-    0,0,115,12,0,0,0,100,1,160,0,124,0,106,1,161,
-    1,83,0,41,2,78,122,20,95,78,97,109,101,115,112,97,
-    99,101,80,97,116,104,40,123,33,114,125,41,41,2,114,48,
-    0,0,0,114,234,0,0,0,41,1,114,102,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,8,
-    95,95,114,101,112,114,95,95,67,4,0,0,115,2,0,0,
-    0,0,1,122,23,95,78,97,109,101,115,112,97,99,101,80,
-    97,116,104,46,95,95,114,101,112,114,95,95,99,2,0,0,
-    0,0,0,0,0,2,0,0,0,3,0,0,0,67,0,0,
-    0,115,12,0,0,0,124,1,124,0,160,0,161,0,107,6,
-    83,0,41,1,78,41,1,114,242,0,0,0,41,2,114,102,
-    0,0,0,218,4,105,116,101,109,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,218,12,95,95,99,111,110,116,
-    97,105,110,115,95,95,70,4,0,0,115,2,0,0,0,0,
-    1,122,27,95,78,97,109,101,115,112,97,99,101,80,97,116,
-    104,46,95,95,99,111,110,116,97,105,110,115,95,95,99,2,
-    0,0,0,0,0,0,0,2,0,0,0,3,0,0,0,67,
-    0,0,0,115,16,0,0,0,124,0,106,0,160,1,124,1,
-    161,1,1,0,100,0,83,0,41,1,78,41,2,114,234,0,
-    0,0,114,165,0,0,0,41,2,114,102,0,0,0,114,249,
+    0,218,8,95,95,105,116,101,114,95,95,84,4,0,0,115,
+    2,0,0,0,0,1,122,23,95,78,97,109,101,115,112,97,
+    99,101,80,97,116,104,46,95,95,105,116,101,114,95,95,99,
+    3,0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,
+    67,0,0,0,115,14,0,0,0,124,2,124,0,106,0,124,
+    1,60,0,100,0,83,0,41,1,78,41,1,114,243,0,0,
+    0,41,3,114,102,0,0,0,218,5,105,110,100,101,120,114,
+    35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,218,11,95,95,115,101,116,105,116,101,109,95,95,
+    87,4,0,0,115,2,0,0,0,0,1,122,26,95,78,97,
+    109,101,115,112,97,99,101,80,97,116,104,46,95,95,115,101,
+    116,105,116,101,109,95,95,99,1,0,0,0,0,0,0,0,
+    1,0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,
+    0,116,0,124,0,160,1,161,0,131,1,83,0,41,1,78,
+    41,2,114,31,0,0,0,114,251,0,0,0,41,1,114,102,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,165,0,0,0,73,4,0,0,115,2,0,0,0,
-    0,1,122,21,95,78,97,109,101,115,112,97,99,101,80,97,
-    116,104,46,97,112,112,101,110,100,78,41,14,114,107,0,0,
-    0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,0,
-    114,186,0,0,0,114,240,0,0,0,114,235,0,0,0,114,
-    242,0,0,0,114,244,0,0,0,114,246,0,0,0,114,247,
-    0,0,0,114,248,0,0,0,114,250,0,0,0,114,165,0,
-    0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,
-    0,114,4,0,0,0,114,232,0,0,0,18,4,0,0,115,
-    20,0,0,0,12,7,8,6,8,10,8,4,8,13,8,3,
-    8,3,8,3,8,3,8,3,114,232,0,0,0,99,0,0,
-    0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,0,
-    0,0,115,80,0,0,0,101,0,90,1,100,0,90,2,100,
-    1,100,2,132,0,90,3,101,4,100,3,100,4,132,0,131,
-    1,90,5,100,5,100,6,132,0,90,6,100,7,100,8,132,
-    0,90,7,100,9,100,10,132,0,90,8,100,11,100,12,132,
-    0,90,9,100,13,100,14,132,0,90,10,100,15,100,16,132,
-    0,90,11,100,17,83,0,41,18,218,16,95,78,97,109,101,
-    115,112,97,99,101,76,111,97,100,101,114,99,4,0,0,0,
-    0,0,0,0,4,0,0,0,4,0,0,0,67,0,0,0,
-    115,18,0,0,0,116,0,124,1,124,2,124,3,131,3,124,
-    0,95,1,100,0,83,0,41,1,78,41,2,114,232,0,0,
-    0,114,234,0,0,0,41,4,114,102,0,0,0,114,100,0,
-    0,0,114,35,0,0,0,114,238,0,0,0,114,2,0,0,
-    0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,0,
-    79,4,0,0,115,2,0,0,0,0,1,122,25,95,78,97,
-    109,101,115,112,97,99,101,76,111,97,100,101,114,46,95,95,
-    105,110,105,116,95,95,99,2,0,0,0,0,0,0,0,2,
-    0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,0,
-    100,1,160,0,124,1,106,1,161,1,83,0,41,2,122,115,
-    82,101,116,117,114,110,32,114,101,112,114,32,102,111,114,32,
-    116,104,101,32,109,111,100,117,108,101,46,10,10,32,32,32,
-    32,32,32,32,32,84,104,101,32,109,101,116,104,111,100,32,
-    105,115,32,100,101,112,114,101,99,97,116,101,100,46,32,32,
-    84,104,101,32,105,109,112,111,114,116,32,109,97,99,104,105,
-    110,101,114,121,32,100,111,101,115,32,116,104,101,32,106,111,
-    98,32,105,116,115,101,108,102,46,10,10,32,32,32,32,32,
-    32,32,32,122,25,60,109,111,100,117,108,101,32,123,33,114,
-    125,32,40,110,97,109,101,115,112,97,99,101,41,62,41,2,
-    114,48,0,0,0,114,107,0,0,0,41,2,114,172,0,0,
-    0,114,191,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,218,11,109,111,100,117,108,101,95,114,101,
-    112,114,82,4,0,0,115,2,0,0,0,0,7,122,28,95,
-    78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,
-    109,111,100,117,108,101,95,114,101,112,114,99,2,0,0,0,
-    0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,
-    115,4,0,0,0,100,1,83,0,41,2,78,84,114,2,0,
-    0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,161,0,
-    0,0,91,4,0,0,115,2,0,0,0,0,1,122,27,95,
-    78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,
-    105,115,95,112,97,99,107,97,103,101,99,2,0,0,0,0,
-    0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,115,
-    4,0,0,0,100,1,83,0,41,2,78,114,30,0,0,0,
-    114,2,0,0,0,41,2,114,102,0,0,0,114,121,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,202,0,0,0,94,4,0,0,115,2,0,0,0,0,1,
-    122,27,95,78,97,109,101,115,112,97,99,101,76,111,97,100,
-    101,114,46,103,101,116,95,115,111,117,114,99,101,99,2,0,
-    0,0,0,0,0,0,2,0,0,0,6,0,0,0,67,0,
-    0,0,115,16,0,0,0,116,0,100,1,100,2,100,3,100,
-    4,100,5,141,4,83,0,41,6,78,114,30,0,0,0,122,
-    8,60,115,116,114,105,110,103,62,114,190,0,0,0,84,41,
-    1,114,204,0,0,0,41,1,114,205,0,0,0,41,2,114,
-    102,0,0,0,114,121,0,0,0,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,188,0,0,0,97,4,0,
-    0,115,2,0,0,0,0,1,122,25,95,78,97,109,101,115,
-    112,97,99,101,76,111,97,100,101,114,46,103,101,116,95,99,
-    111,100,101,99,2,0,0,0,0,0,0,0,2,0,0,0,
-    1,0,0,0,67,0,0,0,115,4,0,0,0,100,1,83,
-    0,41,2,122,42,85,115,101,32,100,101,102,97,117,108,116,
-    32,115,101,109,97,110,116,105,99,115,32,102,111,114,32,109,
-    111,100,117,108,101,32,99,114,101,97,116,105,111,110,46,78,
-    114,2,0,0,0,41,2,114,102,0,0,0,114,166,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,187,0,0,0,100,4,0,0,115,0,0,0,0,122,30,
+    0,0,218,7,95,95,108,101,110,95,95,90,4,0,0,115,
+    2,0,0,0,0,1,122,22,95,78,97,109,101,115,112,97,
+    99,101,80,97,116,104,46,95,95,108,101,110,95,95,99,1,
+    0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,
+    0,0,0,115,12,0,0,0,100,1,160,0,124,0,106,1,
+    161,1,83,0,41,2,78,122,20,95,78,97,109,101,115,112,
+    97,99,101,80,97,116,104,40,123,33,114,125,41,41,2,114,
+    48,0,0,0,114,243,0,0,0,41,1,114,102,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
+    8,95,95,114,101,112,114,95,95,93,4,0,0,115,2,0,
+    0,0,0,1,122,23,95,78,97,109,101,115,112,97,99,101,
+    80,97,116,104,46,95,95,114,101,112,114,95,95,99,2,0,
+    0,0,0,0,0,0,2,0,0,0,3,0,0,0,67,0,
+    0,0,115,12,0,0,0,124,1,124,0,160,0,161,0,107,
+    6,83,0,41,1,78,41,1,114,251,0,0,0,41,2,114,
+    102,0,0,0,218,4,105,116,101,109,114,2,0,0,0,114,
+    2,0,0,0,114,4,0,0,0,218,12,95,95,99,111,110,
+    116,97,105,110,115,95,95,96,4,0,0,115,2,0,0,0,
+    0,1,122,27,95,78,97,109,101,115,112,97,99,101,80,97,
+    116,104,46,95,95,99,111,110,116,97,105,110,115,95,95,99,
+    2,0,0,0,0,0,0,0,2,0,0,0,3,0,0,0,
+    67,0,0,0,115,16,0,0,0,124,0,106,0,160,1,124,
+    1,161,1,1,0,100,0,83,0,41,1,78,41,2,114,243,
+    0,0,0,114,165,0,0,0,41,2,114,102,0,0,0,114,
+    1,1,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,114,165,0,0,0,99,4,0,0,115,2,0,0,
+    0,0,1,122,21,95,78,97,109,101,115,112,97,99,101,80,
+    97,116,104,46,97,112,112,101,110,100,78,41,14,114,107,0,
+    0,0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,
+    0,114,186,0,0,0,114,249,0,0,0,114,244,0,0,0,
+    114,251,0,0,0,114,252,0,0,0,114,254,0,0,0,114,
+    255,0,0,0,114,0,1,0,0,114,2,1,0,0,114,165,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,
+    0,0,114,4,0,0,0,114,241,0,0,0,44,4,0,0,
+    115,20,0,0,0,12,7,8,6,8,10,8,4,8,13,8,
+    3,8,3,8,3,8,3,8,3,114,241,0,0,0,99,0,
+    0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,
+    0,0,0,115,80,0,0,0,101,0,90,1,100,0,90,2,
+    100,1,100,2,132,0,90,3,101,4,100,3,100,4,132,0,
+    131,1,90,5,100,5,100,6,132,0,90,6,100,7,100,8,
+    132,0,90,7,100,9,100,10,132,0,90,8,100,11,100,12,
+    132,0,90,9,100,13,100,14,132,0,90,10,100,15,100,16,
+    132,0,90,11,100,17,83,0,41,18,218,16,95,78,97,109,
+    101,115,112,97,99,101,76,111,97,100,101,114,99,4,0,0,
+    0,0,0,0,0,4,0,0,0,4,0,0,0,67,0,0,
+    0,115,18,0,0,0,116,0,124,1,124,2,124,3,131,3,
+    124,0,95,1,100,0,83,0,41,1,78,41,2,114,241,0,
+    0,0,114,243,0,0,0,41,4,114,102,0,0,0,114,100,
+    0,0,0,114,35,0,0,0,114,247,0,0,0,114,2,0,
+    0,0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,
+    0,105,4,0,0,115,2,0,0,0,0,1,122,25,95,78,
+    97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,95,
+    95,105,110,105,116,95,95,99,2,0,0,0,0,0,0,0,
+    2,0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,
+    0,100,1,160,0,124,1,106,1,161,1,83,0,41,2,122,
+    115,82,101,116,117,114,110,32,114,101,112,114,32,102,111,114,
+    32,116,104,101,32,109,111,100,117,108,101,46,10,10,32,32,
+    32,32,32,32,32,32,84,104,101,32,109,101,116,104,111,100,
+    32,105,115,32,100,101,112,114,101,99,97,116,101,100,46,32,
+    32,84,104,101,32,105,109,112,111,114,116,32,109,97,99,104,
+    105,110,101,114,121,32,100,111,101,115,32,116,104,101,32,106,
+    111,98,32,105,116,115,101,108,102,46,10,10,32,32,32,32,
+    32,32,32,32,122,25,60,109,111,100,117,108,101,32,123,33,
+    114,125,32,40,110,97,109,101,115,112,97,99,101,41,62,41,
+    2,114,48,0,0,0,114,107,0,0,0,41,2,114,172,0,
+    0,0,114,191,0,0,0,114,2,0,0,0,114,2,0,0,
+    0,114,4,0,0,0,218,11,109,111,100,117,108,101,95,114,
+    101,112,114,108,4,0,0,115,2,0,0,0,0,7,122,28,
     95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,
-    46,99,114,101,97,116,101,95,109,111,100,117,108,101,99,2,
-    0,0,0,0,0,0,0,2,0,0,0,1,0,0,0,67,
-    0,0,0,115,4,0,0,0,100,0,83,0,41,1,78,114,
-    2,0,0,0,41,2,114,102,0,0,0,114,191,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    192,0,0,0,103,4,0,0,115,2,0,0,0,0,1,122,
-    28,95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,
-    114,46,101,120,101,99,95,109,111,100,117,108,101,99,2,0,
-    0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,0,
-    0,0,115,26,0,0,0,116,0,160,1,100,1,124,0,106,
-    2,161,2,1,0,116,0,160,3,124,0,124,1,161,2,83,
-    0,41,2,122,98,76,111,97,100,32,97,32,110,97,109,101,
-    115,112,97,99,101,32,109,111,100,117,108,101,46,10,10,32,
-    32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,104,
-    111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,100,
-    46,32,32,85,115,101,32,101,120,101,99,95,109,111,100,117,
-    108,101,40,41,32,105,110,115,116,101,97,100,46,10,10,32,
-    32,32,32,32,32,32,32,122,38,110,97,109,101,115,112,97,
-    99,101,32,109,111,100,117,108,101,32,108,111,97,100,101,100,
-    32,119,105,116,104,32,112,97,116,104,32,123,33,114,125,41,
-    4,114,116,0,0,0,114,130,0,0,0,114,234,0,0,0,
-    114,193,0,0,0,41,2,114,102,0,0,0,114,121,0,0,
+    46,109,111,100,117,108,101,95,114,101,112,114,99,2,0,0,
+    0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,
+    0,115,4,0,0,0,100,1,83,0,41,2,78,84,114,2,
+    0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,
+    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,161,
+    0,0,0,117,4,0,0,115,2,0,0,0,0,1,122,27,
+    95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,
+    46,105,115,95,112,97,99,107,97,103,101,99,2,0,0,0,
+    0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,
+    115,4,0,0,0,100,1,83,0,41,2,78,114,30,0,0,
+    0,114,2,0,0,0,41,2,114,102,0,0,0,114,121,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,202,0,0,0,120,4,0,0,115,2,0,0,0,0,
+    1,122,27,95,78,97,109,101,115,112,97,99,101,76,111,97,
+    100,101,114,46,103,101,116,95,115,111,117,114,99,101,99,2,
+    0,0,0,0,0,0,0,2,0,0,0,6,0,0,0,67,
+    0,0,0,115,16,0,0,0,116,0,100,1,100,2,100,3,
+    100,4,100,5,141,4,83,0,41,6,78,114,30,0,0,0,
+    122,8,60,115,116,114,105,110,103,62,114,190,0,0,0,84,
+    41,1,114,204,0,0,0,41,1,114,205,0,0,0,41,2,
+    114,102,0,0,0,114,121,0,0,0,114,2,0,0,0,114,
+    2,0,0,0,114,4,0,0,0,114,188,0,0,0,123,4,
+    0,0,115,2,0,0,0,0,1,122,25,95,78,97,109,101,
+    115,112,97,99,101,76,111,97,100,101,114,46,103,101,116,95,
+    99,111,100,101,99,2,0,0,0,0,0,0,0,2,0,0,
+    0,1,0,0,0,67,0,0,0,115,4,0,0,0,100,1,
+    83,0,41,2,122,42,85,115,101,32,100,101,102,97,117,108,
+    116,32,115,101,109,97,110,116,105,99,115,32,102,111,114,32,
+    109,111,100,117,108,101,32,99,114,101,97,116,105,111,110,46,
+    78,114,2,0,0,0,41,2,114,102,0,0,0,114,166,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,187,0,0,0,126,4,0,0,115,0,0,0,0,122,
+    30,95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,
+    114,46,99,114,101,97,116,101,95,109,111,100,117,108,101,99,
+    2,0,0,0,0,0,0,0,2,0,0,0,1,0,0,0,
+    67,0,0,0,115,4,0,0,0,100,0,83,0,41,1,78,
+    114,2,0,0,0,41,2,114,102,0,0,0,114,191,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,194,0,0,0,106,4,0,0,115,6,0,0,0,0,7,
-    6,1,8,1,122,28,95,78,97,109,101,115,112,97,99,101,
-    76,111,97,100,101,114,46,108,111,97,100,95,109,111,100,117,
-    108,101,78,41,12,114,107,0,0,0,114,106,0,0,0,114,
-    108,0,0,0,114,186,0,0,0,114,184,0,0,0,114,252,
-    0,0,0,114,161,0,0,0,114,202,0,0,0,114,188,0,
-    0,0,114,187,0,0,0,114,192,0,0,0,114,194,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,251,0,0,0,78,4,0,0,115,16,
-    0,0,0,8,1,8,3,12,9,8,3,8,3,8,3,8,
-    3,8,3,114,251,0,0,0,99,0,0,0,0,0,0,0,
-    0,0,0,0,0,4,0,0,0,64,0,0,0,115,106,0,
-    0,0,101,0,90,1,100,0,90,2,100,1,90,3,101,4,
-    100,2,100,3,132,0,131,1,90,5,101,4,100,4,100,5,
-    132,0,131,1,90,6,101,4,100,6,100,7,132,0,131,1,
-    90,7,101,4,100,8,100,9,132,0,131,1,90,8,101,4,
-    100,17,100,11,100,12,132,1,131,1,90,9,101,4,100,18,
-    100,13,100,14,132,1,131,1,90,10,101,4,100,19,100,15,
-    100,16,132,1,131,1,90,11,100,10,83,0,41,20,218,10,
-    80,97,116,104,70,105,110,100,101,114,122,62,77,101,116,97,
-    32,112,97,116,104,32,102,105,110,100,101,114,32,102,111,114,
-    32,115,121,115,46,112,97,116,104,32,97,110,100,32,112,97,
-    99,107,97,103,101,32,95,95,112,97,116,104,95,95,32,97,
-    116,116,114,105,98,117,116,101,115,46,99,1,0,0,0,0,
-    0,0,0,2,0,0,0,4,0,0,0,67,0,0,0,115,
-    42,0,0,0,120,36,116,0,106,1,160,2,161,0,68,0,
-    93,22,125,1,116,3,124,1,100,1,131,2,114,12,124,1,
-    160,4,161,0,1,0,113,12,87,0,100,2,83,0,41,3,
-    122,125,67,97,108,108,32,116,104,101,32,105,110,118,97,108,
-    105,100,97,116,101,95,99,97,99,104,101,115,40,41,32,109,
-    101,116,104,111,100,32,111,110,32,97,108,108,32,112,97,116,
-    104,32,101,110,116,114,121,32,102,105,110,100,101,114,115,10,
-    32,32,32,32,32,32,32,32,115,116,111,114,101,100,32,105,
-    110,32,115,121,115,46,112,97,116,104,95,105,109,112,111,114,
-    116,101,114,95,99,97,99,104,101,115,32,40,119,104,101,114,
-    101,32,105,109,112,108,101,109,101,110,116,101,100,41,46,218,
-    17,105,110,118,97,108,105,100,97,116,101,95,99,97,99,104,
-    101,115,78,41,5,114,6,0,0,0,218,19,112,97,116,104,
-    95,105,109,112,111,114,116,101,114,95,99,97,99,104,101,218,
-    6,118,97,108,117,101,115,114,110,0,0,0,114,254,0,0,
-    0,41,2,114,172,0,0,0,218,6,102,105,110,100,101,114,
+    114,192,0,0,0,129,4,0,0,115,2,0,0,0,0,1,
+    122,28,95,78,97,109,101,115,112,97,99,101,76,111,97,100,
+    101,114,46,101,120,101,99,95,109,111,100,117,108,101,99,2,
+    0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,
+    0,0,0,115,26,0,0,0,116,0,160,1,100,1,124,0,
+    106,2,161,2,1,0,116,0,160,3,124,0,124,1,161,2,
+    83,0,41,2,122,98,76,111,97,100,32,97,32,110,97,109,
+    101,115,112,97,99,101,32,109,111,100,117,108,101,46,10,10,
+    32,32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,
+    104,111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,
+    100,46,32,32,85,115,101,32,101,120,101,99,95,109,111,100,
+    117,108,101,40,41,32,105,110,115,116,101,97,100,46,10,10,
+    32,32,32,32,32,32,32,32,122,38,110,97,109,101,115,112,
+    97,99,101,32,109,111,100,117,108,101,32,108,111,97,100,101,
+    100,32,119,105,116,104,32,112,97,116,104,32,123,33,114,125,
+    41,4,114,116,0,0,0,114,130,0,0,0,114,243,0,0,
+    0,114,193,0,0,0,41,2,114,102,0,0,0,114,121,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
+    0,114,194,0,0,0,132,4,0,0,115,6,0,0,0,0,
+    7,6,1,8,1,122,28,95,78,97,109,101,115,112,97,99,
+    101,76,111,97,100,101,114,46,108,111,97,100,95,109,111,100,
+    117,108,101,78,41,12,114,107,0,0,0,114,106,0,0,0,
+    114,108,0,0,0,114,186,0,0,0,114,184,0,0,0,114,
+    4,1,0,0,114,161,0,0,0,114,202,0,0,0,114,188,
+    0,0,0,114,187,0,0,0,114,192,0,0,0,114,194,0,
+    0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,
+    0,114,4,0,0,0,114,3,1,0,0,104,4,0,0,115,
+    16,0,0,0,8,1,8,3,12,9,8,3,8,3,8,3,
+    8,3,8,3,114,3,1,0,0,99,0,0,0,0,0,0,
+    0,0,0,0,0,0,4,0,0,0,64,0,0,0,115,106,
+    0,0,0,101,0,90,1,100,0,90,2,100,1,90,3,101,
+    4,100,2,100,3,132,0,131,1,90,5,101,4,100,4,100,
+    5,132,0,131,1,90,6,101,4,100,6,100,7,132,0,131,
+    1,90,7,101,4,100,8,100,9,132,0,131,1,90,8,101,
+    4,100,17,100,11,100,12,132,1,131,1,90,9,101,4,100,
+    18,100,13,100,14,132,1,131,1,90,10,101,4,100,19,100,
+    15,100,16,132,1,131,1,90,11,100,10,83,0,41,20,218,
+    10,80,97,116,104,70,105,110,100,101,114,122,62,77,101,116,
+    97,32,112,97,116,104,32,102,105,110,100,101,114,32,102,111,
+    114,32,115,121,115,46,112,97,116,104,32,97,110,100,32,112,
+    97,99,107,97,103,101,32,95,95,112,97,116,104,95,95,32,
+    97,116,116,114,105,98,117,116,101,115,46,99,1,0,0,0,
+    0,0,0,0,2,0,0,0,4,0,0,0,67,0,0,0,
+    115,42,0,0,0,120,36,116,0,106,1,160,2,161,0,68,
+    0,93,22,125,1,116,3,124,1,100,1,131,2,114,12,124,
+    1,160,4,161,0,1,0,113,12,87,0,100,2,83,0,41,
+    3,122,125,67,97,108,108,32,116,104,101,32,105,110,118,97,
+    108,105,100,97,116,101,95,99,97,99,104,101,115,40,41,32,
+    109,101,116,104,111,100,32,111,110,32,97,108,108,32,112,97,
+    116,104,32,101,110,116,114,121,32,102,105,110,100,101,114,115,
+    10,32,32,32,32,32,32,32,32,115,116,111,114,101,100,32,
+    105,110,32,115,121,115,46,112,97,116,104,95,105,109,112,111,
+    114,116,101,114,95,99,97,99,104,101,115,32,40,119,104,101,
+    114,101,32,105,109,112,108,101,109,101,110,116,101,100,41,46,
+    218,17,105,110,118,97,108,105,100,97,116,101,95,99,97,99,
+    104,101,115,78,41,5,114,6,0,0,0,218,19,112,97,116,
+    104,95,105,109,112,111,114,116,101,114,95,99,97,99,104,101,
+    218,6,118,97,108,117,101,115,114,110,0,0,0,114,6,1,
+    0,0,41,2,114,172,0,0,0,218,6,102,105,110,100,101,
+    114,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,6,1,0,0,150,4,0,0,115,6,0,0,0,0,4,
+    16,1,10,1,122,28,80,97,116,104,70,105,110,100,101,114,
+    46,105,110,118,97,108,105,100,97,116,101,95,99,97,99,104,
+    101,115,99,2,0,0,0,0,0,0,0,3,0,0,0,9,
+    0,0,0,67,0,0,0,115,84,0,0,0,116,0,106,1,
+    100,1,107,9,114,28,116,0,106,1,115,28,116,2,160,3,
+    100,2,116,4,161,2,1,0,120,50,116,0,106,1,68,0,
+    93,36,125,2,121,8,124,2,124,1,131,1,83,0,4,0,
+    116,5,107,10,114,70,1,0,1,0,1,0,119,36,89,0,
+    113,36,88,0,113,36,87,0,100,1,83,0,100,1,83,0,
+    41,3,122,46,83,101,97,114,99,104,32,115,121,115,46,112,
+    97,116,104,95,104,111,111,107,115,32,102,111,114,32,97,32,
+    102,105,110,100,101,114,32,102,111,114,32,39,112,97,116,104,
+    39,46,78,122,23,115,121,115,46,112,97,116,104,95,104,111,
+    111,107,115,32,105,115,32,101,109,112,116,121,41,6,114,6,
+    0,0,0,218,10,112,97,116,104,95,104,111,111,107,115,114,
+    61,0,0,0,114,62,0,0,0,114,120,0,0,0,114,101,
+    0,0,0,41,3,114,172,0,0,0,114,35,0,0,0,90,
+    4,104,111,111,107,114,2,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,218,11,95,112,97,116,104,95,104,111,111,107,
+    115,158,4,0,0,115,16,0,0,0,0,3,16,1,12,1,
+    12,1,2,1,8,1,14,1,12,2,122,22,80,97,116,104,
+    70,105,110,100,101,114,46,95,112,97,116,104,95,104,111,111,
+    107,115,99,2,0,0,0,0,0,0,0,3,0,0,0,8,
+    0,0,0,67,0,0,0,115,102,0,0,0,124,1,100,1,
+    107,2,114,42,121,12,116,0,160,1,161,0,125,1,87,0,
+    110,20,4,0,116,2,107,10,114,40,1,0,1,0,1,0,
+    100,2,83,0,88,0,121,14,116,3,106,4,124,1,25,0,
+    125,2,87,0,110,40,4,0,116,5,107,10,114,96,1,0,
+    1,0,1,0,124,0,160,6,124,1,161,1,125,2,124,2,
+    116,3,106,4,124,1,60,0,89,0,110,2,88,0,124,2,
+    83,0,41,3,122,210,71,101,116,32,116,104,101,32,102,105,
+    110,100,101,114,32,102,111,114,32,116,104,101,32,112,97,116,
+    104,32,101,110,116,114,121,32,102,114,111,109,32,115,121,115,
+    46,112,97,116,104,95,105,109,112,111,114,116,101,114,95,99,
+    97,99,104,101,46,10,10,32,32,32,32,32,32,32,32,73,
+    102,32,116,104,101,32,112,97,116,104,32,101,110,116,114,121,
+    32,105,115,32,110,111,116,32,105,110,32,116,104,101,32,99,
+    97,99,104,101,44,32,102,105,110,100,32,116,104,101,32,97,
+    112,112,114,111,112,114,105,97,116,101,32,102,105,110,100,101,
+    114,10,32,32,32,32,32,32,32,32,97,110,100,32,99,97,
+    99,104,101,32,105,116,46,32,73,102,32,110,111,32,102,105,
+    110,100,101,114,32,105,115,32,97,118,97,105,108,97,98,108,
+    101,44,32,115,116,111,114,101,32,78,111,110,101,46,10,10,
+    32,32,32,32,32,32,32,32,114,30,0,0,0,78,41,7,
+    114,1,0,0,0,114,45,0,0,0,114,224,0,0,0,114,
+    6,0,0,0,114,7,1,0,0,218,8,75,101,121,69,114,
+    114,111,114,114,11,1,0,0,41,3,114,172,0,0,0,114,
+    35,0,0,0,114,9,1,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,218,20,95,112,97,116,104,95,
+    105,109,112,111,114,116,101,114,95,99,97,99,104,101,171,4,
+    0,0,115,22,0,0,0,0,8,8,1,2,1,12,1,14,
+    3,6,1,2,1,14,1,14,1,10,1,16,1,122,31,80,
+    97,116,104,70,105,110,100,101,114,46,95,112,97,116,104,95,
+    105,109,112,111,114,116,101,114,95,99,97,99,104,101,99,3,
+    0,0,0,0,0,0,0,6,0,0,0,4,0,0,0,67,
+    0,0,0,115,82,0,0,0,116,0,124,2,100,1,131,2,
+    114,26,124,2,160,1,124,1,161,1,92,2,125,3,125,4,
+    110,14,124,2,160,2,124,1,161,1,125,3,103,0,125,4,
+    124,3,100,0,107,9,114,60,116,3,160,4,124,1,124,3,
+    161,2,83,0,116,3,160,5,124,1,100,0,161,2,125,5,
+    124,4,124,5,95,6,124,5,83,0,41,2,78,114,119,0,
+    0,0,41,7,114,110,0,0,0,114,119,0,0,0,114,183,
+    0,0,0,114,116,0,0,0,114,180,0,0,0,114,162,0,
+    0,0,114,158,0,0,0,41,6,114,172,0,0,0,114,121,
+    0,0,0,114,9,1,0,0,114,122,0,0,0,114,123,0,
+    0,0,114,166,0,0,0,114,2,0,0,0,114,2,0,0,
+    0,114,4,0,0,0,218,16,95,108,101,103,97,99,121,95,
+    103,101,116,95,115,112,101,99,193,4,0,0,115,18,0,0,
+    0,0,4,10,1,16,2,10,1,4,1,8,1,12,1,12,
+    1,6,1,122,27,80,97,116,104,70,105,110,100,101,114,46,
+    95,108,101,103,97,99,121,95,103,101,116,95,115,112,101,99,
+    78,99,4,0,0,0,0,0,0,0,9,0,0,0,5,0,
+    0,0,67,0,0,0,115,170,0,0,0,103,0,125,4,120,
+    160,124,2,68,0,93,130,125,5,116,0,124,5,116,1,116,
+    2,102,2,131,2,115,30,113,10,124,0,160,3,124,5,161,
+    1,125,6,124,6,100,1,107,9,114,10,116,4,124,6,100,
+    2,131,2,114,72,124,6,160,5,124,1,124,3,161,2,125,
+    7,110,12,124,0,160,6,124,1,124,6,161,2,125,7,124,
+    7,100,1,107,8,114,94,113,10,124,7,106,7,100,1,107,
+    9,114,108,124,7,83,0,124,7,106,8,125,8,124,8,100,
+    1,107,8,114,130,116,9,100,3,131,1,130,1,124,4,160,
+    10,124,8,161,1,1,0,113,10,87,0,116,11,160,12,124,
+    1,100,1,161,2,125,7,124,4,124,7,95,8,124,7,83,
+    0,100,1,83,0,41,4,122,63,70,105,110,100,32,116,104,
+    101,32,108,111,97,100,101,114,32,111,114,32,110,97,109,101,
+    115,112,97,99,101,95,112,97,116,104,32,102,111,114,32,116,
+    104,105,115,32,109,111,100,117,108,101,47,112,97,99,107,97,
+    103,101,32,110,97,109,101,46,78,114,182,0,0,0,122,19,
+    115,112,101,99,32,109,105,115,115,105,110,103,32,108,111,97,
+    100,101,114,41,13,114,141,0,0,0,114,71,0,0,0,218,
+    5,98,121,116,101,115,114,13,1,0,0,114,110,0,0,0,
+    114,182,0,0,0,114,14,1,0,0,114,122,0,0,0,114,
+    158,0,0,0,114,101,0,0,0,114,147,0,0,0,114,116,
+    0,0,0,114,162,0,0,0,41,9,114,172,0,0,0,114,
+    121,0,0,0,114,35,0,0,0,114,181,0,0,0,218,14,
+    110,97,109,101,115,112,97,99,101,95,112,97,116,104,90,5,
+    101,110,116,114,121,114,9,1,0,0,114,166,0,0,0,114,
+    123,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
+    0,0,0,218,9,95,103,101,116,95,115,112,101,99,208,4,
+    0,0,115,40,0,0,0,0,5,4,1,10,1,14,1,2,
+    1,10,1,8,1,10,1,14,2,12,1,8,1,2,1,10,
+    1,4,1,6,1,8,1,8,5,14,2,12,1,6,1,122,
+    20,80,97,116,104,70,105,110,100,101,114,46,95,103,101,116,
+    95,115,112,101,99,99,4,0,0,0,0,0,0,0,6,0,
+    0,0,5,0,0,0,67,0,0,0,115,100,0,0,0,124,
+    2,100,1,107,8,114,14,116,0,106,1,125,2,124,0,160,
+    2,124,1,124,2,124,3,161,3,125,4,124,4,100,1,107,
+    8,114,40,100,1,83,0,124,4,106,3,100,1,107,8,114,
+    92,124,4,106,4,125,5,124,5,114,86,100,2,124,4,95,
+    5,116,6,124,1,124,5,124,0,106,2,131,3,124,4,95,
+    4,124,4,83,0,100,1,83,0,110,4,124,4,83,0,100,
+    1,83,0,41,3,122,141,84,114,121,32,116,111,32,102,105,
+    110,100,32,97,32,115,112,101,99,32,102,111,114,32,39,102,
+    117,108,108,110,97,109,101,39,32,111,110,32,115,121,115,46,
+    112,97,116,104,32,111,114,32,39,112,97,116,104,39,46,10,
+    10,32,32,32,32,32,32,32,32,84,104,101,32,115,101,97,
+    114,99,104,32,105,115,32,98,97,115,101,100,32,111,110,32,
+    115,121,115,46,112,97,116,104,95,104,111,111,107,115,32,97,
+    110,100,32,115,121,115,46,112,97,116,104,95,105,109,112,111,
+    114,116,101,114,95,99,97,99,104,101,46,10,32,32,32,32,
+    32,32,32,32,78,90,9,110,97,109,101,115,112,97,99,101,
+    41,7,114,6,0,0,0,114,35,0,0,0,114,17,1,0,
+    0,114,122,0,0,0,114,158,0,0,0,114,160,0,0,0,
+    114,241,0,0,0,41,6,114,172,0,0,0,114,121,0,0,
+    0,114,35,0,0,0,114,181,0,0,0,114,166,0,0,0,
+    114,16,1,0,0,114,2,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,114,182,0,0,0,240,4,0,0,115,26,0,
+    0,0,0,6,8,1,6,1,14,1,8,1,4,1,10,1,
+    6,1,4,3,6,1,16,1,4,2,6,2,122,20,80,97,
+    116,104,70,105,110,100,101,114,46,102,105,110,100,95,115,112,
+    101,99,99,3,0,0,0,0,0,0,0,4,0,0,0,4,
+    0,0,0,67,0,0,0,115,30,0,0,0,124,0,160,0,
+    124,1,124,2,161,2,125,3,124,3,100,1,107,8,114,24,
+    100,1,83,0,124,3,106,1,83,0,41,2,122,170,102,105,
+    110,100,32,116,104,101,32,109,111,100,117,108,101,32,111,110,
+    32,115,121,115,46,112,97,116,104,32,111,114,32,39,112,97,
+    116,104,39,32,98,97,115,101,100,32,111,110,32,115,121,115,
+    46,112,97,116,104,95,104,111,111,107,115,32,97,110,100,10,
+    32,32,32,32,32,32,32,32,115,121,115,46,112,97,116,104,
+    95,105,109,112,111,114,116,101,114,95,99,97,99,104,101,46,
+    10,10,32,32,32,32,32,32,32,32,84,104,105,115,32,109,
+    101,116,104,111,100,32,105,115,32,100,101,112,114,101,99,97,
+    116,101,100,46,32,32,85,115,101,32,102,105,110,100,95,115,
+    112,101,99,40,41,32,105,110,115,116,101,97,100,46,10,10,
+    32,32,32,32,32,32,32,32,78,41,2,114,182,0,0,0,
+    114,122,0,0,0,41,4,114,172,0,0,0,114,121,0,0,
+    0,114,35,0,0,0,114,166,0,0,0,114,2,0,0,0,
+    114,2,0,0,0,114,4,0,0,0,114,183,0,0,0,8,
+    5,0,0,115,8,0,0,0,0,8,12,1,8,1,4,1,
+    122,22,80,97,116,104,70,105,110,100,101,114,46,102,105,110,
+    100,95,109,111,100,117,108,101,41,1,78,41,2,78,78,41,
+    1,78,41,12,114,107,0,0,0,114,106,0,0,0,114,108,
+    0,0,0,114,109,0,0,0,114,184,0,0,0,114,6,1,
+    0,0,114,11,1,0,0,114,13,1,0,0,114,14,1,0,
+    0,114,17,1,0,0,114,182,0,0,0,114,183,0,0,0,
+    114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,114,5,1,0,0,146,4,0,0,115,20,0,
+    0,0,12,4,12,8,12,13,12,22,12,15,2,1,12,31,
+    2,1,12,23,2,1,114,5,1,0,0,99,0,0,0,0,
+    0,0,0,0,0,0,0,0,3,0,0,0,64,0,0,0,
+    115,90,0,0,0,101,0,90,1,100,0,90,2,100,1,90,
+    3,100,2,100,3,132,0,90,4,100,4,100,5,132,0,90,
+    5,101,6,90,7,100,6,100,7,132,0,90,8,100,8,100,
+    9,132,0,90,9,100,19,100,11,100,12,132,1,90,10,100,
+    13,100,14,132,0,90,11,101,12,100,15,100,16,132,0,131,
+    1,90,13,100,17,100,18,132,0,90,14,100,10,83,0,41,
+    20,218,10,70,105,108,101,70,105,110,100,101,114,122,172,70,
+    105,108,101,45,98,97,115,101,100,32,102,105,110,100,101,114,
+    46,10,10,32,32,32,32,73,110,116,101,114,97,99,116,105,
+    111,110,115,32,119,105,116,104,32,116,104,101,32,102,105,108,
+    101,32,115,121,115,116,101,109,32,97,114,101,32,99,97,99,
+    104,101,100,32,102,111,114,32,112,101,114,102,111,114,109,97,
+    110,99,101,44,32,98,101,105,110,103,10,32,32,32,32,114,
+    101,102,114,101,115,104,101,100,32,119,104,101,110,32,116,104,
+    101,32,100,105,114,101,99,116,111,114,121,32,116,104,101,32,
+    102,105,110,100,101,114,32,105,115,32,104,97,110,100,108,105,
+    110,103,32,104,97,115,32,98,101,101,110,32,109,111,100,105,
+    102,105,101,100,46,10,10,32,32,32,32,99,2,0,0,0,
+    0,0,0,0,5,0,0,0,6,0,0,0,7,0,0,0,
+    115,88,0,0,0,103,0,125,3,120,40,124,2,68,0,93,
+    32,92,2,137,0,125,4,124,3,160,0,135,0,102,1,100,
+    1,100,2,132,8,124,4,68,0,131,1,161,1,1,0,113,
+    10,87,0,124,3,124,0,95,1,124,1,112,58,100,3,124,
+    0,95,2,100,4,124,0,95,3,116,4,131,0,124,0,95,
+    5,116,4,131,0,124,0,95,6,100,5,83,0,41,6,122,
+    154,73,110,105,116,105,97,108,105,122,101,32,119,105,116,104,
+    32,116,104,101,32,112,97,116,104,32,116,111,32,115,101,97,
+    114,99,104,32,111,110,32,97,110,100,32,97,32,118,97,114,
+    105,97,98,108,101,32,110,117,109,98,101,114,32,111,102,10,
+    32,32,32,32,32,32,32,32,50,45,116,117,112,108,101,115,
+    32,99,111,110,116,97,105,110,105,110,103,32,116,104,101,32,
+    108,111,97,100,101,114,32,97,110,100,32,116,104,101,32,102,
+    105,108,101,32,115,117,102,102,105,120,101,115,32,116,104,101,
+    32,108,111,97,100,101,114,10,32,32,32,32,32,32,32,32,
+    114,101,99,111,103,110,105,122,101,115,46,99,1,0,0,0,
+    0,0,0,0,2,0,0,0,3,0,0,0,51,0,0,0,
+    115,22,0,0,0,124,0,93,14,125,1,124,1,136,0,102,
+    2,86,0,1,0,113,2,100,0,83,0,41,1,78,114,2,
+    0,0,0,41,2,114,22,0,0,0,114,236,0,0,0,41,
+    1,114,122,0,0,0,114,2,0,0,0,114,4,0,0,0,
+    114,238,0,0,0,37,5,0,0,115,2,0,0,0,4,0,
+    122,38,70,105,108,101,70,105,110,100,101,114,46,95,95,105,
+    110,105,116,95,95,46,60,108,111,99,97,108,115,62,46,60,
+    103,101,110,101,120,112,114,62,114,59,0,0,0,114,89,0,
+    0,0,78,41,7,114,147,0,0,0,218,8,95,108,111,97,
+    100,101,114,115,114,35,0,0,0,218,11,95,112,97,116,104,
+    95,109,116,105,109,101,218,3,115,101,116,218,11,95,112,97,
+    116,104,95,99,97,99,104,101,218,19,95,114,101,108,97,120,
+    101,100,95,112,97,116,104,95,99,97,99,104,101,41,5,114,
+    102,0,0,0,114,35,0,0,0,218,14,108,111,97,100,101,
+    114,95,100,101,116,97,105,108,115,90,7,108,111,97,100,101,
+    114,115,114,168,0,0,0,114,2,0,0,0,41,1,114,122,
+    0,0,0,114,4,0,0,0,114,186,0,0,0,31,5,0,
+    0,115,16,0,0,0,0,4,4,1,14,1,28,1,6,2,
+    10,1,6,1,8,1,122,19,70,105,108,101,70,105,110,100,
+    101,114,46,95,95,105,110,105,116,95,95,99,1,0,0,0,
+    0,0,0,0,1,0,0,0,2,0,0,0,67,0,0,0,
+    115,10,0,0,0,100,1,124,0,95,0,100,2,83,0,41,
+    3,122,31,73,110,118,97,108,105,100,97,116,101,32,116,104,
+    101,32,100,105,114,101,99,116,111,114,121,32,109,116,105,109,
+    101,46,114,89,0,0,0,78,41,1,114,20,1,0,0,41,
+    1,114,102,0,0,0,114,2,0,0,0,114,2,0,0,0,
+    114,4,0,0,0,114,6,1,0,0,45,5,0,0,115,2,
+    0,0,0,0,2,122,28,70,105,108,101,70,105,110,100,101,
+    114,46,105,110,118,97,108,105,100,97,116,101,95,99,97,99,
+    104,101,115,99,2,0,0,0,0,0,0,0,3,0,0,0,
+    3,0,0,0,67,0,0,0,115,42,0,0,0,124,0,160,
+    0,124,1,161,1,125,2,124,2,100,1,107,8,114,26,100,
+    1,103,0,102,2,83,0,124,2,106,1,124,2,106,2,112,
+    38,103,0,102,2,83,0,41,2,122,197,84,114,121,32,116,
+    111,32,102,105,110,100,32,97,32,108,111,97,100,101,114,32,
+    102,111,114,32,116,104,101,32,115,112,101,99,105,102,105,101,
+    100,32,109,111,100,117,108,101,44,32,111,114,32,116,104,101,
+    32,110,97,109,101,115,112,97,99,101,10,32,32,32,32,32,
+    32,32,32,112,97,99,107,97,103,101,32,112,111,114,116,105,
+    111,110,115,46,32,82,101,116,117,114,110,115,32,40,108,111,
+    97,100,101,114,44,32,108,105,115,116,45,111,102,45,112,111,
+    114,116,105,111,110,115,41,46,10,10,32,32,32,32,32,32,
+    32,32,84,104,105,115,32,109,101,116,104,111,100,32,105,115,
+    32,100,101,112,114,101,99,97,116,101,100,46,32,32,85,115,
+    101,32,102,105,110,100,95,115,112,101,99,40,41,32,105,110,
+    115,116,101,97,100,46,10,10,32,32,32,32,32,32,32,32,
+    78,41,3,114,182,0,0,0,114,122,0,0,0,114,158,0,
+    0,0,41,3,114,102,0,0,0,114,121,0,0,0,114,166,
+    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
+    0,0,114,119,0,0,0,51,5,0,0,115,8,0,0,0,
+    0,7,10,1,8,1,8,1,122,22,70,105,108,101,70,105,
+    110,100,101,114,46,102,105,110,100,95,108,111,97,100,101,114,
+    99,6,0,0,0,0,0,0,0,7,0,0,0,6,0,0,
+    0,67,0,0,0,115,26,0,0,0,124,1,124,2,124,3,
+    131,2,125,6,116,0,124,2,124,3,124,6,124,4,100,1,
+    141,4,83,0,41,2,78,41,2,114,122,0,0,0,114,158,
+    0,0,0,41,1,114,169,0,0,0,41,7,114,102,0,0,
+    0,114,167,0,0,0,114,121,0,0,0,114,35,0,0,0,
+    90,4,115,109,115,108,114,181,0,0,0,114,122,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    254,0,0,0,124,4,0,0,115,6,0,0,0,0,4,16,
-    1,10,1,122,28,80,97,116,104,70,105,110,100,101,114,46,
-    105,110,118,97,108,105,100,97,116,101,95,99,97,99,104,101,
-    115,99,2,0,0,0,0,0,0,0,3,0,0,0,9,0,
-    0,0,67,0,0,0,115,84,0,0,0,116,0,106,1,100,
-    1,107,9,114,28,116,0,106,1,115,28,116,2,160,3,100,
-    2,116,4,161,2,1,0,120,50,116,0,106,1,68,0,93,
-    36,125,2,121,8,124,2,124,1,131,1,83,0,4,0,116,
-    5,107,10,114,70,1,0,1,0,1,0,119,36,89,0,113,
-    36,88,0,113,36,87,0,100,1,83,0,100,1,83,0,41,
-    3,122,46,83,101,97,114,99,104,32,115,121,115,46,112,97,
-    116,104,95,104,111,111,107,115,32,102,111,114,32,97,32,102,
-    105,110,100,101,114,32,102,111,114,32,39,112,97,116,104,39,
-    46,78,122,23,115,121,115,46,112,97,116,104,95,104,111,111,
-    107,115,32,105,115,32,101,109,112,116,121,41,6,114,6,0,
-    0,0,218,10,112,97,116,104,95,104,111,111,107,115,114,61,
-    0,0,0,114,62,0,0,0,114,120,0,0,0,114,101,0,
-    0,0,41,3,114,172,0,0,0,114,35,0,0,0,90,4,
-    104,111,111,107,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,218,11,95,112,97,116,104,95,104,111,111,107,115,
-    132,4,0,0,115,16,0,0,0,0,3,16,1,12,1,12,
-    1,2,1,8,1,14,1,12,2,122,22,80,97,116,104,70,
-    105,110,100,101,114,46,95,112,97,116,104,95,104,111,111,107,
-    115,99,2,0,0,0,0,0,0,0,3,0,0,0,8,0,
-    0,0,67,0,0,0,115,102,0,0,0,124,1,100,1,107,
-    2,114,42,121,12,116,0,160,1,161,0,125,1,87,0,110,
-    20,4,0,116,2,107,10,114,40,1,0,1,0,1,0,100,
-    2,83,0,88,0,121,14,116,3,106,4,124,1,25,0,125,
-    2,87,0,110,40,4,0,116,5,107,10,114,96,1,0,1,
-    0,1,0,124,0,160,6,124,1,161,1,125,2,124,2,116,
-    3,106,4,124,1,60,0,89,0,110,2,88,0,124,2,83,
-    0,41,3,122,210,71,101,116,32,116,104,101,32,102,105,110,
-    100,101,114,32,102,111,114,32,116,104,101,32,112,97,116,104,
-    32,101,110,116,114,121,32,102,114,111,109,32,115,121,115,46,
-    112,97,116,104,95,105,109,112,111,114,116,101,114,95,99,97,
-    99,104,101,46,10,10,32,32,32,32,32,32,32,32,73,102,
-    32,116,104,101,32,112,97,116,104,32,101,110,116,114,121,32,
-    105,115,32,110,111,116,32,105,110,32,116,104,101,32,99,97,
-    99,104,101,44,32,102,105,110,100,32,116,104,101,32,97,112,
-    112,114,111,112,114,105,97,116,101,32,102,105,110,100,101,114,
-    10,32,32,32,32,32,32,32,32,97,110,100,32,99,97,99,
-    104,101,32,105,116,46,32,73,102,32,110,111,32,102,105,110,
-    100,101,114,32,105,115,32,97,118,97,105,108,97,98,108,101,
-    44,32,115,116,111,114,101,32,78,111,110,101,46,10,10,32,
-    32,32,32,32,32,32,32,114,30,0,0,0,78,41,7,114,
-    1,0,0,0,114,45,0,0,0,218,17,70,105,108,101,78,
-    111,116,70,111,117,110,100,69,114,114,111,114,114,6,0,0,
-    0,114,255,0,0,0,218,8,75,101,121,69,114,114,111,114,
-    114,3,1,0,0,41,3,114,172,0,0,0,114,35,0,0,
-    0,114,1,1,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,218,20,95,112,97,116,104,95,105,109,112,
-    111,114,116,101,114,95,99,97,99,104,101,145,4,0,0,115,
-    22,0,0,0,0,8,8,1,2,1,12,1,14,3,6,1,
-    2,1,14,1,14,1,10,1,16,1,122,31,80,97,116,104,
-    70,105,110,100,101,114,46,95,112,97,116,104,95,105,109,112,
-    111,114,116,101,114,95,99,97,99,104,101,99,3,0,0,0,
-    0,0,0,0,6,0,0,0,4,0,0,0,67,0,0,0,
-    115,82,0,0,0,116,0,124,2,100,1,131,2,114,26,124,
-    2,160,1,124,1,161,1,92,2,125,3,125,4,110,14,124,
-    2,160,2,124,1,161,1,125,3,103,0,125,4,124,3,100,
-    0,107,9,114,60,116,3,160,4,124,1,124,3,161,2,83,
-    0,116,3,160,5,124,1,100,0,161,2,125,5,124,4,124,
-    5,95,6,124,5,83,0,41,2,78,114,119,0,0,0,41,
-    7,114,110,0,0,0,114,119,0,0,0,114,183,0,0,0,
-    114,116,0,0,0,114,180,0,0,0,114,162,0,0,0,114,
-    158,0,0,0,41,6,114,172,0,0,0,114,121,0,0,0,
-    114,1,1,0,0,114,122,0,0,0,114,123,0,0,0,114,
+    17,1,0,0,63,5,0,0,115,6,0,0,0,0,1,10,
+    1,8,1,122,20,70,105,108,101,70,105,110,100,101,114,46,
+    95,103,101,116,95,115,112,101,99,78,99,3,0,0,0,0,
+    0,0,0,14,0,0,0,8,0,0,0,67,0,0,0,115,
+    98,1,0,0,100,1,125,3,124,1,160,0,100,2,161,1,
+    100,3,25,0,125,4,121,24,116,1,124,0,106,2,112,34,
+    116,3,160,4,161,0,131,1,106,5,125,5,87,0,110,24,
+    4,0,116,6,107,10,114,66,1,0,1,0,1,0,100,4,
+    125,5,89,0,110,2,88,0,124,5,124,0,106,7,107,3,
+    114,92,124,0,160,8,161,0,1,0,124,5,124,0,95,7,
+    116,9,131,0,114,114,124,0,106,10,125,6,124,4,160,11,
+    161,0,125,7,110,10,124,0,106,12,125,6,124,4,125,7,
+    124,7,124,6,107,6,114,218,116,13,124,0,106,2,124,4,
+    131,2,125,8,120,72,124,0,106,14,68,0,93,54,92,2,
+    125,9,125,10,100,5,124,9,23,0,125,11,116,13,124,8,
+    124,11,131,2,125,12,116,15,124,12,131,1,114,152,124,0,
+    160,16,124,10,124,1,124,12,124,8,103,1,124,2,161,5,
+    83,0,113,152,87,0,116,17,124,8,131,1,125,3,120,88,
+    124,0,106,14,68,0,93,78,92,2,125,9,125,10,116,13,
+    124,0,106,2,124,4,124,9,23,0,131,2,125,12,116,18,
+    106,19,100,6,124,12,100,3,100,7,141,3,1,0,124,7,
+    124,9,23,0,124,6,107,6,114,226,116,15,124,12,131,1,
+    114,226,124,0,160,16,124,10,124,1,124,12,100,8,124,2,
+    161,5,83,0,113,226,87,0,124,3,144,1,114,94,116,18,
+    160,19,100,9,124,8,161,2,1,0,116,18,160,20,124,1,
+    100,8,161,2,125,13,124,8,103,1,124,13,95,21,124,13,
+    83,0,100,8,83,0,41,10,122,111,84,114,121,32,116,111,
+    32,102,105,110,100,32,97,32,115,112,101,99,32,102,111,114,
+    32,116,104,101,32,115,112,101,99,105,102,105,101,100,32,109,
+    111,100,117,108,101,46,10,10,32,32,32,32,32,32,32,32,
+    82,101,116,117,114,110,115,32,116,104,101,32,109,97,116,99,
+    104,105,110,103,32,115,112,101,99,44,32,111,114,32,78,111,
+    110,101,32,105,102,32,110,111,116,32,102,111,117,110,100,46,
+    10,32,32,32,32,32,32,32,32,70,114,59,0,0,0,114,
+    57,0,0,0,114,89,0,0,0,114,186,0,0,0,122,9,
+    116,114,121,105,110,103,32,123,125,41,1,90,9,118,101,114,
+    98,111,115,105,116,121,78,122,25,112,111,115,115,105,98,108,
+    101,32,110,97,109,101,115,112,97,99,101,32,102,111,114,32,
+    123,125,41,22,114,32,0,0,0,114,39,0,0,0,114,35,
+    0,0,0,114,1,0,0,0,114,45,0,0,0,114,230,0,
+    0,0,114,40,0,0,0,114,20,1,0,0,218,11,95,102,
+    105,108,108,95,99,97,99,104,101,114,5,0,0,0,114,23,
+    1,0,0,114,90,0,0,0,114,22,1,0,0,114,28,0,
+    0,0,114,19,1,0,0,114,44,0,0,0,114,17,1,0,
+    0,114,46,0,0,0,114,116,0,0,0,114,130,0,0,0,
+    114,162,0,0,0,114,158,0,0,0,41,14,114,102,0,0,
+    0,114,121,0,0,0,114,181,0,0,0,90,12,105,115,95,
+    110,97,109,101,115,112,97,99,101,90,11,116,97,105,108,95,
+    109,111,100,117,108,101,114,149,0,0,0,90,5,99,97,99,
+    104,101,90,12,99,97,99,104,101,95,109,111,100,117,108,101,
+    90,9,98,97,115,101,95,112,97,116,104,114,236,0,0,0,
+    114,167,0,0,0,90,13,105,110,105,116,95,102,105,108,101,
+    110,97,109,101,90,9,102,117,108,108,95,112,97,116,104,114,
     166,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,218,16,95,108,101,103,97,99,121,95,103,101,116,
-    95,115,112,101,99,167,4,0,0,115,18,0,0,0,0,4,
-    10,1,16,2,10,1,4,1,8,1,12,1,12,1,6,1,
-    122,27,80,97,116,104,70,105,110,100,101,114,46,95,108,101,
-    103,97,99,121,95,103,101,116,95,115,112,101,99,78,99,4,
-    0,0,0,0,0,0,0,9,0,0,0,5,0,0,0,67,
-    0,0,0,115,170,0,0,0,103,0,125,4,120,160,124,2,
-    68,0,93,130,125,5,116,0,124,5,116,1,116,2,102,2,
-    131,2,115,30,113,10,124,0,160,3,124,5,161,1,125,6,
-    124,6,100,1,107,9,114,10,116,4,124,6,100,2,131,2,
-    114,72,124,6,160,5,124,1,124,3,161,2,125,7,110,12,
-    124,0,160,6,124,1,124,6,161,2,125,7,124,7,100,1,
-    107,8,114,94,113,10,124,7,106,7,100,1,107,9,114,108,
-    124,7,83,0,124,7,106,8,125,8,124,8,100,1,107,8,
-    114,130,116,9,100,3,131,1,130,1,124,4,160,10,124,8,
-    161,1,1,0,113,10,87,0,116,11,160,12,124,1,100,1,
-    161,2,125,7,124,4,124,7,95,8,124,7,83,0,100,1,
-    83,0,41,4,122,63,70,105,110,100,32,116,104,101,32,108,
-    111,97,100,101,114,32,111,114,32,110,97,109,101,115,112,97,
-    99,101,95,112,97,116,104,32,102,111,114,32,116,104,105,115,
-    32,109,111,100,117,108,101,47,112,97,99,107,97,103,101,32,
-    110,97,109,101,46,78,114,182,0,0,0,122,19,115,112,101,
-    99,32,109,105,115,115,105,110,103,32,108,111,97,100,101,114,
-    41,13,114,141,0,0,0,114,71,0,0,0,218,5,98,121,
-    116,101,115,114,6,1,0,0,114,110,0,0,0,114,182,0,
-    0,0,114,7,1,0,0,114,122,0,0,0,114,158,0,0,
-    0,114,101,0,0,0,114,147,0,0,0,114,116,0,0,0,
-    114,162,0,0,0,41,9,114,172,0,0,0,114,121,0,0,
-    0,114,35,0,0,0,114,181,0,0,0,218,14,110,97,109,
-    101,115,112,97,99,101,95,112,97,116,104,90,5,101,110,116,
-    114,121,114,1,1,0,0,114,166,0,0,0,114,123,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,9,95,103,101,116,95,115,112,101,99,182,4,0,0,115,
-    40,0,0,0,0,5,4,1,10,1,14,1,2,1,10,1,
-    8,1,10,1,14,2,12,1,8,1,2,1,10,1,4,1,
-    6,1,8,1,8,5,14,2,12,1,6,1,122,20,80,97,
-    116,104,70,105,110,100,101,114,46,95,103,101,116,95,115,112,
-    101,99,99,4,0,0,0,0,0,0,0,6,0,0,0,5,
-    0,0,0,67,0,0,0,115,100,0,0,0,124,2,100,1,
-    107,8,114,14,116,0,106,1,125,2,124,0,160,2,124,1,
-    124,2,124,3,161,3,125,4,124,4,100,1,107,8,114,40,
-    100,1,83,0,124,4,106,3,100,1,107,8,114,92,124,4,
-    106,4,125,5,124,5,114,86,100,2,124,4,95,5,116,6,
-    124,1,124,5,124,0,106,2,131,3,124,4,95,4,124,4,
-    83,0,100,1,83,0,110,4,124,4,83,0,100,1,83,0,
-    41,3,122,141,84,114,121,32,116,111,32,102,105,110,100,32,
-    97,32,115,112,101,99,32,102,111,114,32,39,102,117,108,108,
-    110,97,109,101,39,32,111,110,32,115,121,115,46,112,97,116,
-    104,32,111,114,32,39,112,97,116,104,39,46,10,10,32,32,
-    32,32,32,32,32,32,84,104,101,32,115,101,97,114,99,104,
-    32,105,115,32,98,97,115,101,100,32,111,110,32,115,121,115,
-    46,112,97,116,104,95,104,111,111,107,115,32,97,110,100,32,
-    115,121,115,46,112,97,116,104,95,105,109,112,111,114,116,101,
-    114,95,99,97,99,104,101,46,10,32,32,32,32,32,32,32,
-    32,78,90,9,110,97,109,101,115,112,97,99,101,41,7,114,
-    6,0,0,0,114,35,0,0,0,114,10,1,0,0,114,122,
-    0,0,0,114,158,0,0,0,114,160,0,0,0,114,232,0,
-    0,0,41,6,114,172,0,0,0,114,121,0,0,0,114,35,
-    0,0,0,114,181,0,0,0,114,166,0,0,0,114,9,1,
+    0,0,0,114,182,0,0,0,68,5,0,0,115,70,0,0,
+    0,0,5,4,1,14,1,2,1,24,1,14,1,10,1,10,
+    1,8,1,6,2,6,1,6,1,10,2,6,1,4,2,8,
+    1,12,1,16,1,8,1,10,1,8,1,24,4,8,2,16,
+    1,16,1,16,1,12,1,8,1,10,1,12,1,6,1,12,
+    1,12,1,8,1,4,1,122,20,70,105,108,101,70,105,110,
+    100,101,114,46,102,105,110,100,95,115,112,101,99,99,1,0,
+    0,0,0,0,0,0,9,0,0,0,10,0,0,0,67,0,
+    0,0,115,194,0,0,0,124,0,106,0,125,1,121,22,116,
+    1,160,2,124,1,112,22,116,1,160,3,161,0,161,1,125,
+    2,87,0,110,30,4,0,116,4,116,5,116,6,102,3,107,
+    10,114,58,1,0,1,0,1,0,103,0,125,2,89,0,110,
+    2,88,0,116,7,106,8,160,9,100,1,161,1,115,84,116,
+    10,124,2,131,1,124,0,95,11,110,78,116,10,131,0,125,
+    3,120,64,124,2,68,0,93,56,125,4,124,4,160,12,100,
+    2,161,1,92,3,125,5,125,6,125,7,124,6,114,138,100,
+    3,160,13,124,5,124,7,160,14,161,0,161,2,125,8,110,
+    4,124,5,125,8,124,3,160,15,124,8,161,1,1,0,113,
+    96,87,0,124,3,124,0,95,11,116,7,106,8,160,9,116,
+    16,161,1,114,190,100,4,100,5,132,0,124,2,68,0,131,
+    1,124,0,95,17,100,6,83,0,41,7,122,68,70,105,108,
+    108,32,116,104,101,32,99,97,99,104,101,32,111,102,32,112,
+    111,116,101,110,116,105,97,108,32,109,111,100,117,108,101,115,
+    32,97,110,100,32,112,97,99,107,97,103,101,115,32,102,111,
+    114,32,116,104,105,115,32,100,105,114,101,99,116,111,114,121,
+    46,114,0,0,0,0,114,59,0,0,0,122,5,123,125,46,
+    123,125,99,1,0,0,0,0,0,0,0,2,0,0,0,4,
+    0,0,0,83,0,0,0,115,20,0,0,0,104,0,124,0,
+    93,12,125,1,124,1,160,0,161,0,146,2,113,4,83,0,
+    114,2,0,0,0,41,1,114,90,0,0,0,41,2,114,22,
+    0,0,0,90,2,102,110,114,2,0,0,0,114,2,0,0,
+    0,114,4,0,0,0,250,9,60,115,101,116,99,111,109,112,
+    62,145,5,0,0,115,2,0,0,0,6,0,122,41,70,105,
+    108,101,70,105,110,100,101,114,46,95,102,105,108,108,95,99,
+    97,99,104,101,46,60,108,111,99,97,108,115,62,46,60,115,
+    101,116,99,111,109,112,62,78,41,18,114,35,0,0,0,114,
+    1,0,0,0,114,227,0,0,0,114,45,0,0,0,114,224,
+    0,0,0,218,15,80,101,114,109,105,115,115,105,111,110,69,
+    114,114,111,114,218,18,78,111,116,65,68,105,114,101,99,116,
+    111,114,121,69,114,114,111,114,114,6,0,0,0,114,7,0,
+    0,0,114,8,0,0,0,114,21,1,0,0,114,22,1,0,
+    0,114,85,0,0,0,114,48,0,0,0,114,90,0,0,0,
+    218,3,97,100,100,114,9,0,0,0,114,23,1,0,0,41,
+    9,114,102,0,0,0,114,35,0,0,0,114,228,0,0,0,
+    90,21,108,111,119,101,114,95,115,117,102,102,105,120,95,99,
+    111,110,116,101,110,116,115,114,1,1,0,0,114,100,0,0,
+    0,114,248,0,0,0,114,236,0,0,0,90,8,110,101,119,
+    95,110,97,109,101,114,2,0,0,0,114,2,0,0,0,114,
+    4,0,0,0,114,25,1,0,0,116,5,0,0,115,34,0,
+    0,0,0,2,6,1,2,1,22,1,20,3,10,3,12,1,
+    12,7,6,1,10,1,16,1,4,1,18,2,4,1,14,1,
+    6,1,12,1,122,22,70,105,108,101,70,105,110,100,101,114,
+    46,95,102,105,108,108,95,99,97,99,104,101,99,1,0,0,
+    0,0,0,0,0,3,0,0,0,3,0,0,0,7,0,0,
+    0,115,18,0,0,0,135,0,135,1,102,2,100,1,100,2,
+    132,8,125,2,124,2,83,0,41,3,97,20,1,0,0,65,
+    32,99,108,97,115,115,32,109,101,116,104,111,100,32,119,104,
+    105,99,104,32,114,101,116,117,114,110,115,32,97,32,99,108,
+    111,115,117,114,101,32,116,111,32,117,115,101,32,111,110,32,
+    115,121,115,46,112,97,116,104,95,104,111,111,107,10,32,32,
+    32,32,32,32,32,32,119,104,105,99,104,32,119,105,108,108,
+    32,114,101,116,117,114,110,32,97,110,32,105,110,115,116,97,
+    110,99,101,32,117,115,105,110,103,32,116,104,101,32,115,112,
+    101,99,105,102,105,101,100,32,108,111,97,100,101,114,115,32,
+    97,110,100,32,116,104,101,32,112,97,116,104,10,32,32,32,
+    32,32,32,32,32,99,97,108,108,101,100,32,111,110,32,116,
+    104,101,32,99,108,111,115,117,114,101,46,10,10,32,32,32,
+    32,32,32,32,32,73,102,32,116,104,101,32,112,97,116,104,
+    32,99,97,108,108,101,100,32,111,110,32,116,104,101,32,99,
+    108,111,115,117,114,101,32,105,115,32,110,111,116,32,97,32,
+    100,105,114,101,99,116,111,114,121,44,32,73,109,112,111,114,
+    116,69,114,114,111,114,32,105,115,10,32,32,32,32,32,32,
+    32,32,114,97,105,115,101,100,46,10,10,32,32,32,32,32,
+    32,32,32,99,1,0,0,0,0,0,0,0,1,0,0,0,
+    4,0,0,0,19,0,0,0,115,34,0,0,0,116,0,124,
+    0,131,1,115,20,116,1,100,1,124,0,100,2,141,2,130,
+    1,136,0,124,0,102,1,136,1,158,2,142,0,83,0,41,
+    3,122,45,80,97,116,104,32,104,111,111,107,32,102,111,114,
+    32,105,109,112,111,114,116,108,105,98,46,109,97,99,104,105,
+    110,101,114,121,46,70,105,108,101,70,105,110,100,101,114,46,
+    122,30,111,110,108,121,32,100,105,114,101,99,116,111,114,105,
+    101,115,32,97,114,101,32,115,117,112,112,111,114,116,101,100,
+    41,1,114,35,0,0,0,41,2,114,46,0,0,0,114,101,
+    0,0,0,41,1,114,35,0,0,0,41,2,114,172,0,0,
+    0,114,24,1,0,0,114,2,0,0,0,114,4,0,0,0,
+    218,24,112,97,116,104,95,104,111,111,107,95,102,111,114,95,
+    70,105,108,101,70,105,110,100,101,114,157,5,0,0,115,6,
+    0,0,0,0,2,8,1,12,1,122,54,70,105,108,101,70,
+    105,110,100,101,114,46,112,97,116,104,95,104,111,111,107,46,
+    60,108,111,99,97,108,115,62,46,112,97,116,104,95,104,111,
+    111,107,95,102,111,114,95,70,105,108,101,70,105,110,100,101,
+    114,114,2,0,0,0,41,3,114,172,0,0,0,114,24,1,
+    0,0,114,30,1,0,0,114,2,0,0,0,41,2,114,172,
+    0,0,0,114,24,1,0,0,114,4,0,0,0,218,9,112,
+    97,116,104,95,104,111,111,107,147,5,0,0,115,4,0,0,
+    0,0,10,14,6,122,20,70,105,108,101,70,105,110,100,101,
+    114,46,112,97,116,104,95,104,111,111,107,99,1,0,0,0,
+    0,0,0,0,1,0,0,0,3,0,0,0,67,0,0,0,
+    115,12,0,0,0,100,1,160,0,124,0,106,1,161,1,83,
+    0,41,2,78,122,16,70,105,108,101,70,105,110,100,101,114,
+    40,123,33,114,125,41,41,2,114,48,0,0,0,114,35,0,
+    0,0,41,1,114,102,0,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,4,0,0,0,114,0,1,0,0,165,5,0,
+    0,115,2,0,0,0,0,1,122,19,70,105,108,101,70,105,
+    110,100,101,114,46,95,95,114,101,112,114,95,95,41,1,78,
+    41,15,114,107,0,0,0,114,106,0,0,0,114,108,0,0,
+    0,114,109,0,0,0,114,186,0,0,0,114,6,1,0,0,
+    114,125,0,0,0,114,183,0,0,0,114,119,0,0,0,114,
+    17,1,0,0,114,182,0,0,0,114,25,1,0,0,114,184,
+    0,0,0,114,31,1,0,0,114,0,1,0,0,114,2,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,182,0,0,0,214,4,0,0,115,26,0,0,0,0,
-    6,8,1,6,1,14,1,8,1,4,1,10,1,6,1,4,
-    3,6,1,16,1,4,2,6,2,122,20,80,97,116,104,70,
-    105,110,100,101,114,46,102,105,110,100,95,115,112,101,99,99,
-    3,0,0,0,0,0,0,0,4,0,0,0,4,0,0,0,
-    67,0,0,0,115,30,0,0,0,124,0,160,0,124,1,124,
-    2,161,2,125,3,124,3,100,1,107,8,114,24,100,1,83,
-    0,124,3,106,1,83,0,41,2,122,170,102,105,110,100,32,
-    116,104,101,32,109,111,100,117,108,101,32,111,110,32,115,121,
-    115,46,112,97,116,104,32,111,114,32,39,112,97,116,104,39,
-    32,98,97,115,101,100,32,111,110,32,115,121,115,46,112,97,
-    116,104,95,104,111,111,107,115,32,97,110,100,10,32,32,32,
-    32,32,32,32,32,115,121,115,46,112,97,116,104,95,105,109,
-    112,111,114,116,101,114,95,99,97,99,104,101,46,10,10,32,
-    32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,104,
-    111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,100,
-    46,32,32,85,115,101,32,102,105,110,100,95,115,112,101,99,
-    40,41,32,105,110,115,116,101,97,100,46,10,10,32,32,32,
-    32,32,32,32,32,78,41,2,114,182,0,0,0,114,122,0,
-    0,0,41,4,114,172,0,0,0,114,121,0,0,0,114,35,
+    0,114,18,1,0,0,22,5,0,0,115,18,0,0,0,12,
+    9,8,14,8,4,4,2,8,12,8,5,10,48,8,31,12,
+    18,114,18,1,0,0,99,4,0,0,0,0,0,0,0,6,
+    0,0,0,8,0,0,0,67,0,0,0,115,146,0,0,0,
+    124,0,160,0,100,1,161,1,125,4,124,0,160,0,100,2,
+    161,1,125,5,124,4,115,66,124,5,114,36,124,5,106,1,
+    125,4,110,30,124,2,124,3,107,2,114,56,116,2,124,1,
+    124,2,131,2,125,4,110,10,116,3,124,1,124,2,131,2,
+    125,4,124,5,115,84,116,4,124,1,124,2,124,4,100,3,
+    141,3,125,5,121,36,124,5,124,0,100,2,60,0,124,4,
+    124,0,100,1,60,0,124,2,124,0,100,4,60,0,124,3,
+    124,0,100,5,60,0,87,0,110,20,4,0,116,5,107,10,
+    114,140,1,0,1,0,1,0,89,0,110,2,88,0,100,0,
+    83,0,41,6,78,218,10,95,95,108,111,97,100,101,114,95,
+    95,218,8,95,95,115,112,101,99,95,95,41,1,114,122,0,
+    0,0,90,8,95,95,102,105,108,101,95,95,90,10,95,95,
+    99,97,99,104,101,100,95,95,41,6,218,3,103,101,116,114,
+    122,0,0,0,114,234,0,0,0,114,229,0,0,0,114,169,
+    0,0,0,218,9,69,120,99,101,112,116,105,111,110,41,6,
+    90,2,110,115,114,100,0,0,0,90,8,112,97,116,104,110,
+    97,109,101,90,9,99,112,97,116,104,110,97,109,101,114,122,
     0,0,0,114,166,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,114,183,0,0,0,238,4,0,0,
-    115,8,0,0,0,0,8,12,1,8,1,4,1,122,22,80,
-    97,116,104,70,105,110,100,101,114,46,102,105,110,100,95,109,
-    111,100,117,108,101,41,1,78,41,2,78,78,41,1,78,41,
-    12,114,107,0,0,0,114,106,0,0,0,114,108,0,0,0,
-    114,109,0,0,0,114,184,0,0,0,114,254,0,0,0,114,
-    3,1,0,0,114,6,1,0,0,114,7,1,0,0,114,10,
-    1,0,0,114,182,0,0,0,114,183,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,253,0,0,0,120,4,0,0,115,20,0,0,0,12,
-    4,12,8,12,13,12,22,12,15,2,1,12,31,2,1,12,
-    23,2,1,114,253,0,0,0,99,0,0,0,0,0,0,0,
-    0,0,0,0,0,3,0,0,0,64,0,0,0,115,90,0,
-    0,0,101,0,90,1,100,0,90,2,100,1,90,3,100,2,
-    100,3,132,0,90,4,100,4,100,5,132,0,90,5,101,6,
-    90,7,100,6,100,7,132,0,90,8,100,8,100,9,132,0,
-    90,9,100,19,100,11,100,12,132,1,90,10,100,13,100,14,
-    132,0,90,11,101,12,100,15,100,16,132,0,131,1,90,13,
-    100,17,100,18,132,0,90,14,100,10,83,0,41,20,218,10,
-    70,105,108,101,70,105,110,100,101,114,122,172,70,105,108,101,
-    45,98,97,115,101,100,32,102,105,110,100,101,114,46,10,10,
-    32,32,32,32,73,110,116,101,114,97,99,116,105,111,110,115,
-    32,119,105,116,104,32,116,104,101,32,102,105,108,101,32,115,
-    121,115,116,101,109,32,97,114,101,32,99,97,99,104,101,100,
-    32,102,111,114,32,112,101,114,102,111,114,109,97,110,99,101,
-    44,32,98,101,105,110,103,10,32,32,32,32,114,101,102,114,
-    101,115,104,101,100,32,119,104,101,110,32,116,104,101,32,100,
-    105,114,101,99,116,111,114,121,32,116,104,101,32,102,105,110,
-    100,101,114,32,105,115,32,104,97,110,100,108,105,110,103,32,
-    104,97,115,32,98,101,101,110,32,109,111,100,105,102,105,101,
-    100,46,10,10,32,32,32,32,99,2,0,0,0,0,0,0,
-    0,5,0,0,0,6,0,0,0,7,0,0,0,115,88,0,
-    0,0,103,0,125,3,120,40,124,2,68,0,93,32,92,2,
-    137,0,125,4,124,3,160,0,135,0,102,1,100,1,100,2,
-    132,8,124,4,68,0,131,1,161,1,1,0,113,10,87,0,
-    124,3,124,0,95,1,124,1,112,58,100,3,124,0,95,2,
-    100,4,124,0,95,3,116,4,131,0,124,0,95,5,116,4,
-    131,0,124,0,95,6,100,5,83,0,41,6,122,154,73,110,
-    105,116,105,97,108,105,122,101,32,119,105,116,104,32,116,104,
-    101,32,112,97,116,104,32,116,111,32,115,101,97,114,99,104,
-    32,111,110,32,97,110,100,32,97,32,118,97,114,105,97,98,
-    108,101,32,110,117,109,98,101,114,32,111,102,10,32,32,32,
-    32,32,32,32,32,50,45,116,117,112,108,101,115,32,99,111,
-    110,116,97,105,110,105,110,103,32,116,104,101,32,108,111,97,
-    100,101,114,32,97,110,100,32,116,104,101,32,102,105,108,101,
-    32,115,117,102,102,105,120,101,115,32,116,104,101,32,108,111,
-    97,100,101,114,10,32,32,32,32,32,32,32,32,114,101,99,
-    111,103,110,105,122,101,115,46,99,1,0,0,0,0,0,0,
-    0,2,0,0,0,3,0,0,0,51,0,0,0,115,22,0,
-    0,0,124,0,93,14,125,1,124,1,136,0,102,2,86,0,
-    1,0,113,2,100,0,83,0,41,1,78,114,2,0,0,0,
-    41,2,114,22,0,0,0,114,227,0,0,0,41,1,114,122,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,114,229,0,
-    0,0,11,5,0,0,115,2,0,0,0,4,0,122,38,70,
-    105,108,101,70,105,110,100,101,114,46,95,95,105,110,105,116,
-    95,95,46,60,108,111,99,97,108,115,62,46,60,103,101,110,
-    101,120,112,114,62,114,59,0,0,0,114,89,0,0,0,78,
-    41,7,114,147,0,0,0,218,8,95,108,111,97,100,101,114,
-    115,114,35,0,0,0,218,11,95,112,97,116,104,95,109,116,
-    105,109,101,218,3,115,101,116,218,11,95,112,97,116,104,95,
-    99,97,99,104,101,218,19,95,114,101,108,97,120,101,100,95,
-    112,97,116,104,95,99,97,99,104,101,41,5,114,102,0,0,
-    0,114,35,0,0,0,218,14,108,111,97,100,101,114,95,100,
-    101,116,97,105,108,115,90,7,108,111,97,100,101,114,115,114,
-    168,0,0,0,114,2,0,0,0,41,1,114,122,0,0,0,
-    114,4,0,0,0,114,186,0,0,0,5,5,0,0,115,16,
-    0,0,0,0,4,4,1,14,1,28,1,6,2,10,1,6,
-    1,8,1,122,19,70,105,108,101,70,105,110,100,101,114,46,
-    95,95,105,110,105,116,95,95,99,1,0,0,0,0,0,0,
-    0,1,0,0,0,2,0,0,0,67,0,0,0,115,10,0,
-    0,0,100,1,124,0,95,0,100,2,83,0,41,3,122,31,
-    73,110,118,97,108,105,100,97,116,101,32,116,104,101,32,100,
-    105,114,101,99,116,111,114,121,32,109,116,105,109,101,46,114,
-    89,0,0,0,78,41,1,114,13,1,0,0,41,1,114,102,
-    0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,254,0,0,0,19,5,0,0,115,2,0,0,0,
-    0,2,122,28,70,105,108,101,70,105,110,100,101,114,46,105,
-    110,118,97,108,105,100,97,116,101,95,99,97,99,104,101,115,
-    99,2,0,0,0,0,0,0,0,3,0,0,0,3,0,0,
-    0,67,0,0,0,115,42,0,0,0,124,0,160,0,124,1,
-    161,1,125,2,124,2,100,1,107,8,114,26,100,1,103,0,
-    102,2,83,0,124,2,106,1,124,2,106,2,112,38,103,0,
-    102,2,83,0,41,2,122,197,84,114,121,32,116,111,32,102,
-    105,110,100,32,97,32,108,111,97,100,101,114,32,102,111,114,
-    32,116,104,101,32,115,112,101,99,105,102,105,101,100,32,109,
-    111,100,117,108,101,44,32,111,114,32,116,104,101,32,110,97,
-    109,101,115,112,97,99,101,10,32,32,32,32,32,32,32,32,
-    112,97,99,107,97,103,101,32,112,111,114,116,105,111,110,115,
-    46,32,82,101,116,117,114,110,115,32,40,108,111,97,100,101,
-    114,44,32,108,105,115,116,45,111,102,45,112,111,114,116,105,
-    111,110,115,41,46,10,10,32,32,32,32,32,32,32,32,84,
-    104,105,115,32,109,101,116,104,111,100,32,105,115,32,100,101,
-    112,114,101,99,97,116,101,100,46,32,32,85,115,101,32,102,
-    105,110,100,95,115,112,101,99,40,41,32,105,110,115,116,101,
-    97,100,46,10,10,32,32,32,32,32,32,32,32,78,41,3,
-    114,182,0,0,0,114,122,0,0,0,114,158,0,0,0,41,
-    3,114,102,0,0,0,114,121,0,0,0,114,166,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    119,0,0,0,25,5,0,0,115,8,0,0,0,0,7,10,
-    1,8,1,8,1,122,22,70,105,108,101,70,105,110,100,101,
-    114,46,102,105,110,100,95,108,111,97,100,101,114,99,6,0,
-    0,0,0,0,0,0,7,0,0,0,6,0,0,0,67,0,
-    0,0,115,26,0,0,0,124,1,124,2,124,3,131,2,125,
-    6,116,0,124,2,124,3,124,6,124,4,100,1,141,4,83,
-    0,41,2,78,41,2,114,122,0,0,0,114,158,0,0,0,
-    41,1,114,169,0,0,0,41,7,114,102,0,0,0,114,167,
-    0,0,0,114,121,0,0,0,114,35,0,0,0,90,4,115,
-    109,115,108,114,181,0,0,0,114,122,0,0,0,114,2,0,
-    0,0,114,2,0,0,0,114,4,0,0,0,114,10,1,0,
-    0,37,5,0,0,115,6,0,0,0,0,1,10,1,8,1,
-    122,20,70,105,108,101,70,105,110,100,101,114,46,95,103,101,
-    116,95,115,112,101,99,78,99,3,0,0,0,0,0,0,0,
-    14,0,0,0,8,0,0,0,67,0,0,0,115,98,1,0,
-    0,100,1,125,3,124,1,160,0,100,2,161,1,100,3,25,
-    0,125,4,121,24,116,1,124,0,106,2,112,34,116,3,160,
-    4,161,0,131,1,106,5,125,5,87,0,110,24,4,0,116,
-    6,107,10,114,66,1,0,1,0,1,0,100,4,125,5,89,
-    0,110,2,88,0,124,5,124,0,106,7,107,3,114,92,124,
-    0,160,8,161,0,1,0,124,5,124,0,95,7,116,9,131,
-    0,114,114,124,0,106,10,125,6,124,4,160,11,161,0,125,
-    7,110,10,124,0,106,12,125,6,124,4,125,7,124,7,124,
-    6,107,6,114,218,116,13,124,0,106,2,124,4,131,2,125,
-    8,120,72,124,0,106,14,68,0,93,54,92,2,125,9,125,
-    10,100,5,124,9,23,0,125,11,116,13,124,8,124,11,131,
-    2,125,12,116,15,124,12,131,1,114,152,124,0,160,16,124,
-    10,124,1,124,12,124,8,103,1,124,2,161,5,83,0,113,
-    152,87,0,116,17,124,8,131,1,125,3,120,88,124,0,106,
-    14,68,0,93,78,92,2,125,9,125,10,116,13,124,0,106,
-    2,124,4,124,9,23,0,131,2,125,12,116,18,106,19,100,
-    6,124,12,100,3,100,7,141,3,1,0,124,7,124,9,23,
-    0,124,6,107,6,114,226,116,15,124,12,131,1,114,226,124,
-    0,160,16,124,10,124,1,124,12,100,8,124,2,161,5,83,
-    0,113,226,87,0,124,3,144,1,114,94,116,18,160,19,100,
-    9,124,8,161,2,1,0,116,18,160,20,124,1,100,8,161,
-    2,125,13,124,8,103,1,124,13,95,21,124,13,83,0,100,
-    8,83,0,41,10,122,111,84,114,121,32,116,111,32,102,105,
-    110,100,32,97,32,115,112,101,99,32,102,111,114,32,116,104,
-    101,32,115,112,101,99,105,102,105,101,100,32,109,111,100,117,
-    108,101,46,10,10,32,32,32,32,32,32,32,32,82,101,116,
-    117,114,110,115,32,116,104,101,32,109,97,116,99,104,105,110,
-    103,32,115,112,101,99,44,32,111,114,32,78,111,110,101,32,
-    105,102,32,110,111,116,32,102,111,117,110,100,46,10,32,32,
-    32,32,32,32,32,32,70,114,59,0,0,0,114,57,0,0,
-    0,114,89,0,0,0,114,186,0,0,0,122,9,116,114,121,
-    105,110,103,32,123,125,41,1,90,9,118,101,114,98,111,115,
-    105,116,121,78,122,25,112,111,115,115,105,98,108,101,32,110,
-    97,109,101,115,112,97,99,101,32,102,111,114,32,123,125,41,
-    22,114,32,0,0,0,114,39,0,0,0,114,35,0,0,0,
-    114,1,0,0,0,114,45,0,0,0,114,221,0,0,0,114,
-    40,0,0,0,114,13,1,0,0,218,11,95,102,105,108,108,
-    95,99,97,99,104,101,114,5,0,0,0,114,16,1,0,0,
-    114,90,0,0,0,114,15,1,0,0,114,28,0,0,0,114,
-    12,1,0,0,114,44,0,0,0,114,10,1,0,0,114,46,
-    0,0,0,114,116,0,0,0,114,130,0,0,0,114,162,0,
-    0,0,114,158,0,0,0,41,14,114,102,0,0,0,114,121,
-    0,0,0,114,181,0,0,0,90,12,105,115,95,110,97,109,
-    101,115,112,97,99,101,90,11,116,97,105,108,95,109,111,100,
-    117,108,101,114,149,0,0,0,90,5,99,97,99,104,101,90,
-    12,99,97,99,104,101,95,109,111,100,117,108,101,90,9,98,
-    97,115,101,95,112,97,116,104,114,227,0,0,0,114,167,0,
-    0,0,90,13,105,110,105,116,95,102,105,108,101,110,97,109,
-    101,90,9,102,117,108,108,95,112,97,116,104,114,166,0,0,
-    0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,182,0,0,0,42,5,0,0,115,70,0,0,0,0,5,
-    4,1,14,1,2,1,24,1,14,1,10,1,10,1,8,1,
-    6,2,6,1,6,1,10,2,6,1,4,2,8,1,12,1,
-    16,1,8,1,10,1,8,1,24,4,8,2,16,1,16,1,
-    16,1,12,1,8,1,10,1,12,1,6,1,12,1,12,1,
-    8,1,4,1,122,20,70,105,108,101,70,105,110,100,101,114,
-    46,102,105,110,100,95,115,112,101,99,99,1,0,0,0,0,
-    0,0,0,9,0,0,0,10,0,0,0,67,0,0,0,115,
-    194,0,0,0,124,0,106,0,125,1,121,22,116,1,160,2,
-    124,1,112,22,116,1,160,3,161,0,161,1,125,2,87,0,
-    110,30,4,0,116,4,116,5,116,6,102,3,107,10,114,58,
-    1,0,1,0,1,0,103,0,125,2,89,0,110,2,88,0,
-    116,7,106,8,160,9,100,1,161,1,115,84,116,10,124,2,
-    131,1,124,0,95,11,110,78,116,10,131,0,125,3,120,64,
-    124,2,68,0,93,56,125,4,124,4,160,12,100,2,161,1,
-    92,3,125,5,125,6,125,7,124,6,114,138,100,3,160,13,
-    124,5,124,7,160,14,161,0,161,2,125,8,110,4,124,5,
-    125,8,124,3,160,15,124,8,161,1,1,0,113,96,87,0,
-    124,3,124,0,95,11,116,7,106,8,160,9,116,16,161,1,
-    114,190,100,4,100,5,132,0,124,2,68,0,131,1,124,0,
-    95,17,100,6,83,0,41,7,122,68,70,105,108,108,32,116,
-    104,101,32,99,97,99,104,101,32,111,102,32,112,111,116,101,
-    110,116,105,97,108,32,109,111,100,117,108,101,115,32,97,110,
-    100,32,112,97,99,107,97,103,101,115,32,102,111,114,32,116,
-    104,105,115,32,100,105,114,101,99,116,111,114,121,46,114,0,
-    0,0,0,114,59,0,0,0,122,5,123,125,46,123,125,99,
-    1,0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,
-    83,0,0,0,115,20,0,0,0,104,0,124,0,93,12,125,
-    1,124,1,160,0,161,0,146,2,113,4,83,0,114,2,0,
-    0,0,41,1,114,90,0,0,0,41,2,114,22,0,0,0,
-    90,2,102,110,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,250,9,60,115,101,116,99,111,109,112,62,119,5,
-    0,0,115,2,0,0,0,6,0,122,41,70,105,108,101,70,
-    105,110,100,101,114,46,95,102,105,108,108,95,99,97,99,104,
-    101,46,60,108,111,99,97,108,115,62,46,60,115,101,116,99,
-    111,109,112,62,78,41,18,114,35,0,0,0,114,1,0,0,
-    0,90,7,108,105,115,116,100,105,114,114,45,0,0,0,114,
-    4,1,0,0,218,15,80,101,114,109,105,115,115,105,111,110,
-    69,114,114,111,114,218,18,78,111,116,65,68,105,114,101,99,
-    116,111,114,121,69,114,114,111,114,114,6,0,0,0,114,7,
-    0,0,0,114,8,0,0,0,114,14,1,0,0,114,15,1,
-    0,0,114,85,0,0,0,114,48,0,0,0,114,90,0,0,
-    0,218,3,97,100,100,114,9,0,0,0,114,16,1,0,0,
-    41,9,114,102,0,0,0,114,35,0,0,0,90,8,99,111,
-    110,116,101,110,116,115,90,21,108,111,119,101,114,95,115,117,
-    102,102,105,120,95,99,111,110,116,101,110,116,115,114,249,0,
-    0,0,114,100,0,0,0,114,239,0,0,0,114,227,0,0,
-    0,90,8,110,101,119,95,110,97,109,101,114,2,0,0,0,
-    114,2,0,0,0,114,4,0,0,0,114,18,1,0,0,90,
-    5,0,0,115,34,0,0,0,0,2,6,1,2,1,22,1,
-    20,3,10,3,12,1,12,7,6,1,10,1,16,1,4,1,
-    18,2,4,1,14,1,6,1,12,1,122,22,70,105,108,101,
-    70,105,110,100,101,114,46,95,102,105,108,108,95,99,97,99,
-    104,101,99,1,0,0,0,0,0,0,0,3,0,0,0,3,
-    0,0,0,7,0,0,0,115,18,0,0,0,135,0,135,1,
-    102,2,100,1,100,2,132,8,125,2,124,2,83,0,41,3,
-    97,20,1,0,0,65,32,99,108,97,115,115,32,109,101,116,
-    104,111,100,32,119,104,105,99,104,32,114,101,116,117,114,110,
-    115,32,97,32,99,108,111,115,117,114,101,32,116,111,32,117,
-    115,101,32,111,110,32,115,121,115,46,112,97,116,104,95,104,
-    111,111,107,10,32,32,32,32,32,32,32,32,119,104,105,99,
-    104,32,119,105,108,108,32,114,101,116,117,114,110,32,97,110,
-    32,105,110,115,116,97,110,99,101,32,117,115,105,110,103,32,
-    116,104,101,32,115,112,101,99,105,102,105,101,100,32,108,111,
-    97,100,101,114,115,32,97,110,100,32,116,104,101,32,112,97,
-    116,104,10,32,32,32,32,32,32,32,32,99,97,108,108,101,
-    100,32,111,110,32,116,104,101,32,99,108,111,115,117,114,101,
-    46,10,10,32,32,32,32,32,32,32,32,73,102,32,116,104,
-    101,32,112,97,116,104,32,99,97,108,108,101,100,32,111,110,
-    32,116,104,101,32,99,108,111,115,117,114,101,32,105,115,32,
-    110,111,116,32,97,32,100,105,114,101,99,116,111,114,121,44,
-    32,73,109,112,111,114,116,69,114,114,111,114,32,105,115,10,
-    32,32,32,32,32,32,32,32,114,97,105,115,101,100,46,10,
-    10,32,32,32,32,32,32,32,32,99,1,0,0,0,0,0,
-    0,0,1,0,0,0,4,0,0,0,19,0,0,0,115,34,
-    0,0,0,116,0,124,0,131,1,115,20,116,1,100,1,124,
-    0,100,2,141,2,130,1,136,0,124,0,102,1,136,1,158,
-    2,142,0,83,0,41,3,122,45,80,97,116,104,32,104,111,
-    111,107,32,102,111,114,32,105,109,112,111,114,116,108,105,98,
-    46,109,97,99,104,105,110,101,114,121,46,70,105,108,101,70,
-    105,110,100,101,114,46,122,30,111,110,108,121,32,100,105,114,
-    101,99,116,111,114,105,101,115,32,97,114,101,32,115,117,112,
-    112,111,114,116,101,100,41,1,114,35,0,0,0,41,2,114,
-    46,0,0,0,114,101,0,0,0,41,1,114,35,0,0,0,
-    41,2,114,172,0,0,0,114,17,1,0,0,114,2,0,0,
-    0,114,4,0,0,0,218,24,112,97,116,104,95,104,111,111,
-    107,95,102,111,114,95,70,105,108,101,70,105,110,100,101,114,
-    131,5,0,0,115,6,0,0,0,0,2,8,1,12,1,122,
-    54,70,105,108,101,70,105,110,100,101,114,46,112,97,116,104,
-    95,104,111,111,107,46,60,108,111,99,97,108,115,62,46,112,
-    97,116,104,95,104,111,111,107,95,102,111,114,95,70,105,108,
-    101,70,105,110,100,101,114,114,2,0,0,0,41,3,114,172,
-    0,0,0,114,17,1,0,0,114,23,1,0,0,114,2,0,
-    0,0,41,2,114,172,0,0,0,114,17,1,0,0,114,4,
-    0,0,0,218,9,112,97,116,104,95,104,111,111,107,121,5,
-    0,0,115,4,0,0,0,0,10,14,6,122,20,70,105,108,
-    101,70,105,110,100,101,114,46,112,97,116,104,95,104,111,111,
-    107,99,1,0,0,0,0,0,0,0,1,0,0,0,3,0,
-    0,0,67,0,0,0,115,12,0,0,0,100,1,160,0,124,
-    0,106,1,161,1,83,0,41,2,78,122,16,70,105,108,101,
-    70,105,110,100,101,114,40,123,33,114,125,41,41,2,114,48,
-    0,0,0,114,35,0,0,0,41,1,114,102,0,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,248,
-    0,0,0,139,5,0,0,115,2,0,0,0,0,1,122,19,
-    70,105,108,101,70,105,110,100,101,114,46,95,95,114,101,112,
-    114,95,95,41,1,78,41,15,114,107,0,0,0,114,106,0,
-    0,0,114,108,0,0,0,114,109,0,0,0,114,186,0,0,
-    0,114,254,0,0,0,114,125,0,0,0,114,183,0,0,0,
-    114,119,0,0,0,114,10,1,0,0,114,182,0,0,0,114,
-    18,1,0,0,114,184,0,0,0,114,24,1,0,0,114,248,
-    0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,114,11,1,0,0,252,4,0,0,
-    115,18,0,0,0,12,9,8,14,8,4,4,2,8,12,8,
-    5,10,48,8,31,12,18,114,11,1,0,0,99,4,0,0,
-    0,0,0,0,0,6,0,0,0,8,0,0,0,67,0,0,
-    0,115,146,0,0,0,124,0,160,0,100,1,161,1,125,4,
-    124,0,160,0,100,2,161,1,125,5,124,4,115,66,124,5,
-    114,36,124,5,106,1,125,4,110,30,124,2,124,3,107,2,
-    114,56,116,2,124,1,124,2,131,2,125,4,110,10,116,3,
-    124,1,124,2,131,2,125,4,124,5,115,84,116,4,124,1,
-    124,2,124,4,100,3,141,3,125,5,121,36,124,5,124,0,
-    100,2,60,0,124,4,124,0,100,1,60,0,124,2,124,0,
-    100,4,60,0,124,3,124,0,100,5,60,0,87,0,110,20,
-    4,0,116,5,107,10,114,140,1,0,1,0,1,0,89,0,
-    110,2,88,0,100,0,83,0,41,6,78,218,10,95,95,108,
-    111,97,100,101,114,95,95,218,8,95,95,115,112,101,99,95,
-    95,41,1,114,122,0,0,0,90,8,95,95,102,105,108,101,
-    95,95,90,10,95,95,99,97,99,104,101,100,95,95,41,6,
-    218,3,103,101,116,114,122,0,0,0,114,225,0,0,0,114,
-    220,0,0,0,114,169,0,0,0,218,9,69,120,99,101,112,
-    116,105,111,110,41,6,90,2,110,115,114,100,0,0,0,90,
-    8,112,97,116,104,110,97,109,101,90,9,99,112,97,116,104,
-    110,97,109,101,114,122,0,0,0,114,166,0,0,0,114,2,
-    0,0,0,114,2,0,0,0,114,4,0,0,0,218,14,95,
-    102,105,120,95,117,112,95,109,111,100,117,108,101,145,5,0,
-    0,115,34,0,0,0,0,2,10,1,10,1,4,1,4,1,
-    8,1,8,1,12,2,10,1,4,1,14,1,2,1,8,1,
-    8,1,8,1,12,1,14,2,114,29,1,0,0,99,0,0,
-    0,0,0,0,0,0,3,0,0,0,3,0,0,0,67,0,
-    0,0,115,38,0,0,0,116,0,116,1,160,2,161,0,102,
-    2,125,0,116,3,116,4,102,2,125,1,116,5,116,6,102,
-    2,125,2,124,0,124,1,124,2,103,3,83,0,41,1,122,
-    95,82,101,116,117,114,110,115,32,97,32,108,105,115,116,32,
-    111,102,32,102,105,108,101,45,98,97,115,101,100,32,109,111,
-    100,117,108,101,32,108,111,97,100,101,114,115,46,10,10,32,
-    32,32,32,69,97,99,104,32,105,116,101,109,32,105,115,32,
-    97,32,116,117,112,108,101,32,40,108,111,97,100,101,114,44,
-    32,115,117,102,102,105,120,101,115,41,46,10,32,32,32,32,
-    41,7,114,226,0,0,0,114,143,0,0,0,218,18,101,120,
-    116,101,110,115,105,111,110,95,115,117,102,102,105,120,101,115,
-    114,220,0,0,0,114,86,0,0,0,114,225,0,0,0,114,
-    76,0,0,0,41,3,90,10,101,120,116,101,110,115,105,111,
-    110,115,90,6,115,111,117,114,99,101,90,8,98,121,116,101,
-    99,111,100,101,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,163,0,0,0,168,5,0,0,115,8,0,0,
-    0,0,5,12,1,8,1,8,1,114,163,0,0,0,99,1,
-    0,0,0,0,0,0,0,12,0,0,0,9,0,0,0,67,
-    0,0,0,115,156,1,0,0,124,0,97,0,116,0,106,1,
-    97,1,116,0,106,2,97,2,116,1,106,3,116,4,25,0,
-    125,1,120,56,100,1,68,0,93,48,125,2,124,2,116,1,
-    106,3,107,7,114,58,116,0,160,5,124,2,161,1,125,3,
-    110,10,116,1,106,3,124,2,25,0,125,3,116,6,124,1,
-    124,2,124,3,131,3,1,0,113,32,87,0,100,2,100,3,
-    103,1,102,2,100,4,100,5,100,3,103,2,102,2,102,2,
-    125,4,120,118,124,4,68,0,93,102,92,2,125,5,125,6,
-    116,7,100,6,100,7,132,0,124,6,68,0,131,1,131,1,
-    115,142,116,8,130,1,124,6,100,8,25,0,125,7,124,5,
-    116,1,106,3,107,6,114,174,116,1,106,3,124,5,25,0,
-    125,8,80,0,113,112,121,16,116,0,160,5,124,5,161,1,
-    125,8,80,0,87,0,113,112,4,0,116,9,107,10,114,212,
-    1,0,1,0,1,0,119,112,89,0,113,112,88,0,113,112,
-    87,0,116,9,100,9,131,1,130,1,116,6,124,1,100,10,
-    124,8,131,3,1,0,116,6,124,1,100,11,124,7,131,3,
-    1,0,116,6,124,1,100,12,100,13,160,10,124,6,161,1,
-    131,3,1,0,116,0,160,5,100,14,161,1,125,9,116,6,
-    124,1,100,14,124,9,131,3,1,0,116,0,160,5,100,15,
-    161,1,125,10,116,6,124,1,100,15,124,10,131,3,1,0,
-    124,5,100,4,107,2,144,1,114,88,116,0,160,5,100,16,
-    161,1,125,11,116,6,124,1,100,17,124,11,131,3,1,0,
-    116,6,124,1,100,18,116,11,131,0,131,3,1,0,116,12,
-    160,13,116,2,160,14,161,0,161,1,1,0,124,5,100,4,
-    107,2,144,1,114,152,116,15,160,16,100,19,161,1,1,0,
-    100,20,116,12,107,6,144,1,114,152,100,21,116,17,95,18,
-    100,22,83,0,41,23,122,205,83,101,116,117,112,32,116,104,
-    101,32,112,97,116,104,45,98,97,115,101,100,32,105,109,112,
-    111,114,116,101,114,115,32,102,111,114,32,105,109,112,111,114,
-    116,108,105,98,32,98,121,32,105,109,112,111,114,116,105,110,
-    103,32,110,101,101,100,101,100,10,32,32,32,32,98,117,105,
-    108,116,45,105,110,32,109,111,100,117,108,101,115,32,97,110,
-    100,32,105,110,106,101,99,116,105,110,103,32,116,104,101,109,
-    32,105,110,116,111,32,116,104,101,32,103,108,111,98,97,108,
-    32,110,97,109,101,115,112,97,99,101,46,10,10,32,32,32,
-    32,79,116,104,101,114,32,99,111,109,112,111,110,101,110,116,
-    115,32,97,114,101,32,101,120,116,114,97,99,116,101,100,32,
-    102,114,111,109,32,116,104,101,32,99,111,114,101,32,98,111,
-    111,116,115,116,114,97,112,32,109,111,100,117,108,101,46,10,
-    10,32,32,32,32,41,4,114,50,0,0,0,114,61,0,0,
-    0,218,8,98,117,105,108,116,105,110,115,114,140,0,0,0,
-    90,5,112,111,115,105,120,250,1,47,90,2,110,116,250,1,
-    92,99,1,0,0,0,0,0,0,0,2,0,0,0,3,0,
-    0,0,115,0,0,0,115,26,0,0,0,124,0,93,18,125,
-    1,116,0,124,1,131,1,100,0,107,2,86,0,1,0,113,
-    2,100,1,83,0,41,2,114,29,0,0,0,78,41,1,114,
-    31,0,0,0,41,2,114,22,0,0,0,114,79,0,0,0,
-    114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    229,0,0,0,204,5,0,0,115,2,0,0,0,4,0,122,
-    25,95,115,101,116,117,112,46,60,108,111,99,97,108,115,62,
-    46,60,103,101,110,101,120,112,114,62,114,60,0,0,0,122,
-    30,105,109,112,111,114,116,108,105,98,32,114,101,113,117,105,
-    114,101,115,32,112,111,115,105,120,32,111,114,32,110,116,114,
-    1,0,0,0,114,25,0,0,0,114,21,0,0,0,114,30,
-    0,0,0,90,7,95,116,104,114,101,97,100,90,8,95,119,
-    101,97,107,114,101,102,90,6,119,105,110,114,101,103,114,171,
-    0,0,0,114,5,0,0,0,122,4,46,112,121,119,122,6,
-    95,100,46,112,121,100,84,78,41,19,114,116,0,0,0,114,
-    6,0,0,0,114,143,0,0,0,114,241,0,0,0,114,107,
-    0,0,0,90,18,95,98,117,105,108,116,105,110,95,102,114,
-    111,109,95,110,97,109,101,114,111,0,0,0,218,3,97,108,
-    108,114,151,0,0,0,114,101,0,0,0,114,26,0,0,0,
-    114,11,0,0,0,114,231,0,0,0,114,147,0,0,0,114,
-    30,1,0,0,114,86,0,0,0,114,165,0,0,0,114,170,
-    0,0,0,114,174,0,0,0,41,12,218,17,95,98,111,111,
-    116,115,116,114,97,112,95,109,111,100,117,108,101,90,11,115,
-    101,108,102,95,109,111,100,117,108,101,90,12,98,117,105,108,
-    116,105,110,95,110,97,109,101,90,14,98,117,105,108,116,105,
-    110,95,109,111,100,117,108,101,90,10,111,115,95,100,101,116,
-    97,105,108,115,90,10,98,117,105,108,116,105,110,95,111,115,
-    114,21,0,0,0,114,25,0,0,0,90,9,111,115,95,109,
-    111,100,117,108,101,90,13,116,104,114,101,97,100,95,109,111,
-    100,117,108,101,90,14,119,101,97,107,114,101,102,95,109,111,
-    100,117,108,101,90,13,119,105,110,114,101,103,95,109,111,100,
-    117,108,101,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,218,6,95,115,101,116,117,112,179,5,0,0,115,76,
-    0,0,0,0,8,4,1,6,1,6,3,10,1,10,1,10,
-    1,12,2,10,1,16,3,22,1,14,2,22,1,8,1,10,
-    1,10,1,4,2,2,1,10,1,6,1,14,1,12,2,8,
-    1,12,1,12,1,18,3,10,1,12,3,10,1,12,3,10,
-    1,10,1,12,3,14,1,14,1,10,1,10,1,10,1,114,
-    36,1,0,0,99,1,0,0,0,0,0,0,0,2,0,0,
-    0,4,0,0,0,67,0,0,0,115,50,0,0,0,116,0,
-    124,0,131,1,1,0,116,1,131,0,125,1,116,2,106,3,
-    160,4,116,5,106,6,124,1,142,0,103,1,161,1,1,0,
-    116,2,106,7,160,8,116,9,161,1,1,0,100,1,83,0,
-    41,2,122,41,73,110,115,116,97,108,108,32,116,104,101,32,
-    112,97,116,104,45,98,97,115,101,100,32,105,109,112,111,114,
-    116,32,99,111,109,112,111,110,101,110,116,115,46,78,41,10,
-    114,36,1,0,0,114,163,0,0,0,114,6,0,0,0,114,
-    2,1,0,0,114,147,0,0,0,114,11,1,0,0,114,24,
-    1,0,0,218,9,109,101,116,97,95,112,97,116,104,114,165,
-    0,0,0,114,253,0,0,0,41,2,114,35,1,0,0,90,
-    17,115,117,112,112,111,114,116,101,100,95,108,111,97,100,101,
-    114,115,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,8,95,105,110,115,116,97,108,108,243,5,0,0,115,
-    8,0,0,0,0,2,8,1,6,1,20,1,114,38,1,0,
-    0,41,1,114,47,0,0,0,41,1,78,41,3,78,78,78,
-    41,2,114,60,0,0,0,114,60,0,0,0,41,1,84,41,
-    1,78,41,1,78,41,61,114,109,0,0,0,114,10,0,0,
-    0,90,37,95,67,65,83,69,95,73,78,83,69,78,83,73,
-    84,73,86,69,95,80,76,65,84,70,79,82,77,83,95,66,
-    89,84,69,83,95,75,69,89,114,9,0,0,0,114,11,0,
-    0,0,114,17,0,0,0,114,19,0,0,0,114,28,0,0,
-    0,114,38,0,0,0,114,39,0,0,0,114,43,0,0,0,
-    114,44,0,0,0,114,46,0,0,0,114,56,0,0,0,218,
-    4,116,121,112,101,218,8,95,95,99,111,100,101,95,95,114,
-    142,0,0,0,114,15,0,0,0,114,129,0,0,0,114,14,
-    0,0,0,114,18,0,0,0,114,209,0,0,0,114,75,0,
-    0,0,114,74,0,0,0,114,86,0,0,0,114,76,0,0,
-    0,90,23,68,69,66,85,71,95,66,89,84,69,67,79,68,
-    69,95,83,85,70,70,73,88,69,83,90,27,79,80,84,73,
-    77,73,90,69,68,95,66,89,84,69,67,79,68,69,95,83,
-    85,70,70,73,88,69,83,114,81,0,0,0,114,87,0,0,
-    0,114,93,0,0,0,114,97,0,0,0,114,99,0,0,0,
-    114,118,0,0,0,114,125,0,0,0,114,133,0,0,0,114,
-    137,0,0,0,114,139,0,0,0,114,145,0,0,0,114,150,
-    0,0,0,114,152,0,0,0,114,157,0,0,0,218,6,111,
-    98,106,101,99,116,114,164,0,0,0,114,169,0,0,0,114,
-    170,0,0,0,114,185,0,0,0,114,195,0,0,0,114,212,
-    0,0,0,114,220,0,0,0,114,225,0,0,0,114,231,0,
-    0,0,114,226,0,0,0,114,232,0,0,0,114,251,0,0,
-    0,114,253,0,0,0,114,11,1,0,0,114,29,1,0,0,
-    114,163,0,0,0,114,36,1,0,0,114,38,1,0,0,114,
-    2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,218,8,60,109,111,100,117,108,101,62,24,0,0,
-    0,115,116,0,0,0,4,0,4,1,4,1,2,1,6,3,
-    8,17,8,5,8,5,8,6,8,12,8,10,8,9,8,5,
-    8,7,10,22,10,126,16,1,12,2,4,1,4,2,6,2,
-    6,2,8,2,16,45,8,34,8,19,8,12,8,12,8,28,
-    8,17,8,33,8,28,8,24,10,13,10,10,10,11,8,14,
-    6,3,4,1,14,67,14,64,14,29,16,127,0,17,14,41,
-    18,45,18,26,4,3,18,53,14,60,14,42,14,127,0,5,
-    14,127,0,22,10,23,8,11,8,64,
+    0,0,114,4,0,0,0,218,14,95,102,105,120,95,117,112,
+    95,109,111,100,117,108,101,171,5,0,0,115,34,0,0,0,
+    0,2,10,1,10,1,4,1,4,1,8,1,8,1,12,2,
+    10,1,4,1,14,1,2,1,8,1,8,1,8,1,12,1,
+    14,2,114,36,1,0,0,99,0,0,0,0,0,0,0,0,
+    3,0,0,0,3,0,0,0,67,0,0,0,115,38,0,0,
+    0,116,0,116,1,160,2,161,0,102,2,125,0,116,3,116,
+    4,102,2,125,1,116,5,116,6,102,2,125,2,124,0,124,
+    1,124,2,103,3,83,0,41,1,122,95,82,101,116,117,114,
+    110,115,32,97,32,108,105,115,116,32,111,102,32,102,105,108,
+    101,45,98,97,115,101,100,32,109,111,100,117,108,101,32,108,
+    111,97,100,101,114,115,46,10,10,32,32,32,32,69,97,99,
+    104,32,105,116,101,109,32,105,115,32,97,32,116,117,112,108,
+    101,32,40,108,111,97,100,101,114,44,32,115,117,102,102,105,
+    120,101,115,41,46,10,32,32,32,32,41,7,114,235,0,0,
+    0,114,143,0,0,0,218,18,101,120,116,101,110,115,105,111,
+    110,95,115,117,102,102,105,120,101,115,114,229,0,0,0,114,
+    86,0,0,0,114,234,0,0,0,114,76,0,0,0,41,3,
+    90,10,101,120,116,101,110,115,105,111,110,115,90,6,115,111,
+    117,114,99,101,90,8,98,121,116,101,99,111,100,101,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,114,163,0,
+    0,0,194,5,0,0,115,8,0,0,0,0,5,12,1,8,
+    1,8,1,114,163,0,0,0,99,1,0,0,0,0,0,0,
+    0,12,0,0,0,9,0,0,0,67,0,0,0,115,156,1,
+    0,0,124,0,97,0,116,0,106,1,97,1,116,0,106,2,
+    97,2,116,1,106,3,116,4,25,0,125,1,120,56,100,1,
+    68,0,93,48,125,2,124,2,116,1,106,3,107,7,114,58,
+    116,0,160,5,124,2,161,1,125,3,110,10,116,1,106,3,
+    124,2,25,0,125,3,116,6,124,1,124,2,124,3,131,3,
+    1,0,113,32,87,0,100,2,100,3,103,1,102,2,100,4,
+    100,5,100,3,103,2,102,2,102,2,125,4,120,118,124,4,
+    68,0,93,102,92,2,125,5,125,6,116,7,100,6,100,7,
+    132,0,124,6,68,0,131,1,131,1,115,142,116,8,130,1,
+    124,6,100,8,25,0,125,7,124,5,116,1,106,3,107,6,
+    114,174,116,1,106,3,124,5,25,0,125,8,80,0,113,112,
+    121,16,116,0,160,5,124,5,161,1,125,8,80,0,87,0,
+    113,112,4,0,116,9,107,10,114,212,1,0,1,0,1,0,
+    119,112,89,0,113,112,88,0,113,112,87,0,116,9,100,9,
+    131,1,130,1,116,6,124,1,100,10,124,8,131,3,1,0,
+    116,6,124,1,100,11,124,7,131,3,1,0,116,6,124,1,
+    100,12,100,13,160,10,124,6,161,1,131,3,1,0,116,0,
+    160,5,100,14,161,1,125,9,116,6,124,1,100,14,124,9,
+    131,3,1,0,116,0,160,5,100,15,161,1,125,10,116,6,
+    124,1,100,15,124,10,131,3,1,0,124,5,100,4,107,2,
+    144,1,114,88,116,0,160,5,100,16,161,1,125,11,116,6,
+    124,1,100,17,124,11,131,3,1,0,116,6,124,1,100,18,
+    116,11,131,0,131,3,1,0,116,12,160,13,116,2,160,14,
+    161,0,161,1,1,0,124,5,100,4,107,2,144,1,114,152,
+    116,15,160,16,100,19,161,1,1,0,100,20,116,12,107,6,
+    144,1,114,152,100,21,116,17,95,18,100,22,83,0,41,23,
+    122,205,83,101,116,117,112,32,116,104,101,32,112,97,116,104,
+    45,98,97,115,101,100,32,105,109,112,111,114,116,101,114,115,
+    32,102,111,114,32,105,109,112,111,114,116,108,105,98,32,98,
+    121,32,105,109,112,111,114,116,105,110,103,32,110,101,101,100,
+    101,100,10,32,32,32,32,98,117,105,108,116,45,105,110,32,
+    109,111,100,117,108,101,115,32,97,110,100,32,105,110,106,101,
+    99,116,105,110,103,32,116,104,101,109,32,105,110,116,111,32,
+    116,104,101,32,103,108,111,98,97,108,32,110,97,109,101,115,
+    112,97,99,101,46,10,10,32,32,32,32,79,116,104,101,114,
+    32,99,111,109,112,111,110,101,110,116,115,32,97,114,101,32,
+    101,120,116,114,97,99,116,101,100,32,102,114,111,109,32,116,
+    104,101,32,99,111,114,101,32,98,111,111,116,115,116,114,97,
+    112,32,109,111,100,117,108,101,46,10,10,32,32,32,32,41,
+    4,114,50,0,0,0,114,61,0,0,0,218,8,98,117,105,
+    108,116,105,110,115,114,140,0,0,0,90,5,112,111,115,105,
+    120,250,1,47,90,2,110,116,250,1,92,99,1,0,0,0,
+    0,0,0,0,2,0,0,0,3,0,0,0,115,0,0,0,
+    115,26,0,0,0,124,0,93,18,125,1,116,0,124,1,131,
+    1,100,0,107,2,86,0,1,0,113,2,100,1,83,0,41,
+    2,114,29,0,0,0,78,41,1,114,31,0,0,0,41,2,
+    114,22,0,0,0,114,79,0,0,0,114,2,0,0,0,114,
+    2,0,0,0,114,4,0,0,0,114,238,0,0,0,230,5,
+    0,0,115,2,0,0,0,4,0,122,25,95,115,101,116,117,
+    112,46,60,108,111,99,97,108,115,62,46,60,103,101,110,101,
+    120,112,114,62,114,60,0,0,0,122,30,105,109,112,111,114,
+    116,108,105,98,32,114,101,113,117,105,114,101,115,32,112,111,
+    115,105,120,32,111,114,32,110,116,114,1,0,0,0,114,25,
+    0,0,0,114,21,0,0,0,114,30,0,0,0,90,7,95,
+    116,104,114,101,97,100,90,8,95,119,101,97,107,114,101,102,
+    90,6,119,105,110,114,101,103,114,171,0,0,0,114,5,0,
+    0,0,122,4,46,112,121,119,122,6,95,100,46,112,121,100,
+    84,78,41,19,114,116,0,0,0,114,6,0,0,0,114,143,
+    0,0,0,114,250,0,0,0,114,107,0,0,0,90,18,95,
+    98,117,105,108,116,105,110,95,102,114,111,109,95,110,97,109,
+    101,114,111,0,0,0,218,3,97,108,108,114,151,0,0,0,
+    114,101,0,0,0,114,26,0,0,0,114,11,0,0,0,114,
+    240,0,0,0,114,147,0,0,0,114,37,1,0,0,114,86,
+    0,0,0,114,165,0,0,0,114,170,0,0,0,114,174,0,
+    0,0,41,12,218,17,95,98,111,111,116,115,116,114,97,112,
+    95,109,111,100,117,108,101,90,11,115,101,108,102,95,109,111,
+    100,117,108,101,90,12,98,117,105,108,116,105,110,95,110,97,
+    109,101,90,14,98,117,105,108,116,105,110,95,109,111,100,117,
+    108,101,90,10,111,115,95,100,101,116,97,105,108,115,90,10,
+    98,117,105,108,116,105,110,95,111,115,114,21,0,0,0,114,
+    25,0,0,0,90,9,111,115,95,109,111,100,117,108,101,90,
+    13,116,104,114,101,97,100,95,109,111,100,117,108,101,90,14,
+    119,101,97,107,114,101,102,95,109,111,100,117,108,101,90,13,
+    119,105,110,114,101,103,95,109,111,100,117,108,101,114,2,0,
+    0,0,114,2,0,0,0,114,4,0,0,0,218,6,95,115,
+    101,116,117,112,205,5,0,0,115,76,0,0,0,0,8,4,
+    1,6,1,6,3,10,1,10,1,10,1,12,2,10,1,16,
+    3,22,1,14,2,22,1,8,1,10,1,10,1,4,2,2,
+    1,10,1,6,1,14,1,12,2,8,1,12,1,12,1,18,
+    3,10,1,12,3,10,1,12,3,10,1,10,1,12,3,14,
+    1,14,1,10,1,10,1,10,1,114,43,1,0,0,99,1,
+    0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,
+    0,0,0,115,50,0,0,0,116,0,124,0,131,1,1,0,
+    116,1,131,0,125,1,116,2,106,3,160,4,116,5,106,6,
+    124,1,142,0,103,1,161,1,1,0,116,2,106,7,160,8,
+    116,9,161,1,1,0,100,1,83,0,41,2,122,41,73,110,
+    115,116,97,108,108,32,116,104,101,32,112,97,116,104,45,98,
+    97,115,101,100,32,105,109,112,111,114,116,32,99,111,109,112,
+    111,110,101,110,116,115,46,78,41,10,114,43,1,0,0,114,
+    163,0,0,0,114,6,0,0,0,114,10,1,0,0,114,147,
+    0,0,0,114,18,1,0,0,114,31,1,0,0,218,9,109,
+    101,116,97,95,112,97,116,104,114,165,0,0,0,114,5,1,
+    0,0,41,2,114,42,1,0,0,90,17,115,117,112,112,111,
+    114,116,101,100,95,108,111,97,100,101,114,115,114,2,0,0,
+    0,114,2,0,0,0,114,4,0,0,0,218,8,95,105,110,
+    115,116,97,108,108,13,6,0,0,115,8,0,0,0,0,2,
+    8,1,6,1,20,1,114,45,1,0,0,41,1,114,47,0,
+    0,0,41,1,78,41,3,78,78,78,41,2,114,60,0,0,
+    0,114,60,0,0,0,41,1,84,41,1,78,41,1,78,41,
+    61,114,109,0,0,0,114,10,0,0,0,90,37,95,67,65,
+    83,69,95,73,78,83,69,78,83,73,84,73,86,69,95,80,
+    76,65,84,70,79,82,77,83,95,66,89,84,69,83,95,75,
+    69,89,114,9,0,0,0,114,11,0,0,0,114,17,0,0,
+    0,114,19,0,0,0,114,28,0,0,0,114,38,0,0,0,
+    114,39,0,0,0,114,43,0,0,0,114,44,0,0,0,114,
+    46,0,0,0,114,56,0,0,0,218,4,116,121,112,101,218,
+    8,95,95,99,111,100,101,95,95,114,142,0,0,0,114,15,
+    0,0,0,114,129,0,0,0,114,14,0,0,0,114,18,0,
+    0,0,114,209,0,0,0,114,75,0,0,0,114,74,0,0,
+    0,114,86,0,0,0,114,76,0,0,0,90,23,68,69,66,
+    85,71,95,66,89,84,69,67,79,68,69,95,83,85,70,70,
+    73,88,69,83,90,27,79,80,84,73,77,73,90,69,68,95,
+    66,89,84,69,67,79,68,69,95,83,85,70,70,73,88,69,
+    83,114,81,0,0,0,114,87,0,0,0,114,93,0,0,0,
+    114,97,0,0,0,114,99,0,0,0,114,118,0,0,0,114,
+    125,0,0,0,114,133,0,0,0,114,137,0,0,0,114,139,
+    0,0,0,114,145,0,0,0,114,150,0,0,0,114,152,0,
+    0,0,114,157,0,0,0,218,6,111,98,106,101,99,116,114,
+    164,0,0,0,114,169,0,0,0,114,170,0,0,0,114,185,
+    0,0,0,114,195,0,0,0,114,212,0,0,0,114,229,0,
+    0,0,114,234,0,0,0,114,240,0,0,0,114,235,0,0,
+    0,114,241,0,0,0,114,3,1,0,0,114,5,1,0,0,
+    114,18,1,0,0,114,36,1,0,0,114,163,0,0,0,114,
+    43,1,0,0,114,45,1,0,0,114,2,0,0,0,114,2,
+    0,0,0,114,2,0,0,0,114,4,0,0,0,218,8,60,
+    109,111,100,117,108,101,62,23,0,0,0,115,116,0,0,0,
+    4,0,4,1,4,1,2,1,6,3,8,17,8,5,8,5,
+    8,6,8,12,8,10,8,9,8,5,8,7,10,22,10,126,
+    16,1,12,2,4,1,4,2,6,2,6,2,8,2,16,45,
+    8,34,8,19,8,12,8,12,8,28,8,17,8,33,8,28,
+    8,24,10,13,10,10,10,11,8,14,6,3,4,1,14,67,
+    14,64,14,29,16,127,0,17,14,68,18,45,18,26,4,3,
+    18,53,14,60,14,42,14,127,0,5,14,127,0,22,10,23,
+    8,11,8,64,
 };


From webhook-mailer at python.org  Mon Jan 15 18:27:19 2018
From: webhook-mailer at python.org (Antoine Pitrou)
Date: Mon, 15 Jan 2018 23:27:19 -0000
Subject: [Python-checkins] bpo-14976: Reentrant simple queue (#3346)
Message-ID: <mailman.29.1516058840.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/94e1696d04c65e19ea52e5c8664079c9d9aa0e54
commit: 94e1696d04c65e19ea52e5c8664079c9d9aa0e54
branch: master
author: Antoine Pitrou <pitrou at free.fr>
committer: GitHub <noreply at github.com>
date: 2018-01-16T00:27:16+01:00
summary:

bpo-14976: Reentrant simple queue (#3346)

Add a queue.SimpleQueue class, an unbounded FIFO queue with a reentrant C implementation of put().

files:
A Misc/NEWS.d/next/Library/2017-09-07-19-12-47.bpo-14976.dx0Zxb.rst
A Modules/_queuemodule.c
A Modules/clinic/_queuemodule.c.h
A PCbuild/_queue.vcxproj
A PCbuild/_queue.vcxproj.filters
M Doc/library/queue.rst
M Lib/queue.py
M Lib/test/test_queue.py
M PCbuild/pcbuild.proj
M PCbuild/pcbuild.sln
M PCbuild/pythoncore.vcxproj.filters
M setup.py

diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst
index f9a43bbac3b..1520faa9b83 100644
--- a/Doc/library/queue.rst
+++ b/Doc/library/queue.rst
@@ -23,8 +23,14 @@ the first retrieved (operating like a stack).  With a priority queue,
 the entries are kept sorted (using the :mod:`heapq` module) and the
 lowest valued entry is retrieved first.
 
-Internally, the module uses locks to temporarily block competing threads;
-however, it is not designed to handle reentrancy within a thread.
+Internally, those three types of queues use locks to temporarily block
+competing threads; however, they are not designed to handle reentrancy
+within a thread.
+
+In addition, the module implements a "simple"
+:abbr:`FIFO (first-in, first-out)` queue type where
+specific implementations can provide additional guarantees
+in exchange for the smaller functionality.
 
 The :mod:`queue` module defines the following classes and exceptions:
 
@@ -67,6 +73,14 @@ The :mod:`queue` module defines the following classes and exceptions:
             priority: int
             item: Any=field(compare=False)
 
+.. class:: SimpleQueue()
+
+   Constructor for an unbounded :abbr:`FIFO (first-in, first-out)` queue.
+   Simple queues lack advanced functionality such as task tracking.
+
+   .. versionadded:: 3.7
+
+
 .. exception:: Empty
 
    Exception raised when non-blocking :meth:`~Queue.get` (or
@@ -201,6 +215,60 @@ Example of how to wait for enqueued tasks to be completed::
         t.join()
 
 
+SimpleQueue Objects
+-------------------
+
+:class:`SimpleQueue` objects provide the public methods described below.
+
+.. method:: SimpleQueue.qsize()
+
+   Return the approximate size of the queue.  Note, qsize() > 0 doesn't
+   guarantee that a subsequent get() will not block.
+
+
+.. method:: SimpleQueue.empty()
+
+   Return ``True`` if the queue is empty, ``False`` otherwise. If empty()
+   returns ``False`` it doesn't guarantee that a subsequent call to get()
+   will not block.
+
+
+.. method:: SimpleQueue.put(item, block=True, timeout=None)
+
+   Put *item* into the queue.  The method never blocks and always succeeds
+   (except for potential low-level errors such as failure to allocate memory).
+   The optional args *block* and *timeout* are ignored and only provided
+   for compatibility with :meth:`Queue.put`.
+
+   .. impl-detail::
+      This method has a C implementation which is reentrant.  That is, a
+      ``put()`` or ``get()`` call can be interrupted by another ``put()``
+      call in the same thread without deadlocking or corrupting internal
+      state inside the queue.  This makes it appropriate for use in
+      destructors such as ``__del__`` methods or :mod:`weakref` callbacks.
+
+
+.. method:: SimpleQueue.put_nowait(item)
+
+   Equivalent to ``put(item)``, provided for compatibility with
+   :meth:`Queue.put_nowait`.
+
+
+.. method:: SimpleQueue.get(block=True, timeout=None)
+
+   Remove and return an item from the queue.  If optional args *block* is true and
+   *timeout* is ``None`` (the default), block if necessary until an item is available.
+   If *timeout* is a positive number, it blocks at most *timeout* seconds and
+   raises the :exc:`Empty` exception if no item was available within that time.
+   Otherwise (*block* is false), return an item if one is immediately available,
+   else raise the :exc:`Empty` exception (*timeout* is ignored in that case).
+
+
+.. method:: SimpleQueue.get_nowait()
+
+   Equivalent to ``get(False)``.
+
+
 .. seealso::
 
    Class :class:`multiprocessing.Queue`
@@ -210,4 +278,3 @@ Example of how to wait for enqueued tasks to be completed::
    :class:`collections.deque` is an alternative implementation of unbounded
    queues with fast atomic :meth:`~collections.deque.append` and
    :meth:`~collections.deque.popleft` operations that do not require locking.
-
diff --git a/Lib/queue.py b/Lib/queue.py
index c803b96deb0..ef07957781a 100644
--- a/Lib/queue.py
+++ b/Lib/queue.py
@@ -4,17 +4,26 @@
 from collections import deque
 from heapq import heappush, heappop
 from time import monotonic as time
+try:
+    from _queue import SimpleQueue
+except ImportError:
+    SimpleQueue = None
 
-__all__ = ['Empty', 'Full', 'Queue', 'PriorityQueue', 'LifoQueue']
+__all__ = ['Empty', 'Full', 'Queue', 'PriorityQueue', 'LifoQueue', 'SimpleQueue']
 
-class Empty(Exception):
-    'Exception raised by Queue.get(block=0)/get_nowait().'
-    pass
+
+try:
+    from _queue import Empty
+except AttributeError:
+    class Empty(Exception):
+        'Exception raised by Queue.get(block=0)/get_nowait().'
+        pass
 
 class Full(Exception):
     'Exception raised by Queue.put(block=0)/put_nowait().'
     pass
 
+
 class Queue:
     '''Create a queue object with a given maximum size.
 
@@ -241,3 +250,72 @@ def _put(self, item):
 
     def _get(self):
         return self.queue.pop()
+
+
+class _PySimpleQueue:
+    '''Simple, unbounded FIFO queue.
+
+    This pure Python implementation is not reentrant.
+    '''
+    # Note: while this pure Python version provides fairness
+    # (by using a threading.Semaphore which is itself fair, being based
+    #  on threading.Condition), fairness is not part of the API contract.
+    # This allows the C version to use a different implementation.
+
+    def __init__(self):
+        self._queue = deque()
+        self._count = threading.Semaphore(0)
+
+    def put(self, item, block=True, timeout=None):
+        '''Put the item on the queue.
+
+        The optional 'block' and 'timeout' arguments are ignored, as this method
+        never blocks.  They are provided for compatibility with the Queue class.
+        '''
+        self._queue.append(item)
+        self._count.release()
+
+    def get(self, block=True, timeout=None):
+        '''Remove and return an item from the queue.
+
+        If optional args 'block' is true and 'timeout' is None (the default),
+        block if necessary until an item is available. If 'timeout' is
+        a non-negative number, it blocks at most 'timeout' seconds and raises
+        the Empty exception if no item was available within that time.
+        Otherwise ('block' is false), return an item if one is immediately
+        available, else raise the Empty exception ('timeout' is ignored
+        in that case).
+        '''
+        if timeout is not None and timeout < 0:
+            raise ValueError("'timeout' must be a non-negative number")
+        if not self._count.acquire(block, timeout):
+            raise Empty
+        return self._queue.popleft()
+
+    def put_nowait(self, item):
+        '''Put an item into the queue without blocking.
+
+        This is exactly equivalent to `put(item)` and is only provided
+        for compatibility with the Queue class.
+        '''
+        return self.put(item, block=False)
+
+    def get_nowait(self):
+        '''Remove and return an item from the queue without blocking.
+
+        Only get an item if one is immediately available. Otherwise
+        raise the Empty exception.
+        '''
+        return self.get(block=False)
+
+    def empty(self):
+        '''Return True if the queue is empty, False otherwise (not reliable!).'''
+        return len(self._queue) == 0
+
+    def qsize(self):
+        '''Return the approximate size of the queue (not reliable!).'''
+        return len(self._queue)
+
+
+if SimpleQueue is None:
+    SimpleQueue = _PySimpleQueue
diff --git a/Lib/test/test_queue.py b/Lib/test/test_queue.py
index 6ee906c4d26..1a8d5f8856c 100644
--- a/Lib/test/test_queue.py
+++ b/Lib/test/test_queue.py
@@ -1,12 +1,22 @@
 # Some simple queue module tests, plus some failure conditions
 # to ensure the Queue locks remain stable.
+import collections
+import itertools
 import queue
+import random
+import sys
 import threading
 import time
 import unittest
+import weakref
 from test import support
 
 
+try:
+    import _queue
+except ImportError:
+    _queue = None
+
 QUEUE_SIZE = 5
 
 def qfull(q):
@@ -84,7 +94,7 @@ def setUp(self):
         self.cum = 0
         self.cumlock = threading.Lock()
 
-    def simple_queue_test(self, q):
+    def basic_queue_test(self, q):
         if q.qsize():
             raise RuntimeError("Call this function with an empty queue")
         self.assertTrue(q.empty())
@@ -192,12 +202,12 @@ def test_queue_join(self):
         else:
             self.fail("Did not detect task count going negative")
 
-    def test_simple_queue(self):
+    def test_basic(self):
         # Do it a couple of times on the same queue.
         # Done twice to make sure works with same instance reused.
         q = self.type2test(QUEUE_SIZE)
-        self.simple_queue_test(q)
-        self.simple_queue_test(q)
+        self.basic_queue_test(q)
+        self.basic_queue_test(q)
 
     def test_negative_timeout_raises_exception(self):
         q = self.type2test(QUEUE_SIZE)
@@ -353,5 +363,227 @@ def test_failing_queue(self):
         self.failing_queue_test(q)
 
 
+class BaseSimpleQueueTest:
+
+    def setUp(self):
+        self.q = self.type2test()
+
+    def feed(self, q, seq, rnd):
+        while True:
+            try:
+                val = seq.pop()
+            except IndexError:
+                return
+            q.put(val)
+            if rnd.random() > 0.5:
+                time.sleep(rnd.random() * 1e-3)
+
+    def consume(self, q, results, sentinel):
+        while True:
+            val = q.get()
+            if val == sentinel:
+                return
+            results.append(val)
+
+    def consume_nonblock(self, q, results, sentinel):
+        while True:
+            while True:
+                try:
+                    val = q.get(block=False)
+                except queue.Empty:
+                    time.sleep(1e-5)
+                else:
+                    break
+            if val == sentinel:
+                return
+            results.append(val)
+
+    def consume_timeout(self, q, results, sentinel):
+        while True:
+            while True:
+                try:
+                    val = q.get(timeout=1e-5)
+                except queue.Empty:
+                    pass
+                else:
+                    break
+            if val == sentinel:
+                return
+            results.append(val)
+
+    def run_threads(self, n_feeders, n_consumers, q, inputs,
+                    feed_func, consume_func):
+        results = []
+        sentinel = None
+        seq = inputs + [sentinel] * n_consumers
+        seq.reverse()
+        rnd = random.Random(42)
+
+        exceptions = []
+        def log_exceptions(f):
+            def wrapper(*args, **kwargs):
+                try:
+                    f(*args, **kwargs)
+                except BaseException as e:
+                    exceptions.append(e)
+            return wrapper
+
+        feeders = [threading.Thread(target=log_exceptions(feed_func),
+                                    args=(q, seq, rnd))
+                   for i in range(n_feeders)]
+        consumers = [threading.Thread(target=log_exceptions(consume_func),
+                                      args=(q, results, sentinel))
+                     for i in range(n_consumers)]
+
+        with support.start_threads(feeders + consumers):
+            pass
+
+        self.assertFalse(exceptions)
+        self.assertTrue(q.empty())
+        self.assertEqual(q.qsize(), 0)
+
+        return results
+
+    def test_basic(self):
+        # Basic tests for get(), put() etc.
+        q = self.q
+        self.assertTrue(q.empty())
+        self.assertEqual(q.qsize(), 0)
+        q.put(1)
+        self.assertFalse(q.empty())
+        self.assertEqual(q.qsize(), 1)
+        q.put(2)
+        q.put_nowait(3)
+        q.put(4)
+        self.assertFalse(q.empty())
+        self.assertEqual(q.qsize(), 4)
+
+        self.assertEqual(q.get(), 1)
+        self.assertEqual(q.qsize(), 3)
+
+        self.assertEqual(q.get_nowait(), 2)
+        self.assertEqual(q.qsize(), 2)
+
+        self.assertEqual(q.get(block=False), 3)
+        self.assertFalse(q.empty())
+        self.assertEqual(q.qsize(), 1)
+
+        self.assertEqual(q.get(timeout=0.1), 4)
+        self.assertTrue(q.empty())
+        self.assertEqual(q.qsize(), 0)
+
+        with self.assertRaises(queue.Empty):
+            q.get(block=False)
+        with self.assertRaises(queue.Empty):
+            q.get(timeout=1e-3)
+        with self.assertRaises(queue.Empty):
+            q.get_nowait()
+        self.assertTrue(q.empty())
+        self.assertEqual(q.qsize(), 0)
+
+    def test_negative_timeout_raises_exception(self):
+        q = self.q
+        q.put(1)
+        with self.assertRaises(ValueError):
+            q.get(timeout=-1)
+
+    def test_order(self):
+        # Test a pair of concurrent put() and get()
+        q = self.q
+        inputs = list(range(100))
+        results = self.run_threads(1, 1, q, inputs, self.feed, self.consume)
+
+        # One producer, one consumer => results appended in well-defined order
+        self.assertEqual(results, inputs)
+
+    def test_many_threads(self):
+        # Test multiple concurrent put() and get()
+        N = 50
+        q = self.q
+        inputs = list(range(10000))
+        results = self.run_threads(N, N, q, inputs, self.feed, self.consume)
+
+        # Multiple consumers without synchronization append the
+        # results in random order
+        self.assertEqual(sorted(results), inputs)
+
+    def test_many_threads_nonblock(self):
+        # Test multiple concurrent put() and get(block=False)
+        N = 50
+        q = self.q
+        inputs = list(range(10000))
+        results = self.run_threads(N, N, q, inputs,
+                                   self.feed, self.consume_nonblock)
+
+        self.assertEqual(sorted(results), inputs)
+
+    def test_many_threads_timeout(self):
+        # Test multiple concurrent put() and get(timeout=...)
+        N = 50
+        q = self.q
+        inputs = list(range(1000))
+        results = self.run_threads(N, N, q, inputs,
+                                   self.feed, self.consume_timeout)
+
+        self.assertEqual(sorted(results), inputs)
+
+    def test_references(self):
+        # The queue should lose references to each item as soon as
+        # it leaves the queue.
+        class C:
+            pass
+
+        N = 20
+        q = self.q
+        for i in range(N):
+            q.put(C())
+        for i in range(N):
+            wr = weakref.ref(q.get())
+            self.assertIsNone(wr())
+
+
+class PySimpleQueueTest(BaseSimpleQueueTest, unittest.TestCase):
+    type2test = queue._PySimpleQueue
+
+
+ at unittest.skipIf(_queue is None, "No _queue module found")
+class CSimpleQueueTest(BaseSimpleQueueTest, unittest.TestCase):
+
+    def setUp(self):
+        self.type2test = _queue.SimpleQueue
+        super().setUp()
+
+    def test_is_default(self):
+        self.assertIs(self.type2test, queue.SimpleQueue)
+
+    def test_reentrancy(self):
+        # bpo-14976: put() may be called reentrantly in an asynchronous
+        # callback.
+        q = self.q
+        gen = itertools.count()
+        N = 10000
+        results = []
+
+        # This test exploits the fact that __del__ in a reference cycle
+        # can be called any time the GC may run.
+
+        class Circular(object):
+            def __init__(self):
+                self.circular = self
+
+            def __del__(self):
+                q.put(next(gen))
+
+        while True:
+            o = Circular()
+            q.put(next(gen))
+            del o
+            results.append(q.get())
+            if results[-1] >= N:
+                break
+
+        self.assertEqual(results, list(range(N + 1)))
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/Misc/NEWS.d/next/Library/2017-09-07-19-12-47.bpo-14976.dx0Zxb.rst b/Misc/NEWS.d/next/Library/2017-09-07-19-12-47.bpo-14976.dx0Zxb.rst
new file mode 100644
index 00000000000..a4087551322
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-09-07-19-12-47.bpo-14976.dx0Zxb.rst
@@ -0,0 +1,2 @@
+Add a queue.SimpleQueue class, an unbounded FIFO queue with a reentrant C
+implementation of put().
diff --git a/Modules/_queuemodule.c b/Modules/_queuemodule.c
new file mode 100644
index 00000000000..8715337fb51
--- /dev/null
+++ b/Modules/_queuemodule.c
@@ -0,0 +1,400 @@
+#include "Python.h"
+#include "structmember.h" /* offsetof */
+#include "pythread.h"
+
+/*[clinic input]
+module _queue
+class _queue.SimpleQueue "simplequeueobject *" "&PySimpleQueueType"
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=cf49af81bcbbbea6]*/
+
+extern PyTypeObject PySimpleQueueType;  /* forward decl */
+
+static PyObject *EmptyError;
+
+
+typedef struct {
+    PyObject_HEAD
+    PyThread_type_lock lock;
+    int locked;
+    PyObject *lst;
+    Py_ssize_t lst_pos;
+    PyObject *weakreflist;
+} simplequeueobject;
+
+
+static void
+simplequeue_dealloc(simplequeueobject *self)
+{
+    _PyObject_GC_UNTRACK(self);
+    if (self->lock != NULL) {
+        /* Unlock the lock so it's safe to free it */
+        if (self->locked > 0)
+            PyThread_release_lock(self->lock);
+        PyThread_free_lock(self->lock);
+    }
+    Py_XDECREF(self->lst);
+    if (self->weakreflist != NULL)
+        PyObject_ClearWeakRefs((PyObject *) self);
+    Py_TYPE(self)->tp_free(self);
+}
+
+static int
+simplequeue_traverse(simplequeueobject *self, visitproc visit, void *arg)
+{
+    Py_VISIT(self->lst);
+    return 0;
+}
+
+/*[clinic input]
+ at classmethod
+_queue.SimpleQueue.__new__ as simplequeue_new
+
+Simple, unbounded, reentrant FIFO queue.
+[clinic start generated code]*/
+
+static PyObject *
+simplequeue_new_impl(PyTypeObject *type)
+/*[clinic end generated code: output=ba97740608ba31cd input=a0674a1643e3e2fb]*/
+{
+    simplequeueobject *self;
+
+    self = (simplequeueobject *) type->tp_alloc(type, 0);
+    if (self != NULL) {
+        self->weakreflist = NULL;
+        self->lst = PyList_New(0);
+        self->lock = PyThread_allocate_lock();
+        self->lst_pos = 0;
+        if (self->lock == NULL) {
+            Py_DECREF(self);
+            PyErr_SetString(PyExc_MemoryError, "can't allocate lock");
+            return NULL;
+        }
+        if (self->lst == NULL) {
+            Py_DECREF(self);
+            return NULL;
+        }
+    }
+
+    return (PyObject *) self;
+}
+
+/*[clinic input]
+_queue.SimpleQueue.put
+    item: object
+    block: bool = True
+    timeout: object = None
+
+Put the item on the queue.
+
+The optional 'block' and 'timeout' arguments are ignored, as this method
+never blocks.  They are provided for compatibility with the Queue class.
+
+[clinic start generated code]*/
+
+static PyObject *
+_queue_SimpleQueue_put_impl(simplequeueobject *self, PyObject *item,
+                            int block, PyObject *timeout)
+/*[clinic end generated code: output=4333136e88f90d8b input=6e601fa707a782d5]*/
+{
+    /* BEGIN GIL-protected critical section */
+    if (PyList_Append(self->lst, item) < 0)
+        return NULL;
+    if (self->locked) {
+        /* A get() may be waiting, wake it up */
+        self->locked = 0;
+        PyThread_release_lock(self->lock);
+    }
+    /* END GIL-protected critical section */
+    Py_RETURN_NONE;
+}
+
+/*[clinic input]
+_queue.SimpleQueue.put_nowait
+    item: object
+
+Put an item into the queue without blocking.
+
+This is exactly equivalent to `put(item)` and is only provided
+for compatibility with the Queue class.
+
+[clinic start generated code]*/
+
+static PyObject *
+_queue_SimpleQueue_put_nowait_impl(simplequeueobject *self, PyObject *item)
+/*[clinic end generated code: output=0990536715efb1f1 input=36b1ea96756b2ece]*/
+{
+    return _queue_SimpleQueue_put_impl(self, item, 0, Py_None);
+}
+
+static PyObject *
+simplequeue_pop_item(simplequeueobject *self)
+{
+    Py_ssize_t count, n;
+    PyObject *item;
+
+    n = PyList_GET_SIZE(self->lst);
+    assert(self->lst_pos < n);
+
+    item = PyList_GET_ITEM(self->lst, self->lst_pos);
+    Py_INCREF(Py_None);
+    PyList_SET_ITEM(self->lst, self->lst_pos, Py_None);
+    self->lst_pos += 1;
+    count = n - self->lst_pos;
+    if (self->lst_pos > count) {
+        /* The list is more than 50% empty, reclaim space at the beginning */
+        if (PyList_SetSlice(self->lst, 0, self->lst_pos, NULL)) {
+            /* Undo pop */
+            self->lst_pos -= 1;
+            PyList_SET_ITEM(self->lst, self->lst_pos, item);
+            return NULL;
+        }
+        self->lst_pos = 0;
+    }
+    return item;
+}
+
+/*[clinic input]
+_queue.SimpleQueue.get
+    block: bool = True
+    timeout: object = None
+
+Remove and return an item from the queue.
+
+If optional args 'block' is true and 'timeout' is None (the default),
+block if necessary until an item is available. If 'timeout' is
+a non-negative number, it blocks at most 'timeout' seconds and raises
+the Empty exception if no item was available within that time.
+Otherwise ('block' is false), return an item if one is immediately
+available, else raise the Empty exception ('timeout' is ignored
+in that case).
+
+[clinic start generated code]*/
+
+static PyObject *
+_queue_SimpleQueue_get_impl(simplequeueobject *self, int block,
+                            PyObject *timeout)
+/*[clinic end generated code: output=ec82a7157dcccd1a input=4bf691f9f01fa297]*/
+{
+    _PyTime_t endtime = 0;
+    _PyTime_t timeout_val;
+    PyObject *item;
+    PyLockStatus r;
+    PY_TIMEOUT_T microseconds;
+
+    if (block == 0) {
+        /* Non-blocking */
+        microseconds = 0;
+    }
+    else if (timeout != Py_None) {
+        /* With timeout */
+        if (_PyTime_FromSecondsObject(&timeout_val,
+                                      timeout, _PyTime_ROUND_CEILING) < 0)
+            return NULL;
+        if (timeout_val < 0) {
+            PyErr_SetString(PyExc_ValueError,
+                            "'timeout' must be a non-negative number");
+            return NULL;
+        }
+        microseconds = _PyTime_AsMicroseconds(timeout_val,
+                                              _PyTime_ROUND_CEILING);
+        if (microseconds >= PY_TIMEOUT_MAX) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "timeout value is too large");
+            return NULL;
+        }
+        endtime = _PyTime_GetMonotonicClock() + timeout_val;
+    }
+    else {
+        /* Infinitely blocking */
+        microseconds = -1;
+    }
+
+    /* put() signals the queue to be non-empty by releasing the lock.
+     * So we simply try to acquire the lock in a loop, until the condition
+     * (queue non-empty) becomes true.
+     */
+    while (self->lst_pos == PyList_GET_SIZE(self->lst)) {
+        /* First a simple non-blocking try without releasing the GIL */
+        r = PyThread_acquire_lock_timed(self->lock, 0, 0);
+        if (r == PY_LOCK_FAILURE && microseconds != 0) {
+            Py_BEGIN_ALLOW_THREADS
+            r = PyThread_acquire_lock_timed(self->lock, microseconds, 1);
+            Py_END_ALLOW_THREADS
+        }
+        if (r == PY_LOCK_INTR && Py_MakePendingCalls() < 0) {
+            return NULL;
+        }
+        if (r == PY_LOCK_FAILURE) {
+            /* Timed out */
+            PyErr_SetNone(EmptyError);
+            return NULL;
+        }
+        self->locked = 1;
+        /* Adjust timeout for next iteration (if any) */
+        if (endtime > 0) {
+            timeout_val = endtime - _PyTime_GetMonotonicClock();
+            microseconds = _PyTime_AsMicroseconds(timeout_val, _PyTime_ROUND_CEILING);
+        }
+    }
+    /* BEGIN GIL-protected critical section */
+    assert(self->lst_pos < PyList_GET_SIZE(self->lst));
+    item = simplequeue_pop_item(self);
+    if (self->locked) {
+        PyThread_release_lock(self->lock);
+        self->locked = 0;
+    }
+    /* END GIL-protected critical section */
+
+    return item;
+}
+
+/*[clinic input]
+_queue.SimpleQueue.get_nowait
+
+Remove and return an item from the queue without blocking.
+
+Only get an item if one is immediately available. Otherwise
+raise the Empty exception.
+[clinic start generated code]*/
+
+static PyObject *
+_queue_SimpleQueue_get_nowait_impl(simplequeueobject *self)
+/*[clinic end generated code: output=a89731a75dbe4937 input=6fe5102db540a1b9]*/
+{
+    return _queue_SimpleQueue_get_impl(self, 0, Py_None);
+}
+
+/*[clinic input]
+_queue.SimpleQueue.empty -> bool
+
+Return True if the queue is empty, False otherwise (not reliable!).
+[clinic start generated code]*/
+
+static int
+_queue_SimpleQueue_empty_impl(simplequeueobject *self)
+/*[clinic end generated code: output=1a02a1b87c0ef838 input=1a98431c45fd66f9]*/
+{
+    return self->lst_pos == PyList_GET_SIZE(self->lst);
+}
+
+/*[clinic input]
+_queue.SimpleQueue.qsize -> Py_ssize_t
+
+Return the approximate size of the queue (not reliable!).
+[clinic start generated code]*/
+
+static Py_ssize_t
+_queue_SimpleQueue_qsize_impl(simplequeueobject *self)
+/*[clinic end generated code: output=f9dcd9d0a90e121e input=7a74852b407868a1]*/
+{
+    return PyList_GET_SIZE(self->lst) - self->lst_pos;
+}
+
+
+#include "clinic/_queuemodule.c.h"
+
+
+static PyMethodDef simplequeue_methods[] = {
+    _QUEUE_SIMPLEQUEUE_EMPTY_METHODDEF
+    _QUEUE_SIMPLEQUEUE_GET_METHODDEF
+    _QUEUE_SIMPLEQUEUE_GET_NOWAIT_METHODDEF
+    _QUEUE_SIMPLEQUEUE_PUT_METHODDEF
+    _QUEUE_SIMPLEQUEUE_PUT_NOWAIT_METHODDEF
+    _QUEUE_SIMPLEQUEUE_QSIZE_METHODDEF
+    {NULL,           NULL}              /* sentinel */
+};
+
+
+PyTypeObject PySimpleQueueType = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "_queue.SimpleQueue",               /*tp_name*/
+    sizeof(simplequeueobject),          /*tp_size*/
+    0,                                  /*tp_itemsize*/
+    /* methods */
+    (destructor)simplequeue_dealloc,    /*tp_dealloc*/
+    0,                                  /*tp_print*/
+    0,                                  /*tp_getattr*/
+    0,                                  /*tp_setattr*/
+    0,                                  /*tp_reserved*/
+    0,                                  /*tp_repr*/
+    0,                                  /*tp_as_number*/
+    0,                                  /*tp_as_sequence*/
+    0,                                  /*tp_as_mapping*/
+    0,                                  /*tp_hash*/
+    0,                                  /*tp_call*/
+    0,                                  /*tp_str*/
+    0,                                  /*tp_getattro*/
+    0,                                  /*tp_setattro*/
+    0,                                  /*tp_as_buffer*/
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+        | Py_TPFLAGS_HAVE_GC,           /* tp_flags */
+    simplequeue_new__doc__,             /*tp_doc*/
+    (traverseproc)simplequeue_traverse, /*tp_traverse*/
+    0,                                  /*tp_clear*/
+    0,                                  /*tp_richcompare*/
+    offsetof(simplequeueobject, weakreflist), /*tp_weaklistoffset*/
+    0,                                  /*tp_iter*/
+    0,                                  /*tp_iternext*/
+    simplequeue_methods,                /*tp_methods*/
+    0,                                  /* tp_members */
+    0,                                  /* tp_getset */
+    0,                                  /* tp_base */
+    0,                                  /* tp_dict */
+    0,                                  /* tp_descr_get */
+    0,                                  /* tp_descr_set */
+    0,                                  /* tp_dictoffset */
+    0,                                  /* tp_init */
+    0,                                  /* tp_alloc */
+    simplequeue_new                     /* tp_new */
+};
+
+
+/* Initialization function */
+
+PyDoc_STRVAR(queue_module_doc,
+"C implementation of the Python queue module.\n\
+This module is an implementation detail, please do not use it directly.");
+
+static struct PyModuleDef queuemodule = {
+    PyModuleDef_HEAD_INIT,
+    "_queue",
+    queue_module_doc,
+    -1,
+    NULL,
+    NULL,
+    NULL,
+    NULL,
+    NULL
+};
+
+
+PyMODINIT_FUNC
+PyInit__queue(void)
+{
+    PyObject *m;
+
+    /* Create the module */
+    m = PyModule_Create(&queuemodule);
+    if (m == NULL)
+        return NULL;
+
+    EmptyError = PyErr_NewExceptionWithDoc(
+        "_queue.Empty",
+        "Exception raised by Queue.get(block=0)/get_nowait().",
+        NULL, NULL);
+    if (EmptyError == NULL)
+        return NULL;
+
+    Py_INCREF(EmptyError);
+    if (PyModule_AddObject(m, "Empty", EmptyError) < 0)
+        return NULL;
+
+    if (PyType_Ready(&PySimpleQueueType) < 0)
+        return NULL;
+    Py_INCREF(&PySimpleQueueType);
+    if (PyModule_AddObject(m, "SimpleQueue", (PyObject *)&PySimpleQueueType) < 0)
+        return NULL;
+
+    return m;
+}
diff --git a/Modules/clinic/_queuemodule.c.h b/Modules/clinic/_queuemodule.c.h
new file mode 100644
index 00000000000..97247fd8a12
--- /dev/null
+++ b/Modules/clinic/_queuemodule.c.h
@@ -0,0 +1,218 @@
+/*[clinic input]
+preserve
+[clinic start generated code]*/
+
+PyDoc_STRVAR(simplequeue_new__doc__,
+"SimpleQueue()\n"
+"--\n"
+"\n"
+"Simple, unbounded, reentrant FIFO queue.");
+
+static PyObject *
+simplequeue_new_impl(PyTypeObject *type);
+
+static PyObject *
+simplequeue_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
+{
+    PyObject *return_value = NULL;
+
+    if ((type == &PySimpleQueueType) &&
+        !_PyArg_NoPositional("SimpleQueue", args)) {
+        goto exit;
+    }
+    if ((type == &PySimpleQueueType) &&
+        !_PyArg_NoKeywords("SimpleQueue", kwargs)) {
+        goto exit;
+    }
+    return_value = simplequeue_new_impl(type);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_queue_SimpleQueue_put__doc__,
+"put($self, /, item, block=True, timeout=None)\n"
+"--\n"
+"\n"
+"Put the item on the queue.\n"
+"\n"
+"The optional \'block\' and \'timeout\' arguments are ignored, as this method\n"
+"never blocks.  They are provided for compatibility with the Queue class.");
+
+#define _QUEUE_SIMPLEQUEUE_PUT_METHODDEF    \
+    {"put", (PyCFunction)_queue_SimpleQueue_put, METH_FASTCALL|METH_KEYWORDS, _queue_SimpleQueue_put__doc__},
+
+static PyObject *
+_queue_SimpleQueue_put_impl(simplequeueobject *self, PyObject *item,
+                            int block, PyObject *timeout);
+
+static PyObject *
+_queue_SimpleQueue_put(simplequeueobject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"item", "block", "timeout", NULL};
+    static _PyArg_Parser _parser = {"O|pO:put", _keywords, 0};
+    PyObject *item;
+    int block = 1;
+    PyObject *timeout = Py_None;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &item, &block, &timeout)) {
+        goto exit;
+    }
+    return_value = _queue_SimpleQueue_put_impl(self, item, block, timeout);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_queue_SimpleQueue_put_nowait__doc__,
+"put_nowait($self, /, item)\n"
+"--\n"
+"\n"
+"Put an item into the queue without blocking.\n"
+"\n"
+"This is exactly equivalent to `put(item)` and is only provided\n"
+"for compatibility with the Queue class.");
+
+#define _QUEUE_SIMPLEQUEUE_PUT_NOWAIT_METHODDEF    \
+    {"put_nowait", (PyCFunction)_queue_SimpleQueue_put_nowait, METH_FASTCALL|METH_KEYWORDS, _queue_SimpleQueue_put_nowait__doc__},
+
+static PyObject *
+_queue_SimpleQueue_put_nowait_impl(simplequeueobject *self, PyObject *item);
+
+static PyObject *
+_queue_SimpleQueue_put_nowait(simplequeueobject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"item", NULL};
+    static _PyArg_Parser _parser = {"O:put_nowait", _keywords, 0};
+    PyObject *item;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &item)) {
+        goto exit;
+    }
+    return_value = _queue_SimpleQueue_put_nowait_impl(self, item);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_queue_SimpleQueue_get__doc__,
+"get($self, /, block=True, timeout=None)\n"
+"--\n"
+"\n"
+"Remove and return an item from the queue.\n"
+"\n"
+"If optional args \'block\' is true and \'timeout\' is None (the default),\n"
+"block if necessary until an item is available. If \'timeout\' is\n"
+"a non-negative number, it blocks at most \'timeout\' seconds and raises\n"
+"the Empty exception if no item was available within that time.\n"
+"Otherwise (\'block\' is false), return an item if one is immediately\n"
+"available, else raise the Empty exception (\'timeout\' is ignored\n"
+"in that case).");
+
+#define _QUEUE_SIMPLEQUEUE_GET_METHODDEF    \
+    {"get", (PyCFunction)_queue_SimpleQueue_get, METH_FASTCALL|METH_KEYWORDS, _queue_SimpleQueue_get__doc__},
+
+static PyObject *
+_queue_SimpleQueue_get_impl(simplequeueobject *self, int block,
+                            PyObject *timeout);
+
+static PyObject *
+_queue_SimpleQueue_get(simplequeueobject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"block", "timeout", NULL};
+    static _PyArg_Parser _parser = {"|pO:get", _keywords, 0};
+    int block = 1;
+    PyObject *timeout = Py_None;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &block, &timeout)) {
+        goto exit;
+    }
+    return_value = _queue_SimpleQueue_get_impl(self, block, timeout);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_queue_SimpleQueue_get_nowait__doc__,
+"get_nowait($self, /)\n"
+"--\n"
+"\n"
+"Remove and return an item from the queue without blocking.\n"
+"\n"
+"Only get an item if one is immediately available. Otherwise\n"
+"raise the Empty exception.");
+
+#define _QUEUE_SIMPLEQUEUE_GET_NOWAIT_METHODDEF    \
+    {"get_nowait", (PyCFunction)_queue_SimpleQueue_get_nowait, METH_NOARGS, _queue_SimpleQueue_get_nowait__doc__},
+
+static PyObject *
+_queue_SimpleQueue_get_nowait_impl(simplequeueobject *self);
+
+static PyObject *
+_queue_SimpleQueue_get_nowait(simplequeueobject *self, PyObject *Py_UNUSED(ignored))
+{
+    return _queue_SimpleQueue_get_nowait_impl(self);
+}
+
+PyDoc_STRVAR(_queue_SimpleQueue_empty__doc__,
+"empty($self, /)\n"
+"--\n"
+"\n"
+"Return True if the queue is empty, False otherwise (not reliable!).");
+
+#define _QUEUE_SIMPLEQUEUE_EMPTY_METHODDEF    \
+    {"empty", (PyCFunction)_queue_SimpleQueue_empty, METH_NOARGS, _queue_SimpleQueue_empty__doc__},
+
+static int
+_queue_SimpleQueue_empty_impl(simplequeueobject *self);
+
+static PyObject *
+_queue_SimpleQueue_empty(simplequeueobject *self, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    int _return_value;
+
+    _return_value = _queue_SimpleQueue_empty_impl(self);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyBool_FromLong((long)_return_value);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_queue_SimpleQueue_qsize__doc__,
+"qsize($self, /)\n"
+"--\n"
+"\n"
+"Return the approximate size of the queue (not reliable!).");
+
+#define _QUEUE_SIMPLEQUEUE_QSIZE_METHODDEF    \
+    {"qsize", (PyCFunction)_queue_SimpleQueue_qsize, METH_NOARGS, _queue_SimpleQueue_qsize__doc__},
+
+static Py_ssize_t
+_queue_SimpleQueue_qsize_impl(simplequeueobject *self);
+
+static PyObject *
+_queue_SimpleQueue_qsize(simplequeueobject *self, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    Py_ssize_t _return_value;
+
+    _return_value = _queue_SimpleQueue_qsize_impl(self);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyLong_FromSsize_t(_return_value);
+
+exit:
+    return return_value;
+}
+/*[clinic end generated code: output=8badc3bb85263689 input=a9049054013a1b77]*/
diff --git a/PCbuild/_queue.vcxproj b/PCbuild/_queue.vcxproj
new file mode 100644
index 00000000000..81df2dfe514
--- /dev/null
+++ b/PCbuild/_queue.vcxproj
@@ -0,0 +1,77 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup Label="ProjectConfigurations">
+    <ProjectConfiguration Include="Debug|Win32">
+      <Configuration>Debug</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Debug|x64">
+      <Configuration>Debug</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|Win32">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|x64">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|Win32">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|x64">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|Win32">
+      <Configuration>Release</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|x64">
+      <Configuration>Release</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+  </ItemGroup>
+  <PropertyGroup Label="Globals">
+    <ProjectGuid>{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}</ProjectGuid>
+    <RootNamespace>_queue</RootNamespace>
+    <Keyword>Win32Proj</Keyword>
+  </PropertyGroup>
+  <Import Project="python.props" />
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>DynamicLibrary</ConfigurationType>
+    <CharacterSet>NotSet</CharacterSet>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+  <PropertyGroup>
+    <TargetExt>.pyd</TargetExt>
+  </PropertyGroup>
+  <ImportGroup Label="ExtensionSettings">
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="pyproject.props" />
+  </ImportGroup>
+  <PropertyGroup Label="UserMacros" />
+  <PropertyGroup>
+    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
+  </PropertyGroup>
+  <ItemGroup>
+    <ClCompile Include="..\Modules\_queuemodule.c" />
+  </ItemGroup>
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <ProjectReference Include="pythoncore.vcxproj">
+      <Project>{cf7ac3d1-e2df-41d2-bea6-1e2556cdea26}</Project>
+      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
+    </ProjectReference>
+  </ItemGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+  <ImportGroup Label="ExtensionTargets">
+  </ImportGroup>
+</Project>
diff --git a/PCbuild/_queue.vcxproj.filters b/PCbuild/_queue.vcxproj.filters
new file mode 100644
index 00000000000..88b80826adf
--- /dev/null
+++ b/PCbuild/_queue.vcxproj.filters
@@ -0,0 +1,16 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{c56a5dd3-7838-48e9-a781-855d8be7370f}</UniqueIdentifier>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="..\Modules\_queuemodule.c">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
\ No newline at end of file
diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj
index 22cf7b54189..05e885717ff 100644
--- a/PCbuild/pcbuild.proj
+++ b/PCbuild/pcbuild.proj
@@ -49,7 +49,7 @@
     <!-- pyshellext.dll -->
     <Projects Include="pyshellext.vcxproj" />
     <!-- Extension modules -->
-    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;select;unicodedata;winsound" />
+    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;_queue;select;unicodedata;winsound" />
     <!-- Extension modules that require external sources -->
     <ExternalModules Include="_bz2;_lzma;_sqlite3" />
     <!-- _ssl will build _socket as well, which may cause conflicts in parallel builds -->
diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln
index 82cfaf249d7..5ff86a49c7c 100644
--- a/PCbuild/pcbuild.sln
+++ b/PCbuild/pcbuild.sln
@@ -89,6 +89,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testconsole", "_testconsol
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_asyncio", "_asyncio.vcxproj", "{384C224A-7474-476E-A01B-750EA7DE918C}"
 EndProject
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_queue", "_queue.vcxproj", "{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}"
+EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "liblzma", "liblzma.vcxproj", "{12728250-16EC-4DC6-94D7-E21DD88947F8}"
 EndProject
 Global
@@ -659,6 +661,22 @@ Global
 		{384C224A-7474-476E-A01B-750EA7DE918C}.Release|Win32.Build.0 = Release|Win32
 		{384C224A-7474-476E-A01B-750EA7DE918C}.Release|x64.ActiveCfg = Release|x64
 		{384C224A-7474-476E-A01B-750EA7DE918C}.Release|x64.Build.0 = Release|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Debug|Win32.ActiveCfg = Debug|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Debug|Win32.Build.0 = Debug|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Debug|x64.ActiveCfg = Debug|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Debug|x64.Build.0 = Debug|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGInstrument|Win32.Build.0 = PGInstrument|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGInstrument|x64.ActiveCfg = PGInstrument|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGInstrument|x64.Build.0 = PGInstrument|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGUpdate|Win32.Build.0 = PGUpdate|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGUpdate|x64.ActiveCfg = PGUpdate|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.PGUpdate|x64.Build.0 = PGUpdate|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Release|Win32.ActiveCfg = Release|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Release|Win32.Build.0 = Release|Win32
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Release|x64.ActiveCfg = Release|x64
+		{78D80A15-BD8C-44E2-B49E-1F05B0A0A687}.Release|x64.Build.0 = Release|x64
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Debug|Win32.ActiveCfg = Debug|Win32
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Debug|Win32.Build.0 = Debug|Win32
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Debug|x64.ActiveCfg = Debug|x64
diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters
index 42b2fc3ea7c..13600cb5c9f 100644
--- a/PCbuild/pythoncore.vcxproj.filters
+++ b/PCbuild/pythoncore.vcxproj.filters
@@ -506,6 +506,9 @@
     <ClCompile Include="..\Modules\_pickle.c">
       <Filter>Modules</Filter>
     </ClCompile>
+    <ClCompile Include="..\Modules\_queuemodule.c">
+      <Filter>Modules</Filter>
+    </ClCompile>
     <ClCompile Include="..\Modules\_randommodule.c">
       <Filter>Modules</Filter>
     </ClCompile>
diff --git a/setup.py b/setup.py
index 250425ebd31..8e98fdcb7ef 100644
--- a/setup.py
+++ b/setup.py
@@ -699,6 +699,8 @@ def detect_modules(self):
         exts.append( Extension('_opcode', ['_opcode.c']) )
         # asyncio speedups
         exts.append( Extension("_asyncio", ["_asynciomodule.c"]) )
+        # _queue module
+        exts.append( Extension("_queue", ["_queuemodule.c"]) )
 
         # Modules with some UNIX dependencies -- on by default:
         # (If you have a really backward UNIX, select and socket may not be


From solipsis at pitrou.net  Tue Jan 16 04:13:43 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 16 Jan 2018 09:13:43 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=13
Message-ID: <20180116091343.1.D8C1C8C572123920@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, 7] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, 0, 2] memory blocks, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloggO8vXI', '--timeout', '7200']

From webhook-mailer at python.org  Tue Jan 16 04:30:29 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Tue, 16 Jan 2018 09:30:29 -0000
Subject: [Python-checkins] bpo-26163: Frozenset hash improvement (#5194)
Message-ID: <mailman.32.1516095031.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/b44c5169f64178d2ff2914187b315549e7ab0cb6
commit: b44c5169f64178d2ff2914187b315549e7ab0cb6
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-16T01:30:26-08:00
summary:

bpo-26163: Frozenset hash improvement (#5194)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst
M Objects/setobject.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst
new file mode 100644
index 00000000000..163b9e02c5a
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst	
@@ -0,0 +1,2 @@
+Improved frozenset() hash to create more distinct hash values when faced
+with datasets containing many similar values.
diff --git a/Objects/setobject.c b/Objects/setobject.c
index c920fb98534..4bc1020d56f 100644
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -795,6 +795,7 @@ frozenset_hash(PyObject *self)
     hash ^= ((Py_uhash_t)PySet_GET_SIZE(self) + 1) * 1927868237UL;
 
     /* Disperse patterns arising in nested frozensets */
+    hash ^= (hash >> 11) ^ (~hash >> 25);
     hash = hash * 69069U + 907133923UL;
 
     /* -1 is reserved as an error code */


From webhook-mailer at python.org  Tue Jan 16 05:27:18 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Tue, 16 Jan 2018 10:27:18 -0000
Subject: [Python-checkins] bpo-26163: Frozenset hash improvement (GH-5194)
 (#5198)
Message-ID: <mailman.33.1516098439.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/e7dbd0658304f11daf103d56662656efecad006d
commit: e7dbd0658304f11daf103d56662656efecad006d
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-16T02:27:15-08:00
summary:

bpo-26163: Frozenset hash improvement (GH-5194) (#5198)

(cherry picked from commit b44c5169f64178d2ff2914187b315549e7ab0cb6)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst
M Objects/setobject.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst
new file mode 100644
index 00000000000..163b9e02c5a
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-14-20-32-47.bpo-26163.xv9Iuv.rst	
@@ -0,0 +1,2 @@
+Improved frozenset() hash to create more distinct hash values when faced
+with datasets containing many similar values.
diff --git a/Objects/setobject.c b/Objects/setobject.c
index 24272b4d14a..093a15f367f 100644
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -790,6 +790,7 @@ frozenset_hash(PyObject *self)
     hash ^= ((Py_uhash_t)PySet_GET_SIZE(self) + 1) * 1927868237UL;
 
     /* Disperse patterns arising in nested frozensets */
+    hash ^= (hash >> 11) ^ (~hash >> 25);
     hash = hash * 69069U + 907133923UL;
 
     /* -1 is reserved as an error code */


From webhook-mailer at python.org  Tue Jan 16 06:52:48 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Tue, 16 Jan 2018 11:52:48 -0000
Subject: [Python-checkins] bpo-32544: Speed up hasattr() and getattr()
 (GH-5173)
Message-ID: <mailman.34.1516103571.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/378edee0a3b913d60653dc17dfe61d83405a8135
commit: 378edee0a3b913d60653dc17dfe61d83405a8135
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-16T20:52:41+09:00
summary:

bpo-32544: Speed up hasattr() and getattr() (GH-5173)

AttributeError was raised always when attribute is not found.
This commit skip raising AttributeError when `tp_getattro` is `PyObject_GenericGetAttr`.
It makes hasattr() and getattr() about 4x faster when attribute is not found.

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-16-18-51-58.bpo-32544.ga-cFE.rst
M Include/object.h
M Modules/_threadmodule.c
M Objects/object.c
M Python/bltinmodule.c

diff --git a/Include/object.h b/Include/object.h
index 7db5bfea615..274b67656ee 100644
--- a/Include/object.h
+++ b/Include/object.h
@@ -536,6 +536,8 @@ PyAPI_FUNC(int) PyObject_SetAttr(PyObject *, PyObject *, PyObject *);
 PyAPI_FUNC(int) PyObject_HasAttr(PyObject *, PyObject *);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(int) _PyObject_IsAbstract(PyObject *);
+/* Same as PyObject_GetAttr(), but don't raise AttributeError. */
+PyAPI_FUNC(PyObject *) _PyObject_GetAttrWithoutError(PyObject *, PyObject *);
 PyAPI_FUNC(PyObject *) _PyObject_GetAttrId(PyObject *, struct _Py_Identifier *);
 PyAPI_FUNC(int) _PyObject_SetAttrId(PyObject *, struct _Py_Identifier *, PyObject *);
 PyAPI_FUNC(int) _PyObject_HasAttrId(PyObject *, struct _Py_Identifier *);
@@ -567,7 +569,7 @@ PyAPI_FUNC(int) PyObject_CallFinalizerFromDealloc(PyObject *);
 /* Same as PyObject_Generic{Get,Set}Attr, but passing the attributes
    dict as the last parameter. */
 PyAPI_FUNC(PyObject *)
-_PyObject_GenericGetAttrWithDict(PyObject *, PyObject *, PyObject *);
+_PyObject_GenericGetAttrWithDict(PyObject *, PyObject *, PyObject *, int);
 PyAPI_FUNC(int)
 _PyObject_GenericSetAttrWithDict(PyObject *, PyObject *,
                                  PyObject *, PyObject *);
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-16-18-51-58.bpo-32544.ga-cFE.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-16-18-51-58.bpo-32544.ga-cFE.rst
new file mode 100644
index 00000000000..55a52fa9f1c
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-16-18-51-58.bpo-32544.ga-cFE.rst	
@@ -0,0 +1,3 @@
+``hasattr(obj, name)`` and ``getattr(obj, name, default)`` are about 4 times
+faster than before when ``name`` is not found and ``obj`` doesn't override
+``__getattr__`` or ``__getattribute__``.
diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c
index c9171f52e38..f594bda678a 100644
--- a/Modules/_threadmodule.c
+++ b/Modules/_threadmodule.c
@@ -925,13 +925,15 @@ local_getattro(localobject *self, PyObject *name)
 
     if (Py_TYPE(self) != &localtype)
         /* use generic lookup for subtypes */
-        return _PyObject_GenericGetAttrWithDict((PyObject *)self, name, ldict);
+        return _PyObject_GenericGetAttrWithDict(
+            (PyObject *)self, name, ldict, 0);
 
     /* Optimization: just look in dict ourselves */
     value = PyDict_GetItem(ldict, name);
     if (value == NULL)
         /* Fall back on generic to get __class__ and __dict__ */
-        return _PyObject_GenericGetAttrWithDict((PyObject *)self, name, ldict);
+        return _PyObject_GenericGetAttrWithDict(
+            (PyObject *)self, name, ldict, 0);
 
     Py_INCREF(value);
     return value;
diff --git a/Objects/object.c b/Objects/object.c
index a0d651d0805..62d7fbebf40 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -887,10 +887,41 @@ PyObject_GetAttr(PyObject *v, PyObject *name)
     return NULL;
 }
 
+PyObject *
+_PyObject_GetAttrWithoutError(PyObject *v, PyObject *name)
+{
+    PyTypeObject *tp = Py_TYPE(v);
+    PyObject *ret = NULL;
+
+    if (!PyUnicode_Check(name)) {
+        PyErr_Format(PyExc_TypeError,
+                     "attribute name must be string, not '%.200s'",
+                     name->ob_type->tp_name);
+        return NULL;
+    }
+
+    if (tp->tp_getattro == PyObject_GenericGetAttr) {
+        return _PyObject_GenericGetAttrWithDict(v, name, NULL, 1);
+    }
+    if (tp->tp_getattro != NULL) {
+        ret = (*tp->tp_getattro)(v, name);
+    }
+    else if (tp->tp_getattr != NULL) {
+        const char *name_str = PyUnicode_AsUTF8(name);
+        if (name_str == NULL)
+            return NULL;
+        ret = (*tp->tp_getattr)(v, (char *)name_str);
+    }
+    if (ret == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
+        PyErr_Clear();
+    }
+    return ret;
+}
+
 int
 PyObject_HasAttr(PyObject *v, PyObject *name)
 {
-    PyObject *res = PyObject_GetAttr(v, name);
+    PyObject *res = _PyObject_GetAttrWithoutError(v, name);
     if (res != NULL) {
         Py_DECREF(res);
         return 1;
@@ -1098,10 +1129,13 @@ _PyObject_GetMethod(PyObject *obj, PyObject *name, PyObject **method)
 /* Generic GetAttr functions - put these in your tp_[gs]etattro slot. */
 
 PyObject *
-_PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, PyObject *dict)
+_PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name,
+                                 PyObject *dict, int suppress)
 {
     /* Make sure the logic of _PyObject_GetMethod is in sync with
        this method.
+
+       When suppress=1, this function suppress AttributeError.
     */
 
     PyTypeObject *tp = Py_TYPE(obj);
@@ -1132,6 +1166,10 @@ _PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, PyObject *dict)
         f = descr->ob_type->tp_descr_get;
         if (f != NULL && PyDescr_IsData(descr)) {
             res = f(descr, obj, (PyObject *)obj->ob_type);
+            if (res == NULL && suppress &&
+                    PyErr_ExceptionMatches(PyExc_AttributeError)) {
+                PyErr_Clear();
+            }
             goto done;
         }
     }
@@ -1171,6 +1209,10 @@ _PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, PyObject *dict)
 
     if (f != NULL) {
         res = f(descr, obj, (PyObject *)Py_TYPE(obj));
+        if (res == NULL && suppress &&
+                PyErr_ExceptionMatches(PyExc_AttributeError)) {
+            PyErr_Clear();
+        }
         goto done;
     }
 
@@ -1180,9 +1222,11 @@ _PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, PyObject *dict)
         goto done;
     }
 
-    PyErr_Format(PyExc_AttributeError,
-                 "'%.50s' object has no attribute '%U'",
-                 tp->tp_name, name);
+    if (!suppress) {
+        PyErr_Format(PyExc_AttributeError,
+                     "'%.50s' object has no attribute '%U'",
+                     tp->tp_name, name);
+    }
   done:
     Py_XDECREF(descr);
     Py_DECREF(name);
@@ -1192,7 +1236,7 @@ _PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, PyObject *dict)
 PyObject *
 PyObject_GenericGetAttr(PyObject *obj, PyObject *name)
 {
-    return _PyObject_GenericGetAttrWithDict(obj, name, NULL);
+    return _PyObject_GenericGetAttrWithDict(obj, name, NULL, 0);
 }
 
 int
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
index e702f7c6e9e..844548f75fd 100644
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -1126,13 +1126,15 @@ builtin_getattr(PyObject *self, PyObject *const *args, Py_ssize_t nargs)
                         "getattr(): attribute name must be string");
         return NULL;
     }
-    result = PyObject_GetAttr(v, name);
-    if (result == NULL && dflt != NULL &&
-        PyErr_ExceptionMatches(PyExc_AttributeError))
-    {
-        PyErr_Clear();
-        Py_INCREF(dflt);
-        result = dflt;
+    if (dflt != NULL) {
+        result = _PyObject_GetAttrWithoutError(v, name);
+        if (result == NULL && !PyErr_Occurred()) {
+            Py_INCREF(dflt);
+            return dflt;
+        }
+    }
+    else {
+        result = PyObject_GetAttr(v, name);
     }
     return result;
 }
@@ -1189,10 +1191,9 @@ builtin_hasattr_impl(PyObject *module, PyObject *obj, PyObject *name)
                         "hasattr(): attribute name must be string");
         return NULL;
     }
-    v = PyObject_GetAttr(obj, name);
+    v = _PyObject_GetAttrWithoutError(obj, name);
     if (v == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
+        if (!PyErr_Occurred()) {
             Py_RETURN_FALSE;
         }
         return NULL;


From webhook-mailer at python.org  Tue Jan 16 11:34:27 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 16 Jan 2018 16:34:27 -0000
Subject: [Python-checkins] bpo-31572: Get rid of PyObject_HasAttr() and
 _PyObject_HasAttrId() in the _io module. (#3726)
Message-ID: <mailman.35.1516120468.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/4d9aec022063dcfc4cf40ae46b1c4a968297e664
commit: 4d9aec022063dcfc4cf40ae46b1c4a968297e664
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-16T18:34:21+02:00
summary:

bpo-31572: Get rid of PyObject_HasAttr() and _PyObject_HasAttrId() in the _io module. (#3726)

files:
M Modules/_io/_iomodule.c
M Modules/_io/_iomodule.h
M Modules/_io/bufferedio.c
M Modules/_io/iobase.c
M Modules/_io/textio.c

diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c
index cfd78e2d04a..11d7b018831 100644
--- a/Modules/_io/_iomodule.c
+++ b/Modules/_io/_iomodule.c
@@ -36,6 +36,7 @@ PyObject *_PyIO_str_getstate = NULL;
 PyObject *_PyIO_str_isatty = NULL;
 PyObject *_PyIO_str_newlines = NULL;
 PyObject *_PyIO_str_nl = NULL;
+PyObject *_PyIO_str_peek = NULL;
 PyObject *_PyIO_str_read = NULL;
 PyObject *_PyIO_str_read1 = NULL;
 PyObject *_PyIO_str_readable = NULL;
@@ -740,6 +741,7 @@ PyInit__io(void)
     ADD_INTERNED(getstate)
     ADD_INTERNED(isatty)
     ADD_INTERNED(newlines)
+    ADD_INTERNED(peek)
     ADD_INTERNED(read)
     ADD_INTERNED(read1)
     ADD_INTERNED(readable)
diff --git a/Modules/_io/_iomodule.h b/Modules/_io/_iomodule.h
index db8403774ea..4d318acd0b3 100644
--- a/Modules/_io/_iomodule.h
+++ b/Modules/_io/_iomodule.h
@@ -164,6 +164,7 @@ extern PyObject *_PyIO_str_getstate;
 extern PyObject *_PyIO_str_isatty;
 extern PyObject *_PyIO_str_newlines;
 extern PyObject *_PyIO_str_nl;
+extern PyObject *_PyIO_str_peek;
 extern PyObject *_PyIO_str_read;
 extern PyObject *_PyIO_str_read1;
 extern PyObject *_PyIO_str_readable;
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
index d7e82b9dba1..b81abde5c0a 100644
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -1521,7 +1521,7 @@ static PyObject *
 _bufferedreader_read_all(buffered *self)
 {
     Py_ssize_t current_size;
-    PyObject *res = NULL, *data = NULL, *tmp = NULL, *chunks = NULL;
+    PyObject *res = NULL, *data = NULL, *tmp = NULL, *chunks = NULL, *readall;
 
     /* First copy what we have in the current buffer. */
     current_size = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
@@ -1541,32 +1541,28 @@ _bufferedreader_read_all(buffered *self)
     }
     _bufferedreader_reset_buf(self);
 
-    if (PyObject_HasAttr(self->raw, _PyIO_str_readall)) {
-        tmp = PyObject_CallMethodObjArgs(self->raw, _PyIO_str_readall, NULL);
+    readall = _PyObject_GetAttrWithoutError(self->raw, _PyIO_str_readall);
+    if (readall) {
+        tmp = _PyObject_CallNoArg(readall);
+        Py_DECREF(readall);
         if (tmp == NULL)
             goto cleanup;
         if (tmp != Py_None && !PyBytes_Check(tmp)) {
             PyErr_SetString(PyExc_TypeError, "readall() should return bytes");
             goto cleanup;
         }
-        if (tmp == Py_None) {
-            if (current_size == 0) {
-                res = Py_None;
-                goto cleanup;
-            } else {
-                res = data;
-                goto cleanup;
+        if (current_size == 0) {
+            res = tmp;
+        } else {
+            if (tmp != Py_None) {
+                PyBytes_Concat(&data, tmp);
             }
-        }
-        else if (current_size) {
-            PyBytes_Concat(&data, tmp);
             res = data;
-            goto cleanup;
-        }
-        else {
-            res = tmp;
-            goto cleanup;
         }
+        goto cleanup;
+    }
+    else if (PyErr_Occurred()) {
+        goto cleanup;
     }
 
     chunks = PyList_New(0);
diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
index bcefb3862c3..348d449a272 100644
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -68,11 +68,9 @@ PyDoc_STRVAR(iobase_doc,
    by whatever subclass. */
 
 _Py_IDENTIFIER(__IOBase_closed);
-#define IS_CLOSED(self) \
-    _PyObject_HasAttrId(self, &PyId___IOBase_closed)
-
 _Py_IDENTIFIER(read);
 
+
 /* Internal methods */
 static PyObject *
 iobase_unsupported(const char *message)
@@ -131,6 +129,24 @@ iobase_truncate(PyObject *self, PyObject *args)
     return iobase_unsupported("truncate");
 }
 
+static int
+iobase_is_closed(PyObject *self)
+{
+    PyObject *res;
+    /* This gets the derived attribute, which is *not* __IOBase_closed
+       in most cases! */
+    res = _PyObject_GetAttrId(self, &PyId___IOBase_closed);
+    if (res == NULL) {
+        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
+            return -1;
+        }
+        PyErr_Clear();
+        return 0;
+    }
+    Py_DECREF(res);
+    return 1;
+}
+
 /* Flush and close methods */
 
 /*[clinic input]
@@ -146,45 +162,60 @@ _io__IOBase_flush_impl(PyObject *self)
 /*[clinic end generated code: output=7cef4b4d54656a3b input=773be121abe270aa]*/
 {
     /* XXX Should this return the number of bytes written??? */
-    if (IS_CLOSED(self)) {
+    int closed = iobase_is_closed(self);
+
+    if (!closed) {
+        Py_RETURN_NONE;
+    }
+    if (closed > 0) {
         PyErr_SetString(PyExc_ValueError, "I/O operation on closed file.");
+    }
+    return NULL;
+}
+
+static PyObject *
+iobase_closed_get(PyObject *self, void *context)
+{
+    int closed = iobase_is_closed(self);
+    if (closed < 0) {
         return NULL;
     }
-    Py_RETURN_NONE;
+    return PyBool_FromLong(closed);
 }
 
 static int
-iobase_closed(PyObject *self)
+iobase_check_closed(PyObject *self)
 {
     PyObject *res;
     int closed;
     /* This gets the derived attribute, which is *not* __IOBase_closed
        in most cases! */
-    res = PyObject_GetAttr(self, _PyIO_str_closed);
-    if (res == NULL)
+    res = _PyObject_GetAttrWithoutError(self, _PyIO_str_closed);
+    if (res == NULL) {
+        if (PyErr_Occurred()) {
+            return -1;
+        }
         return 0;
+    }
     closed = PyObject_IsTrue(res);
     Py_DECREF(res);
-    return closed;
-}
-
-static PyObject *
-iobase_closed_get(PyObject *self, void *context)
-{
-    return PyBool_FromLong(IS_CLOSED(self));
+    if (closed <= 0) {
+        return closed;
+    }
+    PyErr_SetString(PyExc_ValueError, "I/O operation on closed file.");
+    return -1;
 }
 
 PyObject *
 _PyIOBase_check_closed(PyObject *self, PyObject *args)
 {
-    if (iobase_closed(self)) {
-        PyErr_SetString(PyExc_ValueError, "I/O operation on closed file.");
+    if (iobase_check_closed(self)) {
         return NULL;
     }
-    if (args == Py_True)
+    if (args == Py_True) {
         return Py_None;
-    else
-        Py_RETURN_NONE;
+    }
+    Py_RETURN_NONE;
 }
 
 /* XXX: IOBase thinks it has to maintain its own internal state in
@@ -204,9 +235,14 @@ _io__IOBase_close_impl(PyObject *self)
 /*[clinic end generated code: output=63c6a6f57d783d6d input=f4494d5c31dbc6b7]*/
 {
     PyObject *res;
+    int closed = iobase_is_closed(self);
 
-    if (IS_CLOSED(self))
+    if (closed < 0) {
+        return NULL;
+    }
+    if (closed) {
         Py_RETURN_NONE;
+    }
 
     res = PyObject_CallMethodObjArgs(self, _PyIO_str_flush, NULL);
 
@@ -237,7 +273,7 @@ iobase_finalize(PyObject *self)
 
     /* If `closed` doesn't exist or can't be evaluated as bool, then the
        object is probably in an unusable state, so ignore. */
-    res = PyObject_GetAttr(self, _PyIO_str_closed);
+    res = _PyObject_GetAttrWithoutError(self, _PyIO_str_closed);
     if (res == NULL) {
         PyErr_Clear();
         closed = -1;
@@ -428,7 +464,7 @@ _PyIOBase_check_writable(PyObject *self, PyObject *args)
 static PyObject *
 iobase_enter(PyObject *self, PyObject *args)
 {
-    if (_PyIOBase_check_closed(self, Py_True) == NULL)
+    if (iobase_check_closed(self))
         return NULL;
 
     Py_INCREF(self);
@@ -472,7 +508,7 @@ static PyObject *
 _io__IOBase_isatty_impl(PyObject *self)
 /*[clinic end generated code: output=60cab77cede41cdd input=9ef76530d368458b]*/
 {
-    if (_PyIOBase_check_closed(self, Py_True) == NULL)
+    if (iobase_check_closed(self))
         return NULL;
     Py_RETURN_FALSE;
 }
@@ -499,24 +535,26 @@ _io__IOBase_readline_impl(PyObject *self, Py_ssize_t limit)
 {
     /* For backwards compatibility, a (slowish) readline(). */
 
-    int has_peek = 0;
-    PyObject *buffer, *result;
+    PyObject *peek, *buffer, *result;
     Py_ssize_t old_size = -1;
-    _Py_IDENTIFIER(peek);
 
-    if (_PyObject_HasAttrId(self, &PyId_peek))
-        has_peek = 1;
+    peek = _PyObject_GetAttrWithoutError(self, _PyIO_str_peek);
+    if (peek == NULL && PyErr_Occurred()) {
+        return NULL;
+    }
 
     buffer = PyByteArray_FromStringAndSize(NULL, 0);
-    if (buffer == NULL)
+    if (buffer == NULL) {
+        Py_XDECREF(peek);
         return NULL;
+    }
 
     while (limit < 0 || PyByteArray_GET_SIZE(buffer) < limit) {
         Py_ssize_t nreadahead = 1;
         PyObject *b;
 
-        if (has_peek) {
-            PyObject *readahead = _PyObject_CallMethodId(self, &PyId_peek, "i", 1);
+        if (peek != NULL) {
+            PyObject *readahead = PyObject_CallFunctionObjArgs(peek, _PyLong_One, NULL);
             if (readahead == NULL) {
                 /* NOTE: PyErr_SetFromErrno() calls PyErr_CheckSignals()
                    when EINTR occurs so we needn't do it ourselves. */
@@ -593,9 +631,11 @@ _io__IOBase_readline_impl(PyObject *self, Py_ssize_t limit)
 
     result = PyBytes_FromStringAndSize(PyByteArray_AS_STRING(buffer),
                                        PyByteArray_GET_SIZE(buffer));
+    Py_XDECREF(peek);
     Py_DECREF(buffer);
     return result;
   fail:
+    Py_XDECREF(peek);
     Py_DECREF(buffer);
     return NULL;
 }
@@ -603,7 +643,7 @@ _io__IOBase_readline_impl(PyObject *self, Py_ssize_t limit)
 static PyObject *
 iobase_iter(PyObject *self)
 {
-    if (_PyIOBase_check_closed(self, Py_True) == NULL)
+    if (iobase_check_closed(self))
         return NULL;
 
     Py_INCREF(self);
@@ -716,7 +756,7 @@ _io__IOBase_writelines(PyObject *self, PyObject *lines)
 {
     PyObject *iter, *res;
 
-    if (_PyIOBase_check_closed(self, Py_True) == NULL)
+    if (iobase_check_closed(self))
         return NULL;
 
     iter = PyObject_GetIter(lines);
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
index d776b5de699..9f3fd2d2346 100644
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -29,7 +29,6 @@ _Py_IDENTIFIER(mode);
 _Py_IDENTIFIER(name);
 _Py_IDENTIFIER(raw);
 _Py_IDENTIFIER(read);
-_Py_IDENTIFIER(read1);
 _Py_IDENTIFIER(readable);
 _Py_IDENTIFIER(replace);
 _Py_IDENTIFIER(reset);
@@ -1202,7 +1201,17 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
         goto error;
     self->seekable = self->telling = r;
 
-    self->has_read1 = _PyObject_HasAttrId(buffer, &PyId_read1);
+    res = _PyObject_GetAttrWithoutError(buffer, _PyIO_str_read1);
+    if (res != NULL) {
+        Py_DECREF(res);
+        self->has_read1 = 1;
+    }
+    else if (!PyErr_Occurred()) {
+        self->has_read1 = 0;
+    }
+    else {
+        goto error;
+    }
 
     self->encoding_start_of_stream = 0;
     if (_textiowrapper_fix_encoder_state(self) < 0) {
@@ -3013,15 +3022,9 @@ textiowrapper_newlines_get(textio *self, void *context)
     CHECK_ATTACHED(self);
     if (self->decoder == NULL)
         Py_RETURN_NONE;
-    res = PyObject_GetAttr(self->decoder, _PyIO_str_newlines);
-    if (res == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-            Py_RETURN_NONE;
-        }
-        else {
-            return NULL;
-        }
+    res = _PyObject_GetAttrWithoutError(self->decoder, _PyIO_str_newlines);
+    if (res == NULL && !PyErr_Occurred()) {
+        Py_RETURN_NONE;
     }
     return res;
 }


From webhook-mailer at python.org  Tue Jan 16 11:34:37 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 16 Jan 2018 16:34:37 -0000
Subject: [Python-checkins] Skip test_readline.test_nonascii() on C locale
 (#5203)
Message-ID: <mailman.36.1516120479.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/c495e799ed376af91ae2ddf6c4bcc592490fe294
commit: c495e799ed376af91ae2ddf6c4bcc592490fe294
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-16T17:34:34+01:00
summary:

Skip test_readline.test_nonascii() on C locale (#5203)

bpo-29240: On FreeBSD, if the LC_CTYPE locale is "C" or "POSIX",
writing and reading non-ASCII bytes into/from a TTY works,
but readline or ncurses ignores non-ASCII bytes on read.

files:
M Lib/test/test_readline.py

diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py
index b4c25dee9d3..67ee9b7f7cf 100644
--- a/Lib/test/test_readline.py
+++ b/Lib/test/test_readline.py
@@ -3,6 +3,7 @@
 """
 from contextlib import ExitStack
 from errno import EIO
+import locale
 import os
 import selectors
 import subprocess
@@ -153,6 +154,13 @@ def test_auto_history_disabled(self):
         self.assertIn(b"History length: 0\r\n", output)
 
     def test_nonascii(self):
+        loc = locale.setlocale(locale.LC_CTYPE, None)
+        if loc in ('C', 'POSIX'):
+            # bpo-29240: On FreeBSD, if the LC_CTYPE locale is C or POSIX,
+            # writing and reading non-ASCII bytes into/from a TTY works, but
+            # readline or ncurses ignores non-ASCII bytes on read.
+            self.skipTest(f"the LC_CTYPE locale is {loc!r}")
+
         try:
             readline.add_history("\xEB\xEF")
         except UnicodeEncodeError as err:


From webhook-mailer at python.org  Tue Jan 16 12:27:36 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 16 Jan 2018 17:27:36 -0000
Subject: [Python-checkins] Skip test_readline.test_nonascii() on C locale
 (GH-5203) (#5204)
Message-ID: <mailman.37.1516123658.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/c2740e8a263e76427a8102a89f4b491a3089b2a1
commit: c2740e8a263e76427a8102a89f4b491a3089b2a1
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-16T18:27:29+01:00
summary:

Skip test_readline.test_nonascii() on C locale (GH-5203) (#5204)

bpo-29240: On FreeBSD, if the LC_CTYPE locale is "C" or "POSIX",
writing and reading non-ASCII bytes into/from a TTY works,
but readline or ncurses ignores non-ASCII bytes on read.
(cherry picked from commit c495e799ed376af91ae2ddf6c4bcc592490fe294)

files:
M Lib/test/test_readline.py

diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py
index 9c979652a1d..dd9d3b57fcb 100644
--- a/Lib/test/test_readline.py
+++ b/Lib/test/test_readline.py
@@ -3,6 +3,7 @@
 """
 from contextlib import ExitStack
 from errno import EIO
+import locale
 import os
 import selectors
 import subprocess
@@ -137,6 +138,13 @@ def test_auto_history_disabled(self):
         self.assertIn(b"History length: 0\r\n", output)
 
     def test_nonascii(self):
+        loc = locale.setlocale(locale.LC_CTYPE, None)
+        if loc in ('C', 'POSIX'):
+            # bpo-29240: On FreeBSD, if the LC_CTYPE locale is C or POSIX,
+            # writing and reading non-ASCII bytes into/from a TTY works, but
+            # readline or ncurses ignores non-ASCII bytes on read.
+            self.skipTest(f"the LC_CTYPE locale is {loc!r}")
+
         try:
             readline.add_history("\xEB\xEF")
         except UnicodeEncodeError as err:


From webhook-mailer at python.org  Tue Jan 16 12:59:39 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Tue, 16 Jan 2018 17:59:39 -0000
Subject: [Python-checkins] bpo-32410: Implement loop.sock_sendfile() (#4976)
Message-ID: <mailman.38.1516125581.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/6b5a27975a415108a5eac12ee302bf2b3233f4d4
commit: 6b5a27975a415108a5eac12ee302bf2b3233f4d4
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-16T19:59:34+02:00
summary:

bpo-32410: Implement loop.sock_sendfile() (#4976)

files:
A Misc/NEWS.d/next/Library/2017-12-22-16-05-01.bpo-32410.8JzhvH.rst
M Doc/library/asyncio-eventloop.rst
M Lib/asyncio/base_events.py
M Lib/asyncio/events.py
M Lib/asyncio/unix_events.py
M Lib/test/test_asyncio/test_base_events.py
M Lib/test/test_asyncio/test_events.py
M Lib/test/test_asyncio/test_unix_events.py

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 4fbbcd86cf4..e63180b750a 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -701,6 +701,36 @@ Low-level socket operations
 
       :meth:`AbstractEventLoop.create_server` and :func:`start_server`.
 
+.. coroutinemethod:: AbstractEventLoop.sock_sendfile(sock, file, \
+                                                     offset=0, count=None, \
+                                                     *, fallback=True)
+
+   Send a file using high-performance :mod:`os.sendfile` if possible
+   and return the total number of bytes which were sent.
+
+   Asynchronous version of :meth:`socket.socket.sendfile`.
+
+   *sock* must be non-blocking :class:`~socket.socket` of
+   :const:`socket.SOCK_STREAM` type.
+
+   *file* must be a regular file object opened in binary mode.
+
+   *offset* tells from where to start reading the file. If specified,
+   *count* is the total number of bytes to transmit as opposed to
+   sending the file until EOF is reached. File position is updated on
+   return or also in case of error in which case :meth:`file.tell()
+   <io.IOBase.tell>` can be used to figure out the number of bytes
+   which were sent.
+
+   *fallback* set to ``True`` makes asyncio to manually read and send
+   the file when the platform does not support the sendfile syscall
+   (e.g. Windows or SSL socket on Unix).
+
+   Raise :exc:`RuntimeError` if the system does not support
+   *sendfile* syscall and *fallback* is ``False``.
+
+   .. versionadded:: 3.7
+
 
 Resolve host name
 -----------------
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index ab002319c18..b6a9384d95b 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -154,6 +154,10 @@ def _run_until_complete_cb(fut):
     futures._get_loop(fut).stop()
 
 
+class _SendfileNotAvailable(RuntimeError):
+    pass
+
+
 class Server(events.AbstractServer):
 
     def __init__(self, loop, sockets):
@@ -647,6 +651,72 @@ def _getaddrinfo_debug(self, host, port, family, type, proto, flags):
         return await self.run_in_executor(
             None, socket.getnameinfo, sockaddr, flags)
 
+    async def sock_sendfile(self, sock, file, offset=0, count=None,
+                            *, fallback=True):
+        if self._debug and sock.gettimeout() != 0:
+            raise ValueError("the socket must be non-blocking")
+        self._check_sendfile_params(sock, file, offset, count)
+        try:
+            return await self._sock_sendfile_native(sock, file,
+                                                    offset, count)
+        except _SendfileNotAvailable as exc:
+            if fallback:
+                return await self._sock_sendfile_fallback(sock, file,
+                                                          offset, count)
+            else:
+                raise RuntimeError(exc.args[0]) from None
+
+    async def _sock_sendfile_native(self, sock, file, offset, count):
+        # NB: sendfile syscall is not supported for SSL sockets and
+        # non-mmap files even if sendfile is supported by OS
+        raise _SendfileNotAvailable(
+            f"syscall sendfile is not available for socket {sock!r} "
+            "and file {file!r} combination")
+
+    async def _sock_sendfile_fallback(self, sock, file, offset, count):
+        if offset:
+            file.seek(offset)
+        blocksize = min(count, 16384) if count else 16384
+        buf = bytearray(blocksize)
+        total_sent = 0
+        try:
+            while True:
+                if count:
+                    blocksize = min(count - total_sent, blocksize)
+                    if blocksize <= 0:
+                        break
+                view = memoryview(buf)[:blocksize]
+                read = file.readinto(view)
+                if not read:
+                    break  # EOF
+                await self.sock_sendall(sock, view)
+                total_sent += read
+            return total_sent
+        finally:
+            if total_sent > 0 and hasattr(file, 'seek'):
+                file.seek(offset + total_sent)
+
+    def _check_sendfile_params(self, sock, file, offset, count):
+        if 'b' not in getattr(file, 'mode', 'b'):
+            raise ValueError("file should be opened in binary mode")
+        if not sock.type == socket.SOCK_STREAM:
+            raise ValueError("only SOCK_STREAM type sockets are supported")
+        if count is not None:
+            if not isinstance(count, int):
+                raise TypeError(
+                    "count must be a positive integer (got {!r})".format(count))
+            if count <= 0:
+                raise ValueError(
+                    "count must be a positive integer (got {!r})".format(count))
+        if not isinstance(offset, int):
+            raise TypeError(
+                "offset must be a non-negative integer (got {!r})".format(
+                    offset))
+        if offset < 0:
+            raise ValueError(
+                "offset must be a non-negative integer (got {!r})".format(
+                    offset))
+
     async def create_connection(
             self, protocol_factory, host=None, port=None,
             *, ssl=None, family=0,
diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index af4545b2cbe..b06721f3b03 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -464,6 +464,10 @@ def remove_writer(self, fd):
     async def sock_accept(self, sock):
         raise NotImplementedError
 
+    async def sock_sendfile(self, sock, file, offset=0, count=None,
+                            *, fallback=None):
+        raise NotImplementedError
+
     # Signal handling.
 
     def add_signal_handler(self, sig, callback, *args):
diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
index 4f6beb43650..f40ef12f265 100644
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -1,6 +1,7 @@
 """Selector event loop for Unix with signal handling."""
 
 import errno
+import io
 import os
 import selectors
 import signal
@@ -308,6 +309,98 @@ def _child_watcher_callback(self, pid, returncode, transp):
                             ssl_handshake_timeout=ssl_handshake_timeout)
         return server
 
+    async def _sock_sendfile_native(self, sock, file, offset, count):
+        try:
+            os.sendfile
+        except AttributeError as exc:
+            raise base_events._SendfileNotAvailable(
+                "os.sendfile() is not available")
+        try:
+            fileno = file.fileno()
+        except (AttributeError, io.UnsupportedOperation) as err:
+            raise base_events._SendfileNotAvailable("not a regular file")
+        try:
+            fsize = os.fstat(fileno).st_size
+        except OSError as err:
+            raise base_events._SendfileNotAvailable("not a regular file")
+        blocksize = count if count else fsize
+        if not blocksize:
+            return 0  # empty file
+
+        fut = self.create_future()
+        self._sock_sendfile_native_impl(fut, None, sock, fileno,
+                                        offset, count, blocksize, 0)
+        return await fut
+
+    def _sock_sendfile_native_impl(self, fut, registered_fd, sock, fileno,
+                                   offset, count, blocksize, total_sent):
+        fd = sock.fileno()
+        if registered_fd is not None:
+            # Remove the callback early.  It should be rare that the
+            # selector says the fd is ready but the call still returns
+            # EAGAIN, and I am willing to take a hit in that case in
+            # order to simplify the common case.
+            self.remove_writer(registered_fd)
+        if fut.cancelled():
+            self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+            return
+        if count:
+            blocksize = count - total_sent
+            if blocksize <= 0:
+                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                fut.set_result(total_sent)
+                return
+
+        try:
+            sent = os.sendfile(fd, fileno, offset, blocksize)
+        except (BlockingIOError, InterruptedError):
+            if registered_fd is None:
+                self._sock_add_cancellation_callback(fut, sock)
+            self.add_writer(fd, self._sock_sendfile_native_impl, fut,
+                            fd, sock, fileno,
+                            offset, count, blocksize, total_sent)
+        except OSError as exc:
+            if total_sent == 0:
+                # We can get here for different reasons, the main
+                # one being 'file' is not a regular mmap(2)-like
+                # file, in which case we'll fall back on using
+                # plain send().
+                err = base_events._SendfileNotAvailable(
+                    "os.sendfile call failed")
+                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                fut.set_exception(err)
+            else:
+                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                fut.set_exception(exc)
+        except Exception as exc:
+            self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+            fut.set_exception(exc)
+        else:
+            if sent == 0:
+                # EOF
+                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                fut.set_result(total_sent)
+            else:
+                offset += sent
+                total_sent += sent
+                if registered_fd is None:
+                    self._sock_add_cancellation_callback(fut, sock)
+                self.add_writer(fd, self._sock_sendfile_native_impl, fut,
+                                fd, sock, fileno,
+                                offset, count, blocksize, total_sent)
+
+    def _sock_sendfile_update_filepos(self, fileno, offset, total_sent):
+        if total_sent > 0:
+            os.lseek(fileno, offset, os.SEEK_SET)
+
+    def _sock_add_cancellation_callback(self, fut, sock):
+        def cb(fut):
+            if fut.cancelled():
+                fd = sock.fileno()
+                if fd != -1:
+                    self.remove_writer(fd)
+        fut.add_done_callback(cb)
+
 
 class _UnixReadPipeTransport(transports.ReadTransport):
 
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index 1fc74737026..085124f5372 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -1787,5 +1787,163 @@ def runner(loop):
             outer_loop.close()
 
 
+class BaseLoopSendfileTests(test_utils.TestCase):
+
+    DATA = b"12345abcde" * 16 * 1024  # 160 KiB
+
+    class MyProto(asyncio.Protocol):
+
+        def __init__(self, loop):
+            self.started = False
+            self.closed = False
+            self.data = bytearray()
+            self.fut = loop.create_future()
+
+        def connection_made(self, transport):
+            self.started = True
+
+        def data_received(self, data):
+            self.data.extend(data)
+
+        def connection_lost(self, exc):
+            self.closed = True
+            self.fut.set_result(None)
+
+        async def wait_closed(self):
+            await self.fut
+
+    @classmethod
+    def setUpClass(cls):
+        with open(support.TESTFN, 'wb') as fp:
+            fp.write(cls.DATA)
+        super().setUpClass()
+
+    @classmethod
+    def tearDownClass(cls):
+        support.unlink(support.TESTFN)
+        super().tearDownClass()
+
+    def setUp(self):
+        from asyncio.selector_events import BaseSelectorEventLoop
+        # BaseSelectorEventLoop() has no native implementation
+        self.loop = BaseSelectorEventLoop()
+        self.set_event_loop(self.loop)
+        self.file = open(support.TESTFN, 'rb')
+        self.addCleanup(self.file.close)
+        super().setUp()
+
+    def make_socket(self, blocking=False):
+        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        sock.setblocking(blocking)
+        self.addCleanup(sock.close)
+        return sock
+
+    def run_loop(self, coro):
+        return self.loop.run_until_complete(coro)
+
+    def prepare(self):
+        sock = self.make_socket()
+        proto = self.MyProto(self.loop)
+        port = support.find_unused_port()
+        server = self.run_loop(self.loop.create_server(
+            lambda: proto, support.HOST, port))
+        self.run_loop(self.loop.sock_connect(sock, (support.HOST, port)))
+
+        def cleanup():
+            server.close()
+            self.run_loop(server.wait_closed())
+
+        self.addCleanup(cleanup)
+
+        return sock, proto
+
+    def test__sock_sendfile_native_failure(self):
+        sock, proto = self.prepare()
+
+        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+                                    "sendfile is not available"):
+            self.run_loop(self.loop._sock_sendfile_native(sock, self.file,
+                                                          0, None))
+
+        self.assertEqual(proto.data, b'')
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sock_sendfile_no_fallback(self):
+        sock, proto = self.prepare()
+
+        with self.assertRaisesRegex(RuntimeError,
+                                    "sendfile is not available"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file,
+                                                  fallback=False))
+
+        self.assertEqual(self.file.tell(), 0)
+        self.assertEqual(proto.data, b'')
+
+    def test_sock_sendfile_fallback(self):
+        sock, proto = self.prepare()
+
+        ret = self.run_loop(self.loop.sock_sendfile(sock, self.file))
+        sock.close()
+        self.run_loop(proto.wait_closed())
+
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(self.file.tell(), len(self.DATA))
+        self.assertEqual(proto.data, self.DATA)
+
+    def test_sock_sendfile_fallback_offset_and_count(self):
+        sock, proto = self.prepare()
+
+        ret = self.run_loop(self.loop.sock_sendfile(sock, self.file,
+                                                    1000, 2000))
+        sock.close()
+        self.run_loop(proto.wait_closed())
+
+        self.assertEqual(ret, 2000)
+        self.assertEqual(self.file.tell(), 3000)
+        self.assertEqual(proto.data, self.DATA[1000:3000])
+
+    def test_blocking_socket(self):
+        self.loop.set_debug(True)
+        sock = self.make_socket(blocking=True)
+        with self.assertRaisesRegex(ValueError, "must be non-blocking"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file))
+
+    def test_nonbinary_file(self):
+        sock = self.make_socket()
+        with open(support.TESTFN, 'r') as f:
+            with self.assertRaisesRegex(ValueError, "binary mode"):
+                self.run_loop(self.loop.sock_sendfile(sock, f))
+
+    def test_nonstream_socket(self):
+        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        self.addCleanup(sock.close)
+        with self.assertRaisesRegex(ValueError, "only SOCK_STREAM type"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file))
+
+    def test_notint_count(self):
+        sock = self.make_socket()
+        with self.assertRaisesRegex(TypeError,
+                                    "count must be a positive integer"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file, 0, 'count'))
+
+    def test_negative_count(self):
+        sock = self.make_socket()
+        with self.assertRaisesRegex(ValueError,
+                                    "count must be a positive integer"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file, 0, -1))
+
+    def test_notint_offset(self):
+        sock = self.make_socket()
+        with self.assertRaisesRegex(TypeError,
+                                    "offset must be a non-negative integer"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file, 'offset'))
+
+    def test_negative_offset(self):
+        sock = self.make_socket()
+        with self.assertRaisesRegex(ValueError,
+                                    "offset must be a non-negative integer"):
+            self.run_loop(self.loop.sock_sendfile(sock, self.file, -1))
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index f63fd3c723a..4140f03967e 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -2555,6 +2555,8 @@ def test_not_implemented_async(self):
                 await loop.sock_connect(f, f)
             with self.assertRaises(NotImplementedError):
                 await loop.sock_accept(f)
+            with self.assertRaises(NotImplementedError):
+                await loop.sock_sendfile(f, mock.Mock())
             with self.assertRaises(NotImplementedError):
                 await loop.connect_read_pipe(f, mock.sentinel.pipe)
             with self.assertRaises(NotImplementedError):
diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py
index 53ed3d94ddb..4e2b76bb472 100644
--- a/Lib/test/test_asyncio/test_unix_events.py
+++ b/Lib/test/test_asyncio/test_unix_events.py
@@ -1,6 +1,7 @@
 """Tests for unix_events.py."""
 
 import collections
+import contextlib
 import errno
 import io
 import os
@@ -21,6 +22,7 @@
 
 import asyncio
 from asyncio import log
+from asyncio import base_events
 from asyncio import unix_events
 from test.test_asyncio import utils as test_utils
 
@@ -417,6 +419,255 @@ def test_create_unix_connection_ssl_timeout_with_plain_sock(self):
             self.loop.run_until_complete(coro)
 
 
+ at unittest.skipUnless(hasattr(os, 'sendfile'),
+                     'sendfile is not supported')
+class SelectorEventLoopUnixSockSendfileTests(test_utils.TestCase):
+    DATA = b"12345abcde" * 16 * 1024  # 160 KiB
+
+    class MyProto(asyncio.Protocol):
+
+        def __init__(self, loop):
+            self.started = False
+            self.closed = False
+            self.data = bytearray()
+            self.fut = loop.create_future()
+            self.transport = None
+
+        def connection_made(self, transport):
+            self.started = True
+            self.transport = transport
+
+        def data_received(self, data):
+            self.data.extend(data)
+
+        def connection_lost(self, exc):
+            self.closed = True
+            self.fut.set_result(None)
+
+        async def wait_closed(self):
+            await self.fut
+
+    @classmethod
+    def setUpClass(cls):
+        with open(support.TESTFN, 'wb') as fp:
+            fp.write(cls.DATA)
+        super().setUpClass()
+
+    @classmethod
+    def tearDownClass(cls):
+        support.unlink(support.TESTFN)
+        super().tearDownClass()
+
+    def setUp(self):
+        self.loop = asyncio.new_event_loop()
+        self.set_event_loop(self.loop)
+        self.file = open(support.TESTFN, 'rb')
+        self.addCleanup(self.file.close)
+        super().setUp()
+
+    def make_socket(self, blocking=False):
+        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        sock.setblocking(blocking)
+        self.addCleanup(sock.close)
+        return sock
+
+    def run_loop(self, coro):
+        return self.loop.run_until_complete(coro)
+
+    def prepare(self):
+        sock = self.make_socket()
+        proto = self.MyProto(self.loop)
+        port = support.find_unused_port()
+        server = self.run_loop(self.loop.create_server(
+            lambda: proto, support.HOST, port))
+        self.run_loop(self.loop.sock_connect(sock, (support.HOST, port)))
+
+        def cleanup():
+            proto.transport.close()
+            self.run_loop(proto.wait_closed())
+
+            server.close()
+            self.run_loop(server.wait_closed())
+
+        self.addCleanup(cleanup)
+
+        return sock, proto
+
+    def test_success(self):
+        sock, proto = self.prepare()
+        ret = self.run_loop(self.loop.sock_sendfile(sock, self.file))
+        sock.close()
+        self.run_loop(proto.wait_closed())
+
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(proto.data, self.DATA)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_with_offset_and_count(self):
+        sock, proto = self.prepare()
+        ret = self.run_loop(self.loop.sock_sendfile(sock, self.file,
+                                                    1000, 2000))
+        sock.close()
+        self.run_loop(proto.wait_closed())
+
+        self.assertEqual(proto.data, self.DATA[1000:3000])
+        self.assertEqual(self.file.tell(), 3000)
+        self.assertEqual(ret, 2000)
+
+    def test_sendfile_not_available(self):
+        sock, proto = self.prepare()
+        with mock.patch('asyncio.unix_events.os', spec=[]):
+            with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+                                        "os[.]sendfile[(][)] is not available"):
+                self.run_loop(self.loop._sock_sendfile_native(sock, self.file,
+                                                              0, None))
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sendfile_not_a_file(self):
+        sock, proto = self.prepare()
+        f = object()
+        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+                                    "not a regular file"):
+            self.run_loop(self.loop._sock_sendfile_native(sock, f,
+                                                          0, None))
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sendfile_iobuffer(self):
+        sock, proto = self.prepare()
+        f = io.BytesIO()
+        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+                                    "not a regular file"):
+            self.run_loop(self.loop._sock_sendfile_native(sock, f,
+                                                          0, None))
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sendfile_not_regular_file(self):
+        sock, proto = self.prepare()
+        f = mock.Mock()
+        f.fileno.return_value = -1
+        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+                                    "not a regular file"):
+            self.run_loop(self.loop._sock_sendfile_native(sock, f,
+                                                          0, None))
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sendfile_zero_size(self):
+        sock, proto = self.prepare()
+        fname = support.TESTFN + '.suffix'
+        with open(fname, 'wb') as f:
+            pass  # make zero sized file
+        f = open(fname, 'rb')
+        self.addCleanup(f.close)
+        self.addCleanup(support.unlink, fname)
+        ret = self.run_loop(self.loop._sock_sendfile_native(sock, f,
+                                                            0, None))
+        sock.close()
+        self.run_loop(proto.wait_closed())
+
+        self.assertEqual(ret, 0)
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_mix_sendfile_and_regular_send(self):
+        buf = b'1234567890' * 1024 * 1024  # 10 MB
+        sock, proto = self.prepare()
+        self.run_loop(self.loop.sock_sendall(sock, buf))
+        ret = self.run_loop(self.loop.sock_sendfile(sock, self.file))
+        self.run_loop(self.loop.sock_sendall(sock, buf))
+        sock.close()
+        self.run_loop(proto.wait_closed())
+
+        self.assertEqual(ret, len(self.DATA))
+        expected = buf + self.DATA + buf
+        self.assertEqual(proto.data, expected)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_cancel1(self):
+        sock, proto = self.prepare()
+
+        fut = self.loop.create_future()
+        fileno = self.file.fileno()
+        self.loop._sock_sendfile_native_impl(fut, None, sock, fileno,
+                                             0, None, len(self.DATA), 0)
+        fut.cancel()
+        with contextlib.suppress(asyncio.CancelledError):
+            self.run_loop(fut)
+        with self.assertRaises(KeyError):
+            self.loop._selector.get_key(sock)
+
+    def test_cancel2(self):
+        sock, proto = self.prepare()
+
+        fut = self.loop.create_future()
+        fileno = self.file.fileno()
+        self.loop._sock_sendfile_native_impl(fut, None, sock, fileno,
+                                             0, None, len(self.DATA), 0)
+        fut.cancel()
+        self.loop._sock_sendfile_native_impl(fut, sock.fileno(), sock, fileno,
+                                             0, None, len(self.DATA), 0)
+        with self.assertRaises(KeyError):
+            self.loop._selector.get_key(sock)
+
+    def test_blocking_error(self):
+        sock, proto = self.prepare()
+
+        fileno = self.file.fileno()
+        fut = mock.Mock()
+        fut.cancelled.return_value = False
+        with mock.patch('os.sendfile', side_effect=BlockingIOError()):
+            self.loop._sock_sendfile_native_impl(fut, None, sock, fileno,
+                                                 0, None, len(self.DATA), 0)
+        key = self.loop._selector.get_key(sock)
+        self.assertIsNotNone(key)
+        fut.add_done_callback.assert_called_once_with(mock.ANY)
+
+    def test_os_error_first_call(self):
+        sock, proto = self.prepare()
+
+        fileno = self.file.fileno()
+        fut = self.loop.create_future()
+        with mock.patch('os.sendfile', side_effect=OSError()):
+            self.loop._sock_sendfile_native_impl(fut, None, sock, fileno,
+                                                 0, None, len(self.DATA), 0)
+        with self.assertRaises(KeyError):
+            self.loop._selector.get_key(sock)
+        exc = fut.exception()
+        self.assertIsInstance(exc, base_events._SendfileNotAvailable)
+        self.assertEqual(0, self.file.tell())
+
+    def test_os_error_next_call(self):
+        sock, proto = self.prepare()
+
+        fileno = self.file.fileno()
+        fut = self.loop.create_future()
+        err = OSError()
+        with mock.patch('os.sendfile', side_effect=err):
+            self.loop._sock_sendfile_native_impl(fut, sock.fileno(),
+                                                 sock, fileno,
+                                                 1000, None, len(self.DATA),
+                                                 1000)
+        with self.assertRaises(KeyError):
+            self.loop._selector.get_key(sock)
+        exc = fut.exception()
+        self.assertIs(exc, err)
+        self.assertEqual(1000, self.file.tell())
+
+    def test_exception(self):
+        sock, proto = self.prepare()
+
+        fileno = self.file.fileno()
+        fut = self.loop.create_future()
+        err = RuntimeError()
+        with mock.patch('os.sendfile', side_effect=err):
+            self.loop._sock_sendfile_native_impl(fut, sock.fileno(),
+                                                 sock, fileno,
+                                                 1000, None, len(self.DATA),
+                                                 1000)
+        with self.assertRaises(KeyError):
+            self.loop._selector.get_key(sock)
+        exc = fut.exception()
+        self.assertIs(exc, err)
+        self.assertEqual(1000, self.file.tell())
+
 
 class UnixReadPipeTransportTests(test_utils.TestCase):
 
diff --git a/Misc/NEWS.d/next/Library/2017-12-22-16-05-01.bpo-32410.8JzhvH.rst b/Misc/NEWS.d/next/Library/2017-12-22-16-05-01.bpo-32410.8JzhvH.rst
new file mode 100644
index 00000000000..968d1da6a3c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-22-16-05-01.bpo-32410.8JzhvH.rst
@@ -0,0 +1 @@
+Implement ``loop.sock_sendfile`` for asyncio event loop.


From webhook-mailer at python.org  Tue Jan 16 13:06:34 2018
From: webhook-mailer at python.org (Alexander Belopolsky)
Date: Tue, 16 Jan 2018 18:06:34 -0000
Subject: [Python-checkins] bpo-32403: Faster date and datetime constructors
 (#4993)
Message-ID: <mailman.39.1516125996.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/9f1b7b93f5f0ef589e7b272e127cacf4ce5d23f1
commit: 9f1b7b93f5f0ef589e7b272e127cacf4ce5d23f1
branch: master
author: Paul Ganssle <pganssle at users.noreply.github.com>
committer: Alexander Belopolsky <abalkin at users.noreply.github.com>
date: 2018-01-16T13:06:31-05:00
summary:

bpo-32403: Faster date and datetime constructors (#4993)

* Add tests for date subclass alternate constructors

* Switch over alternate date constructors to fast path

* Switch datetime constructors to fastpath, fix bpo-32404

* Add fast path for datetime in date subclass constructor

* Set fold in constructor in datetime.combine

* Add news entries.

files:
A Misc/NEWS.d/next/Library/2017-12-23-14-51-46.bpo-32403.CVFapH.rst
A Misc/NEWS.d/next/Library/2017-12-23-14-54-05.bpo-32404.yJqtlJ.rst
M Lib/test/datetimetester.py
M Modules/_datetimemodule.c

diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py
index 1d0c1c5bd23..e8ed79e8b32 100644
--- a/Lib/test/datetimetester.py
+++ b/Lib/test/datetimetester.py
@@ -1552,6 +1552,50 @@ def newmeth(self, start):
         self.assertEqual(dt1.toordinal(), dt2.toordinal())
         self.assertEqual(dt2.newmeth(-7), dt1.year + dt1.month - 7)
 
+    def test_subclass_alternate_constructors(self):
+        # Test that alternate constructors call the constructor
+        class DateSubclass(self.theclass):
+            def __new__(cls, *args, **kwargs):
+                result = self.theclass.__new__(cls, *args, **kwargs)
+                result.extra = 7
+
+                return result
+
+        args = (2003, 4, 14)
+        d_ord = 731319              # Equivalent ordinal date
+        d_isoformat = '2003-04-14'  # Equivalent isoformat()
+
+        base_d = DateSubclass(*args)
+        self.assertIsInstance(base_d, DateSubclass)
+        self.assertEqual(base_d.extra, 7)
+
+        # Timestamp depends on time zone, so we'll calculate the equivalent here
+        ts = datetime.combine(base_d, time(0)).timestamp()
+
+        test_cases = [
+            ('fromordinal', (d_ord,)),
+            ('fromtimestamp', (ts,)),
+            ('fromisoformat', (d_isoformat,)),
+        ]
+
+        for constr_name, constr_args in test_cases:
+            for base_obj in (DateSubclass, base_d):
+                # Test both the classmethod and method
+                with self.subTest(base_obj_type=type(base_obj),
+                                  constr_name=constr_name):
+                    constr = getattr(base_obj, constr_name)
+
+                    dt = constr(*constr_args)
+
+                    # Test that it creates the right subclass
+                    self.assertIsInstance(dt, DateSubclass)
+
+                    # Test that it's equal to the base object
+                    self.assertEqual(dt, base_d)
+
+                    # Test that it called the constructor
+                    self.assertEqual(dt.extra, 7)
+
     def test_pickling_subclass_date(self):
 
         args = 6, 7, 23
@@ -2420,6 +2464,54 @@ def newmeth(self, start):
         self.assertEqual(dt2.newmeth(-7), dt1.year + dt1.month +
                                           dt1.second - 7)
 
+    def test_subclass_alternate_constructors_datetime(self):
+        # Test that alternate constructors call the constructor
+        class DateTimeSubclass(self.theclass):
+            def __new__(cls, *args, **kwargs):
+                result = self.theclass.__new__(cls, *args, **kwargs)
+                result.extra = 7
+
+                return result
+
+        args = (2003, 4, 14, 12, 30, 15, 123456)
+        d_isoformat = '2003-04-14T12:30:15.123456'      # Equivalent isoformat()
+        utc_ts = 1050323415.123456                      # UTC timestamp
+
+        base_d = DateTimeSubclass(*args)
+        self.assertIsInstance(base_d, DateTimeSubclass)
+        self.assertEqual(base_d.extra, 7)
+
+        # Timestamp depends on time zone, so we'll calculate the equivalent here
+        ts = base_d.timestamp()
+
+        test_cases = [
+            ('fromtimestamp', (ts,)),
+            # See https://bugs.python.org/issue32417
+            # ('fromtimestamp', (ts, timezone.utc)),
+            ('utcfromtimestamp', (utc_ts,)),
+            ('fromisoformat', (d_isoformat,)),
+            ('strptime', (d_isoformat, '%Y-%m-%dT%H:%M:%S.%f')),
+            ('combine', (date(*args[0:3]), time(*args[3:]))),
+        ]
+
+        for constr_name, constr_args in test_cases:
+            for base_obj in (DateTimeSubclass, base_d):
+                # Test both the classmethod and method
+                with self.subTest(base_obj_type=type(base_obj),
+                                  constr_name=constr_name):
+                    constr = getattr(base_obj, constr_name)
+
+                    dt = constr(*constr_args)
+
+                    # Test that it creates the right subclass
+                    self.assertIsInstance(dt, DateTimeSubclass)
+
+                    # Test that it's equal to the base object
+                    self.assertEqual(dt, base_d.replace(tzinfo=None))
+
+                    # Test that it called the constructor
+                    self.assertEqual(dt.extra, 7)
+
     def test_fromisoformat_datetime(self):
         # Test that isoformat() is reversible
         base_dates = [
diff --git a/Misc/NEWS.d/next/Library/2017-12-23-14-51-46.bpo-32403.CVFapH.rst b/Misc/NEWS.d/next/Library/2017-12-23-14-51-46.bpo-32403.CVFapH.rst
new file mode 100644
index 00000000000..f05d346948a
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-23-14-51-46.bpo-32403.CVFapH.rst
@@ -0,0 +1,2 @@
+Improved speed of :class:`datetime.date` and :class:`datetime.datetime`
+alternate constructors.
diff --git a/Misc/NEWS.d/next/Library/2017-12-23-14-54-05.bpo-32404.yJqtlJ.rst b/Misc/NEWS.d/next/Library/2017-12-23-14-54-05.bpo-32404.yJqtlJ.rst
new file mode 100644
index 00000000000..5299820429b
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-23-14-54-05.bpo-32404.yJqtlJ.rst
@@ -0,0 +1,2 @@
+Fix bug where :meth:`datetime.datetime.fromtimestamp` did not call __new__
+in :class:`datetime.datetime` subclasses.
diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
index e68c7c0a1c5..d1f48e5bd04 100644
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -847,6 +847,27 @@ new_date_ex(int year, int month, int day, PyTypeObject *type)
 #define new_date(year, month, day) \
     new_date_ex(year, month, day, &PyDateTime_DateType)
 
+// Forward declaration
+static PyObject * new_datetime_ex(int, int, int, int, int, int, int,
+                                  PyObject*, PyTypeObject*);
+
+/* Create date instance with no range checking, or call subclass constructor */
+static PyObject *
+new_date_subclass_ex(int year, int month, int day, PyObject *cls) {
+    PyObject *result;
+    // We have "fast path" constructors for two subclasses: date and datetime
+    if ((PyTypeObject *)cls == &PyDateTime_DateType) {
+        result = new_date_ex(year, month, day, (PyTypeObject *)cls);
+    } else if ((PyTypeObject *)cls == &PyDateTime_DateTimeType) {
+        result = new_datetime_ex(year, month, day, 0, 0, 0, 0, Py_None,
+                                 (PyTypeObject *)cls);
+    } else {
+        result = PyObject_CallFunction(cls, "iii", year, month, day);
+    }
+
+    return result;
+}
+
 /* Create a datetime instance with no range checking. */
 static PyObject *
 new_datetime_ex2(int year, int month, int day, int hour, int minute,
@@ -894,6 +915,40 @@ new_datetime_ex(int year, int month, int day, int hour, int minute,
     new_datetime_ex2(y, m, d, hh, mm, ss, us, tzinfo, fold, \
                     &PyDateTime_DateTimeType)
 
+static PyObject *
+new_datetime_subclass_fold_ex(int year, int month, int day, int hour, int minute,
+                              int second, int usecond, PyObject *tzinfo,
+                              int fold, PyObject *cls) {
+    PyObject* dt;
+    if ((PyTypeObject*)cls == &PyDateTime_DateTimeType) {
+        // Use the fast path constructor
+        dt = new_datetime(year, month, day, hour, minute, second, usecond,
+                          tzinfo, fold);
+    } else {
+        // Subclass
+        dt = PyObject_CallFunction(cls, "iiiiiiiO",
+                                   year,
+                                   month,
+                                   day,
+                                   hour,
+                                   minute,
+                                   second,
+                                   usecond,
+                                   tzinfo);
+    }
+
+    return dt;
+}
+
+static PyObject *
+new_datetime_subclass_ex(int year, int month, int day, int hour, int minute,
+                              int second, int usecond, PyObject *tzinfo,
+                              PyObject *cls) {
+    return new_datetime_subclass_fold_ex(year, month, day, hour, minute,
+                                         second, usecond, tzinfo, 0,
+                                         cls);
+}
+
 /* Create a time instance with no range checking. */
 static PyObject *
 new_time_ex2(int hour, int minute, int second, int usecond,
@@ -2743,10 +2798,10 @@ date_local_from_object(PyObject *cls, PyObject *obj)
     if (_PyTime_localtime(t, &tm) != 0)
         return NULL;
 
-    return PyObject_CallFunction(cls, "iii",
-                                 tm.tm_year + 1900,
-                                 tm.tm_mon + 1,
-                                 tm.tm_mday);
+    return new_date_subclass_ex(tm.tm_year + 1900,
+                                tm.tm_mon + 1,
+                                tm.tm_mday,
+                                cls);
 }
 
 /* Return new date from current time.
@@ -2809,8 +2864,7 @@ date_fromordinal(PyObject *cls, PyObject *args)
                                               ">= 1");
         else {
             ord_to_ymd(ordinal, &year, &month, &day);
-            result = PyObject_CallFunction(cls, "iii",
-                                           year, month, day);
+            result = new_date_subclass_ex(year, month, day, cls);
         }
     }
     return result;
@@ -2845,14 +2899,7 @@ date_fromisoformat(PyObject *cls, PyObject *dtstr) {
         return NULL;
     }
 
-    PyObject *result;
-    if ( (PyTypeObject*)cls == &PyDateTime_DateType ) {
-        result = new_date_ex(year, month, day, (PyTypeObject*)cls);
-    } else {
-        result = PyObject_CallFunction(cls, "iii", year, month, day);
-    }
-
-    return result;
+    return new_date_subclass_ex(year, month, day, cls);
 }
 
 
@@ -4596,9 +4643,8 @@ datetime_from_timet_and_us(PyObject *cls, TM_FUNC f, time_t timet, int us,
                 fold = 1;
         }
     }
-    return new_datetime_ex2(year, month, day, hour,
-                            minute, second, us, tzinfo, fold,
-                            (PyTypeObject *)cls);
+    return new_datetime_subclass_fold_ex(year, month, day, hour, minute,
+                                         second, us, tzinfo, fold, cls);
 }
 
 /* Internal helper.
@@ -4764,17 +4810,16 @@ datetime_combine(PyObject *cls, PyObject *args, PyObject *kw)
             else
                 tzinfo = Py_None;
         }
-        result = PyObject_CallFunction(cls, "iiiiiiiO",
-                                       GET_YEAR(date),
-                                       GET_MONTH(date),
-                                       GET_DAY(date),
-                                       TIME_GET_HOUR(time),
-                                       TIME_GET_MINUTE(time),
-                                       TIME_GET_SECOND(time),
-                                       TIME_GET_MICROSECOND(time),
-                                       tzinfo);
-        if (result)
-            DATE_SET_FOLD(result, TIME_GET_FOLD(time));
+        result = new_datetime_subclass_fold_ex(GET_YEAR(date),
+                                               GET_MONTH(date),
+                                               GET_DAY(date),
+                                               TIME_GET_HOUR(time),
+                                               TIME_GET_MINUTE(time),
+                                               TIME_GET_SECOND(time),
+                                               TIME_GET_MICROSECOND(time),
+                                               tzinfo,
+                                               TIME_GET_FOLD(time),
+                                               cls);
     }
     return result;
 }
@@ -4832,23 +4877,8 @@ datetime_fromisoformat(PyObject* cls, PyObject *dtstr) {
         return NULL;
     }
 
-    PyObject* dt;
-    if ( (PyTypeObject*)cls == &PyDateTime_DateTimeType ) {
-        // Use the fast path constructor
-        dt = new_datetime(year, month, day, hour, minute, second, microsecond,
-                          tzinfo, 0);
-    } else {
-        // Subclass
-        dt = PyObject_CallFunction(cls, "iiiiiiiO",
-                                       year,
-                                       month,
-                                       day,
-                                       hour,
-                                       minute,
-                                       second,
-                                       microsecond,
-                                       tzinfo);
-    }
+    PyObject *dt = new_datetime_subclass_ex(year, month, day, hour, minute,
+                                            second, microsecond, tzinfo, cls);
 
     Py_DECREF(tzinfo);
     return dt;


From webhook-mailer at python.org  Tue Jan 16 15:02:29 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Tue, 16 Jan 2018 20:02:29 -0000
Subject: [Python-checkins] bpo-32549: Compile OpenSSL 1.1.0 on Travis CI
 (#5180)
Message-ID: <mailman.40.1516132950.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ced9cb5303ad1447f84d923e0c7f769f5e0c6297
commit: ced9cb5303ad1447f84d923e0c7f769f5e0c6297
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-16T21:02:26+01:00
summary:

bpo-32549: Compile OpenSSL 1.1.0 on Travis CI (#5180)

Use an improved version of multissl test helper to compile a local copy
of OpenSSL 1.1.0g.

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Tests/2018-01-14-11-40-22.bpo-32549.fLwbVA.rst
M .travis.yml
M Makefile.pre.in
M Tools/ssl/multissltests.py

diff --git a/.travis.yml b/.travis.yml
index c207bd72da2..687d0214ab8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,6 +7,19 @@ group: beta
 cache:
     - pip
     - ccache
+    - directories:
+        - $HOME/multissl
+
+env:
+  global:
+    - OPENSSL=1.1.0g
+    - OPENSSL_DIR="$HOME/multissl/openssl/${OPENSSL}"
+    - PATH="${OPENSSL_DIR}/bin:$PATH"
+    - CFLAGS="-I${OPENSSL_DIR}/include"
+    - LDFLAGS="-L${OPENSSL_DIR}/lib"
+    # Set rpath with env var instead of -Wl,-rpath linker flag
+    # OpenSSL ignores LDFLAGS when linking bin/openssl
+    - LD_RUN_PATH="${OPENSSL_DIR}/lib"
 
 branches:
   only:
@@ -48,6 +61,10 @@ matrix:
               echo "Only docs were updated, stopping build process."
               exit
             fi
+            python3 Tools/ssl/multissltests.py  --steps=library \
+                --base-directory ${HOME}/multissl \
+                --openssl ${OPENSSL} >/dev/null
+            openssl version
             ./configure
             make -s -j4
             # Need a venv that can parse covered code.
@@ -71,6 +88,13 @@ before_script:
         echo "Only docs were updated, stopping build process."
         exit
       fi
+      if [ "${TESTING}" != "docs" ]; then
+        # clang complains about unused-parameter a lot, redirect stderr
+        python3 Tools/ssl/multissltests.py --steps=library \
+            --base-directory ${HOME}/multissl \
+            --openssl ${OPENSSL} >/dev/null 2>&1
+      fi
+      openssl version
       ./configure --with-pydebug
       make -j4
       make -j4 regen-all clinic
diff --git a/Makefile.pre.in b/Makefile.pre.in
index 59ba9d4c1f1..994e8611e13 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1086,7 +1086,7 @@ quicktest:	@DEF_MAKE_RULE@ platform
 # SSL tests
 .PHONY: multisslcompile multissltest
 multisslcompile: build_all
-	$(RUNSHARED) ./$(BUILDPYTHON) Tools/ssl/multissltests.py --compile-only
+	$(RUNSHARED) ./$(BUILDPYTHON) Tools/ssl/multissltests.py --steps=modules
 
 multissltest: build_all
 	$(RUNSHARED) ./$(BUILDPYTHON) Tools/ssl/multissltests.py
diff --git a/Misc/NEWS.d/next/Tests/2018-01-14-11-40-22.bpo-32549.fLwbVA.rst b/Misc/NEWS.d/next/Tests/2018-01-14-11-40-22.bpo-32549.fLwbVA.rst
new file mode 100644
index 00000000000..d98f4661361
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-01-14-11-40-22.bpo-32549.fLwbVA.rst
@@ -0,0 +1 @@
+On Travis CI, Python now Compiles and uses a local copy of OpenSSL 1.1.0g for testing.
diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py
index ce5bbd85308..75874cfca4d 100755
--- a/Tools/ssl/multissltests.py
+++ b/Tools/ssl/multissltests.py
@@ -41,8 +41,8 @@
 log = logging.getLogger("multissl")
 
 OPENSSL_OLD_VERSIONS = [
-     "0.9.8zh",
-     "1.0.1u",
+     # "0.9.8zh",
+     # "1.0.1u",
 ]
 
 OPENSSL_RECENT_VERSIONS = [
@@ -52,8 +52,8 @@
 ]
 
 LIBRESSL_OLD_VERSIONS = [
-    "2.3.10",
-    "2.4.5",
+    # "2.3.10",
+    # "2.4.5",
 ]
 
 LIBRESSL_RECENT_VERSIONS = [
@@ -62,8 +62,10 @@
 ]
 
 # store files in ../multissl
-HERE = os.path.abspath(os.getcwd())
-MULTISSL_DIR = os.path.abspath(os.path.join(HERE, '..', 'multissl'))
+HERE = os.path.dirname(os.path.abspath(__file__))
+PYTHONROOT = os.path.abspath(os.path.join(HERE, '..', '..'))
+MULTISSL_DIR = os.path.abspath(os.path.join(PYTHONROOT, '..', 'multissl'))
+
 
 parser = argparse.ArgumentParser(
     prog='multissl',
@@ -118,9 +120,14 @@
     help="Disable network tests."
 )
 parser.add_argument(
-    '--compile-only',
-    action='store_true',
-    help="Don't run tests, only compile _ssl.c and _hashopenssl.c."
+    '--steps',
+    choices=['library', 'modules', 'tests'],
+    default='tests',
+    help=(
+        "Which steps to perform. 'library' downloads and compiles OpenSSL "
+        "or LibreSSL. 'module' also compiles Python modules. 'tests' builds "
+        "all and runs the test suite."
+    )
 )
 
 
@@ -129,21 +136,21 @@ class AbstractBuilder(object):
     url_template = None
     src_template = None
     build_template = None
+    install_target = 'install'
 
     module_files = ("Modules/_ssl.c",
                     "Modules/_hashopenssl.c")
     module_libs = ("_ssl", "_hashlib")
 
-    def __init__(self, version, compile_args=(),
-                 basedir=MULTISSL_DIR):
+    def __init__(self, version, args):
         self.version = version
-        self.compile_args = compile_args
+        self.args = args
         # installation directory
         self.install_dir = os.path.join(
-            os.path.join(basedir, self.library.lower()), version
+            os.path.join(args.base_directory, self.library.lower()), version
         )
         # source file
-        self.src_dir = os.path.join(basedir, 'src')
+        self.src_dir = os.path.join(args.base_directory, 'src')
         self.src_file = os.path.join(
             self.src_dir, self.src_template.format(version))
         # build directory (removed after install)
@@ -252,13 +259,15 @@ def _build_src(self):
         log.info("Running build in {}".format(self.build_dir))
         cwd = self.build_dir
         cmd = ["./config", "shared", "--prefix={}".format(self.install_dir)]
-        cmd.extend(self.compile_args)
         self._subprocess_call(cmd, cwd=cwd)
         # Old OpenSSL versions do not support parallel builds.
         self._subprocess_call(["make", "-j1"], cwd=cwd)
 
     def _make_install(self, remove=True):
-        self._subprocess_call(["make", "-j1", "install"], cwd=self.build_dir)
+        self._subprocess_call(
+            ["make", "-j1", self.install_target],
+            cwd=self.build_dir
+        )
         if remove:
             shutil.rmtree(self.build_dir)
 
@@ -330,6 +339,8 @@ class BuildOpenSSL(AbstractBuilder):
     url_template = "https://www.openssl.org/source/openssl-{}.tar.gz"
     src_template = "openssl-{}.tar.gz"
     build_template = "openssl-{}"
+    # only install software, skip docs
+    install_target = 'install_sw'
 
 
 class BuildLibreSSL(AbstractBuilder):
@@ -368,57 +379,63 @@ def main():
 
     start = datetime.now()
 
-    for name in ['python', 'setup.py', 'Modules/_ssl.c']:
-        if not os.path.isfile(name):
+    if args.steps in {'modules', 'tests'}:
+        for name in ['setup.py', 'Modules/_ssl.c']:
+            if not os.path.isfile(os.path.join(PYTHONROOT, name)):
+                parser.error(
+                    "Must be executed from CPython build dir"
+                )
+        if not os.path.samefile('python', sys.executable):
             parser.error(
-                "Must be executed from CPython build dir"
+                "Must be executed with ./python from CPython build dir"
             )
-    if not os.path.samefile('python', sys.executable):
-        parser.error(
-            "Must be executed with ./python from CPython build dir"
-        )
-
-    # check for configure and run make
-    configure_make()
+        # check for configure and run make
+        configure_make()
 
     # download and register builder
     builds = []
 
     for version in args.openssl:
-        build = BuildOpenSSL(version)
+        build = BuildOpenSSL(
+            version,
+            args
+        )
         build.install()
         builds.append(build)
 
     for version in args.libressl:
-        build = BuildLibreSSL(version)
+        build = BuildLibreSSL(
+            version,
+            args
+        )
         build.install()
         builds.append(build)
 
-    for build in builds:
-        try:
-            build.recompile_pymods()
-            build.check_pyssl()
-            if not args.compile_only:
-                build.run_python_tests(
-                    tests=args.tests,
-                    network=args.network,
-                )
-        except Exception as e:
-            log.exception("%s failed", build)
-            print("{} failed: {}".format(build, e), file=sys.stderr)
-            sys.exit(2)
-
-    print("\n{} finished in {}".format(
-        "Tests" if not args.compile_only else "Builds",
-        datetime.now() - start
-    ))
+    if args.steps in {'modules', 'tests'}:
+        for build in builds:
+            try:
+                build.recompile_pymods()
+                build.check_pyssl()
+                if args.steps == 'tests':
+                    build.run_python_tests(
+                        tests=args.tests,
+                        network=args.network,
+                    )
+            except Exception as e:
+                log.exception("%s failed", build)
+                print("{} failed: {}".format(build, e), file=sys.stderr)
+                sys.exit(2)
+
+    log.info("\n{} finished in {}".format(
+            args.steps.capitalize(),
+            datetime.now() - start
+        ))
     print('Python: ', sys.version)
-    if args.compile_only:
-        print('Build only')
-    elif args.tests:
-        print('Executed Tests:', ' '.join(args.tests))
-    else:
-        print('Executed all SSL tests.')
+    if args.steps == 'tests':
+        if args.tests:
+            print('Executed Tests:', ' '.join(args.tests))
+        else:
+            print('Executed all SSL tests.')
 
     print('OpenSSL / LibreSSL versions:')
     for build in builds:


From lp_benchmark_robot at intel.com  Tue Jan 16 22:10:42 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Tue, 16 Jan 2018 19:10:42 -0800
Subject: [Python-checkins] [1 down,
 64 flat] Results for Python (master branch) 2018-01-16
Message-ID: <99b7f7af-94eb-4565-99b9-c5ef4badc1b3@fmsmsx107.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-16 15:26:21-08:00.
  - commit:		ced9cb5
  - previous commit:	0a2da50
  - revision date:	2018-01-16 21:02:26+01:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.939% |  -0.681%   |  +6.608%   |  +8.320%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.102% |  +0.005%   |  +21.754%  |  +11.471%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 0.453% |  -0.797%   |  +23.699%  |  +9.900%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.428% |  -1.443%   |  +21.030%  |  +11.726%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.497% |  -0.856%   |  +7.847%   |  +11.785%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.544% |  -0.341%   |  +10.435%  |  +13.087%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.491% |  -0.267%   |  +5.976%   |  +10.012%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.507% |  -0.333%   |  -0.719%   |  +8.621%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 2.980% |  +0.948%   |  +10.337%  |  +17.399%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 5.195% |  +6.806%   |  +18.427%  |  +13.501%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.319% |  +1.749%   |  +5.935%   |  +7.720%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.404% |  -0.756%   |  +5.221%   |  +4.594%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.758% |  +0.338%   |  +2.870%   |  +6.669%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.683% |  +1.357%   |  +14.753%  |  +8.787%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.234% |  +2.378%   |  +11.101%  |  +9.802%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.871% |  +0.307%   |  +6.518%   |  +10.691%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 1.357% |  -0.940%   |  +9.653%   |  +11.418%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 2.778% |  +0.033%   |  +8.725%   |  +10.470%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 0.993% |  -1.329%   |  +1.607%   |  +11.677%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 6.077% |  -2.517%   |  -1.925%   |  +14.725%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.643% |  -3.019%   |  +7.577%   |  +10.853%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.723% |  +0.038%   |  +46.884%  |  +13.621%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.260% |  -1.357%   |  +9.590%   |  +11.610%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.416% |  -0.191%   |  +17.482%  |  +13.931%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 3.023% |  -2.260%   |  +7.920%   |  +10.534%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.341% |  -0.522%   |  +3.856%   |  +5.143%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.581% |  +0.208%   |  -3.786%   |  +1.700%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 0.569% |  +1.302%   |  +5.393%   |  +6.958%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.224% |  -0.505%   |  +3.948%   |  +10.640%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.892% |  +1.583%   |  -1.461%   |  +23.052%  |
+-----+------------------------+--------+------------+------------+------------+
| :-( |             pickle_dict| 0.327% |  -3.419%   |  +0.244%   |  +22.353%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.711% |  -1.715%   |  +1.881%   |  +17.874%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 2.440% |  +0.186%   |  +12.785%  |  +9.372%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.239% |  +0.064%   |  +0.264%   |  +10.324%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.125% |  -0.045%   |  +11.318%  |  +5.810%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.086% |  -0.544%   |  +2.273%   |  +5.770%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.262% |  +0.951%   |  +10.607%  |  +13.316%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 6.419% |  -0.943%   |  +2.157%   |  +11.180%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.259% |  -0.800%   |  -1.962%   |  +12.780%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.271% |  -2.591%   |  -6.047%   |  +5.563%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 0.702% |  -2.097%   |  +4.050%   |  +10.900%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.262% |  +0.636%   |  +8.765%   |  +16.204%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.502% |  -0.354%   |  -3.445%   |  +4.396%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.477% |  +1.133%   |  +27.019%  |  +9.562%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 1.750% |  -0.606%   |  +3.807%   |  +4.950%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.510% |  -0.458%   |  +14.073%  |  +8.735%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.962% |  -4.079%   |  -4.616%   |  -0.266%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.379% |  -0.412%   |  +5.801%   |  +2.264%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 0.870% |  -0.685%   |  +6.976%   |  +6.812%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 2.625% |  +0.349%   |  +6.859%   |  +4.206%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.330% |  -2.152%   |  +16.719%  |  +13.034%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.498% |  +3.339%   |  +15.238%  |  +6.176%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 1.662% |  +0.886%   |  +9.991%   |  +7.113%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.837% |  +3.379%   |  +15.419%  |  +7.977%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.117% |  -1.700%   |  +7.999%   |  +12.559%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 3.959% |  -2.010%   |  +16.941%  |  +16.752%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.116% |  +0.212%   |  +6.560%   |  +7.101%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 0.866% |  +1.939%   |  +2.449%   |  -1.669%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle|10.781% |  -1.799%   |  +1.891%   |  +24.630%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 0.559% |  -1.963%   |  -0.965%   |  +16.077%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 2.244% |  -1.615%   |  +6.982%   |  +8.067%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.004% |  +0.163%   |  +2.161%   |  +12.393%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.332% |  +0.248%   |  +2.165%   |  +9.345%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 1.953% |  -0.248%   |  -7.107%   |  +13.400%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.218% |  +0.170%   |  +3.994%   |  +12.067%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-down-64-flat-results-for-python-master-branch-2018-01-16

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Wed Jan 17 04:11:44 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 17 Jan 2018 09:11:44 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=11
Message-ID: <20180117091144.1.AC39C363303F05CC@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [7, 0, 0] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogwBKqQE', '--timeout', '7200']

From webhook-mailer at python.org  Wed Jan 17 04:26:12 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 17 Jan 2018 09:26:12 -0000
Subject: [Python-checkins] Fix Git attributes of test_importlib data files
 (#5209)
Message-ID: <mailman.43.1516181174.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/44a70e9336765404057b12345465844dc291297a
commit: 44a70e9336765404057b12345465844dc291297a
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-17T10:26:03+01:00
summary:

Fix Git attributes of test_importlib data files (#5209)

bpo-32248: Make sure that Git checks out data files with Unix line
ending on Windows as well.

files:
M .gitattributes

diff --git a/.gitattributes b/.gitattributes
index 9ec17d12406..0a998a5369f 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -26,6 +26,7 @@ Lib/test/decimaltestdata/*.decTest -text
 Lib/test/test_email/data/*.txt -text
 Lib/test/xmltestdata/* -text
 Lib/test/coding20731.py -text
+Lib/test/test_importlib/data01/* -text
 
 # CRLF files
 *.bat text eol=crlf


From webhook-mailer at python.org  Wed Jan 17 09:21:56 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 17 Jan 2018 14:21:56 -0000
Subject: [Python-checkins] bpo-32248: Fix test_importlib.test_open() (#5213)
Message-ID: <mailman.44.1516198918.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/3a0cf931796b9ad1b4833ab89c79d284b4253c91
commit: 3a0cf931796b9ad1b4833ab89c79d284b4253c91
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-17T15:21:50+01:00
summary:

bpo-32248: Fix test_importlib.test_open() (#5213)

Use the binary.file instead of utf-8.file to avoid issues with
Unix newlines vs Windows newlines.

files:
M Lib/test/test_importlib/test_open.py

diff --git a/Lib/test/test_importlib/test_open.py b/Lib/test/test_importlib/test_open.py
index ad236c61716..fd6e84b70f6 100644
--- a/Lib/test/test_importlib/test_open.py
+++ b/Lib/test/test_importlib/test_open.py
@@ -19,9 +19,9 @@ def execute(self, package, path):
 
 class OpenTests:
     def test_open_binary(self):
-        with resources.open_binary(self.data, 'utf-8.file') as fp:
+        with resources.open_binary(self.data, 'binary.file') as fp:
             result = fp.read()
-            self.assertEqual(result, b'Hello, UTF-8 world!\n')
+            self.assertEqual(result, b'\x00\x01\x02\x03')
 
     def test_open_text_default_encoding(self):
         with resources.open_text(self.data, 'utf-8.file') as fp:


From webhook-mailer at python.org  Wed Jan 17 10:35:48 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 17 Jan 2018 15:35:48 -0000
Subject: [Python-checkins] pythoninfo: add time.time and
 datetime.datetime.now (#5214)
Message-ID: <mailman.45.1516203348.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/7d91c0250408e202243c77c1dc9afdb321b6b32f
commit: 7d91c0250408e202243c77c1dc9afdb321b6b32f
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-17T16:35:45+01:00
summary:

pythoninfo: add time.time and datetime.datetime.now (#5214)

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index 9ca26c99708..f1b02336f7e 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -315,6 +315,8 @@ def collect_tkinter(info_add):
 def collect_time(info_add):
     import time
 
+    info_add('time.time', time.time())
+
     attributes = (
         'altzone',
         'daylight',
@@ -326,7 +328,16 @@ def collect_time(info_add):
     if hasattr(time, 'get_clock_info'):
         for clock in ('time', 'perf_counter'):
             tinfo = time.get_clock_info(clock)
-            info_add('time.%s' % clock, tinfo)
+            info_add('time.get_clock_info(%s)' % clock, tinfo)
+
+
+def collect_datetime(info_add):
+    try:
+        import datetime
+    except ImportError:
+        return
+
+    info_add('datetime.datetime.now', datetime.datetime.now())
 
 
 def collect_sysconfig(info_add):
@@ -497,6 +508,7 @@ def collect_info(info):
         collect_sys,
         collect_sysconfig,
         collect_time,
+        collect_datetime,
         collect_tkinter,
         collect_zlib,
         collect_expat,


From webhook-mailer at python.org  Wed Jan 17 11:36:07 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 17 Jan 2018 16:36:07 -0000
Subject: [Python-checkins] pythoninfo: add time.time and
 datetime.datetime.now (GH-5214) (#5219)
Message-ID: <mailman.46.1516206969.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/4e09c0c8589f9152646f2cf403136dc783bcea4c
commit: 4e09c0c8589f9152646f2cf403136dc783bcea4c
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-17T17:36:04+01:00
summary:

pythoninfo: add time.time and datetime.datetime.now (GH-5214) (#5219)

(cherry picked from commit 7d91c0250408e202243c77c1dc9afdb321b6b32f)

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index 9ca26c99708..f1b02336f7e 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -315,6 +315,8 @@ def collect_tkinter(info_add):
 def collect_time(info_add):
     import time
 
+    info_add('time.time', time.time())
+
     attributes = (
         'altzone',
         'daylight',
@@ -326,7 +328,16 @@ def collect_time(info_add):
     if hasattr(time, 'get_clock_info'):
         for clock in ('time', 'perf_counter'):
             tinfo = time.get_clock_info(clock)
-            info_add('time.%s' % clock, tinfo)
+            info_add('time.get_clock_info(%s)' % clock, tinfo)
+
+
+def collect_datetime(info_add):
+    try:
+        import datetime
+    except ImportError:
+        return
+
+    info_add('datetime.datetime.now', datetime.datetime.now())
 
 
 def collect_sysconfig(info_add):
@@ -497,6 +508,7 @@ def collect_info(info):
         collect_sys,
         collect_sysconfig,
         collect_time,
+        collect_datetime,
         collect_tkinter,
         collect_zlib,
         collect_expat,


From webhook-mailer at python.org  Wed Jan 17 11:58:18 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 17 Jan 2018 16:58:18 -0000
Subject: [Python-checkins] pythoninfo: add time.time and
 datetime.datetime.now (GH-5214) (#5220)
Message-ID: <mailman.47.1516208299.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/016f59a71620f5eadbc06ceb79d3d486d5da8ff3
commit: 016f59a71620f5eadbc06ceb79d3d486d5da8ff3
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-17T17:58:16+01:00
summary:

pythoninfo: add time.time and datetime.datetime.now (GH-5214) (#5220)

(cherry picked from commit 7d91c0250408e202243c77c1dc9afdb321b6b32f)

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index 4ce06e00eb9..f40ff8d0eba 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -315,6 +315,8 @@ def collect_tkinter(info_add):
 def collect_time(info_add):
     import time
 
+    info_add('time.time', time.time())
+
     attributes = (
         'altzone',
         'daylight',
@@ -326,7 +328,16 @@ def collect_time(info_add):
     if hasattr(time, 'get_clock_info'):
         for clock in ('time', 'perf_counter'):
             tinfo = time.get_clock_info(clock)
-            info_add('time.%s' % clock, tinfo)
+            info_add('time.get_clock_info(%s)' % clock, tinfo)
+
+
+def collect_datetime(info_add):
+    try:
+        import datetime
+    except ImportError:
+        return
+
+    info_add('datetime.datetime.now', datetime.datetime.now())
 
 
 def collect_sysconfig(info_add):
@@ -497,6 +508,7 @@ def collect_info(info):
         collect_sys,
         collect_sysconfig,
         collect_time,
+        collect_datetime,
         collect_tkinter,
         collect_zlib,
         collect_expat,


From webhook-mailer at python.org  Wed Jan 17 17:08:22 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 17 Jan 2018 22:08:22 -0000
Subject: [Python-checkins] bpo-32226: Fix memory leak in
 generic_alias_dealloc() (#5212)
Message-ID: <mailman.48.1516226905.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/e860089fd93728824cf586fa4d91f08eff3bab73
commit: e860089fd93728824cf586fa4d91f08eff3bab73
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-17T23:08:18+01:00
summary:

bpo-32226: Fix memory leak in generic_alias_dealloc() (#5212)

files:
M Modules/_testcapimodule.c

diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 7b2e2a3e5a4..0d6bf45032d 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -5064,6 +5064,7 @@ static void
 generic_alias_dealloc(PyGenericAliasObject *self)
 {
     Py_CLEAR(self->item);
+    Py_TYPE(self)->tp_free((PyObject *)self);
 }
 
 static PyObject *


From webhook-mailer at python.org  Wed Jan 17 17:58:00 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Wed, 17 Jan 2018 22:58:00 -0000
Subject: [Python-checkins] Use assertCountEqual instead of assertEqual.
 (#5223)
Message-ID: <mailman.49.1516229882.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/3941499d6c2a4d09f2ea476f84b9c4ddc4bdefa0
commit: 3941499d6c2a4d09f2ea476f84b9c4ddc4bdefa0
branch: master
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2018-01-17T14:57:57-08:00
summary:

Use assertCountEqual instead of assertEqual. (#5223)

This test doesn't care about order, the underlying filesystem APIs do not
guarantee directory listings on subsequent calls will be in the same order.

files:
M Lib/test/test_glob.py

diff --git a/Lib/test/test_glob.py b/Lib/test/test_glob.py
index dce64f9fcb1..767bd3764b8 100644
--- a/Lib/test/test_glob.py
+++ b/Lib/test/test_glob.py
@@ -49,10 +49,10 @@ def glob(self, *parts, **kwargs):
             pattern = os.path.join(*parts)
         p = os.path.join(self.tempdir, pattern)
         res = glob.glob(p, **kwargs)
-        self.assertEqual(list(glob.iglob(p, **kwargs)), res)
+        self.assertCountEqual(glob.iglob(p, **kwargs), res)
         bres = [os.fsencode(x) for x in res]
-        self.assertEqual(glob.glob(os.fsencode(p), **kwargs), bres)
-        self.assertEqual(list(glob.iglob(os.fsencode(p), **kwargs)), bres)
+        self.assertCountEqual(glob.glob(os.fsencode(p), **kwargs), bres)
+        self.assertCountEqual(glob.iglob(os.fsencode(p), **kwargs), bres)
         return res
 
     def assertSequencesEqual_noorder(self, l1, l2):


From webhook-mailer at python.org  Wed Jan 17 18:15:50 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Wed, 17 Jan 2018 23:15:50 -0000
Subject: [Python-checkins] Use assertItemsEqual instead of assertEqual.
 (#5224)
Message-ID: <mailman.50.1516230951.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/b1a52b11673f6fd29b357852977282c8b41cfb7d
commit: b1a52b11673f6fd29b357852977282c8b41cfb7d
branch: 2.7
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2018-01-17T15:15:46-08:00
summary:

Use assertItemsEqual instead of assertEqual. (#5224)

This test doesn't care about order, the underlying filesystem APIs do not
guarantee directory listings on subsequent calls will be in the same order.

files:
M Lib/test/test_glob.py

diff --git a/Lib/test/test_glob.py b/Lib/test/test_glob.py
index b360d09dfa6..84b2e7ddd25 100644
--- a/Lib/test/test_glob.py
+++ b/Lib/test/test_glob.py
@@ -49,10 +49,10 @@ def glob(self, *parts):
             pattern = os.path.join(*parts)
         p = os.path.join(self.tempdir, pattern)
         res = glob.glob(p)
-        self.assertEqual(list(glob.iglob(p)), res)
+        self.assertItemsEqual(glob.iglob(p), res)
         ures = [fsdecode(x) for x in res]
-        self.assertEqual(glob.glob(fsdecode(p)), ures)
-        self.assertEqual(list(glob.iglob(fsdecode(p))), ures)
+        self.assertItemsEqual(glob.glob(fsdecode(p)), ures)
+        self.assertItemsEqual(glob.iglob(fsdecode(p)), ures)
         return res
 
     def assertSequencesEqual_noorder(self, l1, l2):


From webhook-mailer at python.org  Wed Jan 17 18:21:04 2018
From: webhook-mailer at python.org (Mariatta)
Date: Wed, 17 Jan 2018 23:21:04 -0000
Subject: [Python-checkins] Fix typo in thread_nt.h code comment (GH-5211)
Message-ID: <mailman.51.1516231265.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/6027802ca7fae118bce6afead51d01a174600d40
commit: 6027802ca7fae118bce6afead51d01a174600d40
branch: master
author: hui shang <shangdahao at gmail.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-17T15:21:01-08:00
summary:

Fix typo in thread_nt.h code comment (GH-5211)

The comment for PyThread_allocate_lock says "It has too be implemented ...".
There was an extra "o" in ".. to be implemented.."

files:
M Python/thread_nt.h

diff --git a/Python/thread_nt.h b/Python/thread_nt.h
index 14213076bcb..61fa8619bc1 100644
--- a/Python/thread_nt.h
+++ b/Python/thread_nt.h
@@ -236,7 +236,7 @@ PyThread_exit_thread(void)
 }
 
 /*
- * Lock support. It has too be implemented as semaphores.
+ * Lock support. It has to be implemented as semaphores.
  * I [Dag] tried to implement it with mutex but I could find a way to
  * tell whether a thread already own the lock or not.
  */


From webhook-mailer at python.org  Wed Jan 17 18:51:30 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Wed, 17 Jan 2018 23:51:30 -0000
Subject: [Python-checkins] Use assertCountEqual instead of assertEqual.
 (GH-5223) (#5225)
Message-ID: <mailman.52.1516233091.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/8d1e41d4148a69b0709fc7fb6bb733451ae1b28d
commit: 8d1e41d4148a69b0709fc7fb6bb733451ae1b28d
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-17T15:51:27-08:00
summary:

Use assertCountEqual instead of assertEqual. (GH-5223) (#5225)

This test doesn't care about order, the underlying filesystem APIs do not
guarantee directory listings on subsequent calls will be in the same order.
(cherry picked from commit 3941499d6c2a4d09f2ea476f84b9c4ddc4bdefa0)

files:
M Lib/test/test_glob.py

diff --git a/Lib/test/test_glob.py b/Lib/test/test_glob.py
index dce64f9fcb1..767bd3764b8 100644
--- a/Lib/test/test_glob.py
+++ b/Lib/test/test_glob.py
@@ -49,10 +49,10 @@ def glob(self, *parts, **kwargs):
             pattern = os.path.join(*parts)
         p = os.path.join(self.tempdir, pattern)
         res = glob.glob(p, **kwargs)
-        self.assertEqual(list(glob.iglob(p, **kwargs)), res)
+        self.assertCountEqual(glob.iglob(p, **kwargs), res)
         bres = [os.fsencode(x) for x in res]
-        self.assertEqual(glob.glob(os.fsencode(p), **kwargs), bres)
-        self.assertEqual(list(glob.iglob(os.fsencode(p), **kwargs)), bres)
+        self.assertCountEqual(glob.glob(os.fsencode(p), **kwargs), bres)
+        self.assertCountEqual(glob.iglob(os.fsencode(p), **kwargs), bres)
         return res
 
     def assertSequencesEqual_noorder(self, l1, l2):


From webhook-mailer at python.org  Wed Jan 17 19:40:41 2018
From: webhook-mailer at python.org (Mariatta)
Date: Thu, 18 Jan 2018 00:40:41 -0000
Subject: [Python-checkins] Fix typo in thread_nt.h code comment (GH-5211)
 (GH-5226)
Message-ID: <mailman.53.1516236042.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/f31c70b0d65510d6d3235816eb95ef43ba6764bf
commit: f31c70b0d65510d6d3235816eb95ef43ba6764bf
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-17T16:40:33-08:00
summary:

Fix typo in thread_nt.h code comment (GH-5211) (GH-5226)

The comment for PyThread_allocate_lock says "It has too be implemented ...".
There was an extra "o" in ".. to be implemented.."
(cherry picked from commit 6027802ca7fae118bce6afead51d01a174600d40)

files:
M Python/thread_nt.h

diff --git a/Python/thread_nt.h b/Python/thread_nt.h
index 74a6ee80298..d87f9614498 100644
--- a/Python/thread_nt.h
+++ b/Python/thread_nt.h
@@ -235,7 +235,7 @@ PyThread_exit_thread(void)
 }
 
 /*
- * Lock support. It has too be implemented as semaphores.
+ * Lock support. It has to be implemented as semaphores.
  * I [Dag] tried to implement it with mutex but I could find a way to
  * tell whether a thread already own the lock or not.
  */


From lp_benchmark_robot at intel.com  Wed Jan 17 21:29:34 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Wed, 17 Jan 2018 18:29:34 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-17
Message-ID: <f3284edb-6b16-4386-a37a-de7fc68523f7@orsmsx106.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-17 03:04:10-08:00.
  - commit:		44a70e9
  - previous commit:	ced9cb5
  - revision date:	2018-01-17 10:26:03+01:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.911% |  +0.134%   |  +6.733%   |  +8.012%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 0.539% |  +0.230%   |  +21.934%  |  +13.743%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.043% |  -0.180%   |  +23.562%  |  +12.614%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.718% |  -0.040%   |  +20.999%  |  +13.615%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.869% |  -0.504%   |  +7.383%   |  +14.702%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.103% |  -0.006%   |  +10.429%  |  +13.081%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 2.280% |  -0.162%   |  +5.824%   |  +11.843%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.498% |  +0.024%   |  -0.695%   |  +8.755%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 3.187% |  -0.283%   |  +10.083%  |  +18.268%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 5.449% |  -1.804%   |  +16.955%  |  +8.463%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.607% |  -0.432%   |  +5.529%   |  +6.372%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.308% |  +0.095%   |  +5.310%   |  +5.256%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.619% |  +0.046%   |  +2.916%   |  +7.322%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.314% |  -0.068%   |  +14.695%  |  +9.597%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.529% |  -0.344%   |  +10.795%  |  +10.216%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.918% |  -0.079%   |  +6.444%   |  +12.709%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.693% |  +0.071%   |  +9.717%   |  +12.797%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.006% |  -0.198%   |  +8.545%   |  +11.115%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.009% |  +0.148%   |  +1.753%   |  +11.800%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 5.660% |  -0.052%   |  -1.978%   |  +13.914%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.570% |  +0.341%   |  +7.892%   |  +13.174%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 3.879% |  +0.170%   |  +46.975%  |  +13.745%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.489% |  -0.203%   |  +9.407%   |  +13.128%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.547% |  -0.050%   |  +17.441%  |  +13.490%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.266% |  +0.128%   |  +8.037%   |  +8.241%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.959% |  +0.579%   |  +4.413%   |  +4.528%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.446% |  +0.140%   |  -3.641%   |  +3.038%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.253% |  -0.179%   |  +5.224%   |  +8.324%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.362% |  -0.729%   |  +3.248%   |  +10.279%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 0.806% |  +0.540%   |  -0.913%   |  +22.103%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.416% |  +0.023%   |  +0.268%   |  +19.239%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.620% |  +0.062%   |  +1.941%   |  +18.233%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 6.474% |  -2.037%   |  +11.008%  |  +11.504%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.084% |  +0.024%   |  +0.288%   |  +9.864%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.123% |  +0.099%   |  +11.406%  |  +5.526%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.081% |  +0.005%   |  +2.278%   |  +5.576%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.922% |  +0.085%   |  +10.683%  |  +13.472%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 6.107% |  +0.956%   |  +3.092%   |  +11.532%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.231% |  +0.048%   |  -1.912%   |  +12.653%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.555% |  -0.551%   |  -6.632%   |  +7.834%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 1.084% |  -0.149%   |  +3.907%   |  +9.282%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.276% |  -0.577%   |  +8.239%   |  +16.212%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.628% |  +0.086%   |  -3.356%   |  +4.158%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.358% |  -0.002%   |  +27.017%  |  +9.340%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.418% |  -0.354%   |  +3.467%   |  +7.901%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 2.267% |  -0.564%   |  +13.588%  |  +10.155%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 2.939% |  -0.519%   |  -5.159%   |  -0.170%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.530% |  -0.165%   |  +5.646%   |  +4.146%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.167% |  -0.176%   |  +6.812%   |  +7.941%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 2.858% |  +0.166%   |  +7.014%   |  +3.771%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 4.028% |  +2.245%   |  +18.588%  |  +9.932%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 3.526% |  -0.838%   |  +14.527%  |  +7.120%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 1.442% |  +0.241%   |  +10.208%  |  +7.019%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 4.110% |  -0.433%   |  +15.053%  |  +9.318%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 5.965% |  +0.394%   |  +8.361%   |  +12.706%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 5.557% |  -1.541%   |  +15.661%  |  +15.145%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.058% |  -0.009%   |  +6.551%   |  +7.989%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 2.728% |  -1.155%   |  +1.322%   |  +1.498%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle|10.552% |  +0.315%   |  +2.200%   |  +25.897%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 3.728% |  -0.627%   |  -1.598%   |  +16.546%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 3.762% |  -0.029%   |  +6.956%   |  +8.714%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.503% |  -0.067%   |  +2.095%   |  +13.120%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.296% |  +0.463%   |  +2.618%   |  +8.465%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 2.818% |  -0.156%   |  -7.275%   |  +13.080%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.404% |  +0.063%   |  +4.054%   |  +12.517%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-17

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Thu Jan 18 04:14:17 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 18 Jan 2018 09:14:17 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=11
Message-ID: <20180118091417.1.85EA66730A190997@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [7, -7, 8] memory blocks, sum=8
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, 1, -2] memory blocks, sum=-1


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogpiRlgm', '--timeout', '7200']

From webhook-mailer at python.org  Thu Jan 18 04:38:12 2018
From: webhook-mailer at python.org (Antoine Pitrou)
Date: Thu, 18 Jan 2018 09:38:12 -0000
Subject: [Python-checkins] bpo-32576: use queue.SimpleQueue in critical
 places (#5216)
Message-ID: <mailman.54.1516268293.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ab74504346a6e2569b3255b7b621c589716888c4
commit: ab74504346a6e2569b3255b7b621c589716888c4
branch: master
author: Antoine Pitrou <pitrou at free.fr>
committer: GitHub <noreply at github.com>
date: 2018-01-18T10:38:03+01:00
summary:

bpo-32576: use queue.SimpleQueue in critical places (#5216)

Where a queue may be invoked from a weakref callback, we need
to use the reentrant SimpleQueue.

files:
A Misc/NEWS.d/next/Library/2018-01-17-13-04-16.bpo-32576.iDL09t.rst
M Lib/concurrent/futures/thread.py
M Lib/multiprocessing/pool.py

diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py
index 2e7100bc352..6e22950a157 100644
--- a/Lib/concurrent/futures/thread.py
+++ b/Lib/concurrent/futures/thread.py
@@ -128,7 +128,7 @@ def __init__(self, max_workers=None, thread_name_prefix='',
             raise TypeError("initializer must be a callable")
 
         self._max_workers = max_workers
-        self._work_queue = queue.Queue()
+        self._work_queue = queue.SimpleQueue()
         self._threads = set()
         self._broken = False
         self._shutdown = False
diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index b1ee725fac6..3e9a0d6b486 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -156,7 +156,7 @@ def __init__(self, processes=None, initializer=None, initargs=(),
                  maxtasksperchild=None, context=None):
         self._ctx = context or get_context()
         self._setup_queues()
-        self._taskqueue = queue.Queue()
+        self._taskqueue = queue.SimpleQueue()
         self._cache = {}
         self._state = RUN
         self._maxtasksperchild = maxtasksperchild
@@ -802,15 +802,18 @@ def __init__(self, processes=None, initializer=None, initargs=()):
         Pool.__init__(self, processes, initializer, initargs)
 
     def _setup_queues(self):
-        self._inqueue = queue.Queue()
-        self._outqueue = queue.Queue()
+        self._inqueue = queue.SimpleQueue()
+        self._outqueue = queue.SimpleQueue()
         self._quick_put = self._inqueue.put
         self._quick_get = self._outqueue.get
 
     @staticmethod
     def _help_stuff_finish(inqueue, task_handler, size):
-        # put sentinels at head of inqueue to make workers finish
-        with inqueue.not_empty:
-            inqueue.queue.clear()
-            inqueue.queue.extend([None] * size)
-            inqueue.not_empty.notify_all()
+        # drain inqueue, and put sentinels at its head to make workers finish
+        try:
+            while True:
+                inqueue.get(block=False)
+        except queue.Empty:
+            pass
+        for i in range(size):
+            inqueue.put(None)
diff --git a/Misc/NEWS.d/next/Library/2018-01-17-13-04-16.bpo-32576.iDL09t.rst b/Misc/NEWS.d/next/Library/2018-01-17-13-04-16.bpo-32576.iDL09t.rst
new file mode 100644
index 00000000000..143a83e0fcc
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-17-13-04-16.bpo-32576.iDL09t.rst
@@ -0,0 +1,2 @@
+Use queue.SimpleQueue() in places where it can be invoked from a weakref
+callback.


From webhook-mailer at python.org  Thu Jan 18 05:15:28 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 18 Jan 2018 10:15:28 -0000
Subject: [Python-checkins] bpo-9566: Fix size_t=>int downcast warnings
 (#5230)
Message-ID: <mailman.55.1516270529.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/05d68a8bd84cb141be9f9335f5b3540f15a989c4
commit: 05d68a8bd84cb141be9f9335f5b3540f15a989c4
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-18T11:15:25+01:00
summary:

bpo-9566: Fix size_t=>int downcast warnings (#5230)

* Use wider types (int => Py_ssize_t) to avoid integer overflows.
* Fix gc.get_freeze_count(): use Py_ssize_t type rather than int, since gc_list_size() returns a Py_ssize_t.

files:
M Modules/clinic/gcmodule.c.h
M Modules/gcmodule.c
M Python/ast_opt.c
M Python/bltinmodule.c
M Python/pathconfig.c

diff --git a/Modules/clinic/gcmodule.c.h b/Modules/clinic/gcmodule.c.h
index c431af79efd..0330c8113bc 100644
--- a/Modules/clinic/gcmodule.c.h
+++ b/Modules/clinic/gcmodule.c.h
@@ -307,22 +307,22 @@ PyDoc_STRVAR(gc_get_freeze_count__doc__,
 #define GC_GET_FREEZE_COUNT_METHODDEF    \
     {"get_freeze_count", (PyCFunction)gc_get_freeze_count, METH_NOARGS, gc_get_freeze_count__doc__},
 
-static int
+static Py_ssize_t
 gc_get_freeze_count_impl(PyObject *module);
 
 static PyObject *
 gc_get_freeze_count(PyObject *module, PyObject *Py_UNUSED(ignored))
 {
     PyObject *return_value = NULL;
-    int _return_value;
+    Py_ssize_t _return_value;
 
     _return_value = gc_get_freeze_count_impl(module);
     if ((_return_value == -1) && PyErr_Occurred()) {
         goto exit;
     }
-    return_value = PyLong_FromLong((long)_return_value);
+    return_value = PyLong_FromSsize_t(_return_value);
 
 exit:
     return return_value;
 }
-/*[clinic end generated code: output=6f9ee4d8dd1f36c1 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=21dc9270b10b7891 input=a9049054013a1b77]*/
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 121eb46012c..ea3c294dcff 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1449,14 +1449,14 @@ gc_unfreeze_impl(PyObject *module)
 }
 
 /*[clinic input]
-gc.get_freeze_count -> int
+gc.get_freeze_count -> Py_ssize_t
 
 Return the number of objects in the permanent generation.
 [clinic start generated code]*/
 
-static int
+static Py_ssize_t
 gc_get_freeze_count_impl(PyObject *module)
-/*[clinic end generated code: output=e4e2ebcc77e5cbf3 input=4b759db880a3c6e4]*/
+/*[clinic end generated code: output=61cbd9f43aa032e1 input=45ffbc65cfe2a6ed]*/
 {
     return gc_list_size(&_PyRuntime.gc.permanent_generation.head);
 }
diff --git a/Python/ast_opt.c b/Python/ast_opt.c
index f98a28aa054..65cf3c12641 100644
--- a/Python/ast_opt.c
+++ b/Python/ast_opt.c
@@ -397,7 +397,7 @@ fold_compare(expr_ty node, PyArena *arena, int optimize)
 {
     asdl_int_seq *ops;
     asdl_seq *args;
-    int i;
+    Py_ssize_t i;
 
     ops = node->v.Compare.ops;
     args = node->v.Compare.comparators;
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
index 844548f75fd..7fc2261ec64 100644
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -52,9 +52,9 @@ _Py_IDENTIFIER(stderr);
 #include "clinic/bltinmodule.c.h"
 
 static PyObject*
-update_bases(PyObject *bases, PyObject *const *args, int nargs)
+update_bases(PyObject *bases, PyObject *const *args, Py_ssize_t nargs)
 {
-    int i, j;
+    Py_ssize_t i, j;
     PyObject *base, *meth, *new_base, *result, *new_bases = NULL;
     PyObject *stack[1] = {bases};
     assert(PyTuple_Check(bases));
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index 7ebd69bf622..6de5481bc8d 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -367,13 +367,12 @@ _Py_FindEnvConfigValue(FILE *env_file, const wchar_t *key,
     fseek(env_file, 0, SEEK_SET);
     while (!feof(env_file)) {
         char * p = fgets(buffer, MAXPATHLEN*2, env_file);
-        wchar_t *tmpbuffer;
-        int n;
 
         if (p == NULL) {
             break;
         }
-        n = strlen(p);
+
+        size_t n = strlen(p);
         if (p[n - 1] != '\n') {
             /* line has overflowed - bail */
             break;
@@ -382,7 +381,8 @@ _Py_FindEnvConfigValue(FILE *env_file, const wchar_t *key,
             /* Comment - skip */
             continue;
         }
-        tmpbuffer = _Py_DecodeUTF8_surrogateescape(buffer, n);
+
+        wchar_t *tmpbuffer = _Py_DecodeUTF8_surrogateescape(buffer, n);
         if (tmpbuffer) {
             wchar_t * state;
             wchar_t * tok = wcstok(tmpbuffer, L" \t\r\n", &state);


From webhook-mailer at python.org  Thu Jan 18 16:23:30 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Thu, 18 Jan 2018 21:23:30 -0000
Subject: [Python-checkins] Removed unnecesssary bit inversion which doesn't
 improve dispersion statistics (#5235)
Message-ID: <mailman.56.1516310611.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/fa7880604191f81cbdddc191216f7b1e39a74d8d
commit: fa7880604191f81cbdddc191216f7b1e39a74d8d
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-18T13:23:27-08:00
summary:

Removed unnecesssary bit inversion which doesn't improve dispersion statistics (#5235)

files:
M Objects/setobject.c

diff --git a/Objects/setobject.c b/Objects/setobject.c
index 4bc1020d56f..47db6b245ca 100644
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -795,7 +795,7 @@ frozenset_hash(PyObject *self)
     hash ^= ((Py_uhash_t)PySet_GET_SIZE(self) + 1) * 1927868237UL;
 
     /* Disperse patterns arising in nested frozensets */
-    hash ^= (hash >> 11) ^ (~hash >> 25);
+    hash ^= (hash >> 11) ^ (hash >> 25);
     hash = hash * 69069U + 907133923UL;
 
     /* -1 is reserved as an error code */


From webhook-mailer at python.org  Thu Jan 18 17:09:43 2018
From: webhook-mailer at python.org (Steve Dower)
Date: Thu, 18 Jan 2018 22:09:43 -0000
Subject: [Python-checkins] bpo-32588: Move _findvs into its own module and
 add missing _queue module to installer (#5227)
Message-ID: <mailman.57.1516313384.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/2507e29a9e4e9dcac6eab46546bd3d34a69342ba
commit: 2507e29a9e4e9dcac6eab46546bd3d34a69342ba
branch: master
author: Steve Dower <steve.dower at microsoft.com>
committer: GitHub <noreply at github.com>
date: 2018-01-19T09:09:36+11:00
summary:

bpo-32588: Move _findvs into its own module and add missing _queue module to installer (#5227)

files:
A Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst
A PCbuild/_distutils_findvs.vcxproj
A PCbuild/_distutils_findvs.vcxproj.filters
M Lib/distutils/_msvccompiler.py
M Lib/distutils/command/bdist_wininst.py
M PC/_findvs.cpp
M PC/config.c
M PCbuild/pcbuild.proj
M PCbuild/pcbuild.sln
M PCbuild/pythoncore.vcxproj
M Tools/msi/lib/lib_files.wxs

diff --git a/Lib/distutils/_msvccompiler.py b/Lib/distutils/_msvccompiler.py
index ef1356b97d6..c9d3c6c6712 100644
--- a/Lib/distutils/_msvccompiler.py
+++ b/Lib/distutils/_msvccompiler.py
@@ -56,7 +56,7 @@ def _find_vc2015():
     return best_version, best_dir
 
 def _find_vc2017():
-    import _findvs
+    import _distutils_findvs
     import threading
 
     best_version = 0,   # tuple for full version comparisons
@@ -66,7 +66,7 @@ def _find_vc2017():
     # initialize COM.
     all_packages = []
     def _getall():
-        all_packages.extend(_findvs.findall())
+        all_packages.extend(_distutils_findvs.findall())
     t = threading.Thread(target=_getall)
     t.start()
     t.join()
diff --git a/Lib/distutils/command/bdist_wininst.py b/Lib/distutils/command/bdist_wininst.py
index 83f00732657..0871a4f7d6b 100644
--- a/Lib/distutils/command/bdist_wininst.py
+++ b/Lib/distutils/command/bdist_wininst.py
@@ -337,11 +337,10 @@ def get_exe_bytes(self):
                 # cross-building, so assume the latest version
                 bv = '14.0'
             else:
-                bv = '.'.join(CRT_ASSEMBLY_VERSION.split('.', 2)[:2])
-                if bv in ('14.11', '14.12'):
-                    # v142, v141 and v140 are binary compatible,
-                    # so keep using the 14.0 stub.
-                    bv = '14.0'
+                # as far as we know, CRT is binary compatible based on
+                # the first field, so assume 'x.0' until proven otherwise
+                major = CRT_ASSEMBLY_VERSION.partition('.')[0]
+                bv = major + '.0'
 
 
         # wininst-x.y.exe is in the same directory as this file
diff --git a/Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst b/Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst
new file mode 100644
index 00000000000..0766a7eb259
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst
@@ -0,0 +1,2 @@
+Create standalone _distutils_findvs module and add missing _queue module to
+installer.
diff --git a/PC/_findvs.cpp b/PC/_findvs.cpp
index dd7c1fcb723..ecc68e82e49 100644
--- a/PC/_findvs.cpp
+++ b/PC/_findvs.cpp
@@ -12,10 +12,6 @@
 #include <Windows.h>
 #include <Strsafe.h>
 #include "external\include\Setup.Configuration.h"
-#pragma comment(lib, "ole32.lib")
-#pragma comment(lib, "oleaut32.lib")
-#pragma comment(lib, "version.lib")
-#pragma comment(lib, "Microsoft.VisualStudio.Setup.Configuration.Native.lib")
 
 #include <Python.h>
 
@@ -236,7 +232,7 @@ static int exec_findvs(PyObject *module)
     return 0; // success
 }
 
-PyDoc_STRVAR(findvs_doc, "The _findvs helper module");
+PyDoc_STRVAR(findvs_doc, "The _distutils_findvs helper module");
 
 static PyModuleDef_Slot findvs_slots[] = {
     { Py_mod_exec, exec_findvs },
@@ -245,7 +241,7 @@ static PyModuleDef_Slot findvs_slots[] = {
 
 static PyModuleDef findvs_def = {
     PyModuleDef_HEAD_INIT,
-    "_findvs",
+    "_distutils_findvs",
     findvs_doc,
     0,              // m_size
     NULL,           // m_methods
@@ -256,8 +252,8 @@ static PyModuleDef findvs_def = {
 };
 
 extern "C" {
-    PyMODINIT_FUNC PyInit__findvs(void)
+    PyMODINIT_FUNC PyInit__distutils_findvs(void)
     {
         return PyModuleDef_Init(&findvs_def);
     }
-}
\ No newline at end of file
+}
diff --git a/PC/config.c b/PC/config.c
index 699e1d07077..91f15b5867d 100644
--- a/PC/config.c
+++ b/PC/config.c
@@ -69,7 +69,6 @@ extern PyObject* _PyWarnings_Init(void);
 extern PyObject* PyInit__string(void);
 extern PyObject* PyInit__stat(void);
 extern PyObject* PyInit__opcode(void);
-extern PyObject* PyInit__findvs(void);
 
 /* tools/freeze/makeconfig.py marker for additional "extern" */
 /* -- ADDMODULE MARKER 1 -- */
@@ -162,8 +161,6 @@ struct _inittab _PyImport_Inittab[] = {
     {"_stat", PyInit__stat},
     {"_opcode", PyInit__opcode},
 
-    {"_findvs", PyInit__findvs},
-
     /* Sentinel */
     {0, 0}
 };
diff --git a/PCbuild/_distutils_findvs.vcxproj b/PCbuild/_distutils_findvs.vcxproj
new file mode 100644
index 00000000000..3db1280ac88
--- /dev/null
+++ b/PCbuild/_distutils_findvs.vcxproj
@@ -0,0 +1,83 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup Label="ProjectConfigurations">
+    <ProjectConfiguration Include="Debug|Win32">
+      <Configuration>Debug</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Debug|x64">
+      <Configuration>Debug</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|Win32">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|x64">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|Win32">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|x64">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|Win32">
+      <Configuration>Release</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|x64">
+      <Configuration>Release</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+  </ItemGroup>
+  <PropertyGroup Label="Globals">
+    <ProjectGuid>{41ADEDF9-11D8-474E-B4D7-BB82332C878E}</ProjectGuid>
+    <RootNamespace>_distutils_findvs</RootNamespace>
+    <Keyword>Win32Proj</Keyword>
+  </PropertyGroup>
+  <Import Project="python.props" />
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>DynamicLibrary</ConfigurationType>
+    <CharacterSet>NotSet</CharacterSet>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+  <PropertyGroup>
+    <TargetExt>.pyd</TargetExt>
+  </PropertyGroup>
+  <ImportGroup Label="ExtensionSettings">
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="pyproject.props" />
+  </ImportGroup>
+  <PropertyGroup Label="UserMacros" />
+  <PropertyGroup>
+    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
+  </PropertyGroup>
+  <ItemDefinitionGroup>
+    <Link>
+      <AdditionalDependencies>version.lib;ole32.lib;oleaut32.lib;Microsoft.VisualStudio.Setup.Configuration.Native.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(PySourcePath)PC\external\$(PlatformToolset)\$(ArchName)</AdditionalLibraryDirectories>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemGroup>
+    <ClCompile Include="..\PC\_findvs.cpp" />
+  </ItemGroup>
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <ProjectReference Include="pythoncore.vcxproj">
+      <Project>{cf7ac3d1-e2df-41d2-bea6-1e2556cdea26}</Project>
+      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
+    </ProjectReference>
+  </ItemGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+  <ImportGroup Label="ExtensionTargets">
+  </ImportGroup>
+</Project>
diff --git a/PCbuild/_distutils_findvs.vcxproj.filters b/PCbuild/_distutils_findvs.vcxproj.filters
new file mode 100644
index 00000000000..f48d74fd69f
--- /dev/null
+++ b/PCbuild/_distutils_findvs.vcxproj.filters
@@ -0,0 +1,16 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{c56a5dd3-7838-48e9-a781-855d8be7370f}</UniqueIdentifier>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="..\PC\_findvs.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
\ No newline at end of file
diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj
index 05e885717ff..848b3b2877f 100644
--- a/PCbuild/pcbuild.proj
+++ b/PCbuild/pcbuild.proj
@@ -49,7 +49,7 @@
     <!-- pyshellext.dll -->
     <Projects Include="pyshellext.vcxproj" />
     <!-- Extension modules -->
-    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;_queue;select;unicodedata;winsound" />
+    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_distutils_findvs;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;_queue;select;unicodedata;winsound" />
     <!-- Extension modules that require external sources -->
     <ExternalModules Include="_bz2;_lzma;_sqlite3" />
     <!-- _ssl will build _socket as well, which may cause conflicts in parallel builds -->
diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln
index 5ff86a49c7c..0443610331e 100644
--- a/PCbuild/pcbuild.sln
+++ b/PCbuild/pcbuild.sln
@@ -1,6 +1,6 @@
 Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio 15
-VisualStudioVersion = 15.0.26621.2
+VisualStudioVersion = 15.0.27130.2024
 MinimumVisualStudioVersion = 10.0.40219.1
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{553EC33E-9816-4996-A660-5D6186A0B0B3}"
 	ProjectSection(SolutionItems) = preProject
@@ -93,6 +93,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_queue", "_queue.vcxproj",
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "liblzma", "liblzma.vcxproj", "{12728250-16EC-4DC6-94D7-E21DD88947F8}"
 EndProject
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_distutils_findvs", "_distutils_findvs.vcxproj", "{41ADEDF9-11D8-474E-B4D7-BB82332C878E}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Win32 = Debug|Win32
@@ -693,6 +695,22 @@ Global
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Release|Win32.Build.0 = Release|Win32
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Release|x64.ActiveCfg = Release|x64
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Release|x64.Build.0 = Release|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|Win32.ActiveCfg = Debug|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|Win32.Build.0 = Debug|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|x64.ActiveCfg = Debug|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|x64.Build.0 = Debug|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|Win32.Build.0 = PGInstrument|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|x64.ActiveCfg = PGInstrument|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|x64.Build.0 = PGInstrument|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|Win32.Build.0 = PGUpdate|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|x64.ActiveCfg = PGUpdate|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|x64.Build.0 = PGUpdate|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|Win32.ActiveCfg = Release|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|Win32.Build.0 = Release|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|x64.ActiveCfg = Release|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|x64.Build.0 = Release|x64
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
index c8a5d7d01b4..bf2ce669f08 100644
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -75,7 +75,6 @@
     </ClCompile>
     <Link>
       <AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(PySourcePath)PC\external\$(PlatformToolset)\$(ArchName)</AdditionalLibraryDirectories>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
@@ -350,7 +349,6 @@
     <ClCompile Include="..\PC\config.c" />
     <ClCompile Include="..\PC\getpathp.c" />
     <ClCompile Include="..\PC\msvcrtmodule.c" />
-    <ClCompile Include="..\PC\_findvs.cpp" />
     <ClCompile Include="..\Python\pyhash.c" />
     <ClCompile Include="..\Python\_warnings.c" />
     <ClCompile Include="..\Python\asdl.c" />
diff --git a/Tools/msi/lib/lib_files.wxs b/Tools/msi/lib/lib_files.wxs
index fa0da3ab43f..5a72612c6a5 100644
--- a/Tools/msi/lib/lib_files.wxs
+++ b/Tools/msi/lib/lib_files.wxs
@@ -1,6 +1,6 @@
 ?<?xml version="1.0" encoding="UTF-8"?>
 <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
-    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3;_asyncio ?>
+    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3;_asyncio;_queue;_distutils_findvs ?>
     <Fragment>
         <ComponentGroup Id="lib_extensions">
             <?foreach ext in $(var.exts)?>


From webhook-mailer at python.org  Thu Jan 18 17:09:51 2018
From: webhook-mailer at python.org (Steve Dower)
Date: Thu, 18 Jan 2018 22:09:51 -0000
Subject: [Python-checkins] [3.6] bpo-32588 Move _distutils_findvs into its
 own module (GH-5227) (#5228)
Message-ID: <mailman.58.1516313394.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ccf7f05c5d3fdea89d857e775d2c6371f3e15b4a
commit: ccf7f05c5d3fdea89d857e775d2c6371f3e15b4a
branch: 3.6
author: Steve Dower <steve.dower at microsoft.com>
committer: GitHub <noreply at github.com>
date: 2018-01-19T09:09:49+11:00
summary:

[3.6] bpo-32588 Move _distutils_findvs into its own module (GH-5227) (#5228)

files:
A Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst
A PCbuild/_distutils_findvs.vcxproj
A PCbuild/_distutils_findvs.vcxproj.filters
M Lib/distutils/_msvccompiler.py
M Lib/distutils/command/bdist_wininst.py
M PC/_findvs.cpp
M PC/config.c
M PCbuild/pcbuild.proj
M PCbuild/pcbuild.sln
M PCbuild/pythoncore.vcxproj
M Tools/msi/lib/lib_files.wxs

diff --git a/Lib/distutils/_msvccompiler.py b/Lib/distutils/_msvccompiler.py
index ef1356b97d6..c9d3c6c6712 100644
--- a/Lib/distutils/_msvccompiler.py
+++ b/Lib/distutils/_msvccompiler.py
@@ -56,7 +56,7 @@ def _find_vc2015():
     return best_version, best_dir
 
 def _find_vc2017():
-    import _findvs
+    import _distutils_findvs
     import threading
 
     best_version = 0,   # tuple for full version comparisons
@@ -66,7 +66,7 @@ def _find_vc2017():
     # initialize COM.
     all_packages = []
     def _getall():
-        all_packages.extend(_findvs.findall())
+        all_packages.extend(_distutils_findvs.findall())
     t = threading.Thread(target=_getall)
     t.start()
     t.join()
diff --git a/Lib/distutils/command/bdist_wininst.py b/Lib/distutils/command/bdist_wininst.py
index 83f00732657..0871a4f7d6b 100644
--- a/Lib/distutils/command/bdist_wininst.py
+++ b/Lib/distutils/command/bdist_wininst.py
@@ -337,11 +337,10 @@ def get_exe_bytes(self):
                 # cross-building, so assume the latest version
                 bv = '14.0'
             else:
-                bv = '.'.join(CRT_ASSEMBLY_VERSION.split('.', 2)[:2])
-                if bv in ('14.11', '14.12'):
-                    # v142, v141 and v140 are binary compatible,
-                    # so keep using the 14.0 stub.
-                    bv = '14.0'
+                # as far as we know, CRT is binary compatible based on
+                # the first field, so assume 'x.0' until proven otherwise
+                major = CRT_ASSEMBLY_VERSION.partition('.')[0]
+                bv = major + '.0'
 
 
         # wininst-x.y.exe is in the same directory as this file
diff --git a/Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst b/Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst
new file mode 100644
index 00000000000..18e717a4257
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2018-01-18-14-56-45.bpo-32588.vHww6F.rst
@@ -0,0 +1 @@
+Create standalone _distutils_findvs module.
diff --git a/PC/_findvs.cpp b/PC/_findvs.cpp
index dd7c1fcb723..ecc68e82e49 100644
--- a/PC/_findvs.cpp
+++ b/PC/_findvs.cpp
@@ -12,10 +12,6 @@
 #include <Windows.h>
 #include <Strsafe.h>
 #include "external\include\Setup.Configuration.h"
-#pragma comment(lib, "ole32.lib")
-#pragma comment(lib, "oleaut32.lib")
-#pragma comment(lib, "version.lib")
-#pragma comment(lib, "Microsoft.VisualStudio.Setup.Configuration.Native.lib")
 
 #include <Python.h>
 
@@ -236,7 +232,7 @@ static int exec_findvs(PyObject *module)
     return 0; // success
 }
 
-PyDoc_STRVAR(findvs_doc, "The _findvs helper module");
+PyDoc_STRVAR(findvs_doc, "The _distutils_findvs helper module");
 
 static PyModuleDef_Slot findvs_slots[] = {
     { Py_mod_exec, exec_findvs },
@@ -245,7 +241,7 @@ static PyModuleDef_Slot findvs_slots[] = {
 
 static PyModuleDef findvs_def = {
     PyModuleDef_HEAD_INIT,
-    "_findvs",
+    "_distutils_findvs",
     findvs_doc,
     0,              // m_size
     NULL,           // m_methods
@@ -256,8 +252,8 @@ static PyModuleDef findvs_def = {
 };
 
 extern "C" {
-    PyMODINIT_FUNC PyInit__findvs(void)
+    PyMODINIT_FUNC PyInit__distutils_findvs(void)
     {
         return PyModuleDef_Init(&findvs_def);
     }
-}
\ No newline at end of file
+}
diff --git a/PC/config.c b/PC/config.c
index f631d73268f..bab9100413f 100644
--- a/PC/config.c
+++ b/PC/config.c
@@ -69,7 +69,6 @@ extern PyObject* _PyWarnings_Init(void);
 extern PyObject* PyInit__string(void);
 extern PyObject* PyInit__stat(void);
 extern PyObject* PyInit__opcode(void);
-extern PyObject* PyInit__findvs(void);
 
 /* tools/freeze/makeconfig.py marker for additional "extern" */
 /* -- ADDMODULE MARKER 1 -- */
@@ -166,8 +165,6 @@ struct _inittab _PyImport_Inittab[] = {
     {"_stat", PyInit__stat},
     {"_opcode", PyInit__opcode},
 
-    {"_findvs", PyInit__findvs},
-
     /* Sentinel */
     {0, 0}
 };
diff --git a/PCbuild/_distutils_findvs.vcxproj b/PCbuild/_distutils_findvs.vcxproj
new file mode 100644
index 00000000000..3db1280ac88
--- /dev/null
+++ b/PCbuild/_distutils_findvs.vcxproj
@@ -0,0 +1,83 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup Label="ProjectConfigurations">
+    <ProjectConfiguration Include="Debug|Win32">
+      <Configuration>Debug</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Debug|x64">
+      <Configuration>Debug</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|Win32">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|x64">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|Win32">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|x64">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|Win32">
+      <Configuration>Release</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|x64">
+      <Configuration>Release</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+  </ItemGroup>
+  <PropertyGroup Label="Globals">
+    <ProjectGuid>{41ADEDF9-11D8-474E-B4D7-BB82332C878E}</ProjectGuid>
+    <RootNamespace>_distutils_findvs</RootNamespace>
+    <Keyword>Win32Proj</Keyword>
+  </PropertyGroup>
+  <Import Project="python.props" />
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>DynamicLibrary</ConfigurationType>
+    <CharacterSet>NotSet</CharacterSet>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+  <PropertyGroup>
+    <TargetExt>.pyd</TargetExt>
+  </PropertyGroup>
+  <ImportGroup Label="ExtensionSettings">
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="pyproject.props" />
+  </ImportGroup>
+  <PropertyGroup Label="UserMacros" />
+  <PropertyGroup>
+    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
+  </PropertyGroup>
+  <ItemDefinitionGroup>
+    <Link>
+      <AdditionalDependencies>version.lib;ole32.lib;oleaut32.lib;Microsoft.VisualStudio.Setup.Configuration.Native.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(PySourcePath)PC\external\$(PlatformToolset)\$(ArchName)</AdditionalLibraryDirectories>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemGroup>
+    <ClCompile Include="..\PC\_findvs.cpp" />
+  </ItemGroup>
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <ProjectReference Include="pythoncore.vcxproj">
+      <Project>{cf7ac3d1-e2df-41d2-bea6-1e2556cdea26}</Project>
+      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
+    </ProjectReference>
+  </ItemGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+  <ImportGroup Label="ExtensionTargets">
+  </ImportGroup>
+</Project>
diff --git a/PCbuild/_distutils_findvs.vcxproj.filters b/PCbuild/_distutils_findvs.vcxproj.filters
new file mode 100644
index 00000000000..f48d74fd69f
--- /dev/null
+++ b/PCbuild/_distutils_findvs.vcxproj.filters
@@ -0,0 +1,16 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{c56a5dd3-7838-48e9-a781-855d8be7370f}</UniqueIdentifier>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="..\PC\_findvs.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
\ No newline at end of file
diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj
index c6b8487c0a2..f7890969c81 100644
--- a/PCbuild/pcbuild.proj
+++ b/PCbuild/pcbuild.proj
@@ -49,7 +49,7 @@
     <!-- pyshellext.dll -->
     <Projects Include="pyshellext.vcxproj" />
     <!-- Extension modules -->
-    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;select;unicodedata;winsound" />
+    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_distutils_findvs;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;select;unicodedata;winsound" />
     <!-- Extension modules that require external sources -->
     <ExternalModules Include="_bz2;_lzma;_sqlite3" />
     <!-- _ssl will build _socket as well, which may cause conflicts in parallel builds -->
diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln
index 736a5997037..94a0089f465 100644
--- a/PCbuild/pcbuild.sln
+++ b/PCbuild/pcbuild.sln
@@ -98,6 +98,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_asyncio", "_asyncio.vcxpro
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "liblzma", "liblzma.vcxproj", "{12728250-16EC-4DC6-94D7-E21DD88947F8}"
 EndProject
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_distutils_findvs", "_distutils_findvs.vcxproj", "{41ADEDF9-11D8-474E-B4D7-BB82332C878E}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Win32 = Debug|Win32
@@ -762,8 +764,27 @@ Global
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Release|Win32.Build.0 = Release|Win32
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Release|x64.ActiveCfg = Release|x64
 		{12728250-16EC-4DC6-94D7-E21DD88947F8}.Release|x64.Build.0 = Release|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|Win32.ActiveCfg = Debug|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|Win32.Build.0 = Debug|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|x64.ActiveCfg = Debug|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Debug|x64.Build.0 = Debug|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|Win32.Build.0 = PGInstrument|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|x64.ActiveCfg = PGInstrument|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGInstrument|x64.Build.0 = PGInstrument|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|Win32.Build.0 = PGUpdate|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|x64.ActiveCfg = PGUpdate|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.PGUpdate|x64.Build.0 = PGUpdate|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|Win32.ActiveCfg = Release|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|Win32.Build.0 = Release|Win32
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|x64.ActiveCfg = Release|x64
+		{41ADEDF9-11D8-474E-B4D7-BB82332C878E}.Release|x64.Build.0 = Release|x64
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
 	EndGlobalSection
+	GlobalSection(ExtensibilityGlobals) = postSolution
+		SolutionGuid = {E8690CFE-326A-430E-82D4-B4CE667CC1BA}
+	EndGlobalSection
 EndGlobal
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
index 0f018525874..cdba3cc0d3c 100644
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -71,7 +71,6 @@
     </ClCompile>
     <Link>
       <AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
-      <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(PySourcePath)PC\external\$(PlatformToolset)\$(ArchName)</AdditionalLibraryDirectories>
       <BaseAddress>0x1e000000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
@@ -350,7 +349,6 @@
     <ClCompile Include="..\PC\config.c" />
     <ClCompile Include="..\PC\getpathp.c" />
     <ClCompile Include="..\PC\msvcrtmodule.c" />
-    <ClCompile Include="..\PC\_findvs.cpp" />
     <ClCompile Include="..\Python\pyhash.c" />
     <ClCompile Include="..\Python\random.c" />
     <ClCompile Include="..\Python\_warnings.c" />
diff --git a/Tools/msi/lib/lib_files.wxs b/Tools/msi/lib/lib_files.wxs
index a83f544db63..6f19006df7e 100644
--- a/Tools/msi/lib/lib_files.wxs
+++ b/Tools/msi/lib/lib_files.wxs
@@ -1,6 +1,6 @@
 ?<?xml version="1.0" encoding="UTF-8"?>
 <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
-    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3;_asyncio ?>
+    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3;_asyncio;_distutils_findvs ?>
     <Fragment>
         <ComponentGroup Id="lib_extensions">
             <?foreach ext in $(var.exts)?>


From webhook-mailer at python.org  Thu Jan 18 17:27:24 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Thu, 18 Jan 2018 22:27:24 -0000
Subject: [Python-checkins] Removed unnecesssary bit inversion which doesn't
 improve dispersion statistics (GH-5235) (#5236)
Message-ID: <mailman.59.1516314445.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/051650ab8de620dd7e3c557f8227c43439ea8c6a
commit: 051650ab8de620dd7e3c557f8227c43439ea8c6a
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-18T14:27:22-08:00
summary:

Removed unnecesssary bit inversion which doesn't improve dispersion statistics (GH-5235) (#5236)

(cherry picked from commit fa7880604191f81cbdddc191216f7b1e39a74d8d)

files:
M Objects/setobject.c

diff --git a/Objects/setobject.c b/Objects/setobject.c
index 093a15f367f..c742041b16d 100644
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -790,7 +790,7 @@ frozenset_hash(PyObject *self)
     hash ^= ((Py_uhash_t)PySet_GET_SIZE(self) + 1) * 1927868237UL;
 
     /* Disperse patterns arising in nested frozensets */
-    hash ^= (hash >> 11) ^ (~hash >> 25);
+    hash ^= (hash >> 11) ^ (hash >> 25);
     hash = hash * 69069U + 907133923UL;
 
     /* -1 is reserved as an error code */


From webhook-mailer at python.org  Thu Jan 18 19:16:16 2018
From: webhook-mailer at python.org (Mariatta)
Date: Fri, 19 Jan 2018 00:16:16 -0000
Subject: [Python-checkins] bpo-32586: Fix code example in urllib2's doc
 (GH-5238)
Message-ID: <mailman.60.1516320977.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/8ca036d4716fc86ff42474ba35d3cd32f0188a15
commit: 8ca036d4716fc86ff42474ba35d3cd32f0188a15
branch: 2.7
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-18T16:16:13-08:00
summary:

bpo-32586: Fix code example in urllib2's doc  (GH-5238)

It should be `urllib2.URLError` instead of just `URLError`.

files:
M Doc/howto/urllib2.rst

diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst
index d697c216d8c..ce6394859d5 100644
--- a/Doc/howto/urllib2.rst
+++ b/Doc/howto/urllib2.rst
@@ -203,7 +203,7 @@ e.g. ::
 
     >>> req = urllib2.Request('http://www.pretend_server.org')
     >>> try: urllib2.urlopen(req)
-    ... except URLError as e:
+    ... except urllib2.URLError as e:
     ...    print e.reason   #doctest: +SKIP
     ...
     (4, 'getaddrinfo failed')


From lp_benchmark_robot at intel.com  Thu Jan 18 21:12:51 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Thu, 18 Jan 2018 18:12:51 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-18
Message-ID: <95463207-cfed-4b7d-8c39-b41e2e61c2d7@orsmsx103.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-18 03:04:13-08:00.
  - commit:		05d68a8
  - previous commit:	44a70e9
  - revision date:	2018-01-18 11:15:25+01:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.867% |  -0.566%   |  +6.205%   |  +8.257%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.381% |  -0.484%   |  +21.556%  |  +10.840%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.621% |  -0.144%   |  +23.451%  |  +9.279%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.733% |  -0.067%   |  +20.946%  |  +11.267%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 3.448% |  -0.565%   |  +6.859%   |  +13.675%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.609% |  -0.603%   |  +9.889%   |  +12.508%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.125% |  -1.165%   |  +4.727%   |  +11.461%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.524% |  -0.524%   |  -1.223%   |  +9.503%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 2.907% |  +0.129%   |  +10.199%  |  +15.992%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 5.226% |  -1.133%   |  +16.014%  |  +13.923%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.649% |  -1.136%   |  +4.456%   |  +7.315%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.296% |  -0.344%   |  +4.984%   |  +5.880%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.757% |  +0.158%   |  +3.069%   |  +4.739%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.949% |  -1.587%   |  +13.342%  |  +10.165%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.267% |  -1.345%   |  +9.596%   |  +10.166%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.846% |  -0.311%   |  +6.153%   |  +11.486%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.411% |  -0.270%   |  +9.472%   |  +11.093%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.221% |  -0.840%   |  +7.776%   |  +10.459%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.050% |  +0.095%   |  +1.846%   |  +11.236%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 5.081% |  +0.311%   |  -1.661%   |  +14.789%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.552% |  -0.870%   |  +7.090%   |  +11.880%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.584% |  +0.344%   |  +47.157%  |  +12.261%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.582% |  -1.230%   |  +8.293%   |  +13.619%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.649% |  -0.119%   |  +17.342%  |  +13.586%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 1.738% |  -0.295%   |  +7.766%   |  +9.856%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 3.213% |  +0.007%   |  +4.419%   |  +4.320%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.532% |  +1.218%   |  -2.378%   |  +2.993%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.342% |  -0.281%   |  +4.957%   |  +6.703%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.546% |  +0.026%   |  +3.273%   |  +10.674%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.187% |  -1.045%   |  -1.967%   |  +24.026%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.331% |  +0.036%   |  +0.304%   |  +23.133%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.936% |  +0.099%   |  +2.038%   |  +19.940%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 4.066% |  +0.653%   |  +11.589%  |  +10.332%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.156% |  +0.008%   |  +0.296%   |  +9.736%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.127% |  -0.113%   |  +11.306%  |  +5.171%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.088% |  -0.004%   |  +2.274%   |  +5.417%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.871% |  -0.325%   |  +10.392%  |  +12.249%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 6.076% |  -1.925%   |  +1.227%   |  +12.173%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.545% |  -0.131%   |  -2.046%   |  +11.944%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 1.745% |  +3.660%   |  -2.729%   |  -0.371%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 0.896% |  -0.181%   |  +3.734%   |  +9.157%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.232% |  +0.458%   |  +8.659%   |  +14.868%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 1.485% |  -0.215%   |  -3.579%   |  +5.015%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 2.223% |  -0.728%   |  +26.486%  |  +10.361%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.224% |  -0.424%   |  +3.057%   |  +5.263%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.744% |  +0.369%   |  +13.907%  |  +8.990%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 4.023% |  +1.241%   |  -3.854%   |  -1.257%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.409% |  -0.253%   |  +5.407%   |  +3.101%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.199% |  -0.671%   |  +6.187%   |  +7.462%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.272% |  -1.768%   |  +5.370%   |  +5.698%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 4.769% |  -1.835%   |  +17.094%  |  +11.725%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 3.119% |  +0.326%   |  +14.806%  |  +5.947%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.492% |  -1.758%   |  +8.629%   |  +8.164%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 4.345% |  -0.865%   |  +14.319%  |  +8.788%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.152% |  -1.020%   |  +7.426%   |  +13.149%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 4.945% |  +1.436%   |  +16.872%  |  +13.967%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.092% |  -0.677%   |  +5.919%   |  +7.700%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.317% |  +0.839%   |  +2.150%   |  -1.016%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 9.484% |  +1.624%   |  +3.789%   |  +24.972%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 0.569% |  +0.070%   |  -1.527%   |  +17.819%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 2.062% |  +1.345%   |  +8.207%   |  +6.289%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.415% |  -0.186%   |  +1.914%   |  +13.036%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.269% |  -0.013%   |  +2.605%   |  +7.677%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 1.300% |  +0.758%   |  -6.461%   |  +12.737%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.267% |  -0.036%   |  +4.019%   |  +12.224%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-18

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Fri Jan 19 04:07:42 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 19 Jan 2018 09:07:42 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=5
Message-ID: <20180119090742.1.6EDDC68CF4FA14F4@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [7, -7, 1] memory blocks, sum=1
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogFoLwUc', '--timeout', '7200']

From webhook-mailer at python.org  Fri Jan 19 13:04:32 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Fri, 19 Jan 2018 18:04:32 -0000
Subject: [Python-checkins] bpo-32410: Make SendfileNotAvailableError
 exception public (#5243)
Message-ID: <mailman.61.1516385074.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/7464e87a6511d3626b04c9833a262a77b1f21e23
commit: 7464e87a6511d3626b04c9833a262a77b1f21e23
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-19T20:04:29+02:00
summary:

bpo-32410: Make SendfileNotAvailableError exception public (#5243)

files:
M Doc/library/asyncio-eventloop.rst
M Lib/asyncio/base_events.py
M Lib/asyncio/events.py
M Lib/asyncio/unix_events.py
M Lib/test/test_asyncio/test_base_events.py
M Lib/test/test_asyncio/test_unix_events.py

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index e63180b750a..6cee171b65a 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -726,7 +726,7 @@ Low-level socket operations
    the file when the platform does not support the sendfile syscall
    (e.g. Windows or SSL socket on Unix).
 
-   Raise :exc:`RuntimeError` if the system does not support
+   Raise :exc:`SendfileNotAvailableError` if the system does not support
    *sendfile* syscall and *fallback* is ``False``.
 
    .. versionadded:: 3.7
@@ -980,6 +980,18 @@ Handle
       .. versionadded:: 3.7
 
 
+SendfileNotAvailableError
+-------------------------
+
+
+.. exception:: SendfileNotAvailableError
+
+   Sendfile syscall is not available, subclass of :exc:`RuntimeError`.
+
+   Raised if the OS does not support senfile syscall for
+   given socket or file type.
+
+
 Event loop examples
 -------------------
 
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index b6a9384d95b..00c84a835c8 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -154,10 +154,6 @@ def _run_until_complete_cb(fut):
     futures._get_loop(fut).stop()
 
 
-class _SendfileNotAvailable(RuntimeError):
-    pass
-
-
 class Server(events.AbstractServer):
 
     def __init__(self, loop, sockets):
@@ -659,17 +655,16 @@ def _getaddrinfo_debug(self, host, port, family, type, proto, flags):
         try:
             return await self._sock_sendfile_native(sock, file,
                                                     offset, count)
-        except _SendfileNotAvailable as exc:
-            if fallback:
-                return await self._sock_sendfile_fallback(sock, file,
-                                                          offset, count)
-            else:
-                raise RuntimeError(exc.args[0]) from None
+        except events.SendfileNotAvailableError as exc:
+            if not fallback:
+                raise
+        return await self._sock_sendfile_fallback(sock, file,
+                                                  offset, count)
 
     async def _sock_sendfile_native(self, sock, file, offset, count):
         # NB: sendfile syscall is not supported for SSL sockets and
         # non-mmap files even if sendfile is supported by OS
-        raise _SendfileNotAvailable(
+        raise events.SendfileNotAvailableError(
             f"syscall sendfile is not available for socket {sock!r} "
             "and file {file!r} combination")
 
diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index b06721f3b03..d5365dc480d 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -3,7 +3,7 @@
 __all__ = (
     'AbstractEventLoopPolicy',
     'AbstractEventLoop', 'AbstractServer',
-    'Handle', 'TimerHandle',
+    'Handle', 'TimerHandle', 'SendfileNotAvailableError',
     'get_event_loop_policy', 'set_event_loop_policy',
     'get_event_loop', 'set_event_loop', 'new_event_loop',
     'get_child_watcher', 'set_child_watcher',
@@ -20,6 +20,14 @@
 from . import format_helpers
 
 
+class SendfileNotAvailableError(RuntimeError):
+    """Sendfile syscall is not available.
+
+    Raised if OS does not support senfile syscall for given socket or
+    file type.
+    """
+
+
 class Handle:
     """Object returned by callback registration methods."""
 
diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
index f40ef12f265..028a0ca8f83 100644
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -313,16 +313,16 @@ def _child_watcher_callback(self, pid, returncode, transp):
         try:
             os.sendfile
         except AttributeError as exc:
-            raise base_events._SendfileNotAvailable(
+            raise events.SendfileNotAvailableError(
                 "os.sendfile() is not available")
         try:
             fileno = file.fileno()
         except (AttributeError, io.UnsupportedOperation) as err:
-            raise base_events._SendfileNotAvailable("not a regular file")
+            raise events.SendfileNotAvailableError("not a regular file")
         try:
             fsize = os.fstat(fileno).st_size
         except OSError as err:
-            raise base_events._SendfileNotAvailable("not a regular file")
+            raise events.SendfileNotAvailableError("not a regular file")
         blocksize = count if count else fsize
         if not blocksize:
             return 0  # empty file
@@ -365,7 +365,7 @@ def _sock_sendfile_native_impl(self, fut, registered_fd, sock, fileno,
                 # one being 'file' is not a regular mmap(2)-like
                 # file, in which case we'll fall back on using
                 # plain send().
-                err = base_events._SendfileNotAvailable(
+                err = events.SendfileNotAvailableError(
                     "os.sendfile call failed")
                 self._sock_sendfile_update_filepos(fileno, offset, total_sent)
                 fut.set_exception(err)
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index 085124f5372..fc3b81096da 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -14,6 +14,7 @@
 import asyncio
 from asyncio import base_events
 from asyncio import constants
+from asyncio import events
 from test.test_asyncio import utils as test_utils
 from test import support
 from test.support.script_helper import assert_python_ok
@@ -1860,7 +1861,7 @@ def cleanup():
     def test__sock_sendfile_native_failure(self):
         sock, proto = self.prepare()
 
-        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+        with self.assertRaisesRegex(events.SendfileNotAvailableError,
                                     "sendfile is not available"):
             self.run_loop(self.loop._sock_sendfile_native(sock, self.file,
                                                           0, None))
@@ -1871,7 +1872,7 @@ def test__sock_sendfile_native_failure(self):
     def test_sock_sendfile_no_fallback(self):
         sock, proto = self.prepare()
 
-        with self.assertRaisesRegex(RuntimeError,
+        with self.assertRaisesRegex(events.SendfileNotAvailableError,
                                     "sendfile is not available"):
             self.run_loop(self.loop.sock_sendfile(sock, self.file,
                                                   fallback=False))
diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py
index 4e2b76bb472..8646bf765e9 100644
--- a/Lib/test/test_asyncio/test_unix_events.py
+++ b/Lib/test/test_asyncio/test_unix_events.py
@@ -23,6 +23,7 @@
 import asyncio
 from asyncio import log
 from asyncio import base_events
+from asyncio import events
 from asyncio import unix_events
 from test.test_asyncio import utils as test_utils
 
@@ -517,7 +518,7 @@ def test_with_offset_and_count(self):
     def test_sendfile_not_available(self):
         sock, proto = self.prepare()
         with mock.patch('asyncio.unix_events.os', spec=[]):
-            with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+            with self.assertRaisesRegex(events.SendfileNotAvailableError,
                                         "os[.]sendfile[(][)] is not available"):
                 self.run_loop(self.loop._sock_sendfile_native(sock, self.file,
                                                               0, None))
@@ -526,7 +527,7 @@ def test_sendfile_not_available(self):
     def test_sendfile_not_a_file(self):
         sock, proto = self.prepare()
         f = object()
-        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+        with self.assertRaisesRegex(events.SendfileNotAvailableError,
                                     "not a regular file"):
             self.run_loop(self.loop._sock_sendfile_native(sock, f,
                                                           0, None))
@@ -535,7 +536,7 @@ def test_sendfile_not_a_file(self):
     def test_sendfile_iobuffer(self):
         sock, proto = self.prepare()
         f = io.BytesIO()
-        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+        with self.assertRaisesRegex(events.SendfileNotAvailableError,
                                     "not a regular file"):
             self.run_loop(self.loop._sock_sendfile_native(sock, f,
                                                           0, None))
@@ -545,7 +546,7 @@ def test_sendfile_not_regular_file(self):
         sock, proto = self.prepare()
         f = mock.Mock()
         f.fileno.return_value = -1
-        with self.assertRaisesRegex(base_events._SendfileNotAvailable,
+        with self.assertRaisesRegex(events.SendfileNotAvailableError,
                                     "not a regular file"):
             self.run_loop(self.loop._sock_sendfile_native(sock, f,
                                                           0, None))
@@ -631,7 +632,7 @@ def test_os_error_first_call(self):
         with self.assertRaises(KeyError):
             self.loop._selector.get_key(sock)
         exc = fut.exception()
-        self.assertIsInstance(exc, base_events._SendfileNotAvailable)
+        self.assertIsInstance(exc, events.SendfileNotAvailableError)
         self.assertEqual(0, self.file.tell())
 
     def test_os_error_next_call(self):
@@ -656,7 +657,7 @@ def test_exception(self):
 
         fileno = self.file.fileno()
         fut = self.loop.create_future()
-        err = RuntimeError()
+        err = events.SendfileNotAvailableError()
         with mock.patch('os.sendfile', side_effect=err):
             self.loop._sock_sendfile_native_impl(fut, sock.fileno(),
                                                  sock, fileno,


From webhook-mailer at python.org  Fri Jan 19 19:25:40 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Sat, 20 Jan 2018 00:25:40 -0000
Subject: [Python-checkins] bpo-25910: Link redirections in docs (#1933)
Message-ID: <mailman.62.1516407942.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/338cd83c5dceaed785f5bf613e2122f871908e2a
commit: 338cd83c5dceaed785f5bf613e2122f871908e2a
branch: master
author: Sanyam Khurana <8039608+CuriousLearner at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-20T01:25:37+01:00
summary:

bpo-25910: Link redirections in docs (#1933)

Fixes some redirection links in docs.

files:
M Doc/distributing/index.rst
M Doc/distutils/index.rst
M Doc/distutils/setupscript.rst
M Doc/extending/extending.rst
M Doc/extending/index.rst
M Doc/faq/general.rst
M Doc/howto/logging.rst
M Doc/howto/pyporting.rst
M Doc/howto/unicode.rst
M Doc/howto/urllib2.rst
M Doc/install/index.rst
M Doc/installing/index.rst
M Doc/library/ast.rst
M Doc/library/colorsys.rst
M Doc/library/hashlib.rst
M Doc/library/mailbox.rst
M Doc/library/math.rst
M Doc/library/plistlib.rst
M Doc/library/ssl.rst
M Doc/library/statistics.rst
M Doc/library/tkinter.rst
M Doc/library/tkinter.ttk.rst
M Doc/library/unittest.mock-examples.rst
M Doc/library/urllib.request.rst
M Doc/library/venv.rst
M Doc/library/wsgiref.rst
M Doc/library/xml.dom.rst
M Doc/using/mac.rst
M Doc/using/unix.rst
M Doc/using/windows.rst
M Doc/whatsnew/2.2.rst
M Doc/whatsnew/2.3.rst
M Doc/whatsnew/2.5.rst
M Doc/whatsnew/2.6.rst
M Doc/whatsnew/2.7.rst
M Doc/whatsnew/3.2.rst
M Doc/whatsnew/3.6.rst

diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst
index 82ecd2c1ef4..aedbe712d3d 100644
--- a/Doc/distributing/index.rst
+++ b/Doc/distributing/index.rst
@@ -62,7 +62,7 @@ Key terms
   locally.
 
 .. _setuptools: https://setuptools.readthedocs.io/en/latest/
-.. _wheel: https://wheel.readthedocs.org
+.. _wheel: https://wheel.readthedocs.io/
 
 Open source licensing and collaboration
 =======================================
@@ -111,7 +111,7 @@ by invoking the ``pip`` module at the command line::
 The Python Packaging User Guide includes more details on the `currently
 recommended tools`_.
 
-.. _currently recommended tools: https://packaging.python.org/en/latest/current/#packaging-tool-recommendations
+.. _currently recommended tools: https://packaging.python.org/guides/tool-recommendations/#packaging-tool-recommendations
 
 Reading the guide
 =================
@@ -124,11 +124,11 @@ involved in creating a project:
 * `Uploading the project to the Python Packaging Index`_
 
 .. _Project structure: \
-   https://packaging.python.org/en/latest/distributing/
+    https://packaging.python.org/tutorials/distributing-packages/
 .. _Building and packaging the project: \
-   https://packaging.python.org/en/latest/distributing/#packaging-your-project
+   https://packaging.python.org/tutorials/distributing-packages/#packaging-your-project
 .. _Uploading the project to the Python Packaging Index: \
-   https://packaging.python.org/en/latest/distributing/#uploading-your-project-to-pypi
+   https://packaging.python.org/tutorials/distributing-packages/#uploading-your-project-to-pypi
 
 
 How do I...?
@@ -160,7 +160,7 @@ Python Packaging User Guide for more information and recommendations.
 .. seealso::
 
    `Python Packaging User Guide: Binary Extensions
-   <https://packaging.python.org/en/latest/extensions>`__
+   <https://packaging.python.org/guides/packaging-binary-extensions/>`__
 
 .. other topics:
 
diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst
index c565bcc5628..d6f7640fcb6 100644
--- a/Doc/distutils/index.rst
+++ b/Doc/distutils/index.rst
@@ -22,7 +22,7 @@ very little overhead for build/release/install mechanics.
    This guide only covers the basic tools for building and distributing
    extensions that are provided as part of this version of Python. Third party
    tools offer easier to use and more secure alternatives. Refer to the `quick
-   recommendations section <https://packaging.python.org/en/latest/current/>`__
+   recommendations section <https://packaging.python.org/guides/tool-recommendations/>`__
    in the Python Packaging User Guide for more information.
 
 .. toctree::
diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst
index 542ad544843..952607a4073 100644
--- a/Doc/distutils/setupscript.rst
+++ b/Doc/distutils/setupscript.rst
@@ -625,7 +625,7 @@ Notes:
 
 (7)
     The valid classifiers are listed on
-    `PyPI <http://pypi.python.org/pypi?:action=list_classifiers>`_.
+    `PyPI <https://pypi.python.org/pypi?:action=list_classifiers>`_.
 
 (8)
     To preserve backward compatibility, this field also accepts a string. If
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index ea1c29a397e..e02f7837b69 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -27,7 +27,8 @@ your system setup; details are given in later chapters.
    avoid writing C extensions and preserve portability to other implementations.
    For example, if your use case is calling C library functions or system calls,
    you should consider using the :mod:`ctypes` module or the `cffi
-   <https://cffi.readthedocs.org>`_ library rather than writing custom C code.
+   <https://cffi.readthedocs.io/>`_ library rather than writing
+   custom C code.
    These modules let you write Python code to interface with C code and are more
    portable between implementations of Python than writing and compiling a C
    extension module.
diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst
index 9eec8c273af..80594e357fd 100644
--- a/Doc/extending/index.rst
+++ b/Doc/extending/index.rst
@@ -32,7 +32,7 @@ approaches to creating C and C++ extensions for Python.
 
 .. seealso::
 
-   `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/en/latest/extensions/>`_
+   `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/guides/packaging-binary-extensions/>`_
       The Python Packaging User Guide not only covers several available
       tools that simplify the creation of binary extensions, but also
       discusses the various reasons why creating an extension module may be
diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst
index d4a97fd81af..0d1cb198da8 100644
--- a/Doc/faq/general.rst
+++ b/Doc/faq/general.rst
@@ -272,7 +272,7 @@ The Python project's infrastructure is located all over the world.
 `www.python.org <https://www.python.org>`_ is graciously hosted by `Rackspace
 <https://www.rackspace.com>`_, with CDN caching provided by `Fastly
 <https://www.fastly.com>`_.  `Upfront Systems
-<http://www.upfrontsystems.co.za/>`_ hosts `bugs.python.org
+<http://www.upfrontsoftware.co.za>`_ hosts `bugs.python.org
 <https://bugs.python.org>`_.  Many other Python services like `the Wiki
 <https://wiki.python.org>`_ are hosted by `Oregon State
 University Open Source Lab <https://osuosl.org>`_.
diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst
index 8074b0f1099..4ee68b4747e 100644
--- a/Doc/howto/logging.rst
+++ b/Doc/howto/logging.rst
@@ -314,7 +314,7 @@ favourite beverage and carry on.
 If your logging needs are simple, then use the above examples to incorporate
 logging into your own scripts, and if you run into problems or don't
 understand something, please post a question on the comp.lang.python Usenet
-group (available at https://groups.google.com/group/comp.lang.python) and you
+group (available at https://groups.google.com/forum/#!forum/comp.lang.python) and you
 should receive help before too long.
 
 Still here? You can carry on reading the next few sections, which provide a
diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst
index bec6a3d72ce..98c81206741 100644
--- a/Doc/howto/pyporting.rst
+++ b/Doc/howto/pyporting.rst
@@ -433,12 +433,12 @@ to make sure everything functions as expected in both versions of Python.
 .. _Futurize: http://python-future.org/automatic_conversion.html
 .. _importlib: https://docs.python.org/3/library/importlib.html#module-importlib
 .. _importlib2: https://pypi.python.org/pypi/importlib2
-.. _Modernize: https://python-modernize.readthedocs.org/en/latest/
+.. _Modernize: https://python-modernize.readthedocs.io/
 .. _mypy: http://mypy-lang.org/
 .. _Porting to Python 3: http://python3porting.com/
 .. _Pylint: https://pypi.python.org/pypi/pylint
 
-.. _Python 3 Q & A: https://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html
+.. _Python 3 Q & A: https://ncoghlan-devs-python-notes.readthedocs.io/en/latest/python3/questions_and_answers.html
 
 .. _pytype: https://github.com/google/pytype
 .. _python-future: http://python-future.org/
diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst
index a83e5a21895..d4b8f8d2204 100644
--- a/Doc/howto/unicode.rst
+++ b/Doc/howto/unicode.rst
@@ -214,7 +214,7 @@ difficult reading.  `A chronology <http://www.unicode.org/history/>`_ of the
 origin and development of Unicode is also available on the site.
 
 To help understand the standard, Jukka Korpela has written `an introductory
-guide <https://www.cs.tut.fi/~jkorpela/unicode/guide.html>`_ to reading the
+guide <http://jkorpela.fi/unicode/guide.html>`_ to reading the
 Unicode character tables.
 
 Another `good introductory article <https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/>`_
diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst
index 8d383e03ee8..c1fd5cf0680 100644
--- a/Doc/howto/urllib2.rst
+++ b/Doc/howto/urllib2.rst
@@ -403,7 +403,7 @@ fetched, particularly the headers sent by the server. It is currently an
 :class:`http.client.HTTPMessage` instance.
 
 Typical headers include 'Content-length', 'Content-type', and so on. See the
-`Quick Reference to HTTP Headers <https://www.cs.tut.fi/~jkorpela/http.html>`_
+`Quick Reference to HTTP Headers <http://jkorpela.fi/http.html>`_
 for a useful listing of HTTP headers with brief explanations of their meaning
 and use.
 
diff --git a/Doc/install/index.rst b/Doc/install/index.rst
index bc080b00c59..0545b8f33d0 100644
--- a/Doc/install/index.rst
+++ b/Doc/install/index.rst
@@ -36,7 +36,7 @@ modules and extensions.
    This guide only covers the basic tools for building and distributing
    extensions that are provided as part of this version of Python. Third party
    tools offer easier to use and more secure alternatives. Refer to the `quick
-   recommendations section <https://packaging.python.org/en/latest/current/>`__
+   recommendations section <https://packaging.python.org/guides/tool-recommendations/>`__
    in the Python Packaging User Guide for more information.
 
 
diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst
index 09bb8251c35..f9a224be92b 100644
--- a/Doc/installing/index.rst
+++ b/Doc/installing/index.rst
@@ -48,7 +48,7 @@ Key terms
   repository of open source licensed packages made available for use by
   other Python users.
 * the `Python Packaging Authority
-  <https://www.pypa.io/en/latest/>`__ are the group of
+  <https://www.pypa.io/>`__ are the group of
   developers and documentation authors responsible for the maintenance and
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation,
diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst
index bda46f46c43..b7f610ba8b2 100644
--- a/Doc/library/ast.rst
+++ b/Doc/library/ast.rst
@@ -261,5 +261,5 @@ and classes for traversing abstract syntax trees:
 
 .. seealso::
 
-    `Green Tree Snakes <https://greentreesnakes.readthedocs.org/>`_, an external documentation resource, has good
+    `Green Tree Snakes <https://greentreesnakes.readthedocs.io/>`_, an external documentation resource, has good
     details on working with Python ASTs.
diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst
index c33f531cc1d..1360c7cc9f1 100644
--- a/Doc/library/colorsys.rst
+++ b/Doc/library/colorsys.rst
@@ -21,7 +21,7 @@ spaces, the coordinates are all between 0 and 1.
 .. seealso::
 
    More information about color spaces can be found at
-   http://www.poynton.com/ColorFAQ.html and
+   http://poynton.ca/ColorFAQ.html and
    https://www.cambridgeincolour.com/tutorials/color-spaces.htm.
 
 The :mod:`colorsys` module defines the following functions:
diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst
index d4916914f79..eda18adc9e5 100644
--- a/Doc/library/hashlib.rst
+++ b/Doc/library/hashlib.rst
@@ -562,7 +562,7 @@ on the hash function used in digital signatures.
     by the signer.
 
     (`NIST SP-800-106 "Randomized Hashing for Digital Signatures"
-    <http://csrc.nist.gov/publications/nistpubs/800-106/NIST-SP-800-106.pdf>`_)
+    <https://csrc.nist.gov/publications/detail/sp/800-106/final>`_)
 
 In BLAKE2 the salt is processed as a one-time input to the hash function during
 initialization, rather than as an input to each compression function.
@@ -728,7 +728,7 @@ Domain Dedication 1.0 Universal:
    https://blake2.net
       Official BLAKE2 website.
 
-   http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf
+   https://csrc.nist.gov/csrc/media/publications/fips/180/2/archive/2002-08-01/documents/fips180-2.pdf
       The FIPS 180-2 publication on Secure Hash Algorithms.
 
    https://en.wikipedia.org/wiki/Cryptographic_hash_function#Cryptographic_hash_algorithms
diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst
index 48d76a83221..d901ad2cc1c 100644
--- a/Doc/library/mailbox.rst
+++ b/Doc/library/mailbox.rst
@@ -620,7 +620,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF.
    `nmh - Message Handling System <http://www.nongnu.org/nmh/>`_
       Home page of :program:`nmh`, an updated version of the original :program:`mh`.
 
-   `MH & nmh: Email for Users & Programmers <http://rand-mh.sourceforge.net/book/>`_
+   `MH & nmh: Email for Users & Programmers <https://rand-mh.sourceforge.io/book/>`_
       A GPL-licensed book on :program:`mh` and :program:`nmh`, with some information
       on the mailbox format.
 
diff --git a/Doc/library/math.rst b/Doc/library/math.rst
index 9e4946323a9..55eb41b86f5 100644
--- a/Doc/library/math.rst
+++ b/Doc/library/math.rst
@@ -461,7 +461,7 @@ Constants
    Tau is a circle constant equal to 2\ *?*, the ratio of a circle's circumference to
    its radius. To learn more about Tau, check out Vi Hart's video `Pi is (still)
    Wrong <https://www.youtube.com/watch?v=jG7vhMMXagQ>`_, and start celebrating
-   `Tau day <http://tauday.com/>`_ by eating twice as much pie!
+   `Tau day <https://tauday.com/>`_ by eating twice as much pie!
 
    .. versionadded:: 3.6
 
diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst
index 7d306a2bcaf..20c086c8b52 100644
--- a/Doc/library/plistlib.rst
+++ b/Doc/library/plistlib.rst
@@ -38,7 +38,7 @@ or :class:`datetime.datetime` objects.
 
 .. seealso::
 
-   `PList manual page <https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/plist.5.html>`_
+   `PList manual page <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/PropertyLists/>`_
       Apple's documentation of the file format.
 
 
diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst
index b40d9584104..4c44ffa23ac 100644
--- a/Doc/library/ssl.rst
+++ b/Doc/library/ssl.rst
@@ -868,9 +868,9 @@ Constants
 .. data:: HAS_NPN
 
    Whether the OpenSSL library has built-in support for *Next Protocol
-   Negotiation* as described in the `NPN draft specification
-   <https://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. When true,
-   you can use the :meth:`SSLContext.set_npn_protocols` method to advertise
+   Negotiation* as described in the `Application Layer Protocol
+   Negotiation <https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation>`_.
+   When true, you can use the :meth:`SSLContext.set_npn_protocols` method to advertise
    which protocols you want to support.
 
    .. versionadded:: 3.3
@@ -1374,7 +1374,7 @@ to speed up repeated connections from the same clients.
    The *capath* string, if present, is
    the path to a directory containing several CA certificates in PEM format,
    following an `OpenSSL specific layout
-   <https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_load_verify_locations.html>`_.
+   <https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html>`_.
 
    The *cadata* object, if present, is either an ASCII string of one or more
    PEM-encoded certificates or a :term:`bytes-like object` of DER-encoded
@@ -1501,8 +1501,8 @@ to speed up repeated connections from the same clients.
    Specify which protocols the socket should advertise during the SSL/TLS
    handshake. It should be a list of strings, like ``['http/1.1', 'spdy/2']``,
    ordered by preference. The selection of a protocol will happen during the
-   handshake, and will play out according to the `NPN draft specification
-   <https://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. After a
+   handshake, and will play out according to the `Application Layer Protocol Negotiation
+   <https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation>`_. After a
    successful handshake, the :meth:`SSLSocket.selected_npn_protocol` method will
    return the agreed-upon protocol.
 
@@ -1663,8 +1663,7 @@ to speed up repeated connections from the same clients.
 .. method:: SSLContext.session_stats()
 
    Get statistics about the SSL sessions created or managed by this context.
-   A dictionary is returned which maps the names of each `piece of information
-   <https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_sess_number.html>`_ to their
+   A dictionary is returned which maps the names of each `piece of information <https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_sess_number.html>`_ to their
    numeric values.  For example, here is the total number of hits and misses
    in the session cache since the context was created::
 
@@ -2365,7 +2364,7 @@ enabled when negotiating a SSL session is possible through the
 :meth:`SSLContext.set_ciphers` method.  Starting from Python 3.2.3, the
 ssl module disables certain weak ciphers by default, but you may want
 to further restrict the cipher choice. Be sure to read OpenSSL's documentation
-about the `cipher list format <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`_.
+about the `cipher list format <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`_.
 If you want to check which ciphers are enabled by a given cipher list, use
 :meth:`SSLContext.get_ciphers` or the ``openssl ciphers`` command on your
 system.
@@ -2393,10 +2392,10 @@ successful call of :func:`~ssl.RAND_add`, :func:`~ssl.RAND_bytes` or
    `RFC 1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management <https://www.ietf.org/rfc/rfc1422>`_
        Steve Kent
 
-   `RFC 4086: Randomness Requirements for Security <http://datatracker.ietf.org/doc/rfc4086/>`_
+   `RFC 4086: Randomness Requirements for Security <https://datatracker.ietf.org/doc/rfc4086/>`_
        Donald E., Jeffrey I. Schiller
 
-   `RFC 5280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile <http://datatracker.ietf.org/doc/rfc5280/>`_
+   `RFC 5280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile <https://datatracker.ietf.org/doc/rfc5280/>`_
        D. Cooper
 
    `RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2 <https://tools.ietf.org/html/rfc5246>`_
diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index 2aa778c4d06..bc3817836b9 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -257,8 +257,6 @@ However, for reading convenience, most of the examples show sorted sequences.
       * "Statistics for the Behavioral Sciences", Frederick J Gravetter and
         Larry B Wallnau (8th Edition).
 
-      * Calculating the `median <https://www.ualberta.ca/~opscan/median.html>`_.
-
       * The `SSMEDIAN
         <https://help.gnome.org/users/gnumeric/stable/gnumeric.html#gnumeric-function-SSMEDIAN>`_
         function in the Gnome Gnumeric spreadsheet, including `this discussion
diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst
index f222fe6f8b2..6d90e43e33b 100644
--- a/Doc/library/tkinter.rst
+++ b/Doc/library/tkinter.rst
@@ -35,7 +35,7 @@ this should open a window demonstrating a simple Tk interface.
    `Tcl/Tk manual <https://www.tcl.tk/man/tcl8.5/>`_
       Official manual for the latest tcl/tk version.
 
-   `Programming Python <http://learning-python.com/books/about-pp4e.html>`_
+   `Programming Python <http://learning-python.com/about-pp4e.html>`_
       Book by Mark Lutz, has excellent coverage of Tkinter.
 
    `Modern Tkinter for Busy Python Developers <https://www.amazon.com/Modern-Tkinter-Python-Developers-ebook/dp/B0071QDNLO/>`_
diff --git a/Doc/library/tkinter.ttk.rst b/Doc/library/tkinter.ttk.rst
index 927f7f3c6c1..debbc81ce1a 100644
--- a/Doc/library/tkinter.ttk.rst
+++ b/Doc/library/tkinter.ttk.rst
@@ -26,7 +26,7 @@ appearance.
 
 .. seealso::
 
-   `Tk Widget Styling Support <https://www.tcl.tk/cgi-bin/tct/tip/48>`_
+   `Tk Widget Styling Support <https://core.tcl.tk/tips/doc/trunk/tip/48.md>`_
       A document introducing theming support for Tk
 
 
diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst
index 05f33740d75..5bf3d575520 100644
--- a/Doc/library/unittest.mock-examples.rst
+++ b/Doc/library/unittest.mock-examples.rst
@@ -1008,7 +1008,7 @@ subclass.
     True
 
 Sometimes this is inconvenient. For example, `one user
-<https://code.google.com/p/mock/issues/detail?id=105>`_ is subclassing mock to
+<https://code.google.com/archive/p/mock/issues/105>`_ is subclassing mock to
 created a `Twisted adaptor
 <https://twistedmatrix.com/documents/11.0.0/api/twisted.python.components.html>`_.
 Having this applied to attributes too actually causes errors.
@@ -1251,7 +1251,7 @@ With a bit of tweaking you could have the comparison function raise the
 :exc:`AssertionError` directly and provide a more useful failure message.
 
 As of version 1.5, the Python testing library `PyHamcrest
-<https://pyhamcrest.readthedocs.org/>`_ provides similar functionality,
+<https://pyhamcrest.readthedocs.io/>`_ provides similar functionality,
 that may be useful here, in the form of its equality matcher
 (`hamcrest.library.integration.match_equality
-<https://pyhamcrest.readthedocs.org/en/release-1.8/integration/#module-hamcrest.library.integration.match_equality>`_).
+<https://pyhamcrest.readthedocs.io/en/release-1.8/integration/#module-hamcrest.library.integration.match_equality>`_).
diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst
index 5a10f9571e9..413d8b6ffdf 100644
--- a/Doc/library/urllib.request.rst
+++ b/Doc/library/urllib.request.rst
@@ -62,7 +62,7 @@ The :mod:`urllib.request` module defines the following functions:
 
    * :meth:`~urllib.response.addinfourl.info` --- return the meta-information of the page, such as headers,
      in the form of an :func:`email.message_from_string` instance (see
-     `Quick Reference to HTTP Headers <https://www.cs.tut.fi/~jkorpela/http.html>`_)
+     `Quick Reference to HTTP Headers <http://jkorpela.fi/http.html>`_)
 
    * :meth:`~urllib.response.addinfourl.getcode` -- return the HTTP status code of the response.
 
diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst
index 17c80c8d982..6707be7fc82 100644
--- a/Doc/library/venv.rst
+++ b/Doc/library/venv.rst
@@ -458,4 +458,4 @@ subclass which installs setuptools and pip into a created virtual environment::
 
 
 This script is also available for download `online
-<https://gist.github.com/4673395>`_.
+<https://gist.github.com/vsajip/4673395>`_.
diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst
index a1d446902b8..b0ef3465b04 100644
--- a/Doc/library/wsgiref.rst
+++ b/Doc/library/wsgiref.rst
@@ -26,8 +26,8 @@ for implementing WSGI servers, a demo HTTP server that serves WSGI applications,
 and a validation tool that checks WSGI servers and applications for conformance
 to the WSGI specification (:pep:`3333`).
 
-See https://wsgi.readthedocs.org/ for more information about WSGI, and links to
-tutorials and other resources.
+See `wsgi.readthedocs.io <https://wsgi.readthedocs.io/>`_ for more information about WSGI, and links
+to tutorials and other resources.
 
 .. XXX If you're just trying to write a web application...
 
diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst
index de334af2363..18519a75a54 100644
--- a/Doc/library/xml.dom.rst
+++ b/Doc/library/xml.dom.rst
@@ -67,13 +67,13 @@ implementations are free to support the strict mapping from IDL).  See section
 
 .. seealso::
 
-   `Document Object Model (DOM) Level 2 Specification <https://www.w3.org/TR/DOM-Level-2-Core/>`_
+   `Document Object Model (DOM) Level 2 Specification <https://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/>`_
       The W3C recommendation upon which the Python DOM API is based.
 
    `Document Object Model (DOM) Level 1 Specification <https://www.w3.org/TR/REC-DOM-Level-1/>`_
       The W3C recommendation for the DOM supported by :mod:`xml.dom.minidom`.
 
-   `Python Language Mapping Specification <http://www.omg.org/spec/PYTH/1.2/PDF>`_
+   `Python Language Mapping Specification <http://www.omg.org/cgi-bin/doc?formal/02-11-05.pdf>`_
       This specifies the mapping from OMG IDL to Python.
 
 
diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst
index b89c98709ea..a386728eaee 100644
--- a/Doc/using/mac.rst
+++ b/Doc/using/mac.rst
@@ -66,7 +66,7 @@ number of standard Unix command line editors, :program:`vim` and
 :program:`BBEdit` or :program:`TextWrangler` from Bare Bones Software (see
 http://www.barebones.com/products/bbedit/index.html) are good choices, as is
 :program:`TextMate` (see https://macromates.com/). Other editors include
-:program:`Gvim` (http://macvim.org) and :program:`Aquamacs`
+:program:`Gvim` (http://macvim-dev.github.io/macvim/) and :program:`Aquamacs`
 (http://aquamacs.org/).
 
 To run your script from the Terminal window you must make sure that
@@ -117,7 +117,7 @@ The IDE
 
 MacPython ships with the standard IDLE development environment. A good
 introduction to using IDLE can be found at
-https://hkn.eecs.berkeley.edu/~dyoo/python/idle_intro/index.html.
+http://www.hashcollision.org/hkn/python/idle_intro/index.html.
 
 
 .. _mac-package-manager:
diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst
index 8b96ebea396..ccdf84dcfa5 100644
--- a/Doc/using/unix.rst
+++ b/Doc/using/unix.rst
@@ -30,7 +30,7 @@ following links:
       for Debian users
    https://en.opensuse.org/Portal:Packaging
       for OpenSuse users
-   https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html
+   https://docs-old.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html
       for Fedora users
    http://www.slackbook.org/html/package-management-making-packages.html
       for Slackware users
diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index cc560d968a1..1c7b8142af2 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -307,11 +307,11 @@ key features:
 `ActivePython <https://www.activestate.com/activepython/>`_
     Installer with multi-platform compatibility, documentation, PyWin32
 
-`Anaconda <https://www.continuum.io/downloads/>`_
+`Anaconda <https://www.anaconda.com/download/>`_
     Popular scientific modules (such as numpy, scipy and pandas) and the
     ``conda`` package manager.
 
-`Canopy <https://www.enthought.com/products/canopy/>`_
+`Canopy <https://www.enthought.com/product/canopy/>`_
     A "comprehensive Python analysis environment" with editors and other
     development tools.
 
@@ -377,7 +377,7 @@ System variables, you need non-restricted access to your machine
 
 .. seealso::
 
-    https://support.microsoft.com/kb/100843
+    https://support.microsoft.com/en-us/help/100843/environment-variables-in-windows-nt
       Environment variables in Windows NT
 
     https://technet.microsoft.com/en-us/library/cc754250.aspx
@@ -386,7 +386,7 @@ System variables, you need non-restricted access to your machine
     https://technet.microsoft.com/en-us/library/cc755104.aspx
       The SETX command, for permanently modifying environment variables
 
-    https://support.microsoft.com/kb/310519
+    https://support.microsoft.com/en-us/help/310519/how-to-manage-environment-variables-in-windows-xp
       How To Manage Environment Variables in Windows XP
 
     https://www.chem.gla.ac.uk/~louis/software/faq/q1.html
diff --git a/Doc/whatsnew/2.2.rst b/Doc/whatsnew/2.2.rst
index a0bb81a9748..0aa6003f4c4 100644
--- a/Doc/whatsnew/2.2.rst
+++ b/Doc/whatsnew/2.2.rst
@@ -957,7 +957,7 @@ New and Improved Modules
      #   'title': 'html2fo 0.3 (Default)'}, ... ]
 
   The :mod:`SimpleXMLRPCServer` module makes it easy to create straightforward
-  XML-RPC servers.  See http://www.xmlrpc.com/ for more information about XML-RPC.
+  XML-RPC servers.  See http://xmlrpc.scripting.com/ for more information about XML-RPC.
 
 * The new :mod:`hmac` module implements the HMAC algorithm described by
   :rfc:`2104`. (Contributed by Gerhard H?ring.)
diff --git a/Doc/whatsnew/2.3.rst b/Doc/whatsnew/2.3.rst
index cebfb21156b..b5628a910ae 100644
--- a/Doc/whatsnew/2.3.rst
+++ b/Doc/whatsnew/2.3.rst
@@ -1955,7 +1955,7 @@ The RPM spec files, found in the :file:`Misc/RPM/` directory in the Python
 source distribution, were updated for 2.3.  (Contributed by Sean Reifschneider.)
 
 Other new platforms now supported by Python include AtheOS
-(http://atheos.cx/), GNU/Hurd, and OpenVMS.
+(http://www.atheos.cx/), GNU/Hurd, and OpenVMS.
 
 .. ======================================================================
 
diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst
index 4d482918313..15d35d384c4 100644
--- a/Doc/whatsnew/2.5.rst
+++ b/Doc/whatsnew/2.5.rst
@@ -330,7 +330,7 @@ statement, only the ``from ... import`` form.
    :pep:`328` - Imports: Multi-Line and Absolute/Relative
       PEP written by Aahz; implemented by Thomas Wouters.
 
-   https://pylib.readthedocs.org/
+   https://pylib.readthedocs.io/
       The py library by Holger Krekel, which contains the :mod:`py.std` package.
 
 .. ======================================================================
diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst
index 45b0ab9f345..33d9f7ce43b 100644
--- a/Doc/whatsnew/2.6.rst
+++ b/Doc/whatsnew/2.6.rst
@@ -172,7 +172,7 @@ this edition of "What's New in Python" links to the bug/patch
 item for each change.
 
 Hosting of the Python bug tracker is kindly provided by
-`Upfront Systems <http://www.upfrontsystems.co.za/>`__
+`Upfront Systems <http://www.upfrontsoftware.co.za>`__
 of Stellenbosch, South Africa.  Martin von L?wis put a
 lot of effort into importing existing bugs and patches from
 SourceForge; his scripts for this import operation are at
diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst
index eec89845395..90ce06c3286 100644
--- a/Doc/whatsnew/2.7.rst
+++ b/Doc/whatsnew/2.7.rst
@@ -1545,7 +1545,7 @@ changes, or look through the Subversion logs for all the details.
   *ciphers* argument that's a string listing the encryption algorithms
   to be allowed; the format of the string is described
   `in the OpenSSL documentation
-  <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__.
+  <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__.
   (Added by Antoine Pitrou; :issue:`8322`.)
 
   Another change makes the extension load all of OpenSSL's ciphers and
diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst
index 78d3105a5a4..2d740006a37 100644
--- a/Doc/whatsnew/3.2.rst
+++ b/Doc/whatsnew/3.2.rst
@@ -1643,7 +1643,7 @@ for secure (encrypted, authenticated) internet connections:
 * The :func:`ssl.wrap_socket` constructor function now takes a *ciphers*
   argument.  The *ciphers* string lists the allowed encryption algorithms using
   the format described in the `OpenSSL documentation
-  <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__.
+  <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__.
 
 * When linked against recent versions of OpenSSL, the :mod:`ssl` module now
   supports the Server Name Indication extension to the TLS protocol, allowing
diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst
index 67c86df0207..a355af7adb8 100644
--- a/Doc/whatsnew/3.6.rst
+++ b/Doc/whatsnew/3.6.rst
@@ -1568,7 +1568,7 @@ mark class variables.  As introduced in :pep:`526`, a variable annotation
 wrapped in ClassVar indicates that a given attribute is intended to be used as
 a class variable and should not be set on instances of that class.
 (Contributed by Ivan Levkivskyi in `Github #280
-<https://github.com/python/typing/issues/280>`_.)
+<https://github.com/python/typing/pull/280>`_.)
 
 A new :const:`~typing.TYPE_CHECKING` constant that is assumed to be
 ``True`` by the static type chekers, but is ``False`` at runtime.


From webhook-mailer at python.org  Fri Jan 19 19:54:45 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Sat, 20 Jan 2018 00:54:45 -0000
Subject: [Python-checkins] bpo-32596: Lazy import concurrent.futures.process
 and thread (GH-5241)
Message-ID: <mailman.63.1516409686.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/6690bb9f17d34eb3dec0aca8919d8d27d6c3c452
commit: 6690bb9f17d34eb3dec0aca8919d8d27d6c3c452
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-20T09:54:42+09:00
summary:

bpo-32596: Lazy import concurrent.futures.process and thread (GH-5241)

files:
A Misc/NEWS.d/next/Library/2018-01-19-19-57-45.bpo-32596.4aVIie.rst
M Lib/concurrent/futures/__init__.py

diff --git a/Lib/concurrent/futures/__init__.py b/Lib/concurrent/futures/__init__.py
index ba8de163905..72aca818d3e 100644
--- a/Lib/concurrent/futures/__init__.py
+++ b/Lib/concurrent/futures/__init__.py
@@ -15,5 +15,36 @@
                                       Executor,
                                       wait,
                                       as_completed)
-from concurrent.futures.process import ProcessPoolExecutor
-from concurrent.futures.thread import ThreadPoolExecutor
+
+__all__ = (
+    'FIRST_COMPLETED',
+    'FIRST_EXCEPTION',
+    'ALL_COMPLETED',
+    'CancelledError',
+    'TimeoutError',
+    'BrokenExecutor',
+    'Future',
+    'Executor',
+    'wait',
+    'as_completed',
+    'ProcessPoolExecutor',
+    'ThreadPoolExecutor',
+)
+
+
+def __dir__():
+    return __all__ + ('__author__', '__doc__')
+
+
+def __getattr__(name):
+    global ProcessPoolExecutor, ThreadPoolExecutor
+
+    if name == 'ProcessPoolExecutor':
+        from .process import ProcessPoolExecutor
+        return ProcessPoolExecutor
+
+    if name == 'ThreadPoolExecutor':
+        from .thread import ThreadPoolExecutor
+        return ThreadPoolExecutor
+
+    raise AttributeError(f"module {__name__} has no attribute {name}")
diff --git a/Misc/NEWS.d/next/Library/2018-01-19-19-57-45.bpo-32596.4aVIie.rst b/Misc/NEWS.d/next/Library/2018-01-19-19-57-45.bpo-32596.4aVIie.rst
new file mode 100644
index 00000000000..a90f7d17653
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-19-19-57-45.bpo-32596.4aVIie.rst
@@ -0,0 +1,4 @@
+``concurrent.futures`` imports ``ThreadPoolExecutor`` and
+``ProcessPoolExecutor`` lazily (using :pep:`562`).
+It makes ``import asyncio`` about 15% faster because asyncio
+uses only ``ThreadPoolExecutor`` by default.


From webhook-mailer at python.org  Fri Jan 19 22:12:25 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sat, 20 Jan 2018 03:12:25 -0000
Subject: [Python-checkins] bpo-32028: Fix suggestions for indented print
 statements (GH-4688)
Message-ID: <mailman.64.1516417945.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/d57f26c753dce61f72b52b96db3a3253d9f2fc3e
commit: d57f26c753dce61f72b52b96db3a3253d9f2fc3e
branch: master
author: Sanyam Khurana <8039608+CuriousLearner at users.noreply.github.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-20T13:12:22+10:00
summary:

bpo-32028: Fix suggestions for indented print statements (GH-4688)

The suggested replacement for print statements previously failed to account
for leading whitespace and hence could end up including unwanted text in
the proposed call to the print builtin.

Patch by Sanyam Khurana.

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst
M Lib/test/test_print.py
M Objects/exceptions.c

diff --git a/Lib/test/test_print.py b/Lib/test/test_print.py
index e6434feaf5e..7bc23cf9fc6 100644
--- a/Lib/test/test_print.py
+++ b/Lib/test/test_print.py
@@ -156,6 +156,15 @@ def test_string_with_excessive_whitespace(self):
 
         self.assertIn('print("Hello World", end=" ")', str(context.exception))
 
+    def test_string_with_leading_whitespace(self):
+        python2_print_str = '''if 1:
+            print "Hello World"
+        '''
+        with self.assertRaises(SyntaxError) as context:
+            exec(python2_print_str)
+
+        self.assertIn('print("Hello World")', str(context.exception))
+
     def test_stream_redirection_hint_for_py2_migration(self):
         # Test correct hint produced for Py2 redirection syntax
         with self.assertRaises(TypeError) as context:
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst
new file mode 100644
index 00000000000..8e2b2e1b350
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst	
@@ -0,0 +1,3 @@
+Leading whitespace is now correctly ignored when generating suggestions
+for converting Py2 print statements to Py3 builtin print function calls.
+Patch by Sanyam Khurana.
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
index 4901eb1cc3c..d59abd1a7f4 100644
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -2846,17 +2846,23 @@ _set_legacy_print_statement_msg(PySyntaxErrorObject *self, Py_ssize_t start)
 
     // PRINT_OFFSET is to remove `print ` word from the data.
     const int PRINT_OFFSET = 6;
-    Py_ssize_t text_len = PyUnicode_GET_LENGTH(self->text);
-    PyObject *data = PyUnicode_Substring(self->text, PRINT_OFFSET, text_len);
-
+    const int STRIP_BOTH = 2;
+    // Issue 32028: Handle case when whitespace is used with print call
+    PyObject *initial_data = _PyUnicode_XStrip(self->text, STRIP_BOTH, strip_sep_obj);
+    if (initial_data == NULL) {
+        Py_DECREF(strip_sep_obj);
+        return -1;
+    }
+    Py_ssize_t text_len = PyUnicode_GET_LENGTH(initial_data);
+    PyObject *data = PyUnicode_Substring(initial_data, PRINT_OFFSET, text_len);
+    Py_DECREF(initial_data);
     if (data == NULL) {
         Py_DECREF(strip_sep_obj);
         return -1;
     }
-    PyObject *new_data = _PyUnicode_XStrip(data, 2, strip_sep_obj);
+    PyObject *new_data = _PyUnicode_XStrip(data, STRIP_BOTH, strip_sep_obj);
     Py_DECREF(data);
     Py_DECREF(strip_sep_obj);
-
     if (new_data == NULL) {
         return -1;
     }


From webhook-mailer at python.org  Fri Jan 19 22:56:34 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sat, 20 Jan 2018 03:56:34 -0000
Subject: [Python-checkins] [3.6] bpo-32028: Fix suggestions for indented
 print statements (GH-5249)
Message-ID: <mailman.65.1516420596.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/4002d5dbf4c058bbf2462f9f5dea057956d1caff
commit: 4002d5dbf4c058bbf2462f9f5dea057956d1caff
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-20T13:56:31+10:00
summary:

[3.6] bpo-32028: Fix suggestions for indented print statements (GH-5249)

The suggested replacement for print statements previously failed to account
for leading whitespace and hence could end up including unwanted text in
the proposed call to the print builtin.

Patch by Sanyam Khurana.
(cherry picked from commit d57f26c753dce61f72b52b96db3a3253d9f2fc3e)

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst
M Lib/test/test_print.py
M Objects/exceptions.c

diff --git a/Lib/test/test_print.py b/Lib/test/test_print.py
index e6434feaf5e..7bc23cf9fc6 100644
--- a/Lib/test/test_print.py
+++ b/Lib/test/test_print.py
@@ -156,6 +156,15 @@ def test_string_with_excessive_whitespace(self):
 
         self.assertIn('print("Hello World", end=" ")', str(context.exception))
 
+    def test_string_with_leading_whitespace(self):
+        python2_print_str = '''if 1:
+            print "Hello World"
+        '''
+        with self.assertRaises(SyntaxError) as context:
+            exec(python2_print_str)
+
+        self.assertIn('print("Hello World")', str(context.exception))
+
     def test_stream_redirection_hint_for_py2_migration(self):
         # Test correct hint produced for Py2 redirection syntax
         with self.assertRaises(TypeError) as context:
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst
new file mode 100644
index 00000000000..8e2b2e1b350
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-03-22-29-13.bpo-32028.KC2w4Q.rst	
@@ -0,0 +1,3 @@
+Leading whitespace is now correctly ignored when generating suggestions
+for converting Py2 print statements to Py3 builtin print function calls.
+Patch by Sanyam Khurana.
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
index df4899372a5..0a75f1b0fe6 100644
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -2865,17 +2865,23 @@ _set_legacy_print_statement_msg(PySyntaxErrorObject *self, Py_ssize_t start)
 
     // PRINT_OFFSET is to remove `print ` word from the data.
     const int PRINT_OFFSET = 6;
-    Py_ssize_t text_len = PyUnicode_GET_LENGTH(self->text);
-    PyObject *data = PyUnicode_Substring(self->text, PRINT_OFFSET, text_len);
-
+    const int STRIP_BOTH = 2;
+    // Issue 32028: Handle case when whitespace is used with print call
+    PyObject *initial_data = _PyUnicode_XStrip(self->text, STRIP_BOTH, strip_sep_obj);
+    if (initial_data == NULL) {
+        Py_DECREF(strip_sep_obj);
+        return -1;
+    }
+    Py_ssize_t text_len = PyUnicode_GET_LENGTH(initial_data);
+    PyObject *data = PyUnicode_Substring(initial_data, PRINT_OFFSET, text_len);
+    Py_DECREF(initial_data);
     if (data == NULL) {
         Py_DECREF(strip_sep_obj);
         return -1;
     }
-    PyObject *new_data = _PyUnicode_XStrip(data, 2, strip_sep_obj);
+    PyObject *new_data = _PyUnicode_XStrip(data, STRIP_BOTH, strip_sep_obj);
     Py_DECREF(data);
     Py_DECREF(strip_sep_obj);
-
     if (new_data == NULL) {
         return -1;
     }


From lp_benchmark_robot at intel.com  Sat Jan 20 00:18:39 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Fri, 19 Jan 2018 21:18:39 -0800
Subject: [Python-checkins] [65 flat] Results for Python (master branch)
 2018-01-19
Message-ID: <905a6897-a3c2-481c-bf7b-5490ab9c6479@orsmsx102.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-19 03:04:15-08:00.
  - commit:		2507e29
  - previous commit:	05d68a8
  - revision date:	2018-01-19 09:09:36+11:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.988% |  -0.144%   |  +6.070%   |  +8.439%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.560% |  -0.037%   |  +21.527%  |  +12.705%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 0.972% |  +0.264%   |  +23.654%  |  +10.164%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.691% |  +0.030%   |  +20.970%  |  +8.979%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.992% |  +0.824%   |  +7.627%   |  +9.343%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.177% |  +0.822%   |  +10.630%  |  +10.548%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.437% |  +0.282%   |  +4.995%   |  +11.847%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.568% |  +0.179%   |  -1.041%   |  +8.139%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 3.468% |  +0.313%   |  +10.480%  |  +15.508%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 5.726% |  +2.047%   |  +17.734%  |  +11.109%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.500% |  +0.251%   |  +4.696%   |  +7.716%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.240% |  +0.259%   |  +5.231%   |  +5.966%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.958% |  +0.029%   |  +3.098%   |  +4.946%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.089% |  +0.756%   |  +13.998%  |  +9.296%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.960% |  +0.489%   |  +10.037%  |  +9.212%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.653% |  +0.155%   |  +6.298%   |  +11.323%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.397% |  +0.142%   |  +9.601%   |  +10.404%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.088% |  +0.415%   |  +8.159%   |  +9.304%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.308% |  +0.107%   |  +1.951%   |  +9.867%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 2.585% |  +0.804%   |  -0.844%   |  +14.820%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.549% |  +1.116%   |  +8.127%   |  +10.329%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 3.074% |  -0.397%   |  +46.947%  |  +10.472%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.502% |  +1.665%   |  +9.820%   |  +11.184%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.510% |  -0.088%   |  +17.270%  |  +13.763%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 1.773% |  -0.139%   |  +7.638%   |  +9.693%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 2.544% |  -0.816%   |  +3.640%   |  +5.255%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.812% |  -0.119%   |  -2.500%   |  +1.336%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 0.738% |  +0.111%   |  +5.063%   |  +7.727%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.397% |  -0.006%   |  +3.267%   |  +9.571%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.085% |  +0.246%   |  -1.717%   |  +22.452%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.483% |  +0.029%   |  +0.332%   |  +22.571%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.722% |  +0.032%   |  +2.070%   |  +21.433%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 2.416% |  +0.962%   |  +12.440%  |  +9.604%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.358% |  -0.088%   |  +0.208%   |  +9.778%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.120% |  -0.016%   |  +11.291%  |  +5.496%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.089% |  -0.040%   |  +2.235%   |  +5.689%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.105% |  -0.108%   |  +10.296%  |  +12.157%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 4.962% |  +2.728%   |  +3.921%   |  +9.194%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.346% |  +0.101%   |  -1.943%   |  +11.908%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 1.203% |  +0.424%   |  -2.293%   |  +3.175%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 0.887% |  -0.009%   |  +3.725%   |  +6.498%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.520% |  -0.158%   |  +8.515%   |  +14.241%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.483% |  +0.157%   |  -3.416%   |  +5.525%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 2.423% |  -0.251%   |  +26.302%  |  +6.324%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.286% |  -0.463%   |  +2.609%   |  +5.185%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 1.648% |  -0.394%   |  +13.568%  |  +8.526%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 1.198% |  +0.881%   |  -2.939%   |  +0.715%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.377% |  +0.087%   |  +5.490%   |  +1.656%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.037% |  +0.530%   |  +6.684%   |  +7.121%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.187% |  +0.921%   |  +6.241%   |  +5.187%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 3.722% |  -0.850%   |  +16.390%  |  +10.794%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 3.182% |  -0.439%   |  +14.431%  |  +7.353%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.051% |  +0.734%   |  +9.301%   |  +7.631%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.917% |  +0.816%   |  +15.018%  |  +8.461%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.506% |  +0.371%   |  +7.770%   |  +12.219%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 4.236% |  -0.223%   |  +16.687%  |  +16.761%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.284% |  +0.232%   |  +6.137%   |  +6.060%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.517% |  -0.016%   |  +2.134%   |  -0.888%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle|11.096% |  -4.252%   |  -0.302%   |  +28.107%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 0.980% |  -0.137%   |  -1.666%   |  +19.014%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 1.873% |  +0.171%   |  +8.364%   |  +4.992%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.116% |  +0.765%   |  +2.664%   |  +10.550%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.754% |  -0.536%   |  +2.083%   |  +9.264%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 2.793% |  -0.919%   |  -7.440%   |  +13.481%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.338% |  +0.293%   |  +4.300%   |  +10.293%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/65-flat-results-for-python-master-branch-2018-01-19

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From solipsis at pitrou.net  Sat Jan 20 04:09:12 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 20 Jan 2018 09:09:12 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-1
Message-ID: <20180120090912.1.C446A0CACA61E99A@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, -7] memory blocks, sum=-7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [1, 0, -2] memory blocks, sum=-1
test_multiprocessing_forkserver leaked [2, 0, 0] memory blocks, sum=2
test_multiprocessing_spawn leaked [2, 0, -1] memory blocks, sum=1


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogiKEu4K', '--timeout', '7200']

From webhook-mailer at python.org  Sat Jan 20 06:24:04 2018
From: webhook-mailer at python.org (Ivan Levkivskyi)
Date: Sat, 20 Jan 2018 11:24:04 -0000
Subject: [Python-checkins] bpo-32226: PEP 560: improve typing module (#4906)
Message-ID: <mailman.66.1516447447.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/d911e40e788fb679723d78b6ea11cabf46caed5a
commit: d911e40e788fb679723d78b6ea11cabf46caed5a
branch: master
author: Ivan Levkivskyi <levkivskyi at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-20T11:23:59Z
summary:

bpo-32226: PEP 560: improve typing module (#4906)

This PR re-designs the internal typing API using the new PEP 560 features.
However, there are only few minor changes in the public API.

files:
M Lib/dataclasses.py
M Lib/test/libregrtest/refleak.py
M Lib/test/test_pydoc.py
M Lib/test/test_typing.py
M Lib/typing.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index db14b0f66c2..7d30da1aacf 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -389,7 +389,8 @@ def _get_field(cls, a_name, a_type):
     if typing is not None:
         # This test uses a typing internal class, but it's the best
         #  way to test if this is a ClassVar.
-        if type(a_type) is typing._ClassVar:
+        if (type(a_type) is typing._GenericAlias and
+                a_type.__origin__ is typing.ClassVar):
             # This field is a ClassVar, so it's not a field.
             f._field_type = _FIELD_CLASSVAR
 
diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 18d5bd0511a..2ca9aa87644 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -135,11 +135,6 @@ def dash_R_cleanup(fs, ps, pic, zdc, abcs):
     # Clear ABC registries, restoring previously saved ABC registries.
     abs_classes = [getattr(collections.abc, a) for a in collections.abc.__all__]
     abs_classes = filter(isabstract, abs_classes)
-    if 'typing' in sys.modules:
-        t = sys.modules['typing']
-        # These classes require special treatment because they do not appear
-        # in direct subclasses of collections.abc classes
-        abs_classes = list(abs_classes) + [t.ChainMap, t.Counter, t.DefaultDict]
     for abc in abs_classes:
         for obj in abc.__subclasses__() + [abc]:
             obj._abc_registry = abcs.get(obj, WeakSet()).copy()
diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py
index 1926cffba26..0058dceed0d 100644
--- a/Lib/test/test_pydoc.py
+++ b/Lib/test/test_pydoc.py
@@ -827,7 +827,7 @@ class C(typing.Generic[T], typing.Mapping[int, str]): ...
                          'f\x08fo\x08oo\x08o(data: List[Any], x: int)'
                          ' -> Iterator[Tuple[int, Any]]')
         self.assertEqual(pydoc.render_doc(C).splitlines()[2],
-                         'class C\x08C(typing.Mapping)')
+                         'class C\x08C(collections.abc.Mapping, typing.Generic)')
 
     def test_builtin(self):
         for name in ('str', 'str.translate', 'builtins.str',
diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py
index 4ae1ebf05f3..3f24faf3762 100644
--- a/Lib/test/test_typing.py
+++ b/Lib/test/test_typing.py
@@ -12,7 +12,7 @@
 from typing import Union, Optional
 from typing import Tuple, List, MutableMapping
 from typing import Callable
-from typing import Generic, ClassVar, GenericMeta
+from typing import Generic, ClassVar
 from typing import cast
 from typing import get_type_hints
 from typing import no_type_check, no_type_check_decorator
@@ -24,20 +24,8 @@
 import abc
 import typing
 import weakref
-try:
-    import collections.abc as collections_abc
-except ImportError:
-    import collections as collections_abc  # Fallback for PY3.2.
-
 
-try:
-    import mod_generics_cache
-except ImportError:
-    # try to use the builtin one, Python 3.5+
-    from test import mod_generics_cache
-
-
-PY36 = sys.version_info[:2] >= (3, 6)
+from test import mod_generics_cache
 
 
 class BaseTestCase(TestCase):
@@ -606,7 +594,10 @@ def test_basics(self):
         Y[str]
         with self.assertRaises(TypeError):
             Y[str, str]
-        self.assertIsSubclass(SimpleMapping[str, int], SimpleMapping)
+        SM1 = SimpleMapping[str, int]
+        with self.assertRaises(TypeError):
+            issubclass(SM1, SimpleMapping)
+        self.assertIsInstance(SM1(), SimpleMapping)
 
     def test_generic_errors(self):
         T = TypeVar('T')
@@ -617,6 +608,8 @@ def test_generic_errors(self):
             Generic[T][T]
         with self.assertRaises(TypeError):
             Generic[T][S]
+        with self.assertRaises(TypeError):
+            class C(Generic[T], Generic[T]): ...
         with self.assertRaises(TypeError):
             isinstance([], List[int])
         with self.assertRaises(TypeError):
@@ -636,7 +629,6 @@ def test_init(self):
         with self.assertRaises(TypeError):
             Generic[T, S, T]
 
-    @skipUnless(PY36, "__init_subclass__ support required")
     def test_init_subclass(self):
         class X(typing.Generic[T]):
             def __init_subclass__(cls, **kwargs):
@@ -659,9 +651,9 @@ class W(X[int]):
 
     def test_repr(self):
         self.assertEqual(repr(SimpleMapping),
-                         __name__ + '.' + 'SimpleMapping')
+                         "<class 'test.test_typing.SimpleMapping'>")
         self.assertEqual(repr(MySimpleMapping),
-                         __name__ + '.' + 'MySimpleMapping')
+                         "<class 'test.test_typing.MySimpleMapping'>")
 
     def test_chain_repr(self):
         T = TypeVar('T')
@@ -713,7 +705,7 @@ def test_new_repr_complex(self):
     def test_new_repr_bare(self):
         T = TypeVar('T')
         self.assertEqual(repr(Generic[T]), 'typing.Generic[~T]')
-        self.assertEqual(repr(typing._Protocol[T]), 'typing.Protocol[~T]')
+        self.assertEqual(repr(typing._Protocol[T]), 'typing._Protocol[~T]')
         class C(typing.Dict[Any, Any]): ...
         # this line should just work
         repr(C.__mro__)
@@ -764,11 +756,13 @@ class D(C[T]):
 
     def test_abc_registry_kept(self):
         T = TypeVar('T')
-        class C(Generic[T]): ...
+        class C(collections.abc.Mapping, Generic[T]): ...
         C.register(int)
         self.assertIsInstance(1, C)
         C[int]
         self.assertIsInstance(1, C)
+        C._abc_registry.clear()
+        C._abc_cache.clear()  # To keep refleak hunting mode clean
 
     def test_false_subclasses(self):
         class MyMapping(MutableMapping[str, str]): pass
@@ -789,18 +783,17 @@ def __len__(self):
                 return 0
         # this should just work
         MM().update()
-        self.assertIsInstance(MM(), collections_abc.MutableMapping)
+        self.assertIsInstance(MM(), collections.abc.MutableMapping)
         self.assertIsInstance(MM(), MutableMapping)
         self.assertNotIsInstance(MM(), List)
         self.assertNotIsInstance({}, MM)
 
     def test_multiple_bases(self):
-        class MM1(MutableMapping[str, str], collections_abc.MutableMapping):
+        class MM1(MutableMapping[str, str], collections.abc.MutableMapping):
             pass
-        with self.assertRaises(TypeError):
-            # consistent MRO not possible
-            class MM2(collections_abc.MutableMapping, MutableMapping[str, str]):
-                pass
+        class MM2(collections.abc.MutableMapping, MutableMapping[str, str]):
+            pass
+        self.assertEqual(MM2.__bases__, (collections.abc.MutableMapping, Generic))
 
     def test_orig_bases(self):
         T = TypeVar('T')
@@ -855,16 +848,17 @@ class D(C, List[T][U][V]): ...
         self.assertEqual(D[int].__parameters__, ())
         self.assertEqual(C[int].__args__, (int,))
         self.assertEqual(D[int].__args__, (int,))
-        self.assertEqual(C.__bases__, (List,))
-        self.assertEqual(D.__bases__, (C, List))
+        self.assertEqual(C.__bases__, (list, Generic))
+        self.assertEqual(D.__bases__, (C, list, Generic))
         self.assertEqual(C.__orig_bases__, (List[T][U][V],))
         self.assertEqual(D.__orig_bases__, (C, List[T][U][V]))
 
     def test_subscript_meta(self):
         T = TypeVar('T')
-        self.assertEqual(Type[GenericMeta], Type[GenericMeta])
-        self.assertEqual(Union[T, int][GenericMeta], Union[GenericMeta, int])
-        self.assertEqual(Callable[..., GenericMeta].__args__, (Ellipsis, GenericMeta))
+        class Meta(type): ...
+        self.assertEqual(Type[Meta], Type[Meta])
+        self.assertEqual(Union[T, int][Meta], Union[Meta, int])
+        self.assertEqual(Callable[..., Meta].__args__, (Ellipsis, Meta))
 
     def test_generic_hashes(self):
         class A(Generic[T]):
@@ -939,7 +933,7 @@ def test_extended_generic_rules_repr(self):
         self.assertEqual(repr(Union[Tuple, Callable]).replace('typing.', ''),
                          'Union[Tuple, Callable]')
         self.assertEqual(repr(Union[Tuple, Tuple[int]]).replace('typing.', ''),
-                         'Tuple')
+                         'Union[Tuple, Tuple[int]]')
         self.assertEqual(repr(Callable[..., Optional[T]][int]).replace('typing.', ''),
                          'Callable[..., Union[int, NoneType]]')
         self.assertEqual(repr(Callable[[], List[T]][int]).replace('typing.', ''),
@@ -980,13 +974,15 @@ def __call__(self):
 
         self.assertEqual(repr(C1[int]).split('.')[-1], 'C1[int]')
         self.assertEqual(C2.__parameters__, ())
-        self.assertIsInstance(C2(), collections_abc.Callable)
-        self.assertIsSubclass(C2, collections_abc.Callable)
-        self.assertIsSubclass(C1, collections_abc.Callable)
+        self.assertIsInstance(C2(), collections.abc.Callable)
+        self.assertIsSubclass(C2, collections.abc.Callable)
+        self.assertIsSubclass(C1, collections.abc.Callable)
         self.assertIsInstance(T1(), tuple)
         self.assertIsSubclass(T2, tuple)
-        self.assertIsSubclass(Tuple[int, ...], typing.Sequence)
-        self.assertIsSubclass(Tuple[int, ...], typing.Iterable)
+        with self.assertRaises(TypeError):
+            issubclass(Tuple[int, ...], typing.Sequence)
+        with self.assertRaises(TypeError):
+            issubclass(Tuple[int, ...], typing.Iterable)
 
     def test_fail_with_bare_union(self):
         with self.assertRaises(TypeError):
@@ -1006,8 +1002,6 @@ def test_fail_with_bare_generic(self):
             Tuple[Generic[T]]
         with self.assertRaises(TypeError):
             List[typing._Protocol]
-        with self.assertRaises(TypeError):
-            isinstance(1, Generic)
 
     def test_type_erasure_special(self):
         T = TypeVar('T')
@@ -1044,21 +1038,6 @@ def test_all_repr_eq_any(self):
                     self.assertNotEqual(repr(base), '')
                     self.assertEqual(base, base)
 
-    def test_substitution_helper(self):
-        T = TypeVar('T')
-        KT = TypeVar('KT')
-        VT = TypeVar('VT')
-        class Map(Generic[KT, VT]):
-            def meth(self, k: KT, v: VT): ...
-        StrMap = Map[str, T]
-        obj = StrMap[int]()
-
-        new_args = typing._subs_tree(obj.__orig_class__)
-        new_annots = {k: typing._replace_arg(v, type(obj).__parameters__, new_args)
-                      for k, v in obj.meth.__annotations__.items()}
-
-        self.assertEqual(new_annots, {'k': str, 'v': int})
-
     def test_pickle(self):
         global C  # pickle wants to reference the class by name
         T = TypeVar('T')
@@ -1078,12 +1057,20 @@ class C(B[int]):
             self.assertEqual(x.foo, 42)
             self.assertEqual(x.bar, 'abc')
             self.assertEqual(x.__dict__, {'foo': 42, 'bar': 'abc'})
-        simples = [Any, Union, Tuple, Callable, ClassVar, List, typing.Iterable]
-        for s in simples:
+        samples = [Any, Union, Tuple, Callable, ClassVar]
+        for s in samples:
             for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                 z = pickle.dumps(s, proto)
                 x = pickle.loads(z)
                 self.assertEqual(s, x)
+        more_samples = [List, typing.Iterable, typing.Type]
+        for s in more_samples:
+            for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+                z = pickle.dumps(s, proto)
+                x = pickle.loads(z)
+                self.assertEqual(repr(s), repr(x))  # TODO: fix this
+                                                # see also comment in test_copy_and_deepcopy
+                                                # the issue is typing/#512
 
     def test_copy_and_deepcopy(self):
         T = TypeVar('T')
@@ -1095,14 +1082,7 @@ class Node(Generic[T]): ...
                   Union['T', int], List['T'], typing.Mapping['T', int]]
         for t in things + [Any]:
             self.assertEqual(t, copy(t))
-            self.assertEqual(t, deepcopy(t))
-            if sys.version_info >= (3, 3):
-                # From copy module documentation:
-                # It does "copy" functions and classes (shallow and deeply), by returning
-                # the original object unchanged; this is compatible with the way these
-                # are treated by the pickle module.
-                self.assertTrue(t is copy(t))
-                self.assertTrue(t is deepcopy(t))
+            self.assertEqual(repr(t), repr(deepcopy(t))) # Use repr() because of TypeVars
 
     def test_copy_generic_instances(self):
         T = TypeVar('T')
@@ -1143,7 +1123,6 @@ class C(Generic[T]):
 
         c = C()
         c_int = C[int]()
-        self.assertEqual(C.__slots__, C[str].__slots__)
 
         c.potato = 0
         c_int.potato = 0
@@ -1154,8 +1133,6 @@ class C(Generic[T]):
 
         def foo(x: C['C']): ...
         self.assertEqual(get_type_hints(foo, globals(), locals())['x'], C[C])
-        self.assertEqual(get_type_hints(foo, globals(), locals())['x'].__slots__,
-                         C.__slots__)
         self.assertEqual(copy(C[int]), deepcopy(C[int]))
 
     def test_parameterized_slots_dict(self):
@@ -1165,7 +1142,6 @@ class D(Generic[T]):
 
         d = D()
         d_int = D[int]()
-        self.assertEqual(D.__slots__, D[str].__slots__)
 
         d.banana = 'yes'
         d_int.banana = 'yes'
@@ -1182,30 +1158,22 @@ class C(Generic[B]):
                 pass
 
     def test_repr_2(self):
-        PY32 = sys.version_info[:2] < (3, 3)
-
         class C(Generic[T]):
             pass
 
         self.assertEqual(C.__module__, __name__)
-        if not PY32:
-            self.assertEqual(C.__qualname__,
-                             'GenericTests.test_repr_2.<locals>.C')
-        self.assertEqual(repr(C).split('.')[-1], 'C')
+        self.assertEqual(C.__qualname__,
+                         'GenericTests.test_repr_2.<locals>.C')
         X = C[int]
         self.assertEqual(X.__module__, __name__)
-        if not PY32:
-            self.assertTrue(X.__qualname__.endswith('.<locals>.C'))
         self.assertEqual(repr(X).split('.')[-1], 'C[int]')
 
         class Y(C[int]):
             pass
 
         self.assertEqual(Y.__module__, __name__)
-        if not PY32:
-            self.assertEqual(Y.__qualname__,
-                             'GenericTests.test_repr_2.<locals>.Y')
-        self.assertEqual(repr(Y).split('.')[-1], 'Y')
+        self.assertEqual(Y.__qualname__,
+                         'GenericTests.test_repr_2.<locals>.Y')
 
     def test_eq_1(self):
         self.assertEqual(Generic, Generic)
@@ -1238,6 +1206,12 @@ class C(A[T, VT], Generic[VT, T, KT], B[KT, T]):
 
         self.assertEqual(C.__parameters__, (VT, T, KT))
 
+    def test_multiple_inheritance_special(self):
+        S = TypeVar('S')
+        class B(Generic[S]): ...
+        class C(List[int], B): ...
+        self.assertEqual(C.__mro__, (C, list, B, Generic, object))
+
     def test_nested(self):
 
         G = Generic
@@ -1408,22 +1382,22 @@ def add_right(self, node: 'Node[T]' = None):
         self.assertEqual(right_hints['node'], Optional[Node[T]])
 
     def test_forwardref_instance_type_error(self):
-        fr = typing._ForwardRef('int')
+        fr = typing.ForwardRef('int')
         with self.assertRaises(TypeError):
             isinstance(42, fr)
 
     def test_forwardref_subclass_type_error(self):
-        fr = typing._ForwardRef('int')
+        fr = typing.ForwardRef('int')
         with self.assertRaises(TypeError):
             issubclass(int, fr)
 
     def test_forward_equality(self):
-        fr = typing._ForwardRef('int')
-        self.assertEqual(fr, typing._ForwardRef('int'))
+        fr = typing.ForwardRef('int')
+        self.assertEqual(fr, typing.ForwardRef('int'))
         self.assertNotEqual(List['int'], List[int])
 
     def test_forward_repr(self):
-        self.assertEqual(repr(List['int']), "typing.List[_ForwardRef('int')]")
+        self.assertEqual(repr(List['int']), "typing.List[ForwardRef('int')]")
 
     def test_union_forward(self):
 
@@ -1579,8 +1553,6 @@ def blah():
         blah()
 
 
-ASYNCIO = sys.version_info[:2] >= (3, 5)
-
 ASYNCIO_TESTS = """
 import asyncio
 
@@ -1618,17 +1590,15 @@ class ACM:
         return None
 """
 
-if ASYNCIO:
-    try:
-        exec(ASYNCIO_TESTS)
-    except ImportError:
-        ASYNCIO = False
+try:
+    exec(ASYNCIO_TESTS)
+except ImportError:
+    ASYNCIO = False  # multithreading is not enabled
 else:
-    # fake names for the sake of static analysis
-    asyncio = None
-    AwaitableWrapper = AsyncIteratorWrapper = ACM = object
+    ASYNCIO = True
+
+# Definitions needed for features introduced in Python 3.6
 
-PY36_TESTS = """
 from test import ann_module, ann_module2, ann_module3
 from typing import AsyncContextManager
 
@@ -1681,15 +1651,6 @@ class HasForeignBaseClass(mod_generics_cache.A):
     g_with(ACM()).send(None)
 except StopIteration as e:
     assert e.args[0] == 42
-"""
-
-if PY36:
-    exec(PY36_TESTS)
-else:
-    # fake names for the sake of static analysis
-    ann_module = ann_module2 = ann_module3 = None
-    A = B = CSub = G = CoolEmployee = CoolEmployeeWithDefault = object
-    XMeth = XRepr = NoneAndForward = object
 
 gth = get_type_hints
 
@@ -1704,14 +1665,12 @@ def test_get_type_hints_from_various_objects(self):
         with self.assertRaises(TypeError):
             gth(None)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_get_type_hints_modules(self):
         ann_module_type_hints = {1: 2, 'f': Tuple[int, int], 'x': int, 'y': str}
         self.assertEqual(gth(ann_module), ann_module_type_hints)
         self.assertEqual(gth(ann_module2), {})
         self.assertEqual(gth(ann_module3), {})
 
-    @skipUnless(PY36, 'Python 3.6 required')
     @expectedFailure
     def test_get_type_hints_modules_forwardref(self):
         # FIXME: This currently exposes a bug in typing. Cached forward references
@@ -1721,7 +1680,6 @@ def test_get_type_hints_modules_forwardref(self):
                      'default_b': Optional[mod_generics_cache.B]}
         self.assertEqual(gth(mod_generics_cache), mgc_hints)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_get_type_hints_classes(self):
         self.assertEqual(gth(ann_module.C),  # gth will find the right globalns
                          {'y': Optional[ann_module.C]})
@@ -1744,7 +1702,6 @@ def test_get_type_hints_classes(self):
                           'my_inner_a2': mod_generics_cache.B.A,
                           'my_outer_a': mod_generics_cache.A})
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_respect_no_type_check(self):
         @no_type_check
         class NoTpCheck:
@@ -1783,7 +1740,6 @@ class B: ...
         b.__annotations__ = {'x': 'A'}
         self.assertEqual(gth(b, locals()), {'x': A})
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_get_type_hints_ClassVar(self):
         self.assertEqual(gth(ann_module2.CV, ann_module2.__dict__),
                          {'var': typing.ClassVar[ann_module2.CV]})
@@ -2082,7 +2038,6 @@ def test_no_generator_instantiation(self):
         with self.assertRaises(TypeError):
             typing.Generator[int, int, int]()
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_async_generator(self):
         ns = {}
         exec("async def f():\n"
@@ -2090,7 +2045,6 @@ def test_async_generator(self):
         g = ns['f']()
         self.assertIsSubclass(type(g), typing.AsyncGenerator)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_no_async_generator_instantiation(self):
         with self.assertRaises(TypeError):
             typing.AsyncGenerator()
@@ -2147,13 +2101,14 @@ def __len__(self):
         self.assertIsSubclass(MMC, typing.Mapping)
 
         self.assertIsInstance(MMB[KT, VT](), typing.Mapping)
-        self.assertIsInstance(MMB[KT, VT](), collections_abc.Mapping)
+        self.assertIsInstance(MMB[KT, VT](), collections.abc.Mapping)
 
-        self.assertIsSubclass(MMA, collections_abc.Mapping)
-        self.assertIsSubclass(MMB, collections_abc.Mapping)
-        self.assertIsSubclass(MMC, collections_abc.Mapping)
+        self.assertIsSubclass(MMA, collections.abc.Mapping)
+        self.assertIsSubclass(MMB, collections.abc.Mapping)
+        self.assertIsSubclass(MMC, collections.abc.Mapping)
 
-        self.assertIsSubclass(MMB[str, str], typing.Mapping)
+        with self.assertRaises(TypeError):
+            issubclass(MMB[str, str], typing.Mapping)
         self.assertIsSubclass(MMC, MMA)
 
         class I(typing.Iterable): ...
@@ -2163,12 +2118,10 @@ class G(typing.Generator[int, int, int]): ...
         def g(): yield 0
         self.assertIsSubclass(G, typing.Generator)
         self.assertIsSubclass(G, typing.Iterable)
-        if hasattr(collections_abc, 'Generator'):
-            self.assertIsSubclass(G, collections_abc.Generator)
-        self.assertIsSubclass(G, collections_abc.Iterable)
+        self.assertIsSubclass(G, collections.abc.Generator)
+        self.assertIsSubclass(G, collections.abc.Iterable)
         self.assertNotIsSubclass(type(g), G)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_subclassing_async_generator(self):
         class G(typing.AsyncGenerator[int, int]):
             def asend(self, value):
@@ -2181,15 +2134,15 @@ def athrow(self, typ, val=None, tb=None):
         g = ns['g']
         self.assertIsSubclass(G, typing.AsyncGenerator)
         self.assertIsSubclass(G, typing.AsyncIterable)
-        self.assertIsSubclass(G, collections_abc.AsyncGenerator)
-        self.assertIsSubclass(G, collections_abc.AsyncIterable)
+        self.assertIsSubclass(G, collections.abc.AsyncGenerator)
+        self.assertIsSubclass(G, collections.abc.AsyncIterable)
         self.assertNotIsSubclass(type(g), G)
 
         instance = G()
         self.assertIsInstance(instance, typing.AsyncGenerator)
         self.assertIsInstance(instance, typing.AsyncIterable)
-        self.assertIsInstance(instance, collections_abc.AsyncGenerator)
-        self.assertIsInstance(instance, collections_abc.AsyncIterable)
+        self.assertIsInstance(instance, collections.abc.AsyncGenerator)
+        self.assertIsInstance(instance, collections.abc.AsyncIterable)
         self.assertNotIsInstance(type(g), G)
         self.assertNotIsInstance(g, G)
 
@@ -2226,23 +2179,23 @@ class D: ...
         self.assertIsSubclass(D, B)
 
         class M(): ...
-        collections_abc.MutableMapping.register(M)
+        collections.abc.MutableMapping.register(M)
         self.assertIsSubclass(M, typing.Mapping)
 
     def test_collections_as_base(self):
 
-        class M(collections_abc.Mapping): ...
+        class M(collections.abc.Mapping): ...
         self.assertIsSubclass(M, typing.Mapping)
         self.assertIsSubclass(M, typing.Iterable)
 
-        class S(collections_abc.MutableSequence): ...
+        class S(collections.abc.MutableSequence): ...
         self.assertIsSubclass(S, typing.MutableSequence)
         self.assertIsSubclass(S, typing.Iterable)
 
-        class I(collections_abc.Iterable): ...
+        class I(collections.abc.Iterable): ...
         self.assertIsSubclass(I, typing.Iterable)
 
-        class A(collections_abc.Mapping, metaclass=abc.ABCMeta): ...
+        class A(collections.abc.Mapping, metaclass=abc.ABCMeta): ...
         class B: ...
         A.register(B)
         self.assertIsSubclass(B, typing.Mapping)
@@ -2363,7 +2316,6 @@ def test_namedtuple_pyversion(self):
                 class NotYet(NamedTuple):
                     whatever = 0
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_annotation_usage(self):
         tim = CoolEmployee('Tim', 9000)
         self.assertIsInstance(tim, CoolEmployee)
@@ -2376,7 +2328,6 @@ def test_annotation_usage(self):
                          collections.OrderedDict(name=str, cool=int))
         self.assertIs(CoolEmployee._field_types, CoolEmployee.__annotations__)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_annotation_usage_with_default(self):
         jelle = CoolEmployeeWithDefault('Jelle')
         self.assertIsInstance(jelle, CoolEmployeeWithDefault)
@@ -2398,7 +2349,6 @@ class NonDefaultAfterDefault(NamedTuple):
     y: int
 """)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_annotation_usage_with_methods(self):
         self.assertEqual(XMeth(1).double(), 2)
         self.assertEqual(XMeth(42).x, XMeth(42)[0])
@@ -2421,7 +2371,6 @@ def _source(self):
         return 'no chance for this as well'
 """)
 
-    @skipUnless(PY36, 'Python 3.6 required')
     def test_namedtuple_keyword_usage(self):
         LocalEmployee = NamedTuple("LocalEmployee", name=str, age=int)
         nick = LocalEmployee('Nick', 25)
@@ -2506,15 +2455,8 @@ def test_alias_equality(self):
         self.assertNotEqual(Pattern[str], str)
 
     def test_errors(self):
-        with self.assertRaises(TypeError):
-            # Doesn't fit AnyStr.
-            Pattern[int]
-        with self.assertRaises(TypeError):
-            # Can't change type vars?
-            Match[T]
         m = Match[Union[str, bytes]]
         with self.assertRaises(TypeError):
-            # Too complicated?
             m[str]
         with self.assertRaises(TypeError):
             # We don't support isinstance().
@@ -2524,12 +2466,12 @@ def test_errors(self):
             issubclass(Pattern[bytes], Pattern[str])
 
     def test_repr(self):
-        self.assertEqual(repr(Pattern), 'Pattern[~AnyStr]')
-        self.assertEqual(repr(Pattern[str]), 'Pattern[str]')
-        self.assertEqual(repr(Pattern[bytes]), 'Pattern[bytes]')
-        self.assertEqual(repr(Match), 'Match[~AnyStr]')
-        self.assertEqual(repr(Match[str]), 'Match[str]')
-        self.assertEqual(repr(Match[bytes]), 'Match[bytes]')
+        self.assertEqual(repr(Pattern), 'typing.Pattern')
+        self.assertEqual(repr(Pattern[str]), 'typing.Pattern[str]')
+        self.assertEqual(repr(Pattern[bytes]), 'typing.Pattern[bytes]')
+        self.assertEqual(repr(Match), 'typing.Match')
+        self.assertEqual(repr(Match[str]), 'typing.Match[str]')
+        self.assertEqual(repr(Match[bytes]), 'typing.Match[bytes]')
 
     def test_re_submodule(self):
         from typing.re import Match, Pattern, __all__, __name__
@@ -2545,7 +2487,7 @@ class A(typing.Match):
                 pass
 
         self.assertEqual(str(ex.exception),
-                         "Cannot subclass typing._TypeAlias")
+                         "type 're.Match' is not an acceptable base type")
 
 
 class AllTests(BaseTestCase):
diff --git a/Lib/typing.py b/Lib/typing.py
index b5564cc29a2..7ca080402e2 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1,24 +1,33 @@
+"""
+The typing module: Support for gradual typing as defined by PEP 484.
+
+At large scale, the structure of the module is following:
+* Imports and exports, all public names should be explicitelly added to __all__.
+* Internal helper functions: these should never be used in code outside this module.
+* _SpecialForm and its instances (special forms): Any, NoReturn, ClassVar, Union, Optional
+* Two classes whose instances can be type arguments in addition to types: ForwardRef and TypeVar
+* The core of internal generics API: _GenericAlias and _VariadicGenericAlias, the latter is
+  currently only used by Tuple and Callable. All subscripted types like X[int], Union[int, str],
+  etc., are instances of either of these classes.
+* The public counterpart of the generics API consists of two classes: Generic and Protocol
+  (the latter is currently private, but will be made public after PEP 544 acceptance).
+* Public helper functions: get_type_hints, overload, cast, no_type_check,
+  no_type_check_decorator.
+* Generic aliases for collections.abc ABCs and few additional protocols.
+* Special types: NewType, NamedTuple, TypedDict (may be added soon).
+* Wrapper submodules for re and io related types.
+"""
+
 import abc
 from abc import abstractmethod, abstractproperty
 import collections
+import collections.abc
 import contextlib
 import functools
 import re as stdlib_re  # Avoid confusion with the re we export.
 import sys
 import types
-try:
-    import collections.abc as collections_abc
-except ImportError:
-    import collections as collections_abc  # Fallback for PY3.2.
-if sys.version_info[:2] >= (3, 6):
-    import _collections_abc  # Needed for private function _check_methods # noqa
-try:
-    from types import WrapperDescriptorType, MethodWrapperType, MethodDescriptorType
-except ImportError:
-    WrapperDescriptorType = type(object.__init__)
-    MethodWrapperType = type(object().__str__)
-    MethodDescriptorType = type(str.join)
-
+from types import WrapperDescriptorType, MethodWrapperType, MethodDescriptorType
 
 # Please keep __all__ alphabetized within each category.
 __all__ = [
@@ -35,8 +44,6 @@
 
     # ABCs (from collections.abc).
     'AbstractSet',  # collections.abc.Set.
-    'GenericMeta',  # subclass of abc.ABCMeta and a metaclass
-                    # for 'Generic' and ABCs below.
     'ByteString',
     'Container',
     'ContextManager',
@@ -53,15 +60,13 @@
     'Sequence',
     'Sized',
     'ValuesView',
-    # The following are added depending on presence
-    # of their non-generic counterparts in stdlib:
-    # Awaitable,
-    # AsyncIterator,
-    # AsyncIterable,
-    # Coroutine,
-    # Collection,
-    # AsyncGenerator,
-    # AsyncContextManager
+    'Awaitable',
+    'AsyncIterator',
+    'AsyncIterable',
+    'Coroutine',
+    'Collection',
+    'AsyncGenerator',
+    'AsyncContextManager',
 
     # Structural checks, a.k.a. protocols.
     'Reversible',
@@ -100,70 +105,199 @@
 # legitimate imports of those modules.
 
 
-def _qualname(x):
-    if sys.version_info[:2] >= (3, 3):
-        return x.__qualname__
-    else:
-        # Fall back to just name.
-        return x.__name__
+def _type_check(arg, msg):
+    """Check that the argument is a type, and return it (internal helper).
 
+    As a special case, accept None and return type(None) instead. Also wrap strings
+    into ForwardRef instances. Consider several corner cases, for example plain
+    special forms like Union are not valid, while Union[int, str] is OK, etc.
+    The msg argument is a human-readable error message, e.g::
 
-def _trim_name(nm):
-    whitelist = ('_TypeAlias', '_ForwardRef', '_TypingBase', '_FinalTypingBase')
-    if nm.startswith('_') and nm not in whitelist:
-        nm = nm[1:]
-    return nm
+        "Union[arg, ...]: arg should be a type."
 
+    We append the repr() of the actual value (truncated to 100 chars).
+    """
+    if arg is None:
+        return type(None)
+    if isinstance(arg, str):
+        return ForwardRef(arg)
+    if (isinstance(arg, _GenericAlias) and
+            arg.__origin__ in (Generic, _Protocol, ClassVar)):
+        raise TypeError(f"{arg} is not valid as type argument")
+    if (isinstance(arg, _SpecialForm) and arg is not Any or
+            arg in (Generic, _Protocol)):
+        raise TypeError(f"Plain {arg} is not valid as type argument")
+    if isinstance(arg, (type, TypeVar, ForwardRef)):
+        return arg
+    if not callable(arg):
+        raise TypeError(f"{msg} Got {arg!r:.100}.")
+    return arg
 
-class TypingMeta(type):
-    """Metaclass for most types defined in typing module
-    (not a part of public API).
 
-    This overrides __new__() to require an extra keyword parameter
-    '_root', which serves as a guard against naive subclassing of the
-    typing classes.  Any legitimate class defined using a metaclass
-    derived from TypingMeta must pass _root=True.
+def _type_repr(obj):
+    """Return the repr() of an object, special-casing types (internal helper).
 
-    This also defines a dummy constructor (all the work for most typing
-    constructs is done in __new__) and a nicer repr().
+    If obj is a type, we return a shorter version than the default
+    type.__repr__, based on the module and qualified name, which is
+    typically enough to uniquely identify a type.  For everything
+    else, we fall back on repr(obj).
     """
+    if isinstance(obj, type):
+        if obj.__module__ == 'builtins':
+            return obj.__qualname__
+        return f'{obj.__module__}.{obj.__qualname__}'
+    if obj is ...:
+        return('...')
+    if isinstance(obj, types.FunctionType):
+        return obj.__name__
+    return repr(obj)
 
-    _is_protocol = False
 
-    def __new__(cls, name, bases, namespace, *, _root=False):
-        if not _root:
-            raise TypeError("Cannot subclass %s" %
-                            (', '.join(map(_type_repr, bases)) or '()'))
-        return super().__new__(cls, name, bases, namespace)
+def _collect_type_vars(types):
+    """Collect all type variable contained in types in order of
+    first appearance (lexicographic order). For example::
 
-    def __init__(self, *args, **kwds):
-        pass
+        _collect_type_vars((T, List[S, T])) == (T, S)
+    """
+    tvars = []
+    for t in types:
+        if isinstance(t, TypeVar) and t not in tvars:
+            tvars.append(t)
+        if isinstance(t, _GenericAlias) and not t._special:
+            tvars.extend([t for t in t.__parameters__ if t not in tvars])
+    return tuple(tvars)
 
-    def _eval_type(self, globalns, localns):
-        """Override this in subclasses to interpret forward references.
 
-        For example, List['C'] is internally stored as
-        List[_ForwardRef('C')], which should evaluate to List[C],
-        where C is an object found in globalns or localns (searching
-        localns first, of course).
-        """
-        return self
+def _subs_tvars(tp, tvars, subs):
+    """Substitute type variables 'tvars' with substitutions 'subs'.
+    These two must have the same length.
+    """
+    if not isinstance(tp, _GenericAlias):
+        return tp
+    new_args = list(tp.__args__)
+    for a, arg in enumerate(tp.__args__):
+        if isinstance(arg, TypeVar):
+            for i, tvar in enumerate(tvars):
+                if arg == tvar:
+                    new_args[a] = subs[i]
+        else:
+            new_args[a] = _subs_tvars(arg, tvars, subs)
+    if tp.__origin__ is Union:
+        return Union[tuple(new_args)]
+    return tp.copy_with(tuple(new_args))
 
-    def _get_type_vars(self, tvars):
-        pass
 
-    def __repr__(self):
-        qname = _trim_name(_qualname(self))
-        return '%s.%s' % (self.__module__, qname)
+def _check_generic(cls, parameters):
+    """Check correct count for parameters of a generic cls (internal helper).
+    This gives a nice error message in case of count mismatch.
+    """
+    if not cls.__parameters__:
+        raise TypeError(f"{cls} is not a generic class")
+    alen = len(parameters)
+    elen = len(cls.__parameters__)
+    if alen != elen:
+        raise TypeError(f"Too {'many' if alen > elen else 'few'} parameters for {cls};"
+                        f" actual {alen}, expected {elen}")
+
+
+def _remove_dups_flatten(parameters):
+    """An internal helper for Union creation and substitution: flatten Union's
+    among parameters, then remove duplicates and strict subclasses.
+    """
+    # Flatten out Union[Union[...], ...].
+    params = []
+    for p in parameters:
+        if isinstance(p, _GenericAlias) and p.__origin__ is Union:
+            params.extend(p.__args__)
+        elif isinstance(p, tuple) and len(p) > 0 and p[0] is Union:
+            params.extend(p[1:])
+        else:
+            params.append(p)
+    # Weed out strict duplicates, preserving the first of each occurrence.
+    all_params = set(params)
+    if len(all_params) < len(params):
+        new_params = []
+        for t in params:
+            if t in all_params:
+                new_params.append(t)
+                all_params.remove(t)
+        params = new_params
+        assert not all_params, all_params
+    # Weed out subclasses.
+    # E.g. Union[int, Employee, Manager] == Union[int, Employee].
+    # If object is present it will be sole survivor among proper classes.
+    # Never discard type variables.
+    # (In particular, Union[str, AnyStr] != AnyStr.)
+    all_params = set(params)
+    for t1 in params:
+        if not isinstance(t1, type):
+            continue
+        if any((isinstance(t2, type) or
+                isinstance(t2, _GenericAlias) and t2._special) and issubclass(t1, t2)
+               for t2 in all_params - {t1}):
+            all_params.remove(t1)
+    return tuple(t for t in params if t in all_params)
+
+
+_cleanups = []
+
+
+def _tp_cache(func):
+    """Internal wrapper caching __getitem__ of generic types with a fallback to
+    original function for non-hashable arguments.
+    """
+    cached = functools.lru_cache()(func)
+    _cleanups.append(cached.cache_clear)
+
+    @functools.wraps(func)
+    def inner(*args, **kwds):
+        try:
+            return cached(*args, **kwds)
+        except TypeError:
+            pass  # All real errors (not unhashable args) are raised below.
+        return func(*args, **kwds)
+    return inner
+
+
+def _eval_type(t, globalns, localns):
+    """Evaluate all forward reverences in the given type t.
+    For use of globalns and localns see the docstring for get_type_hints().
+    """
+    if isinstance(t, ForwardRef):
+        return t._evaluate(globalns, localns)
+    if isinstance(t, _GenericAlias):
+        ev_args = tuple(_eval_type(a, globalns, localns) for a in t.__args__)
+        if ev_args == t.__args__:
+            return t
+        res = t.copy_with(ev_args)
+        res._special = t._special
+        return res
+    return t
 
 
-class _TypingBase(metaclass=TypingMeta, _root=True):
-    """Internal indicator of special typing constructs."""
+class _Final:
+    """Mixin to prohibit subclassing"""
 
     __slots__ = ('__weakref__',)
 
-    def __init__(self, *args, **kwds):
-        pass
+    def __init_subclass__(self, *args, **kwds):
+        if '_root' not in kwds:
+            raise TypeError("Cannot subclass special typing classes")
+
+
+class _SpecialForm(_Final, _root=True):
+    """Internal indicator of special typing constructs.
+    See _doc instance attribute for specific docs.
+    """
+
+    __slots__ = ('_name', '_doc')
+
+    def __getstate__(self):
+        return {'name': self._name, 'doc': self._doc}
+
+    def __setstate__(self, state):
+        self._name = state['name']
+        self._doc = state['doc']
 
     def __new__(cls, *args, **kwds):
         """Constructor.
@@ -175,65 +309,166 @@ def __new__(cls, *args, **kwds):
                 isinstance(args[0], str) and
                 isinstance(args[1], tuple)):
             # Close enough.
-            raise TypeError("Cannot subclass %r" % cls)
+            raise TypeError(f"Cannot subclass {cls!r}")
         return super().__new__(cls)
 
-    # Things that are not classes also need these.
-    def _eval_type(self, globalns, localns):
-        return self
+    def __init__(self, name, doc):
+        self._name = name
+        self._doc = doc
 
-    def _get_type_vars(self, tvars):
-        pass
+    def __eq__(self, other):
+        if not isinstance(other, _SpecialForm):
+            return NotImplemented
+        return self._name == other._name
+
+    def __hash__(self):
+        return hash((self._name,))
 
     def __repr__(self):
-        cls = type(self)
-        qname = _trim_name(_qualname(cls))
-        return '%s.%s' % (cls.__module__, qname)
+        return 'typing.' + self._name
+
+    def __copy__(self):
+        return self  # Special forms are immutable.
 
     def __call__(self, *args, **kwds):
-        raise TypeError("Cannot instantiate %r" % type(self))
+        raise TypeError(f"Cannot instantiate {self!r}")
 
+    def __instancecheck__(self, obj):
+        raise TypeError(f"{self} cannot be used with isinstance()")
 
-class _FinalTypingBase(_TypingBase, _root=True):
-    """Internal mix-in class to prevent instantiation.
+    def __subclasscheck__(self, cls):
+        raise TypeError(f"{self} cannot be used with issubclass()")
 
-    Prevents instantiation unless _root=True is given in class call.
-    It is used to create pseudo-singleton instances Any, Union, Optional, etc.
-    """
+    @_tp_cache
+    def __getitem__(self, parameters):
+        if self._name == 'ClassVar':
+            item = _type_check(parameters, 'ClassVar accepts only single type.')
+            return _GenericAlias(self, (item,))
+        if self._name == 'Union':
+            if parameters == ():
+                raise TypeError("Cannot take a Union of no types.")
+            if not isinstance(parameters, tuple):
+                parameters = (parameters,)
+            msg = "Union[arg, ...]: each arg must be a type."
+            parameters = tuple(_type_check(p, msg) for p in parameters)
+            parameters = _remove_dups_flatten(parameters)
+            if len(parameters) == 1:
+                return parameters[0]
+            return _GenericAlias(self, parameters)
+        if self._name == 'Optional':
+            arg = _type_check(parameters, "Optional[t] requires a single type.")
+            return Union[arg, type(None)]
+        raise TypeError(f"{self} is not subscriptable")
 
-    __slots__ = ()
 
-    def __new__(cls, *args, _root=False, **kwds):
-        self = super().__new__(cls, *args, **kwds)
-        if _root is True:
-            return self
-        raise TypeError("Cannot instantiate %r" % cls)
+Any = _SpecialForm('Any', doc=
+    """Special type indicating an unconstrained type.
+
+    - Any is compatible with every type.
+    - Any assumed to have all methods.
+    - All values assumed to be instances of Any.
+
+    Note that all the above statements are true from the point of view of
+    static type checkers. At runtime, Any should not be used with instance
+    or class checks.
+    """)
+
+NoReturn = _SpecialForm('NoReturn', doc=
+    """Special type indicating functions that never return.
+    Example::
+
+      from typing import NoReturn
+
+      def stop() -> NoReturn:
+          raise Exception('no way')
+
+    This type is invalid in other positions, e.g., ``List[NoReturn]``
+    will fail in static type checkers.
+    """)
+
+ClassVar = _SpecialForm('ClassVar', doc=
+    """Special type construct to mark class variables.
+
+    An annotation wrapped in ClassVar indicates that a given
+    attribute is intended to be used as a class variable and
+    should not be set on instances of that class. Usage::
+
+      class Starship:
+          stats: ClassVar[Dict[str, int]] = {} # class variable
+          damage: int = 10                     # instance variable
+
+    ClassVar accepts only types and cannot be further subscribed.
+
+    Note that ClassVar is not a class itself, and should not
+    be used with isinstance() or issubclass().
+    """)
+
+Union = _SpecialForm('Union', doc=
+    """Union type; Union[X, Y] means either X or Y.
+
+    To define a union, use e.g. Union[int, str].  Details:
+    - The arguments must be types and there must be at least one.
+    - None as an argument is a special case and is replaced by
+      type(None).
+    - Unions of unions are flattened, e.g.::
+
+        Union[Union[int, str], float] == Union[int, str, float]
+
+    - Unions of a single argument vanish, e.g.::
+
+        Union[int] == int  # The constructor actually returns int
+
+    - Redundant arguments are skipped, e.g.::
+
+        Union[int, str, int] == Union[int, str]
+
+    - When comparing unions, the argument order is ignored, e.g.::
+
+        Union[int, str] == Union[str, int]
+
+    - When two arguments have a subclass relationship, the least
+      derived argument is kept, e.g.::
+
+        class Employee: pass
+        class Manager(Employee): pass
+        Union[int, Employee, Manager] == Union[int, Employee]
+        Union[Manager, int, Employee] == Union[int, Employee]
+        Union[Employee, Manager] == Employee
+
+    - Similar for object::
+
+        Union[int, object] == object
+
+    - You cannot subclass or instantiate a union.
+    - You can use Optional[X] as a shorthand for Union[X, None].
+    """)
 
-    def __reduce__(self):
-        return _trim_name(type(self).__name__)
+Optional = _SpecialForm('Optional', doc=
+    """Optional type.
+
+    Optional[X] is equivalent to Union[X, None].
+    """)
 
 
-class _ForwardRef(_TypingBase, _root=True):
+class ForwardRef(_Final, _root=True):
     """Internal wrapper to hold a forward reference."""
 
     __slots__ = ('__forward_arg__', '__forward_code__',
                  '__forward_evaluated__', '__forward_value__')
 
     def __init__(self, arg):
-        super().__init__(arg)
         if not isinstance(arg, str):
-            raise TypeError('Forward reference must be a string -- got %r' % (arg,))
+            raise TypeError(f"Forward reference must be a string -- got {arg!r}")
         try:
             code = compile(arg, '<string>', 'eval')
         except SyntaxError:
-            raise SyntaxError('Forward reference must be an expression -- got %r' %
-                              (arg,))
+            raise SyntaxError(f"Forward reference must be an expression -- got {arg!r}")
         self.__forward_arg__ = arg
         self.__forward_code__ = code
         self.__forward_evaluated__ = False
         self.__forward_value__ = None
 
-    def _eval_type(self, globalns, localns):
+    def _evaluate(self, globalns, localns):
         if not self.__forward_evaluated__ or localns is not globalns:
             if globalns is None and localns is None:
                 globalns = localns = {}
@@ -248,7 +483,7 @@ def _eval_type(self, globalns, localns):
         return self.__forward_value__
 
     def __eq__(self, other):
-        if not isinstance(other, _ForwardRef):
+        if not isinstance(other, ForwardRef):
             return NotImplemented
         return (self.__forward_arg__ == other.__forward_arg__ and
                 self.__forward_value__ == other.__forward_value__)
@@ -256,249 +491,57 @@ def __eq__(self, other):
     def __hash__(self):
         return hash((self.__forward_arg__, self.__forward_value__))
 
-    def __instancecheck__(self, obj):
-        raise TypeError("Forward references cannot be used with isinstance().")
-
-    def __subclasscheck__(self, cls):
-        raise TypeError("Forward references cannot be used with issubclass().")
-
     def __repr__(self):
-        return '_ForwardRef(%r)' % (self.__forward_arg__,)
-
-
-class _TypeAlias(_TypingBase, _root=True):
-    """Internal helper class for defining generic variants of concrete types.
-
-    Note that this is not a type; let's call it a pseudo-type.  It cannot
-    be used in instance and subclass checks in parameterized form, i.e.
-    ``isinstance(42, Match[str])`` raises ``TypeError`` instead of returning
-    ``False``.
-    """
-
-    __slots__ = ('name', 'type_var', 'impl_type', 'type_checker')
+        return f'ForwardRef({self.__forward_arg__!r})'
 
-    def __init__(self, name, type_var, impl_type, type_checker):
-        """Initializer.
 
-        Args:
-            name: The name, e.g. 'Pattern'.
-            type_var: The type parameter, e.g. AnyStr, or the
-                specific type, e.g. str.
-            impl_type: The implementation type.
-            type_checker: Function that takes an impl_type instance.
-                and returns a value that should be a type_var instance.
-        """
-        assert isinstance(name, str), repr(name)
-        assert isinstance(impl_type, type), repr(impl_type)
-        assert not isinstance(impl_type, TypingMeta), repr(impl_type)
-        assert isinstance(type_var, (type, _TypingBase)), repr(type_var)
-        self.name = name
-        self.type_var = type_var
-        self.impl_type = impl_type
-        self.type_checker = type_checker
-
-    def __repr__(self):
-        return "%s[%s]" % (self.name, _type_repr(self.type_var))
-
-    def __getitem__(self, parameter):
-        if not isinstance(self.type_var, TypeVar):
-            raise TypeError("%s cannot be further parameterized." % self)
-        if self.type_var.__constraints__ and isinstance(parameter, type):
-            if not issubclass(parameter, self.type_var.__constraints__):
-                raise TypeError("%s is not a valid substitution for %s." %
-                                (parameter, self.type_var))
-        if isinstance(parameter, TypeVar) and parameter is not self.type_var:
-            raise TypeError("%s cannot be re-parameterized." % self)
-        return self.__class__(self.name, parameter,
-                              self.impl_type, self.type_checker)
-
-    def __eq__(self, other):
-        if not isinstance(other, _TypeAlias):
-            return NotImplemented
-        return self.name == other.name and self.type_var == other.type_var
+class TypeVar(_Final, _root=True):
+    """Type variable.
 
-    def __hash__(self):
-        return hash((self.name, self.type_var))
+    Usage::
 
-    def __instancecheck__(self, obj):
-        if not isinstance(self.type_var, TypeVar):
-            raise TypeError("Parameterized type aliases cannot be used "
-                            "with isinstance().")
-        return isinstance(obj, self.impl_type)
+      T = TypeVar('T')  # Can be anything
+      A = TypeVar('A', str, bytes)  # Must be str or bytes
 
-    def __subclasscheck__(self, cls):
-        if not isinstance(self.type_var, TypeVar):
-            raise TypeError("Parameterized type aliases cannot be used "
-                            "with issubclass().")
-        return issubclass(cls, self.impl_type)
+    Type variables exist primarily for the benefit of static type
+    checkers.  They serve as the parameters for generic types as well
+    as for generic function definitions.  See class Generic for more
+    information on generic types.  Generic functions work as follows:
 
+      def repeat(x: T, n: int) -> List[T]:
+          '''Return a list containing n references to x.'''
+          return [x]*n
 
-def _get_type_vars(types, tvars):
-    for t in types:
-        if isinstance(t, TypingMeta) or isinstance(t, _TypingBase):
-            t._get_type_vars(tvars)
+      def longest(x: A, y: A) -> A:
+          '''Return the longest of two strings.'''
+          return x if len(x) >= len(y) else y
 
+    The latter example's signature is essentially the overloading
+    of (str, str) -> str and (bytes, bytes) -> bytes.  Also note
+    that if the arguments are instances of some subclass of str,
+    the return type is still plain str.
 
-def _type_vars(types):
-    tvars = []
-    _get_type_vars(types, tvars)
-    return tuple(tvars)
+    At runtime, isinstance(x, T) and issubclass(C, T) will raise TypeError.
 
+    Type variables defined with covariant=True or contravariant=True
+    can be used do declare covariant or contravariant generic types.
+    See PEP 484 for more details. By default generic types are invariant
+    in all type variables.
 
-def _eval_type(t, globalns, localns):
-    if isinstance(t, TypingMeta) or isinstance(t, _TypingBase):
-        return t._eval_type(globalns, localns)
-    return t
+    Type variables can be introspected. e.g.:
 
+      T.__name__ == 'T'
+      T.__constraints__ == ()
+      T.__covariant__ == False
+      T.__contravariant__ = False
+      A.__constraints__ == (str, bytes)
+    """
 
-def _type_check(arg, msg):
-    """Check that the argument is a type, and return it (internal helper).
-
-    As a special case, accept None and return type(None) instead.
-    Also, _TypeAlias instances (e.g. Match, Pattern) are acceptable.
-
-    The msg argument is a human-readable error message, e.g.
-
-        "Union[arg, ...]: arg should be a type."
-
-    We append the repr() of the actual value (truncated to 100 chars).
-    """
-    if arg is None:
-        return type(None)
-    if isinstance(arg, str):
-        arg = _ForwardRef(arg)
-    if (
-        isinstance(arg, _TypingBase) and type(arg).__name__ == '_ClassVar' or
-        not isinstance(arg, (type, _TypingBase)) and not callable(arg)
-    ):
-        raise TypeError(msg + " Got %.100r." % (arg,))
-    # Bare Union etc. are not valid as type arguments
-    if (
-        type(arg).__name__ in ('_Union', '_Optional') and
-        not getattr(arg, '__origin__', None) or
-        isinstance(arg, TypingMeta) and arg._gorg in (Generic, _Protocol)
-    ):
-        raise TypeError("Plain %s is not valid as type argument" % arg)
-    return arg
-
-
-def _type_repr(obj):
-    """Return the repr() of an object, special-casing types (internal helper).
-
-    If obj is a type, we return a shorter version than the default
-    type.__repr__, based on the module and qualified name, which is
-    typically enough to uniquely identify a type.  For everything
-    else, we fall back on repr(obj).
-    """
-    if isinstance(obj, type) and not isinstance(obj, TypingMeta):
-        if obj.__module__ == 'builtins':
-            return _qualname(obj)
-        return '%s.%s' % (obj.__module__, _qualname(obj))
-    if obj is ...:
-        return('...')
-    if isinstance(obj, types.FunctionType):
-        return obj.__name__
-    return repr(obj)
-
-
-class _Any(_FinalTypingBase, _root=True):
-    """Special type indicating an unconstrained type.
-
-    - Any is compatible with every type.
-    - Any assumed to have all methods.
-    - All values assumed to be instances of Any.
-
-    Note that all the above statements are true from the point of view of
-    static type checkers. At runtime, Any should not be used with instance
-    or class checks.
-    """
-
-    __slots__ = ()
-
-    def __instancecheck__(self, obj):
-        raise TypeError("Any cannot be used with isinstance().")
-
-    def __subclasscheck__(self, cls):
-        raise TypeError("Any cannot be used with issubclass().")
-
-
-Any = _Any(_root=True)
-
-
-class _NoReturn(_FinalTypingBase, _root=True):
-    """Special type indicating functions that never return.
-    Example::
-
-      from typing import NoReturn
-
-      def stop() -> NoReturn:
-          raise Exception('no way')
-
-    This type is invalid in other positions, e.g., ``List[NoReturn]``
-    will fail in static type checkers.
-    """
-
-    __slots__ = ()
-
-    def __instancecheck__(self, obj):
-        raise TypeError("NoReturn cannot be used with isinstance().")
-
-    def __subclasscheck__(self, cls):
-        raise TypeError("NoReturn cannot be used with issubclass().")
-
-
-NoReturn = _NoReturn(_root=True)
-
-
-class TypeVar(_TypingBase, _root=True):
-    """Type variable.
-
-    Usage::
-
-      T = TypeVar('T')  # Can be anything
-      A = TypeVar('A', str, bytes)  # Must be str or bytes
-
-    Type variables exist primarily for the benefit of static type
-    checkers.  They serve as the parameters for generic types as well
-    as for generic function definitions.  See class Generic for more
-    information on generic types.  Generic functions work as follows:
-
-      def repeat(x: T, n: int) -> List[T]:
-          '''Return a list containing n references to x.'''
-          return [x]*n
-
-      def longest(x: A, y: A) -> A:
-          '''Return the longest of two strings.'''
-          return x if len(x) >= len(y) else y
-
-    The latter example's signature is essentially the overloading
-    of (str, str) -> str and (bytes, bytes) -> bytes.  Also note
-    that if the arguments are instances of some subclass of str,
-    the return type is still plain str.
-
-    At runtime, isinstance(x, T) and issubclass(C, T) will raise TypeError.
-
-    Type variables defined with covariant=True or contravariant=True
-    can be used do declare covariant or contravariant generic types.
-    See PEP 484 for more details. By default generic types are invariant
-    in all type variables.
-
-    Type variables can be introspected. e.g.:
-
-      T.__name__ == 'T'
-      T.__constraints__ == ()
-      T.__covariant__ == False
-      T.__contravariant__ = False
-      A.__constraints__ == (str, bytes)
-    """
-
-    __slots__ = ('__name__', '__bound__', '__constraints__',
-                 '__covariant__', '__contravariant__')
+    __slots__ = ('__name__', '__bound__', '__constraints__',
+                 '__covariant__', '__contravariant__')
 
     def __init__(self, name, *constraints, bound=None,
                  covariant=False, contravariant=False):
-        super().__init__(name, *constraints, bound=bound,
-                         covariant=covariant, contravariant=contravariant)
         self.__name__ = name
         if covariant and contravariant:
             raise ValueError("Bivariant types are not supported.")
@@ -515,9 +558,19 @@ def __init__(self, name, *constraints, bound=None,
         else:
             self.__bound__ = None
 
-    def _get_type_vars(self, tvars):
-        if self not in tvars:
-            tvars.append(self)
+    def __getstate__(self):
+        return {'name': self.__name__,
+                'bound': self.__bound__,
+                'constraints': self.__constraints__,
+                'co': self.__covariant__,
+                'contra': self.__contravariant__}
+
+    def __setstate__(self, state):
+        self.__name__ = state['name']
+        self.__bound__ = state['bound']
+        self.__constraints__ = state['constraints']
+        self.__covariant__ = state['co']
+        self.__contravariant__ = state['contra']
 
     def __repr__(self):
         if self.__covariant__:
@@ -528,44 +581,6 @@ def __repr__(self):
             prefix = '~'
         return prefix + self.__name__
 
-    def __instancecheck__(self, instance):
-        raise TypeError("Type variables cannot be used with isinstance().")
-
-    def __subclasscheck__(self, cls):
-        raise TypeError("Type variables cannot be used with issubclass().")
-
-
-# Some unconstrained type variables.  These are used by the container types.
-# (These are not for export.)
-T = TypeVar('T')  # Any type.
-KT = TypeVar('KT')  # Key type.
-VT = TypeVar('VT')  # Value type.
-T_co = TypeVar('T_co', covariant=True)  # Any type covariant containers.
-V_co = TypeVar('V_co', covariant=True)  # Any type covariant containers.
-VT_co = TypeVar('VT_co', covariant=True)  # Value type covariant containers.
-T_contra = TypeVar('T_contra', contravariant=True)  # Ditto contravariant.
-
-# A useful type variable with constraints.  This represents string types.
-# (This one *is* for export!)
-AnyStr = TypeVar('AnyStr', bytes, str)
-
-
-def _replace_arg(arg, tvars, args):
-    """An internal helper function: replace arg if it is a type variable
-    found in tvars with corresponding substitution from args or
-    with corresponding substitution sub-tree if arg is a generic type.
-    """
-
-    if tvars is None:
-        tvars = []
-    if hasattr(arg, '_subs_tree') and isinstance(arg, (GenericMeta, _TypingBase)):
-        return arg._subs_tree(tvars, args)
-    if isinstance(arg, TypeVar):
-        for i, tvar in enumerate(tvars):
-            if arg == tvar:
-                return args[i]
-    return arg
-
 
 # Special typing constructs Union, Optional, Generic, Callable and Tuple
 # use three special attributes for internal bookkeeping of generic types:
@@ -576,849 +591,285 @@ def _replace_arg(arg, tvars, args):
 # * __args__ is a tuple of all arguments used in subscripting,
 #   e.g., Dict[T, int].__args__ == (T, int).
 
+def _is_dunder(attr):
+    return attr.startswith('__') and attr.endswith('__')
 
-def _subs_tree(cls, tvars=None, args=None):
-    """An internal helper function: calculate substitution tree
-    for generic cls after replacing its type parameters with
-    substitutions in tvars -> args (if any).
-    Repeat the same following __origin__'s.
 
-    Return a list of arguments with all possible substitutions
-    performed. Arguments that are generic classes themselves are represented
-    as tuples (so that no new classes are created by this function).
-    For example: _subs_tree(List[Tuple[int, T]][str]) == [(Tuple, int, str)]
-    """
+class _GenericAlias(_Final, _root=True):
+    """The central part of internal API.
 
-    if cls.__origin__ is None:
-        return cls
-    # Make of chain of origins (i.e. cls -> cls.__origin__)
-    current = cls.__origin__
-    orig_chain = []
-    while current.__origin__ is not None:
-        orig_chain.append(current)
-        current = current.__origin__
-    # Replace type variables in __args__ if asked ...
-    tree_args = []
-    for arg in cls.__args__:
-        tree_args.append(_replace_arg(arg, tvars, args))
-    # ... then continue replacing down the origin chain.
-    for ocls in orig_chain:
-        new_tree_args = []
-        for arg in ocls.__args__:
-            new_tree_args.append(_replace_arg(arg, ocls.__parameters__, tree_args))
-        tree_args = new_tree_args
-    return tree_args
-
-
-def _remove_dups_flatten(parameters):
-    """An internal helper for Union creation and substitution: flatten Union's
-    among parameters, then remove duplicates and strict subclasses.
+    This represents a generic version of type 'origin' with type arguments 'params'.
+    There are two kind of these aliases: user defined and special. The special ones
+    are wrappers around builtin collections and ABCs in collections.abc. These must
+    have 'name' always set. If 'inst' is False, then the alias can't be instantiated,
+    this is used by e.g. typing.List and typing.Dict.
     """
-
-    # Flatten out Union[Union[...], ...].
-    params = []
-    for p in parameters:
-        if isinstance(p, _Union) and p.__origin__ is Union:
-            params.extend(p.__args__)
-        elif isinstance(p, tuple) and len(p) > 0 and p[0] is Union:
-            params.extend(p[1:])
-        else:
-            params.append(p)
-    # Weed out strict duplicates, preserving the first of each occurrence.
-    all_params = set(params)
-    if len(all_params) < len(params):
-        new_params = []
-        for t in params:
-            if t in all_params:
-                new_params.append(t)
-                all_params.remove(t)
-        params = new_params
-        assert not all_params, all_params
-    # Weed out subclasses.
-    # E.g. Union[int, Employee, Manager] == Union[int, Employee].
-    # If object is present it will be sole survivor among proper classes.
-    # Never discard type variables.
-    # (In particular, Union[str, AnyStr] != AnyStr.)
-    all_params = set(params)
-    for t1 in params:
-        if not isinstance(t1, type):
-            continue
-        if any(isinstance(t2, type) and issubclass(t1, t2)
-               for t2 in all_params - {t1}
-               if not (isinstance(t2, GenericMeta) and
-                       t2.__origin__ is not None)):
-            all_params.remove(t1)
-    return tuple(t for t in params if t in all_params)
-
-
-def _check_generic(cls, parameters):
-    # Check correct count for parameters of a generic cls (internal helper).
-    if not cls.__parameters__:
-        raise TypeError("%s is not a generic class" % repr(cls))
-    alen = len(parameters)
-    elen = len(cls.__parameters__)
-    if alen != elen:
-        raise TypeError("Too %s parameters for %s; actual %s, expected %s" %
-                        ("many" if alen > elen else "few", repr(cls), alen, elen))
-
-
-_cleanups = []
-
-
-def _tp_cache(func):
-    """Internal wrapper caching __getitem__ of generic types with a fallback to
-    original function for non-hashable arguments.
-    """
-
-    cached = functools.lru_cache()(func)
-    _cleanups.append(cached.cache_clear)
-
-    @functools.wraps(func)
-    def inner(*args, **kwds):
-        try:
-            return cached(*args, **kwds)
-        except TypeError:
-            pass  # All real errors (not unhashable args) are raised below.
-        return func(*args, **kwds)
-    return inner
-
-
-class _Union(_FinalTypingBase, _root=True):
-    """Union type; Union[X, Y] means either X or Y.
-
-    To define a union, use e.g. Union[int, str].  Details:
-
-    - The arguments must be types and there must be at least one.
-
-    - None as an argument is a special case and is replaced by
-      type(None).
-
-    - Unions of unions are flattened, e.g.::
-
-        Union[Union[int, str], float] == Union[int, str, float]
-
-    - Unions of a single argument vanish, e.g.::
-
-        Union[int] == int  # The constructor actually returns int
-
-    - Redundant arguments are skipped, e.g.::
-
-        Union[int, str, int] == Union[int, str]
-
-    - When comparing unions, the argument order is ignored, e.g.::
-
-        Union[int, str] == Union[str, int]
-
-    - When two arguments have a subclass relationship, the least
-      derived argument is kept, e.g.::
-
-        class Employee: pass
-        class Manager(Employee): pass
-        Union[int, Employee, Manager] == Union[int, Employee]
-        Union[Manager, int, Employee] == Union[int, Employee]
-        Union[Employee, Manager] == Employee
-
-    - Similar for object::
-
-        Union[int, object] == object
-
-    - You cannot subclass or instantiate a union.
-
-    - You can use Optional[X] as a shorthand for Union[X, None].
-    """
-
-    __slots__ = ('__parameters__', '__args__', '__origin__', '__tree_hash__')
-
-    def __new__(cls, parameters=None, origin=None, *args, _root=False):
-        self = super().__new__(cls, parameters, origin, *args, _root=_root)
-        if origin is None:
-            self.__parameters__ = None
-            self.__args__ = None
-            self.__origin__ = None
-            self.__tree_hash__ = hash(frozenset(('Union',)))
-            return self
-        if not isinstance(parameters, tuple):
-            raise TypeError("Expected parameters=<tuple>")
-        if origin is Union:
-            parameters = _remove_dups_flatten(parameters)
-            # It's not a union if there's only one type left.
-            if len(parameters) == 1:
-                return parameters[0]
-        self.__parameters__ = _type_vars(parameters)
-        self.__args__ = parameters
+    def __init__(self, origin, params, *, inst=True, special=False, name=None):
+        self._inst = inst
+        self._special = special
+        if special and name is None:
+            orig_name = origin.__name__
+            name = orig_name[0].title() + orig_name[1:]
+        self._name = name
+        if not isinstance(params, tuple):
+            params = (params,)
         self.__origin__ = origin
-        # Pre-calculate the __hash__ on instantiation.
-        # This improves speed for complex substitutions.
-        subs_tree = self._subs_tree()
-        if isinstance(subs_tree, tuple):
-            self.__tree_hash__ = hash(frozenset(subs_tree))
-        else:
-            self.__tree_hash__ = hash(subs_tree)
-        return self
-
-    def _eval_type(self, globalns, localns):
-        if self.__args__ is None:
-            return self
-        ev_args = tuple(_eval_type(t, globalns, localns) for t in self.__args__)
-        ev_origin = _eval_type(self.__origin__, globalns, localns)
-        if ev_args == self.__args__ and ev_origin == self.__origin__:
-            # Everything is already evaluated.
-            return self
-        return self.__class__(ev_args, ev_origin, _root=True)
-
-    def _get_type_vars(self, tvars):
-        if self.__origin__ and self.__parameters__:
-            _get_type_vars(self.__parameters__, tvars)
-
-    def __repr__(self):
-        if self.__origin__ is None:
-            return super().__repr__()
-        tree = self._subs_tree()
-        if not isinstance(tree, tuple):
-            return repr(tree)
-        return tree[0]._tree_repr(tree)
-
-    def _tree_repr(self, tree):
-        arg_list = []
-        for arg in tree[1:]:
-            if not isinstance(arg, tuple):
-                arg_list.append(_type_repr(arg))
-            else:
-                arg_list.append(arg[0]._tree_repr(arg))
-        return super().__repr__() + '[%s]' % ', '.join(arg_list)
-
-    @_tp_cache
-    def __getitem__(self, parameters):
-        if parameters == ():
-            raise TypeError("Cannot take a Union of no types.")
-        if not isinstance(parameters, tuple):
-            parameters = (parameters,)
-        if self.__origin__ is None:
-            msg = "Union[arg, ...]: each arg must be a type."
-        else:
-            msg = "Parameters to generic types must be types."
-        parameters = tuple(_type_check(p, msg) for p in parameters)
-        if self is not Union:
-            _check_generic(self, parameters)
-        return self.__class__(parameters, origin=self, _root=True)
-
-    def _subs_tree(self, tvars=None, args=None):
-        if self is Union:
-            return Union  # Nothing to substitute
-        tree_args = _subs_tree(self, tvars, args)
-        tree_args = _remove_dups_flatten(tree_args)
-        if len(tree_args) == 1:
-            return tree_args[0]  # Union of a single type is that type
-        return (Union,) + tree_args
-
-    def __eq__(self, other):
-        if isinstance(other, _Union):
-            return self.__tree_hash__ == other.__tree_hash__
-        elif self is not Union:
-            return self._subs_tree() == other
-        else:
-            return self is other
-
-    def __hash__(self):
-        return self.__tree_hash__
-
-    def __instancecheck__(self, obj):
-        raise TypeError("Unions cannot be used with isinstance().")
-
-    def __subclasscheck__(self, cls):
-        raise TypeError("Unions cannot be used with issubclass().")
-
-
-Union = _Union(_root=True)
-
-
-class _Optional(_FinalTypingBase, _root=True):
-    """Optional type.
-
-    Optional[X] is equivalent to Union[X, None].
-    """
-
-    __slots__ = ()
-
-    @_tp_cache
-    def __getitem__(self, arg):
-        arg = _type_check(arg, "Optional[t] requires a single type.")
-        return Union[arg, type(None)]
-
-
-Optional = _Optional(_root=True)
-
-
-def _next_in_mro(cls):
-    """Helper for Generic.__new__.
-
-    Returns the class after the last occurrence of Generic or
-    Generic[...] in cls.__mro__.
-    """
-    next_in_mro = object
-    # Look for the last occurrence of Generic or Generic[...].
-    for i, c in enumerate(cls.__mro__[:-1]):
-        if isinstance(c, GenericMeta) and c._gorg is Generic:
-            next_in_mro = cls.__mro__[i + 1]
-    return next_in_mro
-
-
-def _make_subclasshook(cls):
-    """Construct a __subclasshook__ callable that incorporates
-    the associated __extra__ class in subclass checks performed
-    against cls.
-    """
-    if isinstance(cls.__extra__, abc.ABCMeta):
-        # The logic mirrors that of ABCMeta.__subclasscheck__.
-        # Registered classes need not be checked here because
-        # cls and its extra share the same _abc_registry.
-        def __extrahook__(subclass):
-            res = cls.__extra__.__subclasshook__(subclass)
-            if res is not NotImplemented:
-                return res
-            if cls.__extra__ in subclass.__mro__:
-                return True
-            for scls in cls.__extra__.__subclasses__():
-                if isinstance(scls, GenericMeta):
-                    continue
-                if issubclass(subclass, scls):
-                    return True
-            return NotImplemented
-    else:
-        # For non-ABC extras we'll just call issubclass().
-        def __extrahook__(subclass):
-            if cls.__extra__ and issubclass(subclass, cls.__extra__):
-                return True
-            return NotImplemented
-    return __extrahook__
-
-
-def _no_slots_copy(dct):
-    """Internal helper: copy class __dict__ and clean slots class variables.
-    (They will be re-created if necessary by normal class machinery.)
-    """
-    dict_copy = dict(dct)
-    if '__slots__' in dict_copy:
-        for slot in dict_copy['__slots__']:
-            dict_copy.pop(slot, None)
-    return dict_copy
-
-
-class GenericMeta(TypingMeta, abc.ABCMeta):
-    """Metaclass for generic types.
-
-    This is a metaclass for typing.Generic and generic ABCs defined in
-    typing module. User defined subclasses of GenericMeta can override
-    __new__ and invoke super().__new__. Note that GenericMeta.__new__
-    has strict rules on what is allowed in its bases argument:
-    * plain Generic is disallowed in bases;
-    * Generic[...] should appear in bases at most once;
-    * if Generic[...] is present, then it should list all type variables
-      that appear in other bases.
-    In addition, type of all generic bases is erased, e.g., C[int] is
-    stripped to plain C.
-    """
-
-    def __new__(cls, name, bases, namespace,
-                tvars=None, args=None, origin=None, extra=None, orig_bases=None):
-        """Create a new generic class. GenericMeta.__new__ accepts
-        keyword arguments that are used for internal bookkeeping, therefore
-        an override should pass unused keyword arguments to super().
-        """
-        if tvars is not None:
-            # Called from __getitem__() below.
-            assert origin is not None
-            assert all(isinstance(t, TypeVar) for t in tvars), tvars
-        else:
-            # Called from class statement.
-            assert tvars is None, tvars
-            assert args is None, args
-            assert origin is None, origin
-
-            # Get the full set of tvars from the bases.
-            tvars = _type_vars(bases)
-            # Look for Generic[T1, ..., Tn].
-            # If found, tvars must be a subset of it.
-            # If not found, tvars is it.
-            # Also check for and reject plain Generic,
-            # and reject multiple Generic[...].
-            gvars = None
-            for base in bases:
-                if base is Generic:
-                    raise TypeError("Cannot inherit from plain Generic")
-                if (isinstance(base, GenericMeta) and
-                        base.__origin__ is Generic):
-                    if gvars is not None:
-                        raise TypeError(
-                            "Cannot inherit from Generic[...] multiple types.")
-                    gvars = base.__parameters__
-            if gvars is None:
-                gvars = tvars
-            else:
-                tvarset = set(tvars)
-                gvarset = set(gvars)
-                if not tvarset <= gvarset:
-                    raise TypeError(
-                        "Some type variables (%s) "
-                        "are not listed in Generic[%s]" %
-                        (", ".join(str(t) for t in tvars if t not in gvarset),
-                         ", ".join(str(g) for g in gvars)))
-                tvars = gvars
-
-        initial_bases = bases
-        if extra is not None and type(extra) is abc.ABCMeta and extra not in bases:
-            bases = (extra,) + bases
-        bases = tuple(b._gorg if isinstance(b, GenericMeta) else b for b in bases)
-
-        # remove bare Generic from bases if there are other generic bases
-        if any(isinstance(b, GenericMeta) and b is not Generic for b in bases):
-            bases = tuple(b for b in bases if b is not Generic)
-        namespace.update({'__origin__': origin, '__extra__': extra,
-                          '_gorg': None if not origin else origin._gorg})
-        self = super().__new__(cls, name, bases, namespace, _root=True)
-        super(GenericMeta, self).__setattr__('_gorg',
-                                             self if not origin else origin._gorg)
-        self.__parameters__ = tvars
-        # Be prepared that GenericMeta will be subclassed by TupleMeta
-        # and CallableMeta, those two allow ..., (), or [] in __args___.
         self.__args__ = tuple(... if a is _TypingEllipsis else
                               () if a is _TypingEmpty else
-                              a for a in args) if args else None
-        # Speed hack (https://github.com/python/typing/issues/196).
-        self.__next_in_mro__ = _next_in_mro(self)
-        # Preserve base classes on subclassing (__bases__ are type erased now).
-        if orig_bases is None:
-            self.__orig_bases__ = initial_bases
-
-        # This allows unparameterized generic collections to be used
-        # with issubclass() and isinstance() in the same way as their
-        # collections.abc counterparts (e.g., isinstance([], Iterable)).
-        if (
-            '__subclasshook__' not in namespace and extra or
-            # allow overriding
-            getattr(self.__subclasshook__, '__name__', '') == '__extrahook__'
-        ):
-            self.__subclasshook__ = _make_subclasshook(self)
-        if isinstance(extra, abc.ABCMeta):
-            self._abc_registry = extra._abc_registry
-            self._abc_cache = extra._abc_cache
-        elif origin is not None:
-            self._abc_registry = origin._abc_registry
-            self._abc_cache = origin._abc_cache
-
-        if origin and hasattr(origin, '__qualname__'):  # Fix for Python 3.2.
-            self.__qualname__ = origin.__qualname__
-        self.__tree_hash__ = (hash(self._subs_tree()) if origin else
-                              super(GenericMeta, self).__hash__())
-        return self
-
-    # _abc_negative_cache and _abc_negative_cache_version
-    # realised as descriptors, since GenClass[t1, t2, ...] always
-    # share subclass info with GenClass.
-    # This is an important memory optimization.
-    @property
-    def _abc_negative_cache(self):
-        if isinstance(self.__extra__, abc.ABCMeta):
-            return self.__extra__._abc_negative_cache
-        return self._gorg._abc_generic_negative_cache
-
-    @_abc_negative_cache.setter
-    def _abc_negative_cache(self, value):
-        if self.__origin__ is None:
-            if isinstance(self.__extra__, abc.ABCMeta):
-                self.__extra__._abc_negative_cache = value
-            else:
-                self._abc_generic_negative_cache = value
-
-    @property
-    def _abc_negative_cache_version(self):
-        if isinstance(self.__extra__, abc.ABCMeta):
-            return self.__extra__._abc_negative_cache_version
-        return self._gorg._abc_generic_negative_cache_version
-
-    @_abc_negative_cache_version.setter
-    def _abc_negative_cache_version(self, value):
-        if self.__origin__ is None:
-            if isinstance(self.__extra__, abc.ABCMeta):
-                self.__extra__._abc_negative_cache_version = value
-            else:
-                self._abc_generic_negative_cache_version = value
-
-    def _get_type_vars(self, tvars):
-        if self.__origin__ and self.__parameters__:
-            _get_type_vars(self.__parameters__, tvars)
-
-    def _eval_type(self, globalns, localns):
-        ev_origin = (self.__origin__._eval_type(globalns, localns)
-                     if self.__origin__ else None)
-        ev_args = tuple(_eval_type(a, globalns, localns) for a
-                        in self.__args__) if self.__args__ else None
-        if ev_origin == self.__origin__ and ev_args == self.__args__:
-            return self
-        return self.__class__(self.__name__,
-                              self.__bases__,
-                              _no_slots_copy(self.__dict__),
-                              tvars=_type_vars(ev_args) if ev_args else None,
-                              args=ev_args,
-                              origin=ev_origin,
-                              extra=self.__extra__,
-                              orig_bases=self.__orig_bases__)
-
-    def __repr__(self):
-        if self.__origin__ is None:
-            return super().__repr__()
-        return self._tree_repr(self._subs_tree())
-
-    def _tree_repr(self, tree):
-        arg_list = []
-        for arg in tree[1:]:
-            if arg == ():
-                arg_list.append('()')
-            elif not isinstance(arg, tuple):
-                arg_list.append(_type_repr(arg))
-            else:
-                arg_list.append(arg[0]._tree_repr(arg))
-        return super().__repr__() + '[%s]' % ', '.join(arg_list)
-
-    def _subs_tree(self, tvars=None, args=None):
-        if self.__origin__ is None:
-            return self
-        tree_args = _subs_tree(self, tvars, args)
-        return (self._gorg,) + tuple(tree_args)
-
-    def __eq__(self, other):
-        if not isinstance(other, GenericMeta):
-            return NotImplemented
-        if self.__origin__ is None or other.__origin__ is None:
-            return self is other
-        return self.__tree_hash__ == other.__tree_hash__
-
-    def __hash__(self):
-        return self.__tree_hash__
+                              a for a in params)
+        self.__parameters__ = _collect_type_vars(params)
+        self.__slots__ = None  # This is not documented.
+        if not name:
+            self.__module__ = origin.__module__
 
     @_tp_cache
     def __getitem__(self, params):
+        if self.__origin__ in (Generic, _Protocol):
+            # Can't subscript Generic[...] or _Protocol[...].
+            raise TypeError(f"Cannot subscript already-subscripted {self}")
         if not isinstance(params, tuple):
             params = (params,)
-        if not params and self._gorg is not Tuple:
-            raise TypeError(
-                "Parameter list to %s[...] cannot be empty" % _qualname(self))
         msg = "Parameters to generic types must be types."
         params = tuple(_type_check(p, msg) for p in params)
-        if self is Generic:
-            # Generic can only be subscripted with unique type variables.
-            if not all(isinstance(p, TypeVar) for p in params):
-                raise TypeError(
-                    "Parameters to Generic[...] must all be type variables")
-            if len(set(params)) != len(params):
-                raise TypeError(
-                    "Parameters to Generic[...] must all be unique")
-            tvars = params
-            args = params
-        elif self in (Tuple, Callable):
-            tvars = _type_vars(params)
-            args = params
-        elif self is _Protocol:
-            # _Protocol is internal, don't check anything.
-            tvars = params
-            args = params
-        elif self.__origin__ in (Generic, _Protocol):
-            # Can't subscript Generic[...] or _Protocol[...].
-            raise TypeError("Cannot subscript already-subscripted %s" %
-                            repr(self))
-        else:
-            # Subscripting a regular Generic subclass.
-            _check_generic(self, params)
-            tvars = _type_vars(params)
-            args = params
-
-        prepend = (self,) if self.__origin__ is None else ()
-        return self.__class__(self.__name__,
-                              prepend + self.__bases__,
-                              _no_slots_copy(self.__dict__),
-                              tvars=tvars,
-                              args=args,
-                              origin=self,
-                              extra=self.__extra__,
-                              orig_bases=self.__orig_bases__)
-
-    def __subclasscheck__(self, cls):
-        if self.__origin__ is not None:
-            if sys._getframe(1).f_globals['__name__'] not in ['abc', 'functools']:
-                raise TypeError("Parameterized generics cannot be used with class "
-                                "or instance checks")
-            return False
-        if self is Generic:
-            raise TypeError("Class %r cannot be used with class "
-                            "or instance checks" % self)
-        return super().__subclasscheck__(cls)
-
-    def __instancecheck__(self, instance):
-        # Since we extend ABC.__subclasscheck__ and
-        # ABC.__instancecheck__ inlines the cache checking done by the
-        # latter, we must extend __instancecheck__ too. For simplicity
-        # we just skip the cache check -- instance checks for generic
-        # classes are supposed to be rare anyways.
-        return issubclass(instance.__class__, self)
-
-    def __setattr__(self, attr, value):
-        # We consider all the subscripted generics as proxies for original class
-        if (
-            attr.startswith('__') and attr.endswith('__') or
-            attr.startswith('_abc_') or
-            self._gorg is None  # The class is not fully created, see #typing/506
-        ):
-            super(GenericMeta, self).__setattr__(attr, value)
-        else:
-            super(GenericMeta, self._gorg).__setattr__(attr, value)
+        _check_generic(self, params)
+        return _subs_tvars(self, self.__parameters__, params)
 
+    def copy_with(self, params):
+        # We don't copy self._special.
+        return _GenericAlias(self.__origin__, params, name=self._name, inst=self._inst)
 
-# Prevent checks for Generic to crash when defining Generic.
-Generic = None
+    def __repr__(self):
+        if (self._name != 'Callable' or
+                len(self.__args__) == 2 and self.__args__[0] is Ellipsis):
+            if self._name:
+                name = 'typing.' + self._name
+            else:
+                name = _type_repr(self.__origin__)
+            if not self._special:
+                args = f'[{", ".join([_type_repr(a) for a in self.__args__])}]'
+            else:
+                args = ''
+            return (f'{name}{args}')
+        if self._special:
+            return 'typing.Callable'
+        return (f'typing.Callable'
+                f'[[{", ".join([_type_repr(a) for a in self.__args__[:-1]])}], '
+                f'{_type_repr(self.__args__[-1])}]')
 
+    def __eq__(self, other):
+        if not isinstance(other, _GenericAlias):
+            return NotImplemented
+        if self.__origin__ != other.__origin__:
+            return False
+        if self.__origin__ is Union and other.__origin__ is Union:
+            return frozenset(self.__args__) == frozenset(other.__args__)
+        return self.__args__ == other.__args__
 
-def _generic_new(base_cls, cls, *args, **kwds):
-    # Assure type is erased on instantiation,
-    # but attempt to store it in __orig_class__
-    if cls.__origin__ is None:
-        return base_cls.__new__(cls)
-    else:
-        origin = cls._gorg
-        obj = base_cls.__new__(origin)
+    def __hash__(self):
+        if self.__origin__ is Union:
+            return hash((Union, frozenset(self.__args__)))
+        return hash((self.__origin__, self.__args__))
+
+    def __call__(self, *args, **kwargs):
+        if not self._inst:
+            raise TypeError(f"Type {self._name} cannot be instantiated; "
+                            f"use {self._name.lower()}() instead")
+        result = self.__origin__(*args, **kwargs)
         try:
-            obj.__orig_class__ = cls
+            result.__orig_class__ = self
         except AttributeError:
             pass
-        obj.__init__(*args, **kwds)
-        return obj
-
-
-class Generic(metaclass=GenericMeta):
-    """Abstract base class for generic types.
-
-    A generic type is typically declared by inheriting from
-    this class parameterized with one or more type variables.
-    For example, a generic mapping type might be defined as::
-
-      class Mapping(Generic[KT, VT]):
-          def __getitem__(self, key: KT) -> VT:
-              ...
-          # Etc.
-
-    This class can then be used as follows::
-
-      def lookup_name(mapping: Mapping[KT, VT], key: KT, default: VT) -> VT:
-          try:
-              return mapping[key]
-          except KeyError:
-              return default
-    """
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Generic:
-            raise TypeError("Type Generic cannot be instantiated; "
-                            "it can be used only as a base class")
-        return _generic_new(cls.__next_in_mro__, cls, *args, **kwds)
-
-
-class _TypingEmpty:
-    """Internal placeholder for () or []. Used by TupleMeta and CallableMeta
-    to allow empty list/tuple in specific places, without allowing them
-    to sneak in where prohibited.
-    """
-
-
-class _TypingEllipsis:
-    """Internal placeholder for ... (ellipsis)."""
-
-
-class TupleMeta(GenericMeta):
-    """Metaclass for Tuple (internal)."""
-
-    @_tp_cache
-    def __getitem__(self, parameters):
-        if self.__origin__ is not None or self._gorg is not Tuple:
-            # Normal generic rules apply if this is not the first subscription
-            # or a subscription of a subclass.
-            return super().__getitem__(parameters)
-        if parameters == ():
-            return super().__getitem__((_TypingEmpty,))
-        if not isinstance(parameters, tuple):
-            parameters = (parameters,)
-        if len(parameters) == 2 and parameters[1] is ...:
-            msg = "Tuple[t, ...]: t must be a type."
-            p = _type_check(parameters[0], msg)
-            return super().__getitem__((p, _TypingEllipsis))
-        msg = "Tuple[t0, t1, ...]: each t must be a type."
-        parameters = tuple(_type_check(p, msg) for p in parameters)
-        return super().__getitem__(parameters)
+        return result
+
+    def __mro_entries__(self, bases):
+        if self._name:  # generic version of an ABC or built-in class
+            res = []
+            if self.__origin__ not in bases:
+                res.append(self.__origin__)
+            i = bases.index(self)
+            if not any(isinstance(b, _GenericAlias) or issubclass(b, Generic)
+                       for b in bases[i+1:]):
+                res.append(Generic)
+            return tuple(res)
+        if self.__origin__ is Generic:
+            i = bases.index(self)
+            for b in bases[i+1:]:
+                if isinstance(b, _GenericAlias) and b is not self:
+                    return ()
+        return (self.__origin__,)
+
+    def __getattr__(self, attr):
+        # We are carefull for copy and pickle.
+        # Also for simplicity we just don't relay all dunder names
+        if '__origin__' in self.__dict__ and not _is_dunder(attr):
+            return getattr(self.__origin__, attr)
+        raise AttributeError(attr)
+
+    def __setattr__(self, attr, val):
+        if _is_dunder(attr) or attr in ('_name', '_inst', '_special'):
+            super().__setattr__(attr, val)
+        else:
+            setattr(self.__origin__, attr, val)
 
     def __instancecheck__(self, obj):
-        if self.__args__ is None:
-            return isinstance(obj, tuple)
-        raise TypeError("Parameterized Tuple cannot be used "
-                        "with isinstance().")
+        return self.__subclasscheck__(type(obj))
 
     def __subclasscheck__(self, cls):
-        if self.__args__ is None:
-            return issubclass(cls, tuple)
-        raise TypeError("Parameterized Tuple cannot be used "
-                        "with issubclass().")
-
-
-class Tuple(tuple, extra=tuple, metaclass=TupleMeta):
-    """Tuple type; Tuple[X, Y] is the cross-product type of X and Y.
-
-    Example: Tuple[T1, T2] is a tuple of two elements corresponding
-    to type variables T1 and T2.  Tuple[int, float, str] is a tuple
-    of an int, a float and a string.
-
-    To specify a variable-length tuple of homogeneous type, use Tuple[T, ...].
+        if self._special:
+            if not isinstance(cls, _GenericAlias):
+                return issubclass(cls, self.__origin__)
+            if cls._special:
+                return issubclass(cls.__origin__, self.__origin__)
+        raise TypeError("Subscripted generics cannot be used with"
+                        " class and instance checks")
+
+
+class _VariadicGenericAlias(_GenericAlias, _root=True):
+    """Same as _GenericAlias above but for variadic aliases. Currently,
+    this is used only by special internal aliases: Tuple and Callable.
     """
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Tuple:
-            raise TypeError("Type Tuple cannot be instantiated; "
-                            "use tuple() instead")
-        return _generic_new(tuple, cls, *args, **kwds)
-
-
-class CallableMeta(GenericMeta):
-    """Metaclass for Callable (internal)."""
-
-    def __repr__(self):
-        if self.__origin__ is None:
-            return super().__repr__()
-        return self._tree_repr(self._subs_tree())
-
-    def _tree_repr(self, tree):
-        if self._gorg is not Callable:
-            return super()._tree_repr(tree)
-        # For actual Callable (not its subclass) we override
-        # super()._tree_repr() for nice formatting.
-        arg_list = []
-        for arg in tree[1:]:
-            if not isinstance(arg, tuple):
-                arg_list.append(_type_repr(arg))
-            else:
-                arg_list.append(arg[0]._tree_repr(arg))
-        if arg_list[0] == '...':
-            return repr(tree[0]) + '[..., %s]' % arg_list[1]
-        return (repr(tree[0]) +
-                '[[%s], %s]' % (', '.join(arg_list[:-1]), arg_list[-1]))
-
-    def __getitem__(self, parameters):
-        """A thin wrapper around __getitem_inner__ to provide the latter
-        with hashable arguments to improve speed.
-        """
-
-        if self.__origin__ is not None or self._gorg is not Callable:
-            return super().__getitem__(parameters)
-        if not isinstance(parameters, tuple) or len(parameters) != 2:
+    def __getitem__(self, params):
+        if self._name != 'Callable' or not self._special:
+            return self.__getitem_inner__(params)
+        if not isinstance(params, tuple) or len(params) != 2:
             raise TypeError("Callable must be used as "
                             "Callable[[arg, ...], result].")
-        args, result = parameters
+        args, result = params
         if args is Ellipsis:
-            parameters = (Ellipsis, result)
+            params = (Ellipsis, result)
         else:
             if not isinstance(args, list):
-                raise TypeError("Callable[args, result]: args must be a list."
-                                " Got %.100r." % (args,))
-            parameters = (tuple(args), result)
-        return self.__getitem_inner__(parameters)
+                raise TypeError(f"Callable[args, result]: args must be a list."
+                                f" Got {args}")
+            params = (tuple(args), result)
+        return self.__getitem_inner__(params)
 
     @_tp_cache
-    def __getitem_inner__(self, parameters):
-        args, result = parameters
-        msg = "Callable[args, result]: result must be a type."
-        result = _type_check(result, msg)
-        if args is Ellipsis:
-            return super().__getitem__((_TypingEllipsis, result))
-        msg = "Callable[[arg, ...], result]: each arg must be a type."
-        args = tuple(_type_check(arg, msg) for arg in args)
-        parameters = args + (result,)
-        return super().__getitem__(parameters)
-
-
-class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
-    """Callable type; Callable[[int], str] is a function of (int) -> str.
-
-    The subscription syntax must always be used with exactly two
-    values: the argument list and the return type.  The argument list
-    must be a list of types or ellipsis; the return type must be a single type.
-
-    There is no syntax to indicate optional or keyword arguments,
-    such function types are rarely used as callback types.
-    """
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Callable:
-            raise TypeError("Type Callable cannot be instantiated; "
-                            "use a non-abstract subclass instead")
-        return _generic_new(cls.__next_in_mro__, cls, *args, **kwds)
-
-
-class _ClassVar(_FinalTypingBase, _root=True):
-    """Special type construct to mark class variables.
+    def __getitem_inner__(self, params):
+        if self.__origin__ is tuple and self._special:
+            if params == ():
+                return self.copy_with((_TypingEmpty,))
+            if not isinstance(params, tuple):
+                params = (params,)
+            if len(params) == 2 and params[1] is ...:
+                msg = "Tuple[t, ...]: t must be a type."
+                p = _type_check(params[0], msg)
+                return self.copy_with((p, _TypingEllipsis))
+            msg = "Tuple[t0, t1, ...]: each t must be a type."
+            params = tuple(_type_check(p, msg) for p in params)
+            return self.copy_with(params)
+        if self.__origin__ is collections.abc.Callable and self._special:
+            args, result = params
+            msg = "Callable[args, result]: result must be a type."
+            result = _type_check(result, msg)
+            if args is Ellipsis:
+                return self.copy_with((_TypingEllipsis, result))
+            msg = "Callable[[arg, ...], result]: each arg must be a type."
+            args = tuple(_type_check(arg, msg) for arg in args)
+            params = args + (result,)
+            return self.copy_with(params)
+        return super().__getitem__(params)
+
+
+class Generic:
+    """Abstract base class for generic types.
 
-    An annotation wrapped in ClassVar indicates that a given
-    attribute is intended to be used as a class variable and
-    should not be set on instances of that class. Usage::
+    A generic type is typically declared by inheriting from
+    this class parameterized with one or more type variables.
+    For example, a generic mapping type might be defined as::
 
-      class Starship:
-          stats: ClassVar[Dict[str, int]] = {} # class variable
-          damage: int = 10                     # instance variable
+      class Mapping(Generic[KT, VT]):
+          def __getitem__(self, key: KT) -> VT:
+              ...
+          # Etc.
 
-    ClassVar accepts only types and cannot be further subscribed.
+    This class can then be used as follows::
 
-    Note that ClassVar is not a class itself, and should not
-    be used with isinstance() or issubclass().
+      def lookup_name(mapping: Mapping[KT, VT], key: KT, default: VT) -> VT:
+          try:
+              return mapping[key]
+          except KeyError:
+              return default
     """
+    __slots__ = ()
 
-    __slots__ = ('__type__',)
-
-    def __init__(self, tp=None, **kwds):
-        self.__type__ = tp
-
-    def __getitem__(self, item):
-        cls = type(self)
-        if self.__type__ is None:
-            return cls(_type_check(item,
-                       '{} accepts only single type.'.format(cls.__name__[1:])),
-                       _root=True)
-        raise TypeError('{} cannot be further subscripted'
-                        .format(cls.__name__[1:]))
+    def __new__(cls, *args, **kwds):
+        if cls is Generic:
+            raise TypeError("Type Generic cannot be instantiated; "
+                            "it can be used only as a base class")
+        return super().__new__(cls)
 
-    def _eval_type(self, globalns, localns):
-        new_tp = _eval_type(self.__type__, globalns, localns)
-        if new_tp == self.__type__:
-            return self
-        return type(self)(new_tp, _root=True)
+    @_tp_cache
+    def __class_getitem__(cls, params):
+        if not isinstance(params, tuple):
+            params = (params,)
+        if not params and cls is not Tuple:
+            raise TypeError(
+                f"Parameter list to {cls.__qualname__}[...] cannot be empty")
+        msg = "Parameters to generic types must be types."
+        params = tuple(_type_check(p, msg) for p in params)
+        if cls is Generic:
+            # Generic can only be subscripted with unique type variables.
+            if not all(isinstance(p, TypeVar) for p in params):
+                raise TypeError(
+                    "Parameters to Generic[...] must all be type variables")
+            if len(set(params)) != len(params):
+                raise TypeError(
+                    "Parameters to Generic[...] must all be unique")
+        elif cls is _Protocol:
+            # _Protocol is internal at the moment, just skip the check
+            pass
+        else:
+            # Subscripting a regular Generic subclass.
+            _check_generic(cls, params)
+        return _GenericAlias(cls, params)
 
-    def __repr__(self):
-        r = super().__repr__()
-        if self.__type__ is not None:
-            r += '[{}]'.format(_type_repr(self.__type__))
-        return r
+    def __init_subclass__(cls, *args, **kwargs):
+        tvars = []
+        if '__orig_bases__' in cls.__dict__:
+            error = Generic in cls.__orig_bases__
+        else:
+            error = Generic in cls.__bases__ and cls.__name__ != '_Protocol'
+        if error:
+            raise TypeError("Cannot inherit from plain Generic")
+        if '__orig_bases__' in cls.__dict__:
+            tvars = _collect_type_vars(cls.__orig_bases__)
+            # Look for Generic[T1, ..., Tn].
+            # If found, tvars must be a subset of it.
+            # If not found, tvars is it.
+            # Also check for and reject plain Generic,
+            # and reject multiple Generic[...].
+            gvars = None
+            for base in cls.__orig_bases__:
+                if (isinstance(base, _GenericAlias) and
+                        base.__origin__ is Generic):
+                    if gvars is not None:
+                        raise TypeError(
+                            "Cannot inherit from Generic[...] multiple types.")
+                    gvars = base.__parameters__
+            if gvars is None:
+                gvars = tvars
+            else:
+                tvarset = set(tvars)
+                gvarset = set(gvars)
+                if not tvarset <= gvarset:
+                    s_vars = ', '.join(str(t) for t in tvars if t not in gvarset)
+                    s_args = ', '.join(str(g) for g in gvars)
+                    raise TypeError(f"Some type variables ({s_vars}) are"
+                                    f" not listed in Generic[{s_args}]")
+                tvars = gvars
+        cls.__parameters__ = tuple(tvars)
 
-    def __hash__(self):
-        return hash((type(self).__name__, self.__type__))
 
-    def __eq__(self, other):
-        if not isinstance(other, _ClassVar):
-            return NotImplemented
-        if self.__type__ is not None:
-            return self.__type__ == other.__type__
-        return self is other
+class _TypingEmpty:
+    """Internal placeholder for () or []. Used by TupleMeta and CallableMeta
+    to allow empty list/tuple in specific places, without allowing them
+    to sneak in where prohibited.
+    """
 
 
-ClassVar = _ClassVar(_root=True)
+class _TypingEllipsis:
+    """Internal placeholder for ... (ellipsis)."""
 
 
 def cast(typ, val):
@@ -1503,7 +954,7 @@ def get_type_hints(obj, globalns=None, localns=None):
                 if value is None:
                     value = type(None)
                 if isinstance(value, str):
-                    value = _ForwardRef(value)
+                    value = ForwardRef(value)
                 value = _eval_type(value, base_globals, localns)
                 hints[name] = value
         return hints
@@ -1531,7 +982,7 @@ def get_type_hints(obj, globalns=None, localns=None):
         if value is None:
             value = type(None)
         if isinstance(value, str):
-            value = _ForwardRef(value)
+            value = ForwardRef(value)
         value = _eval_type(value, globalns, localns)
         if name in defaults and defaults[name] is None:
             value = Optional[value]
@@ -1619,7 +1070,7 @@ def utf8(value):
     return _overload_dummy
 
 
-class _ProtocolMeta(GenericMeta):
+class _ProtocolMeta(type):
     """Internal metaclass for _Protocol.
 
     This exists so _Protocol classes can be generic without deriving
@@ -1687,7 +1138,7 @@ def _get_protocol_attrs(self):
         return attrs
 
 
-class _Protocol(metaclass=_ProtocolMeta):
+class _Protocol(Generic, metaclass=_ProtocolMeta):
     """Internal base class for protocol classes.
 
     This implements a simple-minded structural issubclass check
@@ -1699,47 +1150,111 @@ class _Protocol(metaclass=_ProtocolMeta):
 
     _is_protocol = True
 
+    def __class_getitem__(cls, params):
+        return super().__class_getitem__(params)
 
-# Various ABCs mimicking those in collections.abc.
-# A few are simply re-exported for completeness.
-
-Hashable = collections_abc.Hashable  # Not generic.
 
+# Some unconstrained type variables.  These are used by the container types.
+# (These are not for export.)
+T = TypeVar('T')  # Any type.
+KT = TypeVar('KT')  # Key type.
+VT = TypeVar('VT')  # Value type.
+T_co = TypeVar('T_co', covariant=True)  # Any type covariant containers.
+V_co = TypeVar('V_co', covariant=True)  # Any type covariant containers.
+VT_co = TypeVar('VT_co', covariant=True)  # Value type covariant containers.
+T_contra = TypeVar('T_contra', contravariant=True)  # Ditto contravariant.
+# Internal type variable used for Type[].
+CT_co = TypeVar('CT_co', covariant=True, bound=type)
 
-if hasattr(collections_abc, 'Awaitable'):
-    class Awaitable(Generic[T_co], extra=collections_abc.Awaitable):
-        __slots__ = ()
-
-    __all__.append('Awaitable')
+# A useful type variable with constraints.  This represents string types.
+# (This one *is* for export!)
+AnyStr = TypeVar('AnyStr', bytes, str)
 
 
-if hasattr(collections_abc, 'Coroutine'):
-    class Coroutine(Awaitable[V_co], Generic[T_co, T_contra, V_co],
-                    extra=collections_abc.Coroutine):
-        __slots__ = ()
+# Various ABCs mimicking those in collections.abc.
+def _alias(origin, params, inst=True):
+    return _GenericAlias(origin, params, special=True, inst=inst)
+
+Hashable = _alias(collections.abc.Hashable, ())  # Not generic.
+Awaitable = _alias(collections.abc.Awaitable, T_co)
+Coroutine = _alias(collections.abc.Coroutine, (T_co, T_contra, V_co))
+AsyncIterable = _alias(collections.abc.AsyncIterable, T_co)
+AsyncIterator = _alias(collections.abc.AsyncIterator, T_co)
+Iterable = _alias(collections.abc.Iterable, T_co)
+Iterator = _alias(collections.abc.Iterator, T_co)
+Reversible = _alias(collections.abc.Reversible, T_co)
+Sized = _alias(collections.abc.Sized, ())  # Not generic.
+Container = _alias(collections.abc.Container, T_co)
+Collection = _alias(collections.abc.Collection, T_co)
+Callable = _VariadicGenericAlias(collections.abc.Callable, (), special=True)
+Callable.__doc__ = \
+    """Callable type; Callable[[int], str] is a function of (int) -> str.
 
-    __all__.append('Coroutine')
+    The subscription syntax must always be used with exactly two
+    values: the argument list and the return type.  The argument list
+    must be a list of types or ellipsis; the return type must be a single type.
 
+    There is no syntax to indicate optional or keyword arguments,
+    such function types are rarely used as callback types.
+    """
+AbstractSet = _alias(collections.abc.Set, T_co)
+MutableSet = _alias(collections.abc.MutableSet, T)
+# NOTE: Mapping is only covariant in the value type.
+Mapping = _alias(collections.abc.Mapping, (KT, VT_co))
+MutableMapping = _alias(collections.abc.MutableMapping, (KT, VT))
+Sequence = _alias(collections.abc.Sequence, T_co)
+MutableSequence = _alias(collections.abc.MutableSequence, T)
+ByteString = _alias(collections.abc.ByteString, ())  # Not generic
+Tuple = _VariadicGenericAlias(tuple, (), inst=False, special=True)
+Tuple.__doc__ = \
+    """Tuple type; Tuple[X, Y] is the cross-product type of X and Y.
 
-if hasattr(collections_abc, 'AsyncIterable'):
+    Example: Tuple[T1, T2] is a tuple of two elements corresponding
+    to type variables T1 and T2.  Tuple[int, float, str] is a tuple
+    of an int, a float and a string.
 
-    class AsyncIterable(Generic[T_co], extra=collections_abc.AsyncIterable):
-        __slots__ = ()
+    To specify a variable-length tuple of homogeneous type, use Tuple[T, ...].
+    """
+List = _alias(list, T, inst=False)
+Deque = _alias(collections.deque, T)
+Set = _alias(set, T, inst=False)
+FrozenSet = _alias(frozenset, T_co, inst=False)
+MappingView = _alias(collections.abc.MappingView, T_co)
+KeysView = _alias(collections.abc.KeysView, KT)
+ItemsView = _alias(collections.abc.ItemsView, (KT, VT_co))
+ValuesView = _alias(collections.abc.ValuesView, VT_co)
+ContextManager = _alias(contextlib.AbstractContextManager, T_co)
+AsyncContextManager = _alias(contextlib.AbstractAsyncContextManager, T_co)
+Dict = _alias(dict, (KT, VT), inst=False)
+DefaultDict = _alias(collections.defaultdict, (KT, VT))
+Counter = _alias(collections.Counter, T)
+ChainMap = _alias(collections.ChainMap, (KT, VT))
+Generator = _alias(collections.abc.Generator, (T_co, T_contra, V_co))
+AsyncGenerator = _alias(collections.abc.AsyncGenerator, (T_co, T_contra))
+Type = _alias(type, CT_co, inst=False)
+Type.__doc__ = \
+    """A special construct usable to annotate class objects.
 
-    class AsyncIterator(AsyncIterable[T_co],
-                        extra=collections_abc.AsyncIterator):
-        __slots__ = ()
+    For example, suppose we have the following classes::
 
-    __all__.append('AsyncIterable')
-    __all__.append('AsyncIterator')
+      class User: ...  # Abstract base for User classes
+      class BasicUser(User): ...
+      class ProUser(User): ...
+      class TeamUser(User): ...
 
+    And a function that takes a class argument that's a subclass of
+    User and returns an instance of the corresponding class::
 
-class Iterable(Generic[T_co], extra=collections_abc.Iterable):
-    __slots__ = ()
+      U = TypeVar('U', bound=User)
+      def new_user(user_class: Type[U]) -> U:
+          user = user_class()
+          # (Here we could write the user object to a database)
+          return user
 
+      joe = new_user(BasicUser)
 
-class Iterator(Iterable[T_co], extra=collections_abc.Iterator):
-    __slots__ = ()
+    At this point the type checker knows that joe has type BasicUser.
+    """
 
 
 class SupportsInt(_Protocol):
@@ -1790,316 +1305,6 @@ def __round__(self, ndigits: int = 0) -> T_co:
         pass
 
 
-if hasattr(collections_abc, 'Reversible'):
-    class Reversible(Iterable[T_co], extra=collections_abc.Reversible):
-        __slots__ = ()
-else:
-    class Reversible(_Protocol[T_co]):
-        __slots__ = ()
-
-        @abstractmethod
-        def __reversed__(self) -> 'Iterator[T_co]':
-            pass
-
-
-Sized = collections_abc.Sized  # Not generic.
-
-
-class Container(Generic[T_co], extra=collections_abc.Container):
-    __slots__ = ()
-
-
-if hasattr(collections_abc, 'Collection'):
-    class Collection(Sized, Iterable[T_co], Container[T_co],
-                     extra=collections_abc.Collection):
-        __slots__ = ()
-
-    __all__.append('Collection')
-
-
-# Callable was defined earlier.
-
-if hasattr(collections_abc, 'Collection'):
-    class AbstractSet(Collection[T_co],
-                      extra=collections_abc.Set):
-        __slots__ = ()
-else:
-    class AbstractSet(Sized, Iterable[T_co], Container[T_co],
-                      extra=collections_abc.Set):
-        __slots__ = ()
-
-
-class MutableSet(AbstractSet[T], extra=collections_abc.MutableSet):
-    __slots__ = ()
-
-
-# NOTE: It is only covariant in the value type.
-if hasattr(collections_abc, 'Collection'):
-    class Mapping(Collection[KT], Generic[KT, VT_co],
-                  extra=collections_abc.Mapping):
-        __slots__ = ()
-else:
-    class Mapping(Sized, Iterable[KT], Container[KT], Generic[KT, VT_co],
-                  extra=collections_abc.Mapping):
-        __slots__ = ()
-
-
-class MutableMapping(Mapping[KT, VT], extra=collections_abc.MutableMapping):
-    __slots__ = ()
-
-
-if hasattr(collections_abc, 'Reversible'):
-    if hasattr(collections_abc, 'Collection'):
-        class Sequence(Reversible[T_co], Collection[T_co],
-                       extra=collections_abc.Sequence):
-            __slots__ = ()
-    else:
-        class Sequence(Sized, Reversible[T_co], Container[T_co],
-                       extra=collections_abc.Sequence):
-            __slots__ = ()
-else:
-    class Sequence(Sized, Iterable[T_co], Container[T_co],
-                   extra=collections_abc.Sequence):
-        __slots__ = ()
-
-
-class MutableSequence(Sequence[T], extra=collections_abc.MutableSequence):
-    __slots__ = ()
-
-
-class ByteString(Sequence[int], extra=collections_abc.ByteString):
-    __slots__ = ()
-
-
-class List(list, MutableSequence[T], extra=list):
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is List:
-            raise TypeError("Type List cannot be instantiated; "
-                            "use list() instead")
-        return _generic_new(list, cls, *args, **kwds)
-
-
-class Deque(collections.deque, MutableSequence[T], extra=collections.deque):
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Deque:
-            return collections.deque(*args, **kwds)
-        return _generic_new(collections.deque, cls, *args, **kwds)
-
-
-class Set(set, MutableSet[T], extra=set):
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Set:
-            raise TypeError("Type Set cannot be instantiated; "
-                            "use set() instead")
-        return _generic_new(set, cls, *args, **kwds)
-
-
-class FrozenSet(frozenset, AbstractSet[T_co], extra=frozenset):
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is FrozenSet:
-            raise TypeError("Type FrozenSet cannot be instantiated; "
-                            "use frozenset() instead")
-        return _generic_new(frozenset, cls, *args, **kwds)
-
-
-class MappingView(Sized, Iterable[T_co], extra=collections_abc.MappingView):
-    __slots__ = ()
-
-
-class KeysView(MappingView[KT], AbstractSet[KT],
-               extra=collections_abc.KeysView):
-    __slots__ = ()
-
-
-class ItemsView(MappingView[Tuple[KT, VT_co]],
-                AbstractSet[Tuple[KT, VT_co]],
-                Generic[KT, VT_co],
-                extra=collections_abc.ItemsView):
-    __slots__ = ()
-
-
-class ValuesView(MappingView[VT_co], extra=collections_abc.ValuesView):
-    __slots__ = ()
-
-
-if hasattr(contextlib, 'AbstractContextManager'):
-    class ContextManager(Generic[T_co], extra=contextlib.AbstractContextManager):
-        __slots__ = ()
-else:
-    class ContextManager(Generic[T_co]):
-        __slots__ = ()
-
-        def __enter__(self):
-            return self
-
-        @abc.abstractmethod
-        def __exit__(self, exc_type, exc_value, traceback):
-            return None
-
-        @classmethod
-        def __subclasshook__(cls, C):
-            if cls is ContextManager:
-                # In Python 3.6+, it is possible to set a method to None to
-                # explicitly indicate that the class does not implement an ABC
-                # (https://bugs.python.org/issue25958), but we do not support
-                # that pattern here because this fallback class is only used
-                # in Python 3.5 and earlier.
-                if (any("__enter__" in B.__dict__ for B in C.__mro__) and
-                    any("__exit__" in B.__dict__ for B in C.__mro__)):
-                    return True
-            return NotImplemented
-
-
-if hasattr(contextlib, 'AbstractAsyncContextManager'):
-    class AsyncContextManager(Generic[T_co],
-                              extra=contextlib.AbstractAsyncContextManager):
-        __slots__ = ()
-
-    __all__.append('AsyncContextManager')
-elif sys.version_info[:2] >= (3, 5):
-    exec("""
-class AsyncContextManager(Generic[T_co]):
-    __slots__ = ()
-
-    async def __aenter__(self):
-        return self
-
-    @abc.abstractmethod
-    async def __aexit__(self, exc_type, exc_value, traceback):
-        return None
-
-    @classmethod
-    def __subclasshook__(cls, C):
-        if cls is AsyncContextManager:
-            if sys.version_info[:2] >= (3, 6):
-                return _collections_abc._check_methods(C, "__aenter__", "__aexit__")
-            if (any("__aenter__" in B.__dict__ for B in C.__mro__) and
-                    any("__aexit__" in B.__dict__ for B in C.__mro__)):
-                return True
-        return NotImplemented
-
-__all__.append('AsyncContextManager')
-""")
-
-
-class Dict(dict, MutableMapping[KT, VT], extra=dict):
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Dict:
-            raise TypeError("Type Dict cannot be instantiated; "
-                            "use dict() instead")
-        return _generic_new(dict, cls, *args, **kwds)
-
-
-class DefaultDict(collections.defaultdict, MutableMapping[KT, VT],
-                  extra=collections.defaultdict):
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is DefaultDict:
-            return collections.defaultdict(*args, **kwds)
-        return _generic_new(collections.defaultdict, cls, *args, **kwds)
-
-
-class Counter(collections.Counter, Dict[T, int], extra=collections.Counter):
-
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Counter:
-            return collections.Counter(*args, **kwds)
-        return _generic_new(collections.Counter, cls, *args, **kwds)
-
-
-if hasattr(collections, 'ChainMap'):
-    # ChainMap only exists in 3.3+
-    __all__.append('ChainMap')
-
-    class ChainMap(collections.ChainMap, MutableMapping[KT, VT],
-                   extra=collections.ChainMap):
-
-        __slots__ = ()
-
-        def __new__(cls, *args, **kwds):
-            if cls._gorg is ChainMap:
-                return collections.ChainMap(*args, **kwds)
-            return _generic_new(collections.ChainMap, cls, *args, **kwds)
-
-
-# Determine what base class to use for Generator.
-if hasattr(collections_abc, 'Generator'):
-    # Sufficiently recent versions of 3.5 have a Generator ABC.
-    _G_base = collections_abc.Generator
-else:
-    # Fall back on the exact type.
-    _G_base = types.GeneratorType
-
-
-class Generator(Iterator[T_co], Generic[T_co, T_contra, V_co],
-                extra=_G_base):
-    __slots__ = ()
-
-    def __new__(cls, *args, **kwds):
-        if cls._gorg is Generator:
-            raise TypeError("Type Generator cannot be instantiated; "
-                            "create a subclass instead")
-        return _generic_new(_G_base, cls, *args, **kwds)
-
-
-if hasattr(collections_abc, 'AsyncGenerator'):
-    class AsyncGenerator(AsyncIterator[T_co], Generic[T_co, T_contra],
-                         extra=collections_abc.AsyncGenerator):
-        __slots__ = ()
-
-    __all__.append('AsyncGenerator')
-
-
-# Internal type variable used for Type[].
-CT_co = TypeVar('CT_co', covariant=True, bound=type)
-
-
-# This is not a real generic class.  Don't use outside annotations.
-class Type(Generic[CT_co], extra=type):
-    """A special construct usable to annotate class objects.
-
-    For example, suppose we have the following classes::
-
-      class User: ...  # Abstract base for User classes
-      class BasicUser(User): ...
-      class ProUser(User): ...
-      class TeamUser(User): ...
-
-    And a function that takes a class argument that's a subclass of
-    User and returns an instance of the corresponding class::
-
-      U = TypeVar('U', bound=User)
-      def new_user(user_class: Type[U]) -> U:
-          user = user_class()
-          # (Here we could write the user object to a database)
-          return user
-
-      joe = new_user(BasicUser)
-
-    At this point the type checker knows that joe has type BasicUser.
-    """
-
-    __slots__ = ()
-
-
 def _make_nmtuple(name, types):
     msg = "NamedTuple('Name', [(f0, t0), (f1, t1), ...]); each t must be a type"
     types = [(n, _type_check(t, msg)) for n, t in types]
@@ -2114,8 +1319,6 @@ def _make_nmtuple(name, types):
     return nm_tpl
 
 
-_PY36 = sys.version_info[:2] >= (3, 6)
-
 # attributes prohibited to set in NamedTuple class syntax
 _prohibited = ('__new__', '__init__', '__slots__', '__getnewargs__',
                '_fields', '_field_defaults', '_field_types',
@@ -2129,9 +1332,6 @@ class NamedTupleMeta(type):
     def __new__(cls, typename, bases, ns):
         if ns.get('_root', False):
             return super().__new__(cls, typename, bases, ns)
-        if not _PY36:
-            raise TypeError("Class syntax for NamedTuple is only supported"
-                            " in Python 3.6+")
         types = ns.get('__annotations__', {})
         nm_tpl = _make_nmtuple(typename, types.items())
         defaults = []
@@ -2186,9 +1386,6 @@ class Employee(NamedTuple):
     _root = True
 
     def __new__(self, typename, fields=None, **kwargs):
-        if kwargs and not _PY36:
-            raise TypeError("Keyword syntax for NamedTuple is only supported"
-                            " in Python 3.6+")
         if fields is None:
             fields = kwargs.items()
         elif kwargs:
@@ -2384,12 +1581,8 @@ class io:
 io.__name__ = __name__ + '.io'
 sys.modules[io.__name__] = io
 
-
-Pattern = _TypeAlias('Pattern', AnyStr, type(stdlib_re.compile('')),
-                     lambda p: p.pattern)
-Match = _TypeAlias('Match', AnyStr, type(stdlib_re.match('', '')),
-                   lambda m: m.re.pattern)
-
+Pattern = _alias(stdlib_re.Pattern, AnyStr)
+Match = _alias(stdlib_re.Match, AnyStr)
 
 class re:
     """Wrapper namespace for re type aliases."""


From webhook-mailer at python.org  Sat Jan 20 07:19:24 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 20 Jan 2018 12:19:24 -0000
Subject: [Python-checkins] bpo-32598: Use autoconf to detect usable OpenSSL
 (#5242)
Message-ID: <mailman.67.1516450766.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/ff5be6e8100276647e0077e80869fc022d1bb53f
commit: ff5be6e8100276647e0077e80869fc022d1bb53f
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-20T13:19:21+01:00
summary:

bpo-32598: Use autoconf to detect usable OpenSSL (#5242)

Add https://www.gnu.org/software/autoconf-archive/ax_check_openssl.html
to auto-detect compiler flags, linker flags and libraries to compile
OpenSSL extensions. The M4 macro uses pkg-config and falls back to
manual detection.

Add autoconf magic to detect usable X509_VERIFY_PARAM_set1_host()
and related functions.

Refactor setup.py to use new config vars to compile _ssl and _hashlib
modules.

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Build/2018-01-19-14-50-19.bpo-32598.hP7bMV.rst
A m4/ax_check_openssl.m4
M Makefile.pre.in
M Modules/_ssl.c
M aclocal.m4
M configure
M configure.ac
M pyconfig.h.in
M setup.py

diff --git a/Makefile.pre.in b/Makefile.pre.in
index 994e8611e13..d1512671735 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -181,6 +181,11 @@ RUNSHARED=       @RUNSHARED@
 # ensurepip options
 ENSUREPIP=      @ENSUREPIP@
 
+# OpenSSL options for setup.py so sysconfig can pick up AC_SUBST() vars.
+OPENSSL_INCLUDES=@OPENSSL_INCLUDES@
+OPENSSL_LIBS=@OPENSSL_LIBS@
+OPENSSL_LDFLAGS=@OPENSSL_LDFLAGS@
+
 # Modes for directories, executables and data files created by the
 # install process.  Default to user-only-writable for all file types.
 DIRMODE=	755
diff --git a/Misc/NEWS.d/next/Build/2018-01-19-14-50-19.bpo-32598.hP7bMV.rst b/Misc/NEWS.d/next/Build/2018-01-19-14-50-19.bpo-32598.hP7bMV.rst
new file mode 100644
index 00000000000..4b05e73feef
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2018-01-19-14-50-19.bpo-32598.hP7bMV.rst
@@ -0,0 +1,3 @@
+Use autoconf to detect OpenSSL libs, headers and supported features. The
+ax_check_openssl M4 macro uses pkg-config to locate OpenSSL and falls back
+to manual search.
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index 380dd1b72db..c5eec7eded6 100644
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -64,6 +64,13 @@ static PySocketModule_APIObject PySocketModule;
 #include "openssl/rand.h"
 #include "openssl/bio.h"
 
+/* Set HAVE_X509_VERIFY_PARAM_SET1_HOST for non-autoconf builds */
+#ifndef HAVE_X509_VERIFY_PARAM_SET1_HOST
+#  if !defined(LIBRESSL_VERSION_NUMBER) && OPENSSL_VERSION_NUMBER > 0x1000200fL
+#    define HAVE_X509_VERIFY_PARAM_SET1_HOST
+#  endif
+#endif
+
 /* SSL error object */
 static PyObject *PySSLErrorObject;
 static PyObject *PySSLCertVerificationErrorObject;
diff --git a/aclocal.m4 b/aclocal.m4
index 61b2539d2e9..09694962914 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -12,9 +12,9 @@
 # PARTICULAR PURPOSE.
 
 m4_ifndef([AC_CONFIG_MACRO_DIRS], [m4_defun([_AM_CONFIG_MACRO_DIRS], [])m4_defun([AC_CONFIG_MACRO_DIRS], [_AM_CONFIG_MACRO_DIRS($@)])])
-dnl pkg.m4 - Macros to locate and utilise pkg-config.   -*- Autoconf -*-
-dnl serial 11 (pkg-config-0.29)
-dnl
+# pkg.m4 - Macros to locate and utilise pkg-config.   -*- Autoconf -*-
+# serial 11 (pkg-config-0.29.1)
+
 dnl Copyright ? 2004 Scott James Remnant <scott at netsplit.com>.
 dnl Copyright ? 2012-2015 Dan Nicholson <dbn.lists at gmail.com>
 dnl
@@ -55,7 +55,7 @@ dnl
 dnl See the "Since" comment for each macro you use to see what version
 dnl of the macros you require.
 m4_defun([PKG_PREREQ],
-[m4_define([PKG_MACROS_VERSION], [0.29])
+[m4_define([PKG_MACROS_VERSION], [0.29.1])
 m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1,
     [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])])
 ])dnl PKG_PREREQ
@@ -288,3 +288,72 @@ AS_VAR_COPY([$1], [pkg_cv_][$1])
 AS_VAR_IF([$1], [""], [$5], [$4])dnl
 ])dnl PKG_CHECK_VAR
 
+dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES,
+dnl   [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND],
+dnl   [DESCRIPTION], [DEFAULT])
+dnl ------------------------------------------
+dnl
+dnl Prepare a "--with-" configure option using the lowercase
+dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and
+dnl PKG_CHECK_MODULES in a single macro.
+AC_DEFUN([PKG_WITH_MODULES],
+[
+m4_pushdef([with_arg], m4_tolower([$1]))
+
+m4_pushdef([description],
+           [m4_default([$5], [build with ]with_arg[ support])])
+
+m4_pushdef([def_arg], [m4_default([$6], [auto])])
+m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes])
+m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no])
+
+m4_case(def_arg,
+            [yes],[m4_pushdef([with_without], [--without-]with_arg)],
+            [m4_pushdef([with_without],[--with-]with_arg)])
+
+AC_ARG_WITH(with_arg,
+     AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),,
+    [AS_TR_SH([with_]with_arg)=def_arg])
+
+AS_CASE([$AS_TR_SH([with_]with_arg)],
+            [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)],
+            [auto],[PKG_CHECK_MODULES([$1],[$2],
+                                        [m4_n([def_action_if_found]) $3],
+                                        [m4_n([def_action_if_not_found]) $4])])
+
+m4_popdef([with_arg])
+m4_popdef([description])
+m4_popdef([def_arg])
+
+])dnl PKG_WITH_MODULES
+
+dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES,
+dnl   [DESCRIPTION], [DEFAULT])
+dnl -----------------------------------------------
+dnl
+dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES
+dnl check._[VARIABLE-PREFIX] is exported as make variable.
+AC_DEFUN([PKG_HAVE_WITH_MODULES],
+[
+PKG_WITH_MODULES([$1],[$2],,,[$3],[$4])
+
+AM_CONDITIONAL([HAVE_][$1],
+               [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"])
+])dnl PKG_HAVE_WITH_MODULES
+
+dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES,
+dnl   [DESCRIPTION], [DEFAULT])
+dnl ------------------------------------------------------
+dnl
+dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after
+dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make
+dnl and preprocessor variable.
+AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES],
+[
+PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4])
+
+AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"],
+        [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])])
+])dnl PKG_HAVE_DEFINE_WITH_MODULES
+
+m4_include([m4/ax_check_openssl.m4])
diff --git a/configure b/configure
index 9286443124a..420c07639b5 100755
--- a/configure
+++ b/configure
@@ -623,6 +623,9 @@ ac_includes_default="\
 #endif"
 
 ac_subst_vars='LTLIBOBJS
+OPENSSL_LDFLAGS
+OPENSSL_LIBS
+OPENSSL_INCLUDES
 ENSUREPIP
 SRCDIRS
 THREADHEADERS
@@ -778,7 +781,6 @@ infodir
 docdir
 oldincludedir
 includedir
-runstatedir
 localstatedir
 sharedstatedir
 sysconfdir
@@ -837,6 +839,7 @@ with_libc
 enable_big_digits
 with_computed_gotos
 with_ensurepip
+with_openssl
 '
       ac_precious_vars='build_alias
 host_alias
@@ -889,7 +892,6 @@ datadir='${datarootdir}'
 sysconfdir='${prefix}/etc'
 sharedstatedir='${prefix}/com'
 localstatedir='${prefix}/var'
-runstatedir='${localstatedir}/run'
 includedir='${prefix}/include'
 oldincludedir='/usr/include'
 docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1142,15 +1144,6 @@ do
   | -silent | --silent | --silen | --sile | --sil)
     silent=yes ;;
 
-  -runstatedir | --runstatedir | --runstatedi | --runstated \
-  | --runstate | --runstat | --runsta | --runst | --runs \
-  | --run | --ru | --r)
-    ac_prev=runstatedir ;;
-  -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
-  | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
-  | --run=* | --ru=* | --r=*)
-    runstatedir=$ac_optarg ;;
-
   -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
     ac_prev=sbindir ;;
   -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1288,7 +1281,7 @@ fi
 for ac_var in	exec_prefix prefix bindir sbindir libexecdir datarootdir \
 		datadir sysconfdir sharedstatedir localstatedir includedir \
 		oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
-		libdir localedir mandir runstatedir
+		libdir localedir mandir
 do
   eval ac_val=\$$ac_var
   # Remove trailing slashes.
@@ -1441,7 +1434,6 @@ Fine tuning of the installation directories:
   --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
   --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
   --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
-  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
   --libdir=DIR            object code libraries [EPREFIX/lib]
   --includedir=DIR        C header files [PREFIX/include]
   --oldincludedir=DIR     C header files for non-gcc [/usr/include]
@@ -1545,6 +1537,7 @@ Optional Packages:
                           default on supported compilers)
   --with(out)-ensurepip=[=upgrade]
                           "install" or "upgrade" using bundled pip
+  --with-openssl=DIR      root of the OpenSSL directory
 
 Some influential environment variables:
   MACHDEP     name for machine-dependent library files
@@ -2687,6 +2680,8 @@ ac_compiler_gnu=$ac_cv_c_compiler_gnu
 
 
 
+
+
 if test "$srcdir" != . -a "$srcdir" != "$(pwd)"; then
     # If we're building out-of-tree, we need to make sure the following
     # resources get picked up before their $srcdir counterparts.
@@ -16681,6 +16676,265 @@ $as_echo "#define HAVE_GETRANDOM 1" >>confdefs.h
 
 fi
 
+# Check for usable OpenSSL
+
+    found=false
+
+# Check whether --with-openssl was given.
+if test "${with_openssl+set}" = set; then :
+  withval=$with_openssl;
+            case "$withval" in
+            "" | y | ye | yes | n | no)
+            as_fn_error $? "Invalid --with-openssl value" "$LINENO" 5
+              ;;
+            *) ssldirs="$withval"
+              ;;
+            esac
+
+else
+
+            # if pkg-config is installed and openssl has installed a .pc file,
+            # then use that information and don't search ssldirs
+            if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}pkg-config", so it can be a program name with args.
+set dummy ${ac_tool_prefix}pkg-config; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_prog_PKG_CONFIG+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  if test -n "$PKG_CONFIG"; then
+  ac_cv_prog_PKG_CONFIG="$PKG_CONFIG" # Let the user override the test.
+else
+as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+  IFS=$as_save_IFS
+  test -z "$as_dir" && as_dir=.
+    for ac_exec_ext in '' $ac_executable_extensions; do
+  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+    ac_cv_prog_PKG_CONFIG="${ac_tool_prefix}pkg-config"
+    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+    break 2
+  fi
+done
+  done
+IFS=$as_save_IFS
+
+fi
+fi
+PKG_CONFIG=$ac_cv_prog_PKG_CONFIG
+if test -n "$PKG_CONFIG"; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PKG_CONFIG" >&5
+$as_echo "$PKG_CONFIG" >&6; }
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+fi
+if test -z "$ac_cv_prog_PKG_CONFIG"; then
+  ac_ct_PKG_CONFIG=$PKG_CONFIG
+  # Extract the first word of "pkg-config", so it can be a program name with args.
+set dummy pkg-config; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_prog_ac_ct_PKG_CONFIG+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  if test -n "$ac_ct_PKG_CONFIG"; then
+  ac_cv_prog_ac_ct_PKG_CONFIG="$ac_ct_PKG_CONFIG" # Let the user override the test.
+else
+as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+  IFS=$as_save_IFS
+  test -z "$as_dir" && as_dir=.
+    for ac_exec_ext in '' $ac_executable_extensions; do
+  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+    ac_cv_prog_ac_ct_PKG_CONFIG="pkg-config"
+    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+    break 2
+  fi
+done
+  done
+IFS=$as_save_IFS
+
+fi
+fi
+ac_ct_PKG_CONFIG=$ac_cv_prog_ac_ct_PKG_CONFIG
+if test -n "$ac_ct_PKG_CONFIG"; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_ct_PKG_CONFIG" >&5
+$as_echo "$ac_ct_PKG_CONFIG" >&6; }
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+  if test "x$ac_ct_PKG_CONFIG" = x; then
+    PKG_CONFIG=""
+  else
+    case $cross_compiling:$ac_tool_warned in
+yes:)
+{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5
+$as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;}
+ac_tool_warned=yes ;;
+esac
+    PKG_CONFIG=$ac_ct_PKG_CONFIG
+  fi
+else
+  PKG_CONFIG="$ac_cv_prog_PKG_CONFIG"
+fi
+
+            if test x"$PKG_CONFIG" != x""; then
+                OPENSSL_LDFLAGS=`$PKG_CONFIG openssl --libs-only-L 2>/dev/null`
+                if test $? = 0; then
+                    OPENSSL_LIBS=`$PKG_CONFIG openssl --libs-only-l 2>/dev/null`
+                    OPENSSL_INCLUDES=`$PKG_CONFIG openssl --cflags-only-I 2>/dev/null`
+                    found=true
+                fi
+            fi
+
+            # no such luck; use some default ssldirs
+            if ! $found; then
+                ssldirs="/usr/local/ssl /usr/lib/ssl /usr/ssl /usr/pkg /usr/local /usr"
+            fi
+
+
+fi
+
+
+
+    # note that we #include <openssl/foo.h>, so the OpenSSL headers have to be in
+    # an 'openssl' subdirectory
+
+    if ! $found; then
+        OPENSSL_INCLUDES=
+        for ssldir in $ssldirs; do
+            { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openssl/ssl.h in $ssldir" >&5
+$as_echo_n "checking for openssl/ssl.h in $ssldir... " >&6; }
+            if test -f "$ssldir/include/openssl/ssl.h"; then
+                OPENSSL_INCLUDES="-I$ssldir/include"
+                OPENSSL_LDFLAGS="-L$ssldir/lib"
+                OPENSSL_LIBS="-lssl -lcrypto"
+                found=true
+                { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+                break
+            else
+                { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+            fi
+        done
+
+        # if the file wasn't found, well, go ahead and try the link anyway -- maybe
+        # it will just work!
+    fi
+
+    # try the preprocessor and linker with our new flags,
+    # being careful not to pollute the global LIBS, LDFLAGS, and CPPFLAGS
+
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether compiling and linking against OpenSSL works" >&5
+$as_echo_n "checking whether compiling and linking against OpenSSL works... " >&6; }
+    echo "Trying link with OPENSSL_LDFLAGS=$OPENSSL_LDFLAGS;" \
+        "OPENSSL_LIBS=$OPENSSL_LIBS; OPENSSL_INCLUDES=$OPENSSL_INCLUDES" >&5
+
+    save_LIBS="$LIBS"
+    save_LDFLAGS="$LDFLAGS"
+    save_CPPFLAGS="$CPPFLAGS"
+    LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS"
+    LIBS="$OPENSSL_LIBS $LIBS"
+    CPPFLAGS="$OPENSSL_INCLUDES $CPPFLAGS"
+    cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+#include <openssl/ssl.h>
+int
+main ()
+{
+SSL_new(NULL)
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+
+            { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+            have_openssl=yes
+
+else
+
+            { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+            have_openssl=no
+
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+    CPPFLAGS="$save_CPPFLAGS"
+    LDFLAGS="$save_LDFLAGS"
+    LIBS="$save_LIBS"
+
+
+
+
+
+
+if test "$have_openssl" = yes; then
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for X509_VERIFY_PARAM_set1_host in libssl" >&5
+$as_echo_n "checking for X509_VERIFY_PARAM_set1_host in libssl... " >&6; }
+
+    save_LIBS="$LIBS"
+    save_LDFLAGS="$LDFLAGS"
+    save_CPPFLAGS="$CPPFLAGS"
+    LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS"
+    LIBS="$OPENSSL_LIBS $LIBS"
+    CPPFLAGS="$OPENSSL_INCLUDES $CPPFLAGS"
+
+    cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+        #include <openssl/x509_vfy.h>
+
+int
+main ()
+{
+
+        X509_VERIFY_PARAM *p = X509_VERIFY_PARAM_new();
+        X509_VERIFY_PARAM_set1_host(p, "localhost", 0);
+        X509_VERIFY_PARAM_set1_ip_asc(p, "127.0.0.1");
+        X509_VERIFY_PARAM_set_hostflags(p, 0);
+
+  ;
+  return 0;
+}
+
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+
+        ac_cv_has_x509_verify_param_set1_host=yes
+
+else
+
+        ac_cv_has_x509_verify_param_set1_host=no
+
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+    { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_has_x509_verify_param_set1_host" >&5
+$as_echo "$ac_cv_has_x509_verify_param_set1_host" >&6; }
+    if test "$ac_cv_has_x509_verify_param_set1_host" = "yes"; then
+
+$as_echo "#define HAVE_X509_VERIFY_PARAM_SET1_HOST 1" >>confdefs.h
+
+    fi
+
+    CPPFLAGS="$save_CPPFLAGS"
+    LDFLAGS="$save_LDFLAGS"
+    LIBS="$save_LIBS"
+fi
+
 # generate output files
 ac_config_files="$ac_config_files Makefile.pre Misc/python.pc Misc/python-config.sh"
 
diff --git a/configure.ac b/configure.ac
index 1894e21f304..39e2e8e769e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -9,6 +9,8 @@ AC_PREREQ(2.65)
 
 AC_INIT(python, PYTHON_VERSION, https://bugs.python.org/)
 
+AC_CONFIG_MACRO_DIR(m4)
+
 AC_SUBST(BASECPPFLAGS)
 if test "$srcdir" != . -a "$srcdir" != "$(pwd)"; then
     # If we're building out-of-tree, we need to make sure the following
@@ -5460,6 +5462,45 @@ if test "$have_getrandom" = yes; then
               [Define to 1 if the getrandom() function is available])
 fi
 
+# Check for usable OpenSSL
+AX_CHECK_OPENSSL([have_openssl=yes],[have_openssl=no])
+
+if test "$have_openssl" = yes; then
+    AC_MSG_CHECKING([for X509_VERIFY_PARAM_set1_host in libssl])
+
+    save_LIBS="$LIBS"
+    save_LDFLAGS="$LDFLAGS"
+    save_CPPFLAGS="$CPPFLAGS"
+    LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS"
+    LIBS="$OPENSSL_LIBS $LIBS"
+    CPPFLAGS="$OPENSSL_INCLUDES $CPPFLAGS"
+
+    AC_LINK_IFELSE([AC_LANG_PROGRAM([
+        [#include <openssl/x509_vfy.h>]
+    ], [
+        [X509_VERIFY_PARAM *p = X509_VERIFY_PARAM_new();]
+        [X509_VERIFY_PARAM_set1_host(p, "localhost", 0);]
+        [X509_VERIFY_PARAM_set1_ip_asc(p, "127.0.0.1");]
+        [X509_VERIFY_PARAM_set_hostflags(p, 0);]
+    ])
+    ],
+    [
+        ac_cv_has_x509_verify_param_set1_host=yes
+    ],
+    [
+        ac_cv_has_x509_verify_param_set1_host=no
+    ])
+    AC_MSG_RESULT($ac_cv_has_x509_verify_param_set1_host)
+    if test "$ac_cv_has_x509_verify_param_set1_host" = "yes"; then
+        AC_DEFINE(HAVE_X509_VERIFY_PARAM_SET1_HOST, 1,
+        [Define if libssl has X509_VERIFY_PARAM_set1_host and related function])
+    fi
+
+    CPPFLAGS="$save_CPPFLAGS"
+    LDFLAGS="$save_LDFLAGS"
+    LIBS="$save_LIBS"
+fi
+
 # generate output files
 AC_CONFIG_FILES(Makefile.pre Misc/python.pc Misc/python-config.sh)
 AC_CONFIG_FILES([Modules/ld_so_aix], [chmod +x Modules/ld_so_aix])
diff --git a/m4/ax_check_openssl.m4 b/m4/ax_check_openssl.m4
new file mode 100644
index 00000000000..28e48cbefb6
--- /dev/null
+++ b/m4/ax_check_openssl.m4
@@ -0,0 +1,124 @@
+# ===========================================================================
+#     https://www.gnu.org/software/autoconf-archive/ax_check_openssl.html
+# ===========================================================================
+#
+# SYNOPSIS
+#
+#   AX_CHECK_OPENSSL([action-if-found[, action-if-not-found]])
+#
+# DESCRIPTION
+#
+#   Look for OpenSSL in a number of default spots, or in a user-selected
+#   spot (via --with-openssl).  Sets
+#
+#     OPENSSL_INCLUDES to the include directives required
+#     OPENSSL_LIBS to the -l directives required
+#     OPENSSL_LDFLAGS to the -L or -R flags required
+#
+#   and calls ACTION-IF-FOUND or ACTION-IF-NOT-FOUND appropriately
+#
+#   This macro sets OPENSSL_INCLUDES such that source files should use the
+#   openssl/ directory in include directives:
+#
+#     #include <openssl/hmac.h>
+#
+# LICENSE
+#
+#   Copyright (c) 2009,2010 Zmanda Inc. <http://www.zmanda.com/>
+#   Copyright (c) 2009,2010 Dustin J. Mitchell <dustin at zmanda.com>
+#
+#   Copying and distribution of this file, with or without modification, are
+#   permitted in any medium without royalty provided the copyright notice
+#   and this notice are preserved. This file is offered as-is, without any
+#   warranty.
+
+#serial 10
+
+AU_ALIAS([CHECK_SSL], [AX_CHECK_OPENSSL])
+AC_DEFUN([AX_CHECK_OPENSSL], [
+    found=false
+    AC_ARG_WITH([openssl],
+        [AS_HELP_STRING([--with-openssl=DIR],
+            [root of the OpenSSL directory])],
+        [
+            case "$withval" in
+            "" | y | ye | yes | n | no)
+            AC_MSG_ERROR([Invalid --with-openssl value])
+              ;;
+            *) ssldirs="$withval"
+              ;;
+            esac
+        ], [
+            # if pkg-config is installed and openssl has installed a .pc file,
+            # then use that information and don't search ssldirs
+            AC_CHECK_TOOL([PKG_CONFIG], [pkg-config])
+            if test x"$PKG_CONFIG" != x""; then
+                OPENSSL_LDFLAGS=`$PKG_CONFIG openssl --libs-only-L 2>/dev/null`
+                if test $? = 0; then
+                    OPENSSL_LIBS=`$PKG_CONFIG openssl --libs-only-l 2>/dev/null`
+                    OPENSSL_INCLUDES=`$PKG_CONFIG openssl --cflags-only-I 2>/dev/null`
+                    found=true
+                fi
+            fi
+
+            # no such luck; use some default ssldirs
+            if ! $found; then
+                ssldirs="/usr/local/ssl /usr/lib/ssl /usr/ssl /usr/pkg /usr/local /usr"
+            fi
+        ]
+        )
+
+
+    # note that we #include <openssl/foo.h>, so the OpenSSL headers have to be in
+    # an 'openssl' subdirectory
+
+    if ! $found; then
+        OPENSSL_INCLUDES=
+        for ssldir in $ssldirs; do
+            AC_MSG_CHECKING([for openssl/ssl.h in $ssldir])
+            if test -f "$ssldir/include/openssl/ssl.h"; then
+                OPENSSL_INCLUDES="-I$ssldir/include"
+                OPENSSL_LDFLAGS="-L$ssldir/lib"
+                OPENSSL_LIBS="-lssl -lcrypto"
+                found=true
+                AC_MSG_RESULT([yes])
+                break
+            else
+                AC_MSG_RESULT([no])
+            fi
+        done
+
+        # if the file wasn't found, well, go ahead and try the link anyway -- maybe
+        # it will just work!
+    fi
+
+    # try the preprocessor and linker with our new flags,
+    # being careful not to pollute the global LIBS, LDFLAGS, and CPPFLAGS
+
+    AC_MSG_CHECKING([whether compiling and linking against OpenSSL works])
+    echo "Trying link with OPENSSL_LDFLAGS=$OPENSSL_LDFLAGS;" \
+        "OPENSSL_LIBS=$OPENSSL_LIBS; OPENSSL_INCLUDES=$OPENSSL_INCLUDES" >&AS_MESSAGE_LOG_FD
+
+    save_LIBS="$LIBS"
+    save_LDFLAGS="$LDFLAGS"
+    save_CPPFLAGS="$CPPFLAGS"
+    LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS"
+    LIBS="$OPENSSL_LIBS $LIBS"
+    CPPFLAGS="$OPENSSL_INCLUDES $CPPFLAGS"
+    AC_LINK_IFELSE(
+        [AC_LANG_PROGRAM([#include <openssl/ssl.h>], [SSL_new(NULL)])],
+        [
+            AC_MSG_RESULT([yes])
+            $1
+        ], [
+            AC_MSG_RESULT([no])
+            $2
+        ])
+    CPPFLAGS="$save_CPPFLAGS"
+    LDFLAGS="$save_LDFLAGS"
+    LIBS="$save_LIBS"
+
+    AC_SUBST([OPENSSL_INCLUDES])
+    AC_SUBST([OPENSSL_LIBS])
+    AC_SUBST([OPENSSL_LDFLAGS])
+])
diff --git a/pyconfig.h.in b/pyconfig.h.in
index ff1083ae0b0..dd7c62bad1b 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -1237,6 +1237,9 @@
 /* Define to 1 if you have the `writev' function. */
 #undef HAVE_WRITEV
 
+/* Define if libssl has X509_VERIFY_PARAM_set1_host and related function */
+#undef HAVE_X509_VERIFY_PARAM_SET1_HOST
+
 /* Define if the zlib library has inflateCopy */
 #undef HAVE_ZLIB_COPY
 
diff --git a/setup.py b/setup.py
index 8e98fdcb7ef..c23628a2a94 100644
--- a/setup.py
+++ b/setup.py
@@ -860,74 +860,15 @@ def detect_modules(self):
         exts.append( Extension('_socket', ['socketmodule.c'],
                                depends = ['socketmodule.h']) )
         # Detect SSL support for the socket module (via _ssl)
-        search_for_ssl_incs_in = [
-                              '/usr/local/ssl/include',
-                              '/usr/contrib/ssl/include/'
-                             ]
-        ssl_incs = find_file('openssl/ssl.h', inc_dirs,
-                             search_for_ssl_incs_in
-                             )
-        if ssl_incs is not None:
-            krb5_h = find_file('krb5.h', inc_dirs,
-                               ['/usr/kerberos/include'])
-            if krb5_h:
-                ssl_incs += krb5_h
-        ssl_libs = find_library_file(self.compiler, 'ssl',lib_dirs,
-                                     ['/usr/local/ssl/lib',
-                                      '/usr/contrib/ssl/lib/'
-                                     ] )
-
-        if (ssl_incs is not None and
-            ssl_libs is not None):
-            exts.append( Extension('_ssl', ['_ssl.c'],
-                                   include_dirs = ssl_incs,
-                                   library_dirs = ssl_libs,
-                                   libraries = ['ssl', 'crypto'],
-                                   depends = ['socketmodule.h']), )
+        ssl_ext, hashlib_ext = self._detect_openssl(inc_dirs, lib_dirs)
+        if ssl_ext is not None:
+            exts.append(ssl_ext)
         else:
             missing.append('_ssl')
-
-        # find out which version of OpenSSL we have
-        openssl_ver = 0
-        openssl_ver_re = re.compile(
-            r'^\s*#\s*define\s+OPENSSL_VERSION_NUMBER\s+(0x[0-9a-fA-F]+)' )
-
-        # look for the openssl version header on the compiler search path.
-        opensslv_h = find_file('openssl/opensslv.h', [],
-                inc_dirs + search_for_ssl_incs_in)
-        if opensslv_h:
-            name = os.path.join(opensslv_h[0], 'openssl/opensslv.h')
-            if host_platform == 'darwin' and is_macosx_sdk_path(name):
-                name = os.path.join(macosx_sdk_root(), name[1:])
-            try:
-                with open(name, 'r') as incfile:
-                    for line in incfile:
-                        m = openssl_ver_re.match(line)
-                        if m:
-                            openssl_ver = int(m.group(1), 16)
-                            break
-            except IOError as msg:
-                print("IOError while reading opensshv.h:", msg)
-
-        #print('openssl_ver = 0x%08x' % openssl_ver)
-        min_openssl_ver = 0x00907000
-        have_any_openssl = ssl_incs is not None and ssl_libs is not None
-        have_usable_openssl = (have_any_openssl and
-                               openssl_ver >= min_openssl_ver)
-
-        if have_any_openssl:
-            if have_usable_openssl:
-                # The _hashlib module wraps optimized implementations
-                # of hash functions from the OpenSSL library.
-                exts.append( Extension('_hashlib', ['_hashopenssl.c'],
-                                       depends = ['hashlib.h'],
-                                       include_dirs = ssl_incs,
-                                       library_dirs = ssl_libs,
-                                       libraries = ['ssl', 'crypto']) )
-            else:
-                print("warning: openssl 0x%08x is too old for _hashlib" %
-                      openssl_ver)
-                missing.append('_hashlib')
+        if hashlib_ext is not None:
+            exts.append(hashlib_ext)
+        else:
+            missing.append('_hashlib')
 
         # We always compile these even when OpenSSL is available (issue #14693).
         # It's harmless and the object code is tiny (40-50 KiB per module,
@@ -2183,6 +2124,61 @@ def _decimal_ext(self):
         )
         return ext
 
+    def _detect_openssl(self, inc_dirs, lib_dirs):
+        config_vars = sysconfig.get_config_vars()
+
+        def split_var(name, sep):
+            # poor man's shlex, the re module is not available yet.
+            value = config_vars.get(name)
+            if not value:
+                return ()
+            # This trick works because ax_check_openssl uses --libs-only-L,
+            # --libs-only-l, and --cflags-only-I.
+            value = ' ' + value
+            sep = ' ' + sep
+            return [v.strip() for v in value.split(sep) if v.strip()]
+
+        openssl_includes = split_var('OPENSSL_INCLUDES', '-I')
+        openssl_libdirs = split_var('OPENSSL_LDFLAGS', '-L')
+        openssl_libs = split_var('OPENSSL_LIBS', '-l')
+        if not openssl_libs:
+            # libssl and libcrypto not found
+            return None, None
+
+        # Find OpenSSL includes
+        ssl_incs = find_file(
+            'openssl/ssl.h', inc_dirs, openssl_includes
+        )
+        if ssl_incs is None:
+            return None, None
+
+        # OpenSSL 1.0.2 uses Kerberos for KRB5 ciphers
+        krb5_h = find_file(
+            'krb5.h', inc_dirs,
+            ['/usr/kerberos/include']
+        )
+        if krb5_h:
+            ssl_incs.extend(krb5_h)
+
+        ssl_ext = Extension(
+            '_ssl', ['_ssl.c'],
+            include_dirs=openssl_includes,
+            library_dirs=openssl_libdirs,
+            libraries=openssl_libs,
+            depends=['socketmodule.h']
+        )
+
+        hashlib_ext = Extension(
+            '_hashlib', ['_hashopenssl.c'],
+            depends=['hashlib.h'],
+            include_dirs=openssl_includes,
+            library_dirs=openssl_libdirs,
+            libraries=openssl_libs,
+        )
+
+        return ssl_ext, hashlib_ext
+
+
 class PyBuildInstall(install):
     # Suppress the warning about installation into the lib_dynload
     # directory, which is not in sys.path when running Python during


From webhook-mailer at python.org  Sat Jan 20 09:16:33 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 20 Jan 2018 14:16:33 -0000
Subject: [Python-checkins] bpo-32602: Test ECDSA certs (#5247)
Message-ID: <mailman.68.1516457795.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/bd5c7d238c01b90fbfae8ea45b47bd601900abaf
commit: bd5c7d238c01b90fbfae8ea45b47bd601900abaf
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-20T15:16:30+01:00
summary:

bpo-32602: Test ECDSA certs (#5247)

Add test certs and test for ECDSA cert and EC/RSA dual mode.

I'm also adding certs for IDNA 2003/2008 tests and simplify some test
data handling.

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Lib/test/idnsans.pem
A Lib/test/keycertecc.pem
A Lib/test/secp384r1.pem
A Misc/NEWS.d/next/Tests/2018-01-19-20-47-11.bpo-32602.dz41pq.rst
M Lib/test/allsans.pem
M Lib/test/capath/b1930218.0
M Lib/test/capath/ceff1710.0
M Lib/test/keycert.passwd.pem
M Lib/test/keycert.pem
M Lib/test/keycert2.pem
M Lib/test/keycert3.pem
M Lib/test/keycert4.pem
M Lib/test/make_ssl_certs.py
M Lib/test/pycacert.pem
M Lib/test/pycakey.pem
M Lib/test/revocation.crl
M Lib/test/ssl_cert.pem
M Lib/test/ssl_key.passwd.pem
M Lib/test/ssl_key.pem
M Lib/test/test_asyncio/utils.py
M Lib/test/test_ssl.py

diff --git a/Lib/test/allsans.pem b/Lib/test/allsans.pem
index 3ee4f59513a..bf59f30abaa 100644
--- a/Lib/test/allsans.pem
+++ b/Lib/test/allsans.pem
@@ -1,37 +1,64 @@
 -----BEGIN PRIVATE KEY-----
-MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAOoy7/QOtTjQ0niE
-6uDcTwtkC0R2Tvy1AjVnXohCntZfdzbTGDoYTgXSOLsP8A697jUiJ8VCePGH50xG
-Z4DKnAF3a9O3a9nr2pLXb0iY3XOMv+YEBii7CfI+3oxFYgCl0sMgHzDD2ZTVYAsm
-DWgLUVsE2gHEccRwrM2tPf2EgR+FAgMBAAECgYEA3qyfyYVSeTrTYxO93x6ZaVMu
-A2IZp9zSxMQL9bKiI2GRj+cV2ebSCGbg2btFnD6qBor7FWsmYz+8g6FNN/9sY4az
-61rMqMtQvLBe+7L8w70FeTze4qQ4Y1oQri0qD6tBWhDVlpnbI5Py9bkZKD67yVUk
-elcEA/5x4PrYXkuqsAECQQD80NjT0mDvaY0JOOaQFSEpMv6QiUA8GGX8Xli7IoKb
-tAolPG8rQBa+qSpcWfDMTrWw/aWHuMEEQoP/bVDH9W4FAkEA7SYQbBAKnojZ5A3G
-kOHdV7aeivRQxQk/JN8Fb8oKB9Csvpv/BsuGxPKXHdhFa6CBTTsNRtHQw/szPo4l
-xMIjgQJAPoMxqibR+0EBM6+TKzteSL6oPXsCnBl4Vk/J5vPgkbmR7KUl4+7j8N8J
-b2554TrxKEN/w7CGYZRE6UrRd7ATNQJAWD7Yz41sli+wfPdPU2xo1BHljyl4wMk/
-EPZYbI/PCbdyAH/F935WyQTIjNeEhZc1Zkq6FwdOWw8ns3hrv3rKgQJAHXv1BqUa
-czGPIFxX2TNoqtcl6/En4vrxVB1wzsfzkkDAg98kBl7qsF+S3qujSzKikjeaVbI2
-/CyWR2P3yLtOmA==
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQD59JyhPgYe7nhZ
+Z2IGhaklNgtRkD+5BVs7lEWovYRBlXpPA6PuaHat25rI8EGYHmlufPherg2Qu6sC
+GmZZKo7TgjlmDcwVS4hkebtFH7OZy5Il7Y2ZIdiK7Xp9Z0EPoqwacYowB0a8WhZY
+I2Vm4EzCNKl6/htkwjgn2JXGizxvGt/1kNqP/GBAX+vjgeahOsn8jVh96KpFHJbS
+g83cX4t8M7FJv7yNoDLvORHnvKCOXbQmr6ZMGcZN8PwS8awQ31khZTpEx+hCe+Pi
+GzeOlxpZimXWDAGWA4tZ58Ka/QvO7VQbD5Ci166ODvvs+tEXfBUExtPcS+02IBJV
+tzhBna9VAgMBAAECggEAPar9DccIqY76QEyCYcuOPLEFv9zP6+0HYj6lpQkE3U1s
+vJvQURyS0zgQCy1Dca1nI6xPdsSIckHq4fzzbWJTlJlXYfdbd5GIGAn0iwxUOkiA
+ST0/px0zmKsYgmH8KkhfH7MNfeX9rLCpPJuXA/eo2G03tzGEPqqwQhxsb2ygv2Qs
+M7OqJz6RJu87K1Y+psWIv9+VhNVja0kvsg52QMK9mtp8layb54qLI5R5e09sIudq
+RHegtnSOBo9kt32H9vWUFaF5PpYt4yks4KYI4ulKGWJGXHMDW4uHUaE/tjNQuYAX
+DuDvjN+ECSJvigiUbu2k0xB2KYIb1fpcxlz/YBdADQKBgQD/Z2VtBUjOFnJKz00f
+xN0akp7XPgd1yCb1/wZq9PQiGvzIAMDIplioTvjOjhOzPJaWD0GICNeypzQ48+0P
+UsPIKbazpIZN6bZncr65plSpg0KANq46hbkPHOo8PHDa7yoxBUSPr8F7P1OCRkn6
++QdgcnrAly7yfqO2ahAWOX7iCwKBgQD6ifXSCKfRF1GUb3Ws7S1rLxeBasWq+EmC
+sUnck0S+AyaMkN+kZ5zejbN+NDuUMQ7+3wUIheTclUhzR0LP3+r5jjHsimJuvOml
+wuV37F+Om5lD/Xx27NfbtRKn/bK6o0zDL8JB2eFB0N7Fh7hRYoUMdrpQs5sU91IC
+pNYlAcLwHwKBgGvLK9eTf2LbvmksjRR3dgodD8UwfN2NGESC2iaSM+ehFEclajhF
+XO3MRt6GwHHJhJTY44OSl9bjEvtmmAr7l34HfQDc04JWvZFzsGOSe/D/YTXT3jz8
+61ohjgrWR5tfjaMa4hDy0Oo/k/NLzzWJnT9rkbtvE3VtVZNLuHZo1dB5AoGBAMHO
+wStV6MO1nzUNN+Gqo8zbY/qIJxsH8I26KaIJBk9azpJEa8yZHl+HDEffjgsoHCqL
+STB7qzv7+0y53nRCClo8ZmBN+LEjUDcbWjl3z7/YnCpdR9ATjTP3kdQETCNWucXw
+Bvy72CX6tqnlQG8soDGxEpXlKl2AqJ9E9icwgqUPAoGAL6xTDdgcYTbk9wxCd41l
+NhHTSvLrGXLAzv61PCnlOJEJbuuezb2VW0ibsud5CA4Mi0tf9ET790XSOFd5nCjQ
+6rr06AkjQsoFvjL1dO9EzVFPW0JrZ3C9y8ZOjdeAfPEmFL2T6VqmQ+IcCUNhSr39
+NBdKrboEFfnKanfbstekhAs=
 -----END PRIVATE KEY-----
 -----BEGIN CERTIFICATE-----
-MIIDcjCCAtugAwIBAgIJAN5dc9TOWjB7MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV
+MIIGMDCCBRigAwIBAgIJAJYf8T95ptq5MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV
 BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u
-IFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnMwHhcNMTYwODA1
-MTAyMTExWhcNMjYwODAzMTAyMTExWjBdMQswCQYDVQQGEwJYWTEXMBUGA1UEBwwO
+IFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnMwHhcNMTgwMTE5
+MTkwOTA3WhcNMjgwMTE3MTkwOTA3WjBdMQswCQYDVQQGEwJYWTEXMBUGA1UEBwwO
 Q2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0
-aW9uMRAwDgYDVQQDDAdhbGxzYW5zMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
-gQDqMu/0DrU40NJ4hOrg3E8LZAtEdk78tQI1Z16IQp7WX3c20xg6GE4F0ji7D/AO
-ve41IifFQnjxh+dMRmeAypwBd2vTt2vZ69qS129ImN1zjL/mBAYouwnyPt6MRWIA
-pdLDIB8ww9mU1WALJg1oC1FbBNoBxHHEcKzNrT39hIEfhQIDAQABo4IBODCCATQw
-ggEwBgNVHREEggEnMIIBI4IHYWxsc2Fuc6AeBgMqAwSgFwwVc29tZSBvdGhlciBp
-ZGVudGlmaWVyoDUGBisGAQUCAqArMCmgEBsOS0VSQkVST1MuUkVBTE2hFTAToAMC
-AQGhDDAKGwh1c2VybmFtZYEQdXNlckBleGFtcGxlLm9yZ4IPd3d3LmV4YW1wbGUu
-b3JnpGcwZTELMAkGA1UEBhMCWFkxFzAVBgNVBAcMDkNhc3RsZSBBbnRocmF4MSMw
-IQYDVQQKDBpQeXRob24gU29mdHdhcmUgRm91bmRhdGlvbjEYMBYGA1UEAwwPZGly
-bmFtZSBleGFtcGxlhhdodHRwczovL3d3dy5weXRob24ub3JnL4cEfwAAAYcQAAAA
-AAAAAAAAAAAAAAAAAYgEKgMEBTANBgkqhkiG9w0BAQsFAAOBgQAy16h+F+nOmeiT
-VWR0fc8F/j6FcadbLseAUaogcC15OGxCl4UYpLV88HBkABOoGCpP155qwWTwOrdG
-iYPGJSusf1OnJEbvzFejZf6u078bPd9/ZL4VWLjv+FPGkjd+N+/OaqMvgj8Lu99f
-3Y/C4S7YbHxxwff6C6l2Xli+q6gnuQ==
+aW9uMRAwDgYDVQQDDAdhbGxzYW5zMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
+CgKCAQEA+fScoT4GHu54WWdiBoWpJTYLUZA/uQVbO5RFqL2EQZV6TwOj7mh2rdua
+yPBBmB5pbnz4Xq4NkLurAhpmWSqO04I5Zg3MFUuIZHm7RR+zmcuSJe2NmSHYiu16
+fWdBD6KsGnGKMAdGvFoWWCNlZuBMwjSpev4bZMI4J9iVxos8bxrf9ZDaj/xgQF/r
+44HmoTrJ/I1YfeiqRRyW0oPN3F+LfDOxSb+8jaAy7zkR57ygjl20Jq+mTBnGTfD8
+EvGsEN9ZIWU6RMfoQnvj4hs3jpcaWYpl1gwBlgOLWefCmv0Lzu1UGw+Qoteujg77
+7PrRF3wVBMbT3EvtNiASVbc4QZ2vVQIDAQABo4IC8TCCAu0wggEwBgNVHREEggEn
+MIIBI4IHYWxsc2Fuc6AeBgMqAwSgFwwVc29tZSBvdGhlciBpZGVudGlmaWVyoDUG
+BisGAQUCAqArMCmgEBsOS0VSQkVST1MuUkVBTE2hFTAToAMCAQGhDDAKGwh1c2Vy
+bmFtZYEQdXNlckBleGFtcGxlLm9yZ4IPd3d3LmV4YW1wbGUub3JnpGcwZTELMAkG
+A1UEBhMCWFkxFzAVBgNVBAcMDkNhc3RsZSBBbnRocmF4MSMwIQYDVQQKDBpQeXRo
+b24gU29mdHdhcmUgRm91bmRhdGlvbjEYMBYGA1UEAwwPZGlybmFtZSBleGFtcGxl
+hhdodHRwczovL3d3dy5weXRob24ub3JnL4cEfwAAAYcQAAAAAAAAAAAAAAAAAAAA
+AYgEKgMEBTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
+AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFH9ye3+WhBnHqNhtFu059bzY
+SWM8MIGPBgNVHSMEgYcwgYSAFH9ye3+WhBnHqNhtFu059bzYSWM8oWGkXzBdMQsw
+CQYDVQQGEwJYWTEXMBUGA1UEBwwOQ2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5
+dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMRAwDgYDVQQDDAdhbGxzYW5zggkAlh/x
+P3mm2rkwgYMGCCsGAQUFBwEBBHcwdTA8BggrBgEFBQcwAoYwaHR0cDovL3Rlc3Rj
+YS5weXRob250ZXN0Lm5ldC90ZXN0Y2EvcHljYWNlcnQuY2VyMDUGCCsGAQUFBzAB
+hilodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3RjYS9vY3NwLzBDBgNV
+HR8EPDA6MDigNqA0hjJodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3Rj
+YS9yZXZvY2F0aW9uLmNybDANBgkqhkiG9w0BAQsFAAOCAQEAYwYJcerUPvnsP7e2
+HGp/It0OZ8Cvpt8Qf7A+NSPvJqkyKakl8zK/50iq/qQKH09CnfEae4rfXLdlYsvV
+2PZYK0LDWnyTcHSJWAVJjlSFIFt3ig9FdHv9GYtSWWod66cZ0sEZOoF2IHZUGby+
+Qa+JQpmv5jEuGIZzjcsh6hSOou8ph7LsCsRdVlQqk8rM97vB7DAgh01vedlbolsq
+JxsuPRydNFV/eWq3AgAWgZL3LdYYIAgaVOTnnd3xARw8DlT1q6+Lzc71GBXrRZYh
+qgd+xC/K1812gMPImTX02bxpkhCuIdVd7cztWi8sdQmSgDEFdYMXo4NzlFTK8dlC
+Y4wa3Q==
 -----END CERTIFICATE-----
diff --git a/Lib/test/capath/b1930218.0 b/Lib/test/capath/b1930218.0
index 373349cae05..07556ff9071 100644
--- a/Lib/test/capath/b1930218.0
+++ b/Lib/test/capath/b1930218.0
@@ -1,21 +1,21 @@
 -----BEGIN CERTIFICATE-----
-MIIDbTCCAlWgAwIBAgIJALCSZLHy2iHQMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+MIIDbTCCAlWgAwIBAgIJAILtv0HIgJGbMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
 BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
-MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xMzAxMDQxOTQ3MDdaFw0yMzAxMDIx
-OTQ3MDdaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yODAxMTcx
+OTA5MDZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
 Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI
-hvcNAQEBBQADggEPADCCAQoCggEBAOfe6eMMnwC2of0rW5bSb8zgvoa5IF7sA3pV
-q+qk6flJhdJm1e3HeupWji2P50LiYiipn9Ybjuu1tJyfFKvf5pSLdh0+bSRh7Qy/
-AIphDN9cyDZzFgDNR7ptpKR0iIMjChn8Cac8SkvT5x0t5OpMVCHzJtuJNxjUArtA
-Ml+k/y0c99S77I7PXIKs5nwIbEiFYQd/JeBc4Lw0X+C5BEd1yEcLjbzWyGhfM4Ni
-0iBENbGtgRqKzbw1sFyLR9YY6ZwYl8wBPCnM6B7k5MG43ufCERiHWpM02KYl9xRx
-6+QhotIPLi7UYgA109bvXGBLTKkU4t0VWEY3Mya35y5d7ULkxU0CAwEAAaNQME4w
-HQYDVR0OBBYEFLzdYtl22hvSVGvP4GabHh57VgwLMB8GA1UdIwQYMBaAFLzdYtl2
-2hvSVGvP4GabHh57VgwLMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
-AH0K9cuN0129mY74Kw+668LZpidPLnsvDmTYHDVQTu78kLmNbajFxgawr/Mtvzu4
-QgfdGH1tlVRXhRhgRy/reBv56Bf9Wg2HFyisTGrmvCn09FVwKULeheqrbCMGZDB1
-Ao5TvF4BMzfMHs24pP3K5F9lO4MchvFVAqA6j9uRt0AUtOeN0u5zuuPlNC28lG9O
-JAb3X4sOp45r3l519DKaULFEM5rQBeJ4gv/b2opj66nd0b+gYa3jnookXWIO50yR
-f+/fNDY7L131hLIvxG2TlhpvMCjx2hKaZLRAMx293itTqOq+1rxOlvVE+zIYrtUf
-9mmvtk57HVjsO6lTo15YyJ4=
+hvcNAQEBBQADggEPADCCAQoCggEBAMMYaWvJRymYjrFWwi76Dl68I4CzB2Ik0kJb
+8Uq/qcghdcjj5iwfhzxufBvtOTKVt0CyYEjDmhYI/m1niDQ7d3dwHHBa0R9fBCFU
+uQzjQYUdWO4v7fMO79gjofpz+0wo4OXmTQsCUkmGx75+veZWdotwjgqPBjMgHXtb
+qtDFG6ubzFQJPL/kQGbx+9b3Fp3EGdTD8v8HvG9aniUbAkql7EKWOnDSbJkrzr7o
+0gHv1bqwz5Q+gtAB1ktxgAMKEkWGeYHYS9LotbcsbJpMihAQ5OT1386EkcrRRuCE
+cxdm22lDeICDvhRN8T4a1mz13kXzOa+R1T1URL9BzHNoGvwk25ECAwEAAaNQME4w
+HQYDVR0OBBYEFJrPz27rcT3bPPGuiGtWcgPLCKdIMB8GA1UdIwQYMBaAFJrPz27r
+cT3bPPGuiGtWcgPLCKdIMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
+ABAlyNwMVVzLg25573fsDY4MBsFLDNb3dVIhuBdKOIids3jEQvu4fBQ4EPus2hEA
+W0KHXkWfbU5CpJoYBjkPRaaWiTLWWbPTjuOVtsSiS3QvZ8H7u/lybzdK5/RIM3Hf
+uPXmQT/V1S8mCfgOkv9w6varWPuQBNZDLo+x+waradDcqPhbB/LUZh9j+F3BnkFE
+u8nofeBG5KfIMl8xYuUcXIndt6JPng0TuF+xhFNMH84Z4QEAXr9BVZSppRPb8lnz
+1k65nZ25CtmyGG18sfeWqr32+ZUPSm48fEZb39R47Jrc4uMB5oh3OZOcuipj+SVL
+T6wIeTnGe98HNbrAwlC/WoE=
 -----END CERTIFICATE-----
diff --git a/Lib/test/capath/ceff1710.0 b/Lib/test/capath/ceff1710.0
index 373349cae05..07556ff9071 100644
--- a/Lib/test/capath/ceff1710.0
+++ b/Lib/test/capath/ceff1710.0
@@ -1,21 +1,21 @@
 -----BEGIN CERTIFICATE-----
-MIIDbTCCAlWgAwIBAgIJALCSZLHy2iHQMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+MIIDbTCCAlWgAwIBAgIJAILtv0HIgJGbMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
 BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
-MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xMzAxMDQxOTQ3MDdaFw0yMzAxMDIx
-OTQ3MDdaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yODAxMTcx
+OTA5MDZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
 Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI
-hvcNAQEBBQADggEPADCCAQoCggEBAOfe6eMMnwC2of0rW5bSb8zgvoa5IF7sA3pV
-q+qk6flJhdJm1e3HeupWji2P50LiYiipn9Ybjuu1tJyfFKvf5pSLdh0+bSRh7Qy/
-AIphDN9cyDZzFgDNR7ptpKR0iIMjChn8Cac8SkvT5x0t5OpMVCHzJtuJNxjUArtA
-Ml+k/y0c99S77I7PXIKs5nwIbEiFYQd/JeBc4Lw0X+C5BEd1yEcLjbzWyGhfM4Ni
-0iBENbGtgRqKzbw1sFyLR9YY6ZwYl8wBPCnM6B7k5MG43ufCERiHWpM02KYl9xRx
-6+QhotIPLi7UYgA109bvXGBLTKkU4t0VWEY3Mya35y5d7ULkxU0CAwEAAaNQME4w
-HQYDVR0OBBYEFLzdYtl22hvSVGvP4GabHh57VgwLMB8GA1UdIwQYMBaAFLzdYtl2
-2hvSVGvP4GabHh57VgwLMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
-AH0K9cuN0129mY74Kw+668LZpidPLnsvDmTYHDVQTu78kLmNbajFxgawr/Mtvzu4
-QgfdGH1tlVRXhRhgRy/reBv56Bf9Wg2HFyisTGrmvCn09FVwKULeheqrbCMGZDB1
-Ao5TvF4BMzfMHs24pP3K5F9lO4MchvFVAqA6j9uRt0AUtOeN0u5zuuPlNC28lG9O
-JAb3X4sOp45r3l519DKaULFEM5rQBeJ4gv/b2opj66nd0b+gYa3jnookXWIO50yR
-f+/fNDY7L131hLIvxG2TlhpvMCjx2hKaZLRAMx293itTqOq+1rxOlvVE+zIYrtUf
-9mmvtk57HVjsO6lTo15YyJ4=
+hvcNAQEBBQADggEPADCCAQoCggEBAMMYaWvJRymYjrFWwi76Dl68I4CzB2Ik0kJb
+8Uq/qcghdcjj5iwfhzxufBvtOTKVt0CyYEjDmhYI/m1niDQ7d3dwHHBa0R9fBCFU
+uQzjQYUdWO4v7fMO79gjofpz+0wo4OXmTQsCUkmGx75+veZWdotwjgqPBjMgHXtb
+qtDFG6ubzFQJPL/kQGbx+9b3Fp3EGdTD8v8HvG9aniUbAkql7EKWOnDSbJkrzr7o
+0gHv1bqwz5Q+gtAB1ktxgAMKEkWGeYHYS9LotbcsbJpMihAQ5OT1386EkcrRRuCE
+cxdm22lDeICDvhRN8T4a1mz13kXzOa+R1T1URL9BzHNoGvwk25ECAwEAAaNQME4w
+HQYDVR0OBBYEFJrPz27rcT3bPPGuiGtWcgPLCKdIMB8GA1UdIwQYMBaAFJrPz27r
+cT3bPPGuiGtWcgPLCKdIMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
+ABAlyNwMVVzLg25573fsDY4MBsFLDNb3dVIhuBdKOIids3jEQvu4fBQ4EPus2hEA
+W0KHXkWfbU5CpJoYBjkPRaaWiTLWWbPTjuOVtsSiS3QvZ8H7u/lybzdK5/RIM3Hf
+uPXmQT/V1S8mCfgOkv9w6varWPuQBNZDLo+x+waradDcqPhbB/LUZh9j+F3BnkFE
+u8nofeBG5KfIMl8xYuUcXIndt6JPng0TuF+xhFNMH84Z4QEAXr9BVZSppRPb8lnz
+1k65nZ25CtmyGG18sfeWqr32+ZUPSm48fEZb39R47Jrc4uMB5oh3OZOcuipj+SVL
+T6wIeTnGe98HNbrAwlC/WoE=
 -----END CERTIFICATE-----
diff --git a/Lib/test/idnsans.pem b/Lib/test/idnsans.pem
new file mode 100644
index 00000000000..b4a771ce7ec
--- /dev/null
+++ b/Lib/test/idnsans.pem
@@ -0,0 +1,136 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDeU8YQtyeEjPLA
+SdMBPTS9QcuAZIJjbJRgr8nsRb767pbmWR9C1JuDy/Bz/AprFC6Om950fLn3pOqR
+zDUWZX/qTe+o27i8u0Qzk06bhRkxAdTEoTfRcH/FkJaimJqeTt9rZqc+AGSNKM8o
+4GyPW4IELnavmMB30+7rKIJMIpIn1a1k6MybJYdWNSuVqwArAVvRlj5qOiqX7KAS
+otFRP8pz+Lgw3qREQzgnZz/bcScKd+5Uy4qMFPNOMjgW6nDV60ekNx0GT+59E/+8
+64GRq34rNVu2SN0XXcQh33R3LwwrvAdymaLyr1YyIRM5gLPxugxCIA0SYjG0YoGB
+uUSwtNa7AgMBAAECggEBAJjxUGPXW1wYCja1km5byJgZVwEwI3J6E2igBWyAXm0J
+DM3RqWu0DneQKA3h6NjYvV5lY5cG5nex/5vkuvB5SpHIo4GqBV/wA27ne0AJQ9cu
+x0utDFUL6xnh6X5ZNKSK5a9gotRIOOPSmxAnswa7kKmHvSX3ExBbvxQOffQaJCk5
+0GHl6I/HltqVzMu4ICAo0NY0gw1n+hVKTo28KkJ9PL7X6v6H5yvZ3L6TkMytSvqf
+9iVlYuIN66ToBtxaI4g2RiUJtA2hdT9IP7Wg4YD6Ptyih90zXz2wTzWppFem6UA9
+dePig94R9moj9ucuK0tx3kSATNo0op/XEx1e3OOtcQECgYEA/w7pNOPYgj7VMyYx
+p4Lx4BOllzQts8mIBtUVZVQSJ2miun6DTalZVT2V3ayTuE0qhUHd1SHu9F77a9fQ
+qaSUUY9elwXyfvcNCfhYVRJxyxirI4Z6ZCBwjpWOGSBB59NTeDhVnbkTlfE6guqS
+3KRS1pfIQ6FCvGIrhjRZgHo1TGECgYEA3yXsospbOS7VeBj0UPSB87fp1QM+r48o
+RflIsRzdsN9Ka2j6EiYpgKdbgXr80vkctYTK0dT8jrFSk81Y932CZezH2IWo8Meo
+40qaFWMboNFBIC4yv6RSRxJMQfYsKnXC2trSnXH+qf55Trey4uZNMX7VJ+RFKExS
+ieSWSbTWmJsCgYEAzo3yyoRiiEf+PKgHulLPMtp2VddJ07m30WCrLR5CfWyM/l8K
+UtB8qg1v2s+x6aWEc9p9necXLwvkrNdgAqJoAw0KW1/TnILSKmrWjj6brRBTODfl
+0kR7It128F4xQV7g0BE/NLX3aIytB+yT9t+Uvni5FBv6gbk26j5m5ScTFsECgYEA
+hzrQYQcIqWq8av+Ub8r9Rdlal4BT6Mh0u5MKfmrj3mAzFUyU35LI6/J//cOum5vj
+zg0fbHIKa98CEBgNpk4lS+dmZMz7SI92xedb4UIiaB7nvLzCfGj0g6WPGRo6QbED
+2OVrZYbDsflJQm8ItYCjny8htf8b+gPmsTIZ8ajps6kCgYBnES8waDDAkL98lK28
+dcgnJXN+1UzeI6//If2uvDZEQ9tG/yMk2JYc84qZJLU5bRplMAjIQUVUcFWa+ZzV
+ylnDhagAtiWkHPcElWHym9dH8CRuYM3OTDsApZ7yMB/ArCcZMIA35OvNf6uc4lNV
+VD9VkaygPIg6ilv4npeTceqp8A==
+-----END PRIVATE KEY-----
+Certificate:
+    Data:
+        Version: 3 (0x2)
+        Serial Number:
+            82:ed:bf:41:c8:80:91:9f
+    Signature Algorithm: sha1WithRSAEncryption
+        Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server
+        Validity
+            Not Before: Jan 19 19:09:07 2018 GMT
+            Not After : Nov 28 19:09:07 2027 GMT
+        Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=idnsans
+        Subject Public Key Info:
+            Public Key Algorithm: rsaEncryption
+                Public-Key: (2048 bit)
+                Modulus:
+                    00:de:53:c6:10:b7:27:84:8c:f2:c0:49:d3:01:3d:
+                    34:bd:41:cb:80:64:82:63:6c:94:60:af:c9:ec:45:
+                    be:fa:ee:96:e6:59:1f:42:d4:9b:83:cb:f0:73:fc:
+                    0a:6b:14:2e:8e:9b:de:74:7c:b9:f7:a4:ea:91:cc:
+                    35:16:65:7f:ea:4d:ef:a8:db:b8:bc:bb:44:33:93:
+                    4e:9b:85:19:31:01:d4:c4:a1:37:d1:70:7f:c5:90:
+                    96:a2:98:9a:9e:4e:df:6b:66:a7:3e:00:64:8d:28:
+                    cf:28:e0:6c:8f:5b:82:04:2e:76:af:98:c0:77:d3:
+                    ee:eb:28:82:4c:22:92:27:d5:ad:64:e8:cc:9b:25:
+                    87:56:35:2b:95:ab:00:2b:01:5b:d1:96:3e:6a:3a:
+                    2a:97:ec:a0:12:a2:d1:51:3f:ca:73:f8:b8:30:de:
+                    a4:44:43:38:27:67:3f:db:71:27:0a:77:ee:54:cb:
+                    8a:8c:14:f3:4e:32:38:16:ea:70:d5:eb:47:a4:37:
+                    1d:06:4f:ee:7d:13:ff:bc:eb:81:91:ab:7e:2b:35:
+                    5b:b6:48:dd:17:5d:c4:21:df:74:77:2f:0c:2b:bc:
+                    07:72:99:a2:f2:af:56:32:21:13:39:80:b3:f1:ba:
+                    0c:42:20:0d:12:62:31:b4:62:81:81:b9:44:b0:b4:
+                    d6:bb
+                Exponent: 65537 (0x10001)
+        X509v3 extensions:
+            X509v3 Subject Alternative Name: 
+                DNS:idnsans, DNS:xn--knig-5qa.idn.pythontest.net, DNS:xn--knigsgsschen-lcb0w.idna2003.pythontest.net, DNS:xn--knigsgchen-b4a3dun.idna2008.pythontest.net, DNS:xn--nxasmq6b.idna2003.pythontest.net, DNS:xn--nxasmm1c.idna2008.pythontest.net
+            X509v3 Key Usage: critical
+                Digital Signature, Key Encipherment
+            X509v3 Extended Key Usage: 
+                TLS Web Server Authentication, TLS Web Client Authentication
+            X509v3 Basic Constraints: critical
+                CA:FALSE
+            X509v3 Subject Key Identifier: 
+                3B:F0:22:A0:1E:9B:CE:2A:7C:AE:B1:32:1B:B0:8E:3E:33:40:E3:FA
+            X509v3 Authority Key Identifier: 
+                keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48
+                DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server
+                serial:82:ED:BF:41:C8:80:91:9B
+
+            Authority Information Access: 
+                CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer
+                OCSP - URI:http://testca.pythontest.net/testca/ocsp/
+
+            X509v3 CRL Distribution Points: 
+
+                Full Name:
+                  URI:http://testca.pythontest.net/testca/revocation.crl
+
+    Signature Algorithm: sha1WithRSAEncryption
+         8b:1f:d7:e4:0d:15:76:b4:f5:87:33:de:b9:84:9b:f2:c1:9b:
+         c9:97:50:f7:18:33:ed:b7:60:83:be:bb:94:1c:49:39:ae:54:
+         24:43:f7:85:d8:2a:8c:26:17:56:1e:a6:b7:63:c5:05:f1:6e:
+         f4:79:eb:fd:af:12:84:3c:28:4a:8f:b1:01:97:91:ba:18:2b:
+         ba:54:25:49:1b:5b:2e:1e:6b:33:2d:f5:07:2e:76:04:e0:a8:
+         95:25:3f:cc:c8:26:c0:30:b6:90:d2:2b:e1:e2:13:b0:a8:76:
+         f0:06:90:b9:d5:28:6b:8a:e9:72:1a:ed:4f:7e:3c:37:2e:00:
+         aa:9b:f1:29:44:94:f2:dc:c8:31:5f:4c:2d:00:d3:5e:78:6c:
+         68:fc:0e:1e:46:be:d8:2e:29:88:78:8e:7e:f5:50:c8:5c:5d:
+         5f:4c:09:d5:51:07:40:be:9b:30:ed:a3:29:68:25:6b:88:69:
+         c7:43:35:54:2f:6e:9a:30:f1:d6:87:54:84:20:ef:a5:aa:33:
+         df:00:6a:87:a9:b4:d7:89:1f:e7:60:0d:01:60:66:11:61:3f:
+         d0:9f:86:37:cc:b3:b8:48:7e:1f:d2:7a:0f:02:e7:11:1d:dd:
+         34:c4:0b:45:47:2b:05:37:dd:ee:6e:0e:1c:bd:de:24:42:50:
+         a4:07:af:e5
+-----BEGIN CERTIFICATE-----
+MIIFvTCCBKWgAwIBAgIJAILtv0HIgJGfMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDdaFw0yNzExMjgx
+OTA5MDdaMF0xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj
+MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2lk
+bnNhbnMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDeU8YQtyeEjPLA
+SdMBPTS9QcuAZIJjbJRgr8nsRb767pbmWR9C1JuDy/Bz/AprFC6Om950fLn3pOqR
+zDUWZX/qTe+o27i8u0Qzk06bhRkxAdTEoTfRcH/FkJaimJqeTt9rZqc+AGSNKM8o
+4GyPW4IELnavmMB30+7rKIJMIpIn1a1k6MybJYdWNSuVqwArAVvRlj5qOiqX7KAS
+otFRP8pz+Lgw3qREQzgnZz/bcScKd+5Uy4qMFPNOMjgW6nDV60ekNx0GT+59E/+8
+64GRq34rNVu2SN0XXcQh33R3LwwrvAdymaLyr1YyIRM5gLPxugxCIA0SYjG0YoGB
+uUSwtNa7AgMBAAGjggKOMIICijCB4QYDVR0RBIHZMIHWggdpZG5zYW5zgh94bi0t
+a25pZy01cWEuaWRuLnB5dGhvbnRlc3QubmV0gi54bi0ta25pZ3Nnc3NjaGVuLWxj
+YjB3LmlkbmEyMDAzLnB5dGhvbnRlc3QubmV0gi54bi0ta25pZ3NnY2hlbi1iNGEz
+ZHVuLmlkbmEyMDA4LnB5dGhvbnRlc3QubmV0giR4bi0tbnhhc21xNmIuaWRuYTIw
+MDMucHl0aG9udGVzdC5uZXSCJHhuLS1ueGFzbW0xYy5pZG5hMjAwOC5weXRob250
+ZXN0Lm5ldDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
+AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFDvwIqAem84qfK6xMhuwjj4z
+QOP6MH0GA1UdIwR2MHSAFJrPz27rcT3bPPGuiGtWcgPLCKdIoVGkTzBNMQswCQYD
+VQQGEwJYWTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0Ex
+FjAUBgNVBAMMDW91ci1jYS1zZXJ2ZXKCCQCC7b9ByICRmzCBgwYIKwYBBQUHAQEE
+dzB1MDwGCCsGAQUFBzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rl
+c3RjYS9weWNhY2VydC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2EucHl0
+aG9udGVzdC5uZXQvdGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6
+Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3JsMA0G
+CSqGSIb3DQEBBQUAA4IBAQCLH9fkDRV2tPWHM965hJvywZvJl1D3GDPtt2CDvruU
+HEk5rlQkQ/eF2CqMJhdWHqa3Y8UF8W70eev9rxKEPChKj7EBl5G6GCu6VCVJG1su
+HmszLfUHLnYE4KiVJT/MyCbAMLaQ0ivh4hOwqHbwBpC51ShriulyGu1Pfjw3LgCq
+m/EpRJTy3MgxX0wtANNeeGxo/A4eRr7YLimIeI5+9VDIXF1fTAnVUQdAvpsw7aMp
+aCVriGnHQzVUL26aMPHWh1SEIO+lqjPfAGqHqbTXiR/nYA0BYGYRYT/Qn4Y3zLO4
+SH4f0noPAucRHd00xAtFRysFN93ubg4cvd4kQlCkB6/l
+-----END CERTIFICATE-----
diff --git a/Lib/test/keycert.passwd.pem b/Lib/test/keycert.passwd.pem
index e90574881db..0ad69605519 100644
--- a/Lib/test/keycert.passwd.pem
+++ b/Lib/test/keycert.passwd.pem
@@ -1,33 +1,50 @@
 -----BEGIN RSA PRIVATE KEY-----
 Proc-Type: 4,ENCRYPTED
-DEK-Info: DES-EDE3-CBC,1A8D9D2A02EC698A
+DEK-Info: DES-EDE3-CBC,E74528136B90D2DD
 
-kJYbfZ8L0sfe9Oty3gw0aloNnY5E8fegRfQLZlNoxTl6jNt0nIwI8kDJ36CZgR9c
-u3FDJm/KqrfUoz8vW+qEnWhSG7QPX2wWGPHd4K94Yz/FgrRzZ0DoK7XxXq9gOtVA
-AVGQhnz32p+6WhfGsCr9ArXEwRZrTk/FvzEPaU5fHcoSkrNVAGX8IpSVkSDwEDQr
-Gv17+cfk99UV1OCza6yKHoFkTtrC+PZU71LomBabivS2Oc4B9hYuSR2hF01wTHP+
-YlWNagZOOVtNz4oKK9x9eNQpmfQXQvPPTfusexKIbKfZrMvJoxcm1gfcZ0H/wK6P
-6wmXSG35qMOOztCZNtperjs1wzEBXznyK8QmLcAJBjkfarABJX9vBEzZV0OUKhy+
-noORFwHTllphbmydLhu6ehLUZMHPhzAS5UN7srtpSN81eerDMy0RMUAwA7/PofX1
-94Me85Q8jP0PC9ETdsJcPqLzAPETEYu0ELewKRcrdyWi+tlLFrpE5KT/s5ecbl9l
-7B61U4Kfd1PIXc/siINhU3A3bYK+845YyUArUOnKf1kEox7p1RpD7yFqVT04lRTo
-cibNKATBusXSuBrp2G6GNuhWEOSafWCKJQAzgCYIp6ZTV2khhMUGppc/2H3CF6cO
-zX0KtlPVZC7hLkB6HT8SxYUwF1zqWY7+/XPPdc37MeEZ87Q3UuZwqORLY+Z0hpgt
-L5JXBCoklZhCAaN2GqwFLXtGiRSRFGY7xXIhbDTlE65Wv1WGGgDLMKGE1gOz3yAo
-2jjG1+yAHJUdE69XTFHSqSkvaloA1W03LdMXZ9VuQJ/ySXCie6ABAQ==
+WRHVD2PJXPqjFSHg92HURIsUzvsTE4a9oi0SC5yMBFKNWA5Z933gK3XTifp6jul5
+zpNYi8jBXZ2EqJJBxCuVcefmXSxL0q7CMej25TdIC4BVAFJVveeprHPUFkNB0IM1
+go5Lg4YofYqTCg3OE3k7WvfR3Zg1cRYxksDKO+WNZgWyKBex5X4vjOiyUqDl3GKt
+kQXnkg1VgPV2Vrx93S9XNdELNRTguwf+XG0fkhtYhp/zCto8uKTgy5elK2P/ulGp
+7fe6uj7h/uN9L7EOC6CjRkitywfeBUER739mOcGT4imSFJ9G27TCqPzj2ea3uuaf
+/v1xhkQ4M6lNY/gcRfgVpCXhW43aAQV8XXQRMJTqLmz5Y5hYTKn+Ugq5vJ/ngyRM
+lu1gUJnYYaemBTb4hbm6aBvnYK9mORa891Pmf+vxU9rYuQIdVAhvvXh4KBreSEBI
+1AFy6dFKXl8ZKs6Wrq5wPefmFFkRmZ8OBiiq0fp2ApCRGZw6LsjCgwrRM38JiY7d
+3OdsJpKvRYufgUyuuzUE0xA+E4yMvD48M9pPq2fC8O5giuGL1uEekQWXJuq+6ZRI
+XYKIeSkuQALbX3RAzCPXTUEMtCYXKm/gxrrwJ+Bet4ob2amf3MX0uvWwOuAq++Fk
+J0HFSBxrwvIWOhyQXOPuJdAN8PXA7dWOXfOgOMF0hQYqZCl3T4TiVZJbwVQtg1sN
+dO7oAD5ZPHiKzvveZuB6k1FlBG8j0TyAC+44ChxkPDD3jF4dd6zGe62sDf85p4/d
+W80gxJeD3xnDxG0ePPns+GuKUpUaWS7WvHtDpeFW1JEhvOqf8p1Li9a7RzWVo8ML
+mGTdQgBIYIf6/fk69pFKl0nKtBU75KaunZz4nAmd9bNED4naDurMBg44u5TvODbJ
+vgYIYXIYjNvONbskJatVrrTS8zch2NwVIjCi8L/hecwBXbIXzo1pECpc6BU7sQT8
++i9sDKBeJcRipzfKZNHvnO19mUZaPCY8+a/f9c21DgKXz+bgLcJbohpSaeGM8Gfc
+aZd3Vp9n3OJ3g2zQR1++HO9v1vR/wLELu6MeydkvMduHLmOPCn54gZ9z51ZNPAwa
+qfFIsH+mLh9ks0H74ssF59uIlstkgB9zmZHv/Q0dK9ZfG/VEH6rSgdETWhZxhoMQ
+Z92jXBEFT0zhI3rrIPNY+XS7eJCQIc1wc84Ea3cRk7SP+S1og3JtAxX56ykUwtkM
+LQ/Dwwa6h1aqD0l2d5x1/BSdavtTuSegISRWQ4iOmSvEdlFP7H4g6RZk/okbLzMD
+Evq5gNc7vlXhVawoQU8JCanJ5ZbbWnIRZfiXxBQS4lpYPKvJt4ML9z/x+82XxcXv
+Z93N2Wep7wWW5OwS2LcQcOgZRDSIPompwo/0pMFGOS+5oort0ZDRHdmmGLjvBcCb
+1KQmKQ4+8brI/3rjRzts6uDLjTGNxSCieNsnqhwHUv9Mg9WDSWupcGa+x27L89x3
+rObf6+3umcFLSjIzU8wuv1hx/e/y98Kv7BDBNYpAr6kVMrLnzYjAfJbBmqlxkzkQ
+IgQzgrk2QZoTdgwR+S374NAMO0AE5IlO+/qa6qp2SORGTDX64I3UNw==
 -----END RSA PRIVATE KEY-----
 -----BEGIN CERTIFICATE-----
-MIICVDCCAb2gAwIBAgIJANfHOBkZr8JOMA0GCSqGSIb3DQEBBQUAMF8xCzAJBgNV
-BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u
-IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMDEw
-MDgyMzAxNTZaFw0yMDEwMDUyMzAxNTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH
-Ew5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZvdW5k
-YXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAw
-gYkCgYEA21vT5isq7F68amYuuNpSFlKDPrMUCa4YWYqZRt2OZ+/3NKaZ2xAiSwr7
-6MrQF70t5nLbSPpqE5+5VrS58SY+g/sXLiFd6AplH1wJZwh78DofbFYXUggktFMt
-pTyiX8jtP66bkcPkDADA089RI1TQR6Ca+n7HFa7c1fabVV6i3zkCAwEAAaMYMBYw
-FAYDVR0RBA0wC4IJbG9jYWxob3N0MA0GCSqGSIb3DQEBBQUAA4GBAHPctQBEQ4wd
-BJ6+JcpIraopLn8BGhbjNWj40mmRqWB/NAWF6M5ne7KpGAu7tLeG4hb1zLaldK8G
-lxy2GPSRF6LFS48dpEj2HbMv2nvv6xxalDMJ9+DicWgAKTQ6bcX2j3GUkCR0g/T1
-CRlNBAAlvhKzO7Clpf9l0YKBEfraJByX
+MIIDWTCCAkGgAwIBAgIJAPm6B21bar2bMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV
+BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u
+IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODAx
+MTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH
+DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k
+YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEP
+ADCCAQoCggEBAKvvsX2gEti4shve3iYMc+jE4Se7WHs1Bol2f21H8qNboDOFdeb1
+RKHjmq3exHpajywOUEgne9nKHJY/3f2phR4Y5klqG6liLgiSpVyRlcBGbeT2qEAj
+9oLiLFUXLGfGDds2mTwivQDLJBWi51j7ff5k2Pr58fN5ugYMn24T9FNyn0moT+qj
+SFoBNm58l9jrdkJSlgWfqPlbiMa+mqDn/SFtrwLF2Trbfzu42Sd9UdIzMaSSrzbN
+sGm53pNhCh8KndWUQ8GPP2IsLPoUU4qAtmZuTxCx2S1cXrN9EkmT69tlOH84YfSn
+96Ih9bWRc7M5y5bfVdEVM+fKQl3hBRf05qMCAwEAAaMYMBYwFAYDVR0RBA0wC4IJ
+bG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQAtQ8f37cCEk7/rAcbYR53ce3iK
+Vpihb0U2ni1QjG9Tg9UIExkIGkwTiCm7kwQL+GEStBu9AG/QVrOjeTriRiddhWkk
+ze8kRaI3AC/63t6Vh9Q1x6PESgeE4OtAO9JpJCf4GILglA789Y/b/GF8zJZQxR13
+qpB4ZwWw7gCBhdEW59u6CFeBmfDa58hM8lWvuVoRrTi7bjUeC6PAn5HVMzZSykhu
+4HaUfBp6bKFjuym2+h/VvM1n8C3chjVSmutsLb6ELdD8IK0vPV/yf5+LN256zSsS
+dyUZYd8XwQaioEMKdbhLvnehyzHiWfQIUR3BdhONxoIJhHv/EAo8eCkHHYIF
 -----END CERTIFICATE-----
diff --git a/Lib/test/keycert.pem b/Lib/test/keycert.pem
index 64318aa2e03..9545dcf4b94 100644
--- a/Lib/test/keycert.pem
+++ b/Lib/test/keycert.pem
@@ -1,31 +1,48 @@
 -----BEGIN PRIVATE KEY-----
-MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANtb0+YrKuxevGpm
-LrjaUhZSgz6zFAmuGFmKmUbdjmfv9zSmmdsQIksK++jK0Be9LeZy20j6ahOfuVa0
-ufEmPoP7Fy4hXegKZR9cCWcIe/A6H2xWF1IIJLRTLaU8ol/I7T+um5HD5AwAwNPP
-USNU0Eegmvp+xxWu3NX2m1Veot85AgMBAAECgYA3ZdZ673X0oexFlq7AAmrutkHt
-CL7LvwrpOiaBjhyTxTeSNWzvtQBkIU8DOI0bIazA4UreAFffwtvEuPmonDb3F+Iq
-SMAu42XcGyVZEl+gHlTPU9XRX7nTOXVt+MlRRRxL6t9GkGfUAXI3XxJDXW3c0vBK
-UL9xqD8cORXOfE06rQJBAP8mEX1ERkR64Ptsoe4281vjTlNfIbs7NMPkUnrn9N/Y
-BLhjNIfQ3HFZG8BTMLfX7kCS9D593DW5tV4Z9BP/c6cCQQDcFzCcVArNh2JSywOQ
-ZfTfRbJg/Z5Lt9Fkngv1meeGNPgIMLN8Sg679pAOOWmzdMO3V706rNPzSVMME7E5
-oPIfAkEA8pDddarP5tCvTTgUpmTFbakm0KoTZm2+FzHcnA4jRh+XNTjTOv98Y6Ik
-eO5d1ZnKXseWvkZncQgxfdnMqqpj5wJAcNq/RVne1DbYlwWchT2Si65MYmmJ8t+F
-0mcsULqjOnEMwf5e+ptq5LzwbyrHZYq5FNk7ocufPv/ZQrcSSC+cFwJBAKvOJByS
-x56qyGeZLOQlWS2JS3KJo59XuLFGqcbgN9Om9xFa41Yb4N9NvplFivsvZdw3m1Q/
-SPIXQuT8RMPDVNQ=
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCr77F9oBLYuLIb
+3t4mDHPoxOEnu1h7NQaJdn9tR/KjW6AzhXXm9USh45qt3sR6Wo8sDlBIJ3vZyhyW
+P939qYUeGOZJahupYi4IkqVckZXARm3k9qhAI/aC4ixVFyxnxg3bNpk8Ir0AyyQV
+oudY+33+ZNj6+fHzeboGDJ9uE/RTcp9JqE/qo0haATZufJfY63ZCUpYFn6j5W4jG
+vpqg5/0hba8Cxdk62387uNknfVHSMzGkkq82zbBpud6TYQofCp3VlEPBjz9iLCz6
+FFOKgLZmbk8QsdktXF6zfRJJk+vbZTh/OGH0p/eiIfW1kXOzOcuW31XRFTPnykJd
+4QUX9OajAgMBAAECggEAHppmXDbuw9Z0FVPg9KLIysioTtsgz6VLiZIm8juZK4x2
+glUh/D7xvWL2uDXrgN+3lh7iGUW13LkFx5SMncbbo9TIwI57Z/XKvcnkVwquve+L
+RfLFVc1Q5lD9lROv2rS86KTaN4LzYz3FKXi6dvMkpPAsUtfEQhMLkmISypQQq/1z
+EJaqo7r85OjN7e0wKazlKZpOzJEa5FQLMVRjTRFhLFNbHXX/tAet2jw+umATKbw8
+hYgiuZ44TwSEd9JeIV/oSYWfI/3HetuYW0ru3caiztRF2NySNu8lcsWgNC7fIku9
+mcHjtSNzs91QN1Qlu7GQvvhpt6OWDirNDCW+49WGaQKBgQDg9SDhfF0jRYslgYbH
+cqO4ggaFdHjrAAYpwnAgvanhFZL/zEqm5G1E7l/e2fCkJ9VOSFO0A208chvwMcr+
+dCjHE2tVdE81aQ2v/Eo83VdS1RcOV4Y75yPH48rMhxPaHvxWD/FFDbf0/P2mtPB7
+SZ3kIeZMkE1wxdaO3AKUbQoozwKBgQDDqYgg7kVtygyICE1mB8Hwp6nUxFTczG7y
+4XcsDqMIrKmw+PbQluvkoHoStxeVrsTloDhkTjIrpmYLyAiazg+PUJdkd6xrfLSj
+VV6X93W0S/1egEb1F1CGFxtk8v/PWH4K76EPL2vxXdxjywz3GWlrL9yDYaB2szzS
+DqgwVMqx7QKBgDCD7UF0Bsoyl13RX3XoPXLvZ+SkR+e2q52Z94C4JskKVBeiwX7Y
+yNAS8M4pBoMArDoj0xmBm69rlKbqtjLGbnzwrTdSzDpim7cWnBQgUFLm7gAD1Elb
+AhZ8BCK0Bw4FnLoa2hfga4oEfdfUMgEE0W5/+SEOBgWKRUmuHUhRc911AoGAY2EN
+YmSDYSM5wDIvVb5k9B3EtevOiqNPSw/XnsoEZtiEC/44JnQxdltIBY93bDBrk5IQ
+cmoBM4h91kgQjshQwOMXMhFSwvmBKmCm/hrTbvMVytTutXfVD3ZXFKwT4DW7N0TF
+ElhsxBh/YzRz7mG62JVjtFt2zDN3ld2Z8YpvtXUCgYEA4EJ4ObS5YyvcXAKHJFo6
+Fxmavyrf8LSm3MFA65uSnFvWukMVqqRMReQc5jvpxHKCis+XvnHzyOfL0gW9ZTi7
+tWGGbBi0TRJCa8BkvgngUZxOxUlMfg/7cVxOIB0TPoUSgxFd/+qVz4GZMvr0dPu7
+eAF7J/8ECVvb0wSPTUI1N3c=
 -----END PRIVATE KEY-----
 -----BEGIN CERTIFICATE-----
-MIICVDCCAb2gAwIBAgIJANfHOBkZr8JOMA0GCSqGSIb3DQEBBQUAMF8xCzAJBgNV
-BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u
-IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMDEw
-MDgyMzAxNTZaFw0yMDEwMDUyMzAxNTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH
-Ew5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZvdW5k
-YXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAw
-gYkCgYEA21vT5isq7F68amYuuNpSFlKDPrMUCa4YWYqZRt2OZ+/3NKaZ2xAiSwr7
-6MrQF70t5nLbSPpqE5+5VrS58SY+g/sXLiFd6AplH1wJZwh78DofbFYXUggktFMt
-pTyiX8jtP66bkcPkDADA089RI1TQR6Ca+n7HFa7c1fabVV6i3zkCAwEAAaMYMBYw
-FAYDVR0RBA0wC4IJbG9jYWxob3N0MA0GCSqGSIb3DQEBBQUAA4GBAHPctQBEQ4wd
-BJ6+JcpIraopLn8BGhbjNWj40mmRqWB/NAWF6M5ne7KpGAu7tLeG4hb1zLaldK8G
-lxy2GPSRF6LFS48dpEj2HbMv2nvv6xxalDMJ9+DicWgAKTQ6bcX2j3GUkCR0g/T1
-CRlNBAAlvhKzO7Clpf9l0YKBEfraJByX
+MIIDWTCCAkGgAwIBAgIJAPm6B21bar2bMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV
+BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u
+IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODAx
+MTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH
+DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k
+YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEP
+ADCCAQoCggEBAKvvsX2gEti4shve3iYMc+jE4Se7WHs1Bol2f21H8qNboDOFdeb1
+RKHjmq3exHpajywOUEgne9nKHJY/3f2phR4Y5klqG6liLgiSpVyRlcBGbeT2qEAj
+9oLiLFUXLGfGDds2mTwivQDLJBWi51j7ff5k2Pr58fN5ugYMn24T9FNyn0moT+qj
+SFoBNm58l9jrdkJSlgWfqPlbiMa+mqDn/SFtrwLF2Trbfzu42Sd9UdIzMaSSrzbN
+sGm53pNhCh8KndWUQ8GPP2IsLPoUU4qAtmZuTxCx2S1cXrN9EkmT69tlOH84YfSn
+96Ih9bWRc7M5y5bfVdEVM+fKQl3hBRf05qMCAwEAAaMYMBYwFAYDVR0RBA0wC4IJ
+bG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQAtQ8f37cCEk7/rAcbYR53ce3iK
+Vpihb0U2ni1QjG9Tg9UIExkIGkwTiCm7kwQL+GEStBu9AG/QVrOjeTriRiddhWkk
+ze8kRaI3AC/63t6Vh9Q1x6PESgeE4OtAO9JpJCf4GILglA789Y/b/GF8zJZQxR13
+qpB4ZwWw7gCBhdEW59u6CFeBmfDa58hM8lWvuVoRrTi7bjUeC6PAn5HVMzZSykhu
+4HaUfBp6bKFjuym2+h/VvM1n8C3chjVSmutsLb6ELdD8IK0vPV/yf5+LN256zSsS
+dyUZYd8XwQaioEMKdbhLvnehyzHiWfQIUR3BdhONxoIJhHv/EAo8eCkHHYIF
 -----END CERTIFICATE-----
diff --git a/Lib/test/keycert2.pem b/Lib/test/keycert2.pem
index e8a9e082b31..bb5fa65a8ac 100644
--- a/Lib/test/keycert2.pem
+++ b/Lib/test/keycert2.pem
@@ -1,31 +1,49 @@
 -----BEGIN PRIVATE KEY-----
-MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBAJnsJZVrppL+W5I9
-zGQrrawWwE5QJpBK9nWw17mXrZ03R1cD9BamLGivVISbPlRlAVnZBEyh1ATpsB7d
-CUQ+WHEvALquvx4+Yw5l+fXeiYRjrLRBYZuVy8yNtXzU3iWcGObcYRkUdiXdOyP7
-sLF2YZHRvQZpzgDBKkrraeQ81w21AgMBAAECgYBEm7n07FMHWlE+0kT0sXNsLYfy
-YE+QKZnJw9WkaDN+zFEEPELkhZVt5BjsMraJr6v2fIEqF0gGGJPkbenffVq2B5dC
-lWUOxvJHufMK4sM3Cp6s/gOp3LP+QkzVnvJSfAyZU6l+4PGX5pLdUsXYjPxgzjzL
-S36tF7/2Uv1WePyLUQJBAMsPhYzUXOPRgmbhcJiqi9A9c3GO8kvSDYTCKt3VMnqz
-HBn6MQ4VQasCD1F+7jWTI0FU/3vdw8non/Fj8hhYqZcCQQDCDRdvmZqDiZnpMqDq
-L6ZSrLTVtMvZXZbgwForaAD9uHj51TME7+eYT7EG2YCgJTXJ4YvRJEnPNyskwdKt
-vTSTAkEAtaaN/vyemEJ82BIGStwONNw0ILsSr5cZ9tBHzqiA/tipY+e36HRFiXhP
-QcU9zXlxyWkDH8iz9DSAmE2jbfoqwwJANlMJ65E543cjIlitGcKLMnvtCCLcKpb7
-xSG0XJB6Lo11OKPJ66jp0gcFTSCY1Lx2CXVd+gfJrfwI1Pp562+bhwJBAJ9IfDPU
-R8OpO9v1SGd8x33Owm7uXOpB9d63/T70AD1QOXjKUC4eXYbt0WWfWuny/RNPRuyh
-w7DXSfUF+kPKolU=
+MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQC3ulRNfhbOAey/
+B+wIVYx+d5az7EV4riR6yi/qE6G+bxbTvay2pqySHtDweuaYSh2cVmcasBKKIFJm
+rCD1zR8UmLb5i2XFIina1t3eePCuBZMrvZZwkzlQUSM1AZtjGOO/W0I3FwO6y645
+9xA5PduKI7SMYkH/VL3zE5W1JwMovv6bvNiT+GU5l6mB9ylCTgLpmUqoQhRqz/35
+zCzVyoh+ppDvVcpWYfvXywsXsgQwbAF0QJm8SSFi0TZm5ykv4WE16afQp08yuZS0
+3U4K3MJCa4rxO58edcxBopWYfQ29K3iINM8enRfr5q+u5mAAbALAEEvyFjgLWl/u
+7arxn7bJAgMBAAECggEBAJfMt8KfHzBunrDnVrk8FayYGkfmOzAOkc1yKEx6k/TH
+zFB+Mqlm5MaF95P5t3S0J+r36JBAUdEWC38RUNpF9BwMYYGlDxzlsTdCuGYL/q+J
+o6NMLXQt7/jQUQqGnWAvPFzqhbcGqOo5R2ZVH25sEWv9PDuRI35XAepIkDTwWsfa
+P6UcJJoP+4v9B++fb3sSL4zNwp1BqS4wxR8YTR0t1zQqOxJ5BGPw1J8aBMs1sq5t
+qyosAQAT63kLrdqWotHaM26QxjqEQUMlh12XMWb5GdBXUxbvyGtEabsqskGa/f8B
+RdHE437J8D8l+jxb2mZLzrlaH3dq2tbFGCe1rT8qLRECgYEA5CWIvoD/YnQydLGA
+OlEhCSocqURuqcotg9Ev0nt/C60jkr/NHFLGppz9lhqjIDjixt3sIMGZMFzxRtwM
+pSYal3XiR7rZuHau9iM35yDhpuytEiGbYy1ADakJRzY5jq/Qa8RfPP9Atua5xAeP
+q6DiSnq9vhHv9G+O4MxzHBmrw9sCgYEAziiJWFthcwvuXn3Jv9xFYKEb/06puZAx
+EgQCz/3rPzv5fmGD/sKVo1U/K4z/eA82DNeKG8QRTFJCxT8TCNRxOmGV7HdCYo/B
+4BTNNvbKcdi3l0j75kKoADg+nt5CD5lz6gLG0GrUEnVO1y5HVfCTb3BEAfa36C85
+9i0sfQGiwysCgYEAuus9k8cgdct5oz3iLuVVSark/JGCkT2B+OOkaLChsDFUWeEm
+7TOsaclpwldkmvvAYOplkZjMJ2GelE2pVo1XcAw3LkmaI5WpVyQXoxe/iQGT8qzy
+IFlsh0Scw2lb0tmcyw6CcPk4TiHOxRrkzNrtS9QwLM+JZx0XVHptPPKTVc0CgYAu
+j/VFYY5G/8Dc0qhIjyWUR48dQNUQtkJ/ASzpcT46z/7vznKTjbtiYpSb74KbyUO5
+7sygrM4DYOj3x+Eys1jHiNbly6HQxQtS4x/edCsRP5NntfI+9XsgYZOzKhvdjhki
+F3J0DEzNxnUCIM+311hVaRPTJbgv1srOkTFlIoNydQKBgQC6/OHGaC/OewQqRlRK
+Mg5KZm01/pk4iKrpA5nG7OTAeoa70NzXNtG8J3WnaJ4mWanNwNUOyRMAMrsUAy9q
+EeGqHM5mMFpY4TeVuNLL21lu/x3KYw6mKL3Ctinn+JLAoYoqEy8deZnEA5/tjYlz
+YhFBchnUicjoUN1chdpM6SpV2Q==
 -----END PRIVATE KEY-----
 -----BEGIN CERTIFICATE-----
-MIICXTCCAcagAwIBAgIJAIO3upAG445fMA0GCSqGSIb3DQEBBQUAMGIxCzAJBgNV
-BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u
-IFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMTDGZha2Vob3N0bmFtZTAeFw0x
-MDEwMDkxNTAxMDBaFw0yMDEwMDYxNTAxMDBaMGIxCzAJBgNVBAYTAlhZMRcwFQYD
-VQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZv
-dW5kYXRpb24xFTATBgNVBAMTDGZha2Vob3N0bmFtZTCBnzANBgkqhkiG9w0BAQEF
-AAOBjQAwgYkCgYEAmewllWumkv5bkj3MZCutrBbATlAmkEr2dbDXuZetnTdHVwP0
-FqYsaK9UhJs+VGUBWdkETKHUBOmwHt0JRD5YcS8Auq6/Hj5jDmX59d6JhGOstEFh
-m5XLzI21fNTeJZwY5txhGRR2Jd07I/uwsXZhkdG9BmnOAMEqSutp5DzXDbUCAwEA
-AaMbMBkwFwYDVR0RBBAwDoIMZmFrZWhvc3RuYW1lMA0GCSqGSIb3DQEBBQUAA4GB
-AH+iMClLLGSaKWgwXsmdVo4FhTZZHo8Uprrtg3N9FxEeE50btpDVQysgRt5ias3K
-m+bME9zbKwvbVWD5zZdjus4pDgzwF/iHyccL8JyYhxOvS/9zmvAtFXj/APIIbZFp
-IT75d9f88ScIGEtknZQejnrdhB64tYki/EqluiuKBqKD
+MIIDYjCCAkqgAwIBAgIJALJXRr8qF6oIMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
+BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u
+IFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTAeFw0x
+ODAxMTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMGIxCzAJBgNVBAYTAlhZMRcwFQYD
+VQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZv
+dW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTCCASIwDQYJKoZIhvcNAQEB
+BQADggEPADCCAQoCggEBALe6VE1+Fs4B7L8H7AhVjH53lrPsRXiuJHrKL+oTob5v
+FtO9rLamrJIe0PB65phKHZxWZxqwEoogUmasIPXNHxSYtvmLZcUiKdrW3d548K4F
+kyu9lnCTOVBRIzUBm2MY479bQjcXA7rLrjn3EDk924ojtIxiQf9UvfMTlbUnAyi+
+/pu82JP4ZTmXqYH3KUJOAumZSqhCFGrP/fnMLNXKiH6mkO9VylZh+9fLCxeyBDBs
+AXRAmbxJIWLRNmbnKS/hYTXpp9CnTzK5lLTdTgrcwkJrivE7nx51zEGilZh9Db0r
+eIg0zx6dF+vmr67mYABsAsAQS/IWOAtaX+7tqvGftskCAwEAAaMbMBkwFwYDVR0R
+BBAwDoIMZmFrZWhvc3RuYW1lMA0GCSqGSIb3DQEBCwUAA4IBAQCZhHhGItpkqhEq
+ntMRd6Hv0GoOJixNvgeMwK4NJSRT/no3OirtUTzccn46h+SWibSa2eVssAV+pAVJ
+HbzkN/DH27A1mMx1zJL1ekcOKA1AF6MXhUnrUGXMqW36YNtzHfXJLrwvpLJ13OQg
+/Kxo4Nw68bGzM+PyRtKU/mpgYyfcvwR+ZSeIDh1fvUZK/IEVCf8ub42GPVs5wPfv
+M+k5aHxWTxeif3K1byTRzxHupYNG2yWO4XEdnBGOuOwzzN4/iQyNcsuQKeuKHGrt
+YvIlG/ri04CQ7xISZCj74yjTZ+/A2bXre2mQXAHqKPumHL7cl34+erzbUaxYxbTE
+u5FcOmLQ
 -----END CERTIFICATE-----
diff --git a/Lib/test/keycert3.pem b/Lib/test/keycert3.pem
index 5bfa62c4ca3..621eb08bb0c 100644
--- a/Lib/test/keycert3.pem
+++ b/Lib/test/keycert3.pem
@@ -1,73 +1,132 @@
 -----BEGIN PRIVATE KEY-----
-MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAMLgD0kAKDb5cFyP
-jbwNfR5CtewdXC+kMXAWD8DLxiTTvhMW7qVnlwOm36mZlszHKvsRf05lT4pegiFM
-9z2j1OlaN+ci/X7NU22TNN6crYSiN77FjYJP464j876ndSxyD+rzys386T+1r1aZ
-aggEdkj1TsSsv1zWIYKlPIjlvhuxAgMBAAECgYA0aH+T2Vf3WOPv8KdkcJg6gCRe
-yJKXOWgWRcicx/CUzOEsTxmFIDPLxqAWA3k7v0B+3vjGw5Y9lycV/5XqXNoQI14j
-y09iNsumds13u5AKkGdTJnZhQ7UKdoVHfuP44ZdOv/rJ5/VD6F4zWywpe90pcbK+
-AWDVtusgGQBSieEl1QJBAOyVrUG5l2yoUBtd2zr/kiGm/DYyXlIthQO/A3/LngDW
-5/ydGxVsT7lAVOgCsoT+0L4efTh90PjzW8LPQrPBWVMCQQDS3h/FtYYd5lfz+FNL
-9CEe1F1w9l8P749uNUD0g317zv1tatIqVCsQWHfVHNdVvfQ+vSFw38OORO00Xqs9
-1GJrAkBkoXXEkxCZoy4PteheO/8IWWLGGr6L7di6MzFl1lIqwT6D8L9oaV2vynFT
-DnKop0pa09Unhjyw57KMNmSE2SUJAkEArloTEzpgRmCq4IK2/NpCeGdHS5uqRlbh
-1VIa/xGps7EWQl5Mn8swQDel/YP3WGHTjfx7pgSegQfkyaRtGpZ9OQJAa9Vumj8m
-JAAtI0Bnga8hgQx7BhTQY4CadDxyiRGOGYhwUzYVCqkb2sbVRH9HnwUaJT7cWBY3
-RnJdHOMXWem7/w==
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgV4G+Zzf2DT5n
+oAisIGFhn/bz7Vn5WiXUqbDsxROJOh/7BtOlduZka0pPhFylGbnxS8l1kEWHRI2T
+6hOoWzumB6ItKiN+T5J30lAvSyo7iwdFoAQ/S5nPXQfhNARQe/NEOhRtpcuNdyx4
+BWdPdPuJQA1ASNJCLwcLOoRxaLbKLvb2V5T5FCAkeNPtRvPuT4gKQItMmiHfAhoV
+C8MZWF/GC0RukHINys5MwqeFexam8CznmQPMYrLdhmKTj3DTivCPoh97EDIFGlgZ
+SCaaYDVQA+aqlo/q2pi52PFwC1KzhNEA7EeOqSwC1NQjwjHuhcnf9WbxrgTq2zh3
+rv5YEW2ZAgMBAAECggEAPfSMtTumPcJskIumuXp7yk02EyliZrWZqwBuBwVqHsS5
+nkbFXnXWrLbgn9MrDsFrE5NdgKUmPnQVMVs8sIr5jyGejSCNCs4I4iRn1pfIgwcj
+K/xEEALd6GGF0pDd/CgvB5GOoLVf4KKf2kmLvWrOKJpSzoUN5A8+v8AaYYOMr4sC
+czbvfGomzEIewEG+Rw9zOVUDlmwyEKPQZ47E7PQ+EEA7oeFdR+1Zj6eT9ndegf8B
+54frySYCLRUCk/sHCpWhaJBtBrcpht7Y8CfY7hiH/7x866fvuLnYPz4YALtUb0wN
+7zUCNS9ol3n4LbjFFKfZtiRjKaCBRzMjK0rz6ydFcQKBgQDyLI3oGbnW73vqcDe/
+6eR0w++fiCAVhfMs3AO/gOaJi2la2JHlJ5u+cIHQIOFwEhn6Zq0AtdmnFx1TS5IQ
+C0MdXI0XoQQw7rEF8EJcvfe85Z0QxENVhzydtdb8QpJfnQGfBfLyQlaaRYzRRHB6
+VdYUHF3EIPVIhbjbghal+Qep/QKBgQDtJlRPHkWwTMevu0J0fYbWN1ywtVTFUR//
+k7VyORSf8yuuSnaQRop4cbcqONxmDKH6Or1fl3NYBsAxtXkkOK1E2OZNo2sfQdRa
+wpA7o7mPHRhztQFpT5vflp+8P6+PEFat8D04eBOhNwrwwfhiPjD4gv5KvN4XutRW
+VWv/2pnmzQKBgHPvHGg2mJ7quvm6ixXW1MWJX1eSBToIjCe3lBvDi5nhIaiZ8Q4w
+7gA3QA3xD7tlDwauzLeAVxgEmsdbcCs6GQEfY3QiYy1Bt4FOSZa4YrcNfSmfq1Rw
+j3Y4rRjKjeQz96i3YlzToT3tecJc7zPBj+DEy6au2H3Fdn+vQURneWHJAoGBANG7
+XES8mRVaUh/wlM1BVsaNH8SIGfiHzqzRjV7/bGYpQTBbWpAuUrhCmaMVtpXqBjav
+TFwGLVRkZAWSYRjPpy2ERenT5SE3rv61o6mbGrifGsj6A82HQmtzYsGx8SmtYXtj
+REF0sKebbmmOooUAS379GrguYJzL9o6D7YfRZNrhAoGAVfb/tiFU4S67DSpYpQey
+ULhgfsFpDByICY6Potsg67gVFf9jIaB83NPTx3u/r6sHFgxFw7lQsuZcgSuWMu7t
+glzOXVIP11Y5sl5CJ5OsfeK1/0umMZF5MWPyAQCx/qrPlZL86vXjt24Y/VaOxsAi
+CZYdyJsjgOrJrWoMbo5ta54=
 -----END PRIVATE KEY-----
 Certificate:
     Data:
-        Version: 1 (0x0)
-        Serial Number: 12723342612721443281 (0xb09264b1f2da21d1)
+        Version: 3 (0x2)
+        Serial Number:
+            82:ed:bf:41:c8:80:91:9c
     Signature Algorithm: sha1WithRSAEncryption
         Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server
         Validity
-            Not Before: Jan  4 19:47:07 2013 GMT
-            Not After : Nov 13 19:47:07 2022 GMT
+            Not Before: Jan 19 19:09:06 2018 GMT
+            Not After : Nov 28 19:09:06 2027 GMT
         Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=localhost
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption
-                Public-Key: (1024 bit)
+                Public-Key: (2048 bit)
                 Modulus:
-                    00:c2:e0:0f:49:00:28:36:f9:70:5c:8f:8d:bc:0d:
-                    7d:1e:42:b5:ec:1d:5c:2f:a4:31:70:16:0f:c0:cb:
-                    c6:24:d3:be:13:16:ee:a5:67:97:03:a6:df:a9:99:
-                    96:cc:c7:2a:fb:11:7f:4e:65:4f:8a:5e:82:21:4c:
-                    f7:3d:a3:d4:e9:5a:37:e7:22:fd:7e:cd:53:6d:93:
-                    34:de:9c:ad:84:a2:37:be:c5:8d:82:4f:e3:ae:23:
-                    f3:be:a7:75:2c:72:0f:ea:f3:ca:cd:fc:e9:3f:b5:
-                    af:56:99:6a:08:04:76:48:f5:4e:c4:ac:bf:5c:d6:
-                    21:82:a5:3c:88:e5:be:1b:b1
+                    00:e0:57:81:be:67:37:f6:0d:3e:67:a0:08:ac:20:
+                    61:61:9f:f6:f3:ed:59:f9:5a:25:d4:a9:b0:ec:c5:
+                    13:89:3a:1f:fb:06:d3:a5:76:e6:64:6b:4a:4f:84:
+                    5c:a5:19:b9:f1:4b:c9:75:90:45:87:44:8d:93:ea:
+                    13:a8:5b:3b:a6:07:a2:2d:2a:23:7e:4f:92:77:d2:
+                    50:2f:4b:2a:3b:8b:07:45:a0:04:3f:4b:99:cf:5d:
+                    07:e1:34:04:50:7b:f3:44:3a:14:6d:a5:cb:8d:77:
+                    2c:78:05:67:4f:74:fb:89:40:0d:40:48:d2:42:2f:
+                    07:0b:3a:84:71:68:b6:ca:2e:f6:f6:57:94:f9:14:
+                    20:24:78:d3:ed:46:f3:ee:4f:88:0a:40:8b:4c:9a:
+                    21:df:02:1a:15:0b:c3:19:58:5f:c6:0b:44:6e:90:
+                    72:0d:ca:ce:4c:c2:a7:85:7b:16:a6:f0:2c:e7:99:
+                    03:cc:62:b2:dd:86:62:93:8f:70:d3:8a:f0:8f:a2:
+                    1f:7b:10:32:05:1a:58:19:48:26:9a:60:35:50:03:
+                    e6:aa:96:8f:ea:da:98:b9:d8:f1:70:0b:52:b3:84:
+                    d1:00:ec:47:8e:a9:2c:02:d4:d4:23:c2:31:ee:85:
+                    c9:df:f5:66:f1:ae:04:ea:db:38:77:ae:fe:58:11:
+                    6d:99
                 Exponent: 65537 (0x10001)
+        X509v3 extensions:
+            X509v3 Subject Alternative Name: 
+                DNS:localhost
+            X509v3 Key Usage: critical
+                Digital Signature, Key Encipherment
+            X509v3 Extended Key Usage: 
+                TLS Web Server Authentication, TLS Web Client Authentication
+            X509v3 Basic Constraints: critical
+                CA:FALSE
+            X509v3 Subject Key Identifier: 
+                85:11:BE:16:47:04:D1:30:EE:86:8A:18:70:BE:A8:28:6F:82:3D:CE
+            X509v3 Authority Key Identifier: 
+                keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48
+                DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server
+                serial:82:ED:BF:41:C8:80:91:9B
+
+            Authority Information Access: 
+                CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer
+                OCSP - URI:http://testca.pythontest.net/testca/ocsp/
+
+            X509v3 CRL Distribution Points: 
+
+                Full Name:
+                  URI:http://testca.pythontest.net/testca/revocation.crl
+
     Signature Algorithm: sha1WithRSAEncryption
-         2f:42:5f:a3:09:2c:fa:51:88:c7:37:7f:ea:0e:63:f0:a2:9a:
-         e5:5a:e2:c8:20:f0:3f:60:bc:c8:0f:b6:c6:76:ce:db:83:93:
-         f5:a3:33:67:01:8e:04:cd:00:9a:73:fd:f3:35:86:fa:d7:13:
-         e2:46:c6:9d:c0:29:53:d4:a9:90:b8:77:4b:e6:83:76:e4:92:
-         d6:9c:50:cf:43:d0:c6:01:77:61:9a:de:9b:70:f7:72:cd:59:
-         00:31:69:d9:b4:ca:06:9c:6d:c3:c7:80:8c:68:e6:b5:a2:f8:
-         ef:1d:bb:16:9f:77:77:ef:87:62:22:9b:4d:69:a4:3a:1a:f1:
-         21:5e:8c:32:ac:92:fd:15:6b:18:c2:7f:15:0d:98:30:ca:75:
-         8f:1a:71:df:da:1d:b2:ef:9a:e8:2d:2e:02:fd:4a:3c:aa:96:
-         0b:06:5d:35:b3:3d:24:87:4b:e0:b0:58:60:2f:45:ac:2e:48:
-         8a:b0:99:10:65:27:ff:cc:b1:d8:fd:bd:26:6b:b9:0c:05:2a:
-         f4:45:63:35:51:07:ed:83:85:fe:6f:69:cb:bb:40:a8:ae:b6:
-         3b:56:4a:2d:a4:ed:6d:11:2c:4d:ed:17:24:fd:47:bc:d3:41:
-         a2:d3:06:fe:0c:90:d8:d8:94:26:c4:ff:cc:a1:d8:42:77:eb:
-         fc:a9:94:71
+         7f:a1:7e:3e:68:01:b0:32:b8:57:b8:03:68:13:13:b3:e3:f4:
+         70:2f:15:e5:0f:87:b9:fd:e0:12:e3:16:f2:91:53:c7:4e:25:
+         af:ca:cb:a7:d9:9d:57:4d:bf:a2:80:d4:78:aa:04:31:fd:6d:
+         cc:6d:82:43:e9:62:16:0d:0e:26:8b:e7:f1:3d:57:5c:68:02:
+         9c:2b:b6:c9:fd:62:2f:10:85:88:cc:44:a5:e7:a2:3e:89:f2:
+         1f:02:6a:3f:d0:3c:6c:24:2d:bc:51:62:7a:ec:25:c5:86:87:
+         77:35:8f:f9:7e:d0:17:3d:77:56:bf:1a:0c:be:09:78:ee:ea:
+         73:97:65:60:94:91:35:b3:5c:46:8a:5e:6d:94:52:de:48:b7:
+         1f:6c:28:79:7f:ff:08:8d:e4:7d:d0:b9:0b:7c:ae:c4:1d:2a:
+         a1:b3:50:11:82:03:5e:6c:e7:26:fa:05:32:39:07:83:49:b9:
+         a2:fa:04:da:0d:e5:ff:4c:db:97:d0:c3:a7:43:37:4c:16:de:
+         3c:b5:e9:7e:82:d4:b3:10:df:d1:c1:66:72:9c:15:67:19:3b:
+         7b:91:0a:82:07:67:c5:06:03:5f:80:54:08:81:8a:b1:5c:7c:
+         4c:d2:07:38:92:eb:12:f5:71:ae:de:05:15:c8:e1:33:f0:e4:
+         96:0f:0f:1e
 -----BEGIN CERTIFICATE-----
-MIICpDCCAYwCCQCwkmSx8toh0TANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJY
-WTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNV
-BAMMDW91ci1jYS1zZXJ2ZXIwHhcNMTMwMTA0MTk0NzA3WhcNMjIxMTEzMTk0NzA3
-WjBfMQswCQYDVQQGEwJYWTEXMBUGA1UEBxMOQ2FzdGxlIEFudGhyYXgxIzAhBgNV
-BAoTGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMRIwEAYDVQQDEwlsb2NhbGhv
-c3QwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMLgD0kAKDb5cFyPjbwNfR5C
-tewdXC+kMXAWD8DLxiTTvhMW7qVnlwOm36mZlszHKvsRf05lT4pegiFM9z2j1Ola
-N+ci/X7NU22TNN6crYSiN77FjYJP464j876ndSxyD+rzys386T+1r1aZaggEdkj1
-TsSsv1zWIYKlPIjlvhuxAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAC9CX6MJLPpR
-iMc3f+oOY/CimuVa4sgg8D9gvMgPtsZ2ztuDk/WjM2cBjgTNAJpz/fM1hvrXE+JG
-xp3AKVPUqZC4d0vmg3bkktacUM9D0MYBd2Ga3ptw93LNWQAxadm0ygacbcPHgIxo
-5rWi+O8duxafd3fvh2Iim01ppDoa8SFejDKskv0VaxjCfxUNmDDKdY8acd/aHbLv
-mugtLgL9SjyqlgsGXTWzPSSHS+CwWGAvRawuSIqwmRBlJ//Msdj9vSZruQwFKvRF
-YzVRB+2Dhf5vacu7QKiutjtWSi2k7W0RLE3tFyT9R7zTQaLTBv4MkNjYlCbE/8yh
-2EJ36/yplHE=
+MIIE8TCCA9mgAwIBAgIJAILtv0HIgJGcMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yNzExMjgx
+OTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj
+MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxv
+Y2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBXgb5nN/YN
+PmegCKwgYWGf9vPtWflaJdSpsOzFE4k6H/sG06V25mRrSk+EXKUZufFLyXWQRYdE
+jZPqE6hbO6YHoi0qI35PknfSUC9LKjuLB0WgBD9Lmc9dB+E0BFB780Q6FG2ly413
+LHgFZ090+4lADUBI0kIvBws6hHFotsou9vZXlPkUICR40+1G8+5PiApAi0yaId8C
+GhULwxlYX8YLRG6Qcg3KzkzCp4V7FqbwLOeZA8xist2GYpOPcNOK8I+iH3sQMgUa
+WBlIJppgNVAD5qqWj+ramLnY8XALUrOE0QDsR46pLALU1CPCMe6Fyd/1ZvGuBOrb
+OHeu/lgRbZkCAwEAAaOCAcAwggG8MBQGA1UdEQQNMAuCCWxvY2FsaG9zdDAOBgNV
+HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud
+EwEB/wQCMAAwHQYDVR0OBBYEFIURvhZHBNEw7oaKGHC+qChvgj3OMH0GA1UdIwR2
+MHSAFJrPz27rcT3bPPGuiGtWcgPLCKdIoVGkTzBNMQswCQYDVQQGEwJYWTEmMCQG
+A1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNVBAMMDW91
+ci1jYS1zZXJ2ZXKCCQCC7b9ByICRmzCBgwYIKwYBBQUHAQEEdzB1MDwGCCsGAQUF
+BzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3RjYS9weWNhY2Vy
+dC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQv
+dGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly90ZXN0Y2EucHl0
+aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3JsMA0GCSqGSIb3DQEBBQUA
+A4IBAQB/oX4+aAGwMrhXuANoExOz4/RwLxXlD4e5/eAS4xbykVPHTiWvysun2Z1X
+Tb+igNR4qgQx/W3MbYJD6WIWDQ4mi+fxPVdcaAKcK7bJ/WIvEIWIzESl56I+ifIf
+Amo/0DxsJC28UWJ67CXFhod3NY/5ftAXPXdWvxoMvgl47upzl2VglJE1s1xGil5t
+lFLeSLcfbCh5f/8IjeR90LkLfK7EHSqhs1ARggNebOcm+gUyOQeDSbmi+gTaDeX/
+TNuX0MOnQzdMFt48tel+gtSzEN/RwWZynBVnGTt7kQqCB2fFBgNfgFQIgYqxXHxM
+0gc4kusS9XGu3gUVyOEz8OSWDw8e
 -----END CERTIFICATE-----
diff --git a/Lib/test/keycert4.pem b/Lib/test/keycert4.pem
index 53355c8a50c..b7df7f3f2c7 100644
--- a/Lib/test/keycert4.pem
+++ b/Lib/test/keycert4.pem
@@ -1,73 +1,132 @@
 -----BEGIN PRIVATE KEY-----
-MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAK5UQiMI5VkNs2Qv
-L7gUaiDdFevNUXRjU4DHAe3ZzzYLZNE69h9gO9VCSS16tJ5fT5VEu0EZyGr0e3V2
-NkX0ZoU0Hc/UaY4qx7LHmn5SYZpIxhJnkf7SyHJK1zUaGlU0/LxYqIuGCtF5dqx1
-L2OQhEx1GM6RydHdgX69G64LXcY5AgMBAAECgYAhsRMfJkb9ERLMl/oG/5sLQu9L
-pWDKt6+ZwdxzlZbggQ85CMYshjLKIod2DLL/sLf2x1PRXyRG131M1E3k8zkkz6de
-R1uDrIN/x91iuYzfLQZGh8bMY7Yjd2eoroa6R/7DjpElGejLxOAaDWO0ST2IFQy9
-myTGS2jSM97wcXfsSQJBANP3jelJoS5X6BRjTSneY21wcocxVuQh8pXpErALVNsT
-drrFTeaBuZp7KvbtnIM5g2WRNvaxLZlAY/hXPJvi6ncCQQDSix1cebml6EmPlEZS
-Mm8gwI2F9ufUunwJmBJcz826Do0ZNGByWDAM/JQZH4FX4GfAFNuj8PUb+GQfadkx
-i1DPAkEA0lVsNHojvuDsIo8HGuzarNZQT2beWjJ1jdxh9t7HrTx7LIps6rb/fhOK
-Zs0R6gVAJaEbcWAPZ2tFyECInAdnsQJAUjaeXXjuxFkjOFym5PvqpvhpivEx78Bu
-JPTr3rAKXmfGMxxfuOa0xK1wSyshP6ZR/RBn/+lcXPKubhHQDOegwwJAJF1DBQnN
-+/tLmOPULtDwfP4Zixn+/8GmGOahFoRcu6VIGHmRilJTn6MOButw7Glv2YdeC6l/
-e83Gq6ffLVfKNQ==
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDH/76hZAZH4cSV
+CmVZa5HEqKCjCKrcPwBECs9BS+3ibwN4x9NnFNP+tCeFGgJXl7WGFoeXgg3oK+1p
+FsOWpsRHuF3BdqkCnShSydmT8bLaGHwKeL0cPxJP5T/uW7ezPKW2VWXGMwmwRaRJ
+9dj2VCUu20vDZWSGFr9zjnjoJczBtH3RsVUgpK7euEHuQ5pIM9QSOaCo+5FPR7s7
+1nU7YqbFWtd+NhC8Og1G497B31DQlHciF6BRm6/cNGAmHaAErKUGBFdkGtFPHBn4
+vktoEg9fwxJAZLvGpoTZWrB4HRsRwVTmFdGvK+JXK225xF23AXRXp/snhSuSFeLj
+E5cpyJJ7AgMBAAECggEAQOv527X2e/sDr0XSpHZQuT/r9UBpBlnFIlFH+fBF5k0X
+GWv0ae/O6U1dzs0kmX57xG0n0ry6+vTXeleTYiH8cTOd66EzN9AAOO+hG29IGZf9
+HAEZkkO/FARc/mjzdtFnEYsjIHWM3ZWdwQx3Q28JKu6w51rQiN51g3NqOCGdF/uF
+rE5XPKsKndn+nLHvsNuApFgUYZEwdrozgUueEgRaPTUCNhzotcA9eWoBdA24XNhk
+x8Cm/bZWabXm7gBO75zl3Cu2F21ay+EuwyOZTsx6lZi6YX9/zo1mkO81Zi3tQk50
+NMEI0feLNwsdxTbmOcVJadjOgd+QVghlFyr5HGBWMQKBgQD3AH3rhnAo6tOyNkGN
++IzIU1MhUS452O7IavykUYO9sM24BVChpRtlI9Dpev4yE/q3BAO3+oWT3cJrN7/3
+iyo1dzAkpGvI65XWfElXFM4nLjEiZzx4W9fiPN91Oucpr0ED6+BZXTtz4gVm0TP/
+TUc2xvTB6EKvIyWmKOYEi0snxQKBgQDPSOjbz9jWOrC9XY7PmtLB6QJDDz7XSGVK
+wzD+gDAPpAwhk58BEokdOhBx2Lwl8zMJi0CRHgH2vNvkRyhvUQ4UFzisrqann/Tw
+klp5sw3iWC6ERC8z9zL7GfHs7sK3mOVeAdK6ffowPM3JrZ2vPusVBdr0MN3oZwki
+CtNXqbY1PwKBgGheQNbAW6wubX0kB9chavtKmhm937Z5v4vYCSC1gOEqUAKt3EAx
+L74wwBmn6rjmUE382EVpCgBM99WuHONQXmlxD1qsTw763LlgkuzE0cckcYaD8L06
+saHa7uDuHrcyYlpx1L5t8q0ol/e19i6uTKUMtGcq6OJwC3yGU4sgAIWxAoGBAMVq
+qiQXm2vFL+jafxYoXUvDMJ1PmskMsTP4HOR2j8+FrOwZnVk3HxGP6HOVOPRn4JbZ
+YiAT1Uj6a+7I+rCyINdvmlGUcTK6fFzW9oZryvBkjcD483/pkktmVWwTpa2YV/Ml
+h16IdsyUTGYlDUYHhXtbPUJOfDpIT4F1j/0wrFGfAoGAO82BcUsehEUQE0xvQLIn
+7QaFtUI5z19WW730jVuEobiYlh9Ka4DPbKMvka8MwyOxEwhk39gZQavmfG6+wZm+
+kjERU23LhHziJGWS2Um4yIhC7myKbWaLzjHEq72dszLpQku4BzE5fT60fxI7cURD
+WGm/Z3Q2weS3ZGIoMj1RNPI=
 -----END PRIVATE KEY-----
 Certificate:
     Data:
-        Version: 1 (0x0)
-        Serial Number: 12723342612721443282 (0xb09264b1f2da21d2)
+        Version: 3 (0x2)
+        Serial Number:
+            82:ed:bf:41:c8:80:91:9d
     Signature Algorithm: sha1WithRSAEncryption
         Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server
         Validity
-            Not Before: Jan  4 19:47:07 2013 GMT
-            Not After : Nov 13 19:47:07 2022 GMT
+            Not Before: Jan 19 19:09:06 2018 GMT
+            Not After : Nov 28 19:09:06 2027 GMT
         Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=fakehostname
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption
-                Public-Key: (1024 bit)
+                Public-Key: (2048 bit)
                 Modulus:
-                    00:ae:54:42:23:08:e5:59:0d:b3:64:2f:2f:b8:14:
-                    6a:20:dd:15:eb:cd:51:74:63:53:80:c7:01:ed:d9:
-                    cf:36:0b:64:d1:3a:f6:1f:60:3b:d5:42:49:2d:7a:
-                    b4:9e:5f:4f:95:44:bb:41:19:c8:6a:f4:7b:75:76:
-                    36:45:f4:66:85:34:1d:cf:d4:69:8e:2a:c7:b2:c7:
-                    9a:7e:52:61:9a:48:c6:12:67:91:fe:d2:c8:72:4a:
-                    d7:35:1a:1a:55:34:fc:bc:58:a8:8b:86:0a:d1:79:
-                    76:ac:75:2f:63:90:84:4c:75:18:ce:91:c9:d1:dd:
-                    81:7e:bd:1b:ae:0b:5d:c6:39
+                    00:c7:ff:be:a1:64:06:47:e1:c4:95:0a:65:59:6b:
+                    91:c4:a8:a0:a3:08:aa:dc:3f:00:44:0a:cf:41:4b:
+                    ed:e2:6f:03:78:c7:d3:67:14:d3:fe:b4:27:85:1a:
+                    02:57:97:b5:86:16:87:97:82:0d:e8:2b:ed:69:16:
+                    c3:96:a6:c4:47:b8:5d:c1:76:a9:02:9d:28:52:c9:
+                    d9:93:f1:b2:da:18:7c:0a:78:bd:1c:3f:12:4f:e5:
+                    3f:ee:5b:b7:b3:3c:a5:b6:55:65:c6:33:09:b0:45:
+                    a4:49:f5:d8:f6:54:25:2e:db:4b:c3:65:64:86:16:
+                    bf:73:8e:78:e8:25:cc:c1:b4:7d:d1:b1:55:20:a4:
+                    ae:de:b8:41:ee:43:9a:48:33:d4:12:39:a0:a8:fb:
+                    91:4f:47:bb:3b:d6:75:3b:62:a6:c5:5a:d7:7e:36:
+                    10:bc:3a:0d:46:e3:de:c1:df:50:d0:94:77:22:17:
+                    a0:51:9b:af:dc:34:60:26:1d:a0:04:ac:a5:06:04:
+                    57:64:1a:d1:4f:1c:19:f8:be:4b:68:12:0f:5f:c3:
+                    12:40:64:bb:c6:a6:84:d9:5a:b0:78:1d:1b:11:c1:
+                    54:e6:15:d1:af:2b:e2:57:2b:6d:b9:c4:5d:b7:01:
+                    74:57:a7:fb:27:85:2b:92:15:e2:e3:13:97:29:c8:
+                    92:7b
                 Exponent: 65537 (0x10001)
+        X509v3 extensions:
+            X509v3 Subject Alternative Name: 
+                DNS:fakehostname
+            X509v3 Key Usage: critical
+                Digital Signature, Key Encipherment
+            X509v3 Extended Key Usage: 
+                TLS Web Server Authentication, TLS Web Client Authentication
+            X509v3 Basic Constraints: critical
+                CA:FALSE
+            X509v3 Subject Key Identifier: 
+                F8:76:79:CB:11:85:F0:46:E5:95:E6:7E:69:CB:12:5E:4E:AA:EC:4D
+            X509v3 Authority Key Identifier: 
+                keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48
+                DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server
+                serial:82:ED:BF:41:C8:80:91:9B
+
+            Authority Information Access: 
+                CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer
+                OCSP - URI:http://testca.pythontest.net/testca/ocsp/
+
+            X509v3 CRL Distribution Points: 
+
+                Full Name:
+                  URI:http://testca.pythontest.net/testca/revocation.crl
+
     Signature Algorithm: sha1WithRSAEncryption
-         ad:45:8a:8e:ef:c6:ef:04:41:5c:2c:4a:84:dc:02:76:0c:d0:
-         66:0f:f0:16:04:58:4d:fd:68:b7:b8:d3:a8:41:a5:5c:3c:6f:
-         65:3c:d1:f8:ce:43:35:e7:41:5f:53:3d:c9:2c:c3:7d:fc:56:
-         4a:fa:47:77:38:9d:bb:97:28:0a:3b:91:19:7f:bc:74:ae:15:
-         6b:bd:20:36:67:45:a5:1e:79:d7:75:e6:89:5c:6d:54:84:d1:
-         95:d7:a7:b4:33:3c:af:37:c4:79:8f:5e:75:dc:75:c2:18:fb:
-         61:6f:2d:dc:38:65:5b:ba:67:28:d0:88:d7:8d:b9:23:5a:8e:
-         e8:c6:bb:db:ce:d5:b8:41:2a:ce:93:08:b6:95:ad:34:20:18:
-         d5:3b:37:52:74:50:0b:07:2c:b0:6d:a4:4c:7b:f4:e0:fd:d1:
-         af:17:aa:20:cd:62:e3:f0:9d:37:69:db:41:bd:d4:1c:fb:53:
-         20:da:88:9d:76:26:67:ce:01:90:a7:80:1d:a9:5b:39:73:68:
-         54:0a:d1:2a:03:1b:8f:3c:43:5d:5d:c4:51:f1:a7:e7:11:da:
-         31:2c:49:06:af:04:f4:b8:3c:99:c4:20:b9:06:36:a2:00:92:
-         61:1d:0c:6d:24:05:e2:82:e1:47:db:a0:5f:ba:b9:fb:ba:fa:
-         49:12:1e:ce
+         6d:50:8d:fb:ee:4e:93:8b:eb:47:56:ba:38:cc:80:e1:9d:c7:
+         e1:9e:1f:9c:22:0c:d2:08:9b:ed:bf:31:d9:00:ee:af:8c:56:
+         78:92:d1:7c:ba:4e:81:7f:82:1f:f4:68:99:86:91:c6:cb:57:
+         d3:b9:41:12:fa:75:53:fd:22:32:21:50:af:6b:4c:b1:34:36:
+         d1:a8:25:0a:d0:f0:f8:81:7d:69:58:6e:af:e3:d2:c4:32:87:
+         79:d7:cd:ad:0c:56:f3:15:27:10:0c:f9:57:59:53:00:ed:af:
+         5d:4d:07:86:7a:e5:f3:97:88:bc:86:b4:f1:17:46:33:55:28:
+         66:7b:70:d3:a5:12:b9:4f:c7:ed:e6:13:20:2d:f0:9e:ec:17:
+         64:cf:fd:13:14:1b:76:ba:64:ac:c5:51:b6:cd:13:0a:93:b1:
+         fd:43:09:a0:0b:44:6c:77:45:43:0b:e5:ed:70:b2:76:dc:08:
+         4a:5b:73:5f:c1:fc:7f:63:70:f8:b9:ca:3c:98:06:5f:fd:98:
+         d1:e4:e6:61:5f:09:8f:6c:18:86:98:9c:cb:3f:73:7b:3f:38:
+         f5:a7:09:20:ee:a5:63:1c:ff:8b:a6:d1:8c:e8:f4:84:3d:99:
+         38:0f:cc:e0:52:03:f9:18:05:23:76:39:de:52:ce:8e:fb:a6:
+         6e:f5:4f:c3
 -----BEGIN CERTIFICATE-----
-MIICpzCCAY8CCQCwkmSx8toh0jANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJY
+MIIE9zCCA9+gAwIBAgIJAILtv0HIgJGdMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yNzExMjgx
+OTA5MDZaMGIxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj
+MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMMDGZh
+a2Vob3N0bmFtZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMf/vqFk
+BkfhxJUKZVlrkcSooKMIqtw/AEQKz0FL7eJvA3jH02cU0/60J4UaAleXtYYWh5eC
+Degr7WkWw5amxEe4XcF2qQKdKFLJ2ZPxstoYfAp4vRw/Ek/lP+5bt7M8pbZVZcYz
+CbBFpEn12PZUJS7bS8NlZIYWv3OOeOglzMG0fdGxVSCkrt64Qe5Dmkgz1BI5oKj7
+kU9HuzvWdTtipsVa1342ELw6DUbj3sHfUNCUdyIXoFGbr9w0YCYdoASspQYEV2Qa
+0U8cGfi+S2gSD1/DEkBku8amhNlasHgdGxHBVOYV0a8r4lcrbbnEXbcBdFen+yeF
+K5IV4uMTlynIknsCAwEAAaOCAcMwggG/MBcGA1UdEQQQMA6CDGZha2Vob3N0bmFt
+ZTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
+MAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFPh2ecsRhfBG5ZXmfmnLEl5OquxNMH0G
+A1UdIwR2MHSAFJrPz27rcT3bPPGuiGtWcgPLCKdIoVGkTzBNMQswCQYDVQQGEwJY
 WTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNV
-BAMMDW91ci1jYS1zZXJ2ZXIwHhcNMTMwMTA0MTk0NzA3WhcNMjIxMTEzMTk0NzA3
-WjBiMQswCQYDVQQGEwJYWTEXMBUGA1UEBxMOQ2FzdGxlIEFudGhyYXgxIzAhBgNV
-BAoTGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMRUwEwYDVQQDEwxmYWtlaG9z
-dG5hbWUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAK5UQiMI5VkNs2QvL7gU
-aiDdFevNUXRjU4DHAe3ZzzYLZNE69h9gO9VCSS16tJ5fT5VEu0EZyGr0e3V2NkX0
-ZoU0Hc/UaY4qx7LHmn5SYZpIxhJnkf7SyHJK1zUaGlU0/LxYqIuGCtF5dqx1L2OQ
-hEx1GM6RydHdgX69G64LXcY5AgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAK1Fio7v
-xu8EQVwsSoTcAnYM0GYP8BYEWE39aLe406hBpVw8b2U80fjOQzXnQV9TPcksw338
-Vkr6R3c4nbuXKAo7kRl/vHSuFWu9IDZnRaUeedd15olcbVSE0ZXXp7QzPK83xHmP
-XnXcdcIY+2FvLdw4ZVu6ZyjQiNeNuSNajujGu9vO1bhBKs6TCLaVrTQgGNU7N1J0
-UAsHLLBtpEx79OD90a8XqiDNYuPwnTdp20G91Bz7UyDaiJ12JmfOAZCngB2pWzlz
-aFQK0SoDG488Q11dxFHxp+cR2jEsSQavBPS4PJnEILkGNqIAkmEdDG0kBeKC4Ufb
-oF+6ufu6+kkSHs4=
+BAMMDW91ci1jYS1zZXJ2ZXKCCQCC7b9ByICRmzCBgwYIKwYBBQUHAQEEdzB1MDwG
+CCsGAQUFBzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3RjYS9w
+eWNhY2VydC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2EucHl0aG9udGVz
+dC5uZXQvdGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly90ZXN0
+Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3JsMA0GCSqGSIb3
+DQEBBQUAA4IBAQBtUI377k6Ti+tHVro4zIDhncfhnh+cIgzSCJvtvzHZAO6vjFZ4
+ktF8uk6Bf4If9GiZhpHGy1fTuUES+nVT/SIyIVCva0yxNDbRqCUK0PD4gX1pWG6v
+49LEMod5182tDFbzFScQDPlXWVMA7a9dTQeGeuXzl4i8hrTxF0YzVShme3DTpRK5
+T8ft5hMgLfCe7Bdkz/0TFBt2umSsxVG2zRMKk7H9QwmgC0Rsd0VDC+XtcLJ23AhK
+W3Nfwfx/Y3D4uco8mAZf/ZjR5OZhXwmPbBiGmJzLP3N7Pzj1pwkg7qVjHP+LptGM
+6PSEPZk4D8zgUgP5GAUjdjneUs6O+6Zu9U/D
 -----END CERTIFICATE-----
diff --git a/Lib/test/keycertecc.pem b/Lib/test/keycertecc.pem
new file mode 100644
index 00000000000..deb484f9920
--- /dev/null
+++ b/Lib/test/keycertecc.pem
@@ -0,0 +1,96 @@
+-----BEGIN PRIVATE KEY-----
+MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDDe3QWmhZX07HZbntz4
+CFqAOaoYMdYwD7Z3WPNIc2zR7p4D6BMOa7NAWjLV5A7CUw6hZANiAAQ5IVKzLLz4
+LCfcpy6fMOp+jk5KwywsU3upPtjA6E3UetxPcfnnv+gghRyDAYLN2OVqZgLMEmUo
+F1j1SM1QrbhHIuNcVxI9gPPMdumcNFSz/hqxrBRtA/8Z2gywczdNLjc=
+-----END PRIVATE KEY-----
+Certificate:
+    Data:
+        Version: 3 (0x2)
+        Serial Number:
+            82:ed:bf:41:c8:80:91:9e
+    Signature Algorithm: sha1WithRSAEncryption
+        Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server
+        Validity
+            Not Before: Jan 19 19:09:06 2018 GMT
+            Not After : Nov 28 19:09:06 2027 GMT
+        Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=localhost-ecc
+        Subject Public Key Info:
+            Public Key Algorithm: id-ecPublicKey
+                Public-Key: (384 bit)
+                pub:
+                    04:39:21:52:b3:2c:bc:f8:2c:27:dc:a7:2e:9f:30:
+                    ea:7e:8e:4e:4a:c3:2c:2c:53:7b:a9:3e:d8:c0:e8:
+                    4d:d4:7a:dc:4f:71:f9:e7:bf:e8:20:85:1c:83:01:
+                    82:cd:d8:e5:6a:66:02:cc:12:65:28:17:58:f5:48:
+                    cd:50:ad:b8:47:22:e3:5c:57:12:3d:80:f3:cc:76:
+                    e9:9c:34:54:b3:fe:1a:b1:ac:14:6d:03:ff:19:da:
+                    0c:b0:73:37:4d:2e:37
+                ASN1 OID: secp384r1
+                NIST CURVE: P-384
+        X509v3 extensions:
+            X509v3 Subject Alternative Name: 
+                DNS:localhost-ecc
+            X509v3 Key Usage: critical
+                Digital Signature, Key Encipherment
+            X509v3 Extended Key Usage: 
+                TLS Web Server Authentication, TLS Web Client Authentication
+            X509v3 Basic Constraints: critical
+                CA:FALSE
+            X509v3 Subject Key Identifier: 
+                33:23:0E:15:04:83:2E:3D:BF:DA:81:6D:10:38:80:C3:C2:B0:A4:74
+            X509v3 Authority Key Identifier: 
+                keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48
+                DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server
+                serial:82:ED:BF:41:C8:80:91:9B
+
+            Authority Information Access: 
+                CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer
+                OCSP - URI:http://testca.pythontest.net/testca/ocsp/
+
+            X509v3 CRL Distribution Points: 
+
+                Full Name:
+                  URI:http://testca.pythontest.net/testca/revocation.crl
+
+    Signature Algorithm: sha1WithRSAEncryption
+         3b:6f:97:af:7e:5f:e0:14:34:ed:57:7e:de:ce:c4:85:1e:aa:
+         84:52:94:7c:e5:ce:e9:9c:88:8b:ad:b5:4d:16:ac:af:81:ea:
+         b8:a2:e2:50:2e:cb:e9:11:bd:1b:a6:3f:0c:a2:d7:7b:67:72:
+         b3:43:16:ad:c6:87:ac:6e:ac:47:78:ef:2f:8c:86:e8:9b:d1:
+         43:8c:c1:7a:91:30:e9:14:d6:9f:41:8b:9b:0b:24:9b:78:86:
+         11:8a:fc:2b:cd:c9:13:ee:90:4f:14:33:51:a3:c4:9e:d6:06:
+         48:f5:41:12:af:f0:f2:71:40:78:f5:96:c2:5d:cf:e1:38:ff:
+         bf:10:eb:74:2f:c2:23:21:3e:27:f5:f1:f2:af:2c:62:82:31:
+         00:c8:96:1b:c3:7e:8d:71:89:e7:40:b5:67:1a:33:fb:c0:8b:
+         96:0c:36:78:25:27:82:d8:27:27:52:0f:f7:69:cd:ff:2b:92:
+         10:d3:d2:0a:db:65:ed:af:90:eb:db:76:f3:8a:7a:13:9e:c6:
+         33:57:15:42:06:13:d6:54:49:fa:84:a7:0e:1d:14:72:ca:19:
+         8e:2b:aa:a4:02:54:3c:f6:1c:23:81:7a:59:54:b0:92:65:72:
+         c8:e5:ba:9f:03:4e:30:f2:4d:45:85:e3:35:a8:b1:68:58:b9:
+         3b:20:a3:eb
+-----BEGIN CERTIFICATE-----
+MIIESzCCAzOgAwIBAgIJAILtv0HIgJGeMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yNzExMjgx
+OTA5MDZaMGMxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj
+MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xFjAUBgNVBAMMDWxv
+Y2FsaG9zdC1lY2MwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQ5IVKzLLz4LCfcpy6f
+MOp+jk5KwywsU3upPtjA6E3UetxPcfnnv+gghRyDAYLN2OVqZgLMEmUoF1j1SM1Q
+rbhHIuNcVxI9gPPMdumcNFSz/hqxrBRtA/8Z2gywczdNLjejggHEMIIBwDAYBgNV
+HREEETAPgg1sb2NhbGhvc3QtZWNjMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAU
+BggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUMyMO
+FQSDLj2/2oFtEDiAw8KwpHQwfQYDVR0jBHYwdIAUms/PbutxPds88a6Ia1ZyA8sI
+p0ihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
+Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcoIJAILtv0HIgJGb
+MIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKGMGh0dHA6Ly90ZXN0Y2EucHl0
+aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNlcjA1BggrBgEFBQcwAYYpaHR0
+cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2Evb2NzcC8wQwYDVR0fBDww
+OjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2EvcmV2
+b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQEFBQADggEBADtvl69+X+AUNO1Xft7OxIUe
+qoRSlHzlzumciIuttU0WrK+B6rii4lAuy+kRvRumPwyi13tncrNDFq3Gh6xurEd4
+7y+Mhuib0UOMwXqRMOkU1p9Bi5sLJJt4hhGK/CvNyRPukE8UM1GjxJ7WBkj1QRKv
+8PJxQHj1lsJdz+E4/78Q63QvwiMhPif18fKvLGKCMQDIlhvDfo1xiedAtWcaM/vA
+i5YMNnglJ4LYJydSD/dpzf8rkhDT0grbZe2vkOvbdvOKehOexjNXFUIGE9ZUSfqE
+pw4dFHLKGY4rqqQCVDz2HCOBellUsJJlcsjlup8DTjDyTUWF4zWosWhYuTsgo+s=
+-----END CERTIFICATE-----
diff --git a/Lib/test/make_ssl_certs.py b/Lib/test/make_ssl_certs.py
index 4d9f01ba6a8..b908c40c11e 100644
--- a/Lib/test/make_ssl_certs.py
+++ b/Lib/test/make_ssl_certs.py
@@ -2,14 +2,17 @@
 and friends."""
 
 import os
+import pprint
 import shutil
 import tempfile
 from subprocess import *
 
 req_template = """
+    [ default ]
+    base_url               = http://testca.pythontest.net/testca
+
     [req]
     distinguished_name     = req_distinguished_name
-    x509_extensions        = req_x509_extensions
     prompt                 = no
 
     [req_distinguished_name]
@@ -18,8 +21,25 @@
     O                      = Python Software Foundation
     CN                     = {hostname}
 
-    [req_x509_extensions]
+    [req_x509_extensions_simple]
+    subjectAltName         = @san
+
+    [req_x509_extensions_full]
     subjectAltName         = @san
+    keyUsage               = critical,keyEncipherment,digitalSignature
+    extendedKeyUsage       = serverAuth,clientAuth
+    basicConstraints       = critical,CA:false
+    subjectKeyIdentifier   = hash
+    authorityKeyIdentifier = keyid:always,issuer:always
+    authorityInfoAccess    = @issuer_ocsp_info
+    crlDistributionPoints  = @crl_info
+
+    [ issuer_ocsp_info ]
+    caIssuers;URI.0        = $base_url/pycacert.cer
+    OCSP;URI.0             = $base_url/ocsp/
+
+    [ crl_info ]
+    URI.0                  = $base_url/revocation.crl
 
     [san]
     DNS.1 = {hostname}
@@ -56,7 +76,6 @@
     private_key = pycakey.pem
     serial    = $dir/serial
     RANDFILE  = $dir/.rand
-
     policy          = policy_match
 
     [ policy_match ]
@@ -87,7 +106,9 @@
 
 here = os.path.abspath(os.path.dirname(__file__))
 
-def make_cert_key(hostname, sign=False, extra_san=''):
+
+def make_cert_key(hostname, sign=False, extra_san='',
+                  ext='req_x509_extensions_full', key='rsa:2048'):
     print("creating cert for " + hostname)
     tempnames = []
     for i in range(3):
@@ -99,7 +120,8 @@ def make_cert_key(hostname, sign=False, extra_san=''):
         with open(req_file, 'w') as f:
             f.write(req)
         args = ['req', '-new', '-days', '3650', '-nodes',
-                '-newkey', 'rsa:1024', '-keyout', key_file,
+                '-newkey', key, '-keyout', key_file,
+                '-extensions', ext,
                 '-config', req_file]
         if sign:
             with tempfile.NamedTemporaryFile(delete=False) as f:
@@ -112,8 +134,15 @@ def make_cert_key(hostname, sign=False, extra_san=''):
         check_call(['openssl'] + args)
 
         if sign:
-            args = ['ca', '-config', req_file, '-out', cert_file, '-outdir', 'cadir',
-                    '-policy', 'policy_anything', '-batch', '-infiles', reqfile ]
+            args = [
+                'ca',
+                '-config', req_file,
+                '-extensions', ext,
+                '-out', cert_file,
+                '-outdir', 'cadir',
+                '-policy', 'policy_anything',
+                '-batch', '-infiles', reqfile
+            ]
             check_call(['openssl'] + args)
 
 
@@ -157,9 +186,21 @@ def make_ca():
             args = ['ca', '-config', t.name, '-gencrl', '-out', 'revocation.crl']
             check_call(['openssl'] + args)
 
+    # capath hashes depend on subject!
+    check_call([
+        'openssl', 'x509', '-in', 'pycacert.pem', '-out', 'capath/ceff1710.0'
+    ])
+    shutil.copy('capath/ceff1710.0', 'capath/b1930218.0')
+
+
+def print_cert(path):
+    import _ssl
+    pprint.pprint(_ssl._test_decode_cert(path))
+
+
 if __name__ == '__main__':
     os.chdir(here)
-    cert, key = make_cert_key('localhost')
+    cert, key = make_cert_key('localhost', ext='req_x509_extensions_simple')
     with open('ssl_cert.pem', 'w') as f:
         f.write(cert)
     with open('ssl_key.pem', 'w') as f:
@@ -177,7 +218,7 @@ def make_ca():
 
     # For certificate matching tests
     make_ca()
-    cert, key = make_cert_key('fakehostname')
+    cert, key = make_cert_key('fakehostname', ext='req_x509_extensions_simple')
     with open('keycert2.pem', 'w') as f:
         f.write(key)
         f.write(cert)
@@ -192,6 +233,13 @@ def make_ca():
         f.write(key)
         f.write(cert)
 
+    cert, key = make_cert_key(
+        'localhost-ecc', True, key='param:secp384r1.pem'
+    )
+    with open('keycertecc.pem', 'w') as f:
+        f.write(key)
+        f.write(cert)
+
     extra_san = [
         'otherName.1 = 1.2.3.4;UTF8:some other identifier',
         'otherName.2 = 1.3.6.1.5.2.2;SEQUENCE:princ_name',
@@ -211,6 +259,24 @@ def make_ca():
         f.write(key)
         f.write(cert)
 
+    extra_san = [
+        # k?nig (king)
+        'DNS.2 = xn--knig-5qa.idn.pythontest.net',
+        # k?nigsg??chen (king's alleyway)
+        'DNS.3 = xn--knigsgsschen-lcb0w.idna2003.pythontest.net',
+        'DNS.4 = xn--knigsgchen-b4a3dun.idna2008.pythontest.net',
+        # ????? (marble)
+        'DNS.5 = xn--nxasmq6b.idna2003.pythontest.net',
+        'DNS.6 = xn--nxasmm1c.idna2008.pythontest.net',
+    ]
+
+    # IDN SANS, signed
+    cert, key = make_cert_key('idnsans', True, extra_san='\n'.join(extra_san))
+    with open('idnsans.pem', 'w') as f:
+        f.write(key)
+        f.write(cert)
+
     unmake_ca()
-    print("\n\nPlease change the values in test_ssl.py, test_parse_cert function related to notAfter,notBefore and serialNumber")
-    check_call(['openssl','x509','-in','keycert.pem','-dates','-serial','-noout'])
+    print("update Lib/test/test_ssl.py and Lib/test/test_asyncio/util.py")
+    print_cert('keycert.pem')
+    print_cert('keycert3.pem')
diff --git a/Lib/test/pycacert.pem b/Lib/test/pycacert.pem
index 09b1f3e08ae..850fa32aef7 100644
--- a/Lib/test/pycacert.pem
+++ b/Lib/test/pycacert.pem
@@ -1,78 +1,79 @@
 Certificate:
     Data:
         Version: 3 (0x2)
-        Serial Number: 12723342612721443280 (0xb09264b1f2da21d0)
+        Serial Number:
+            82:ed:bf:41:c8:80:91:9b
     Signature Algorithm: sha1WithRSAEncryption
         Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server
         Validity
-            Not Before: Jan  4 19:47:07 2013 GMT
-            Not After : Jan  2 19:47:07 2023 GMT
+            Not Before: Jan 19 19:09:06 2018 GMT
+            Not After : Jan 17 19:09:06 2028 GMT
         Subject: C=XY, O=Python Software Foundation CA, CN=our-ca-server
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption
                 Public-Key: (2048 bit)
                 Modulus:
-                    00:e7:de:e9:e3:0c:9f:00:b6:a1:fd:2b:5b:96:d2:
-                    6f:cc:e0:be:86:b9:20:5e:ec:03:7a:55:ab:ea:a4:
-                    e9:f9:49:85:d2:66:d5:ed:c7:7a:ea:56:8e:2d:8f:
-                    e7:42:e2:62:28:a9:9f:d6:1b:8e:eb:b5:b4:9c:9f:
-                    14:ab:df:e6:94:8b:76:1d:3e:6d:24:61:ed:0c:bf:
-                    00:8a:61:0c:df:5c:c8:36:73:16:00:cd:47:ba:6d:
-                    a4:a4:74:88:83:23:0a:19:fc:09:a7:3c:4a:4b:d3:
-                    e7:1d:2d:e4:ea:4c:54:21:f3:26:db:89:37:18:d4:
-                    02:bb:40:32:5f:a4:ff:2d:1c:f7:d4:bb:ec:8e:cf:
-                    5c:82:ac:e6:7c:08:6c:48:85:61:07:7f:25:e0:5c:
-                    e0:bc:34:5f:e0:b9:04:47:75:c8:47:0b:8d:bc:d6:
-                    c8:68:5f:33:83:62:d2:20:44:35:b1:ad:81:1a:8a:
-                    cd:bc:35:b0:5c:8b:47:d6:18:e9:9c:18:97:cc:01:
-                    3c:29:cc:e8:1e:e4:e4:c1:b8:de:e7:c2:11:18:87:
-                    5a:93:34:d8:a6:25:f7:14:71:eb:e4:21:a2:d2:0f:
-                    2e:2e:d4:62:00:35:d3:d6:ef:5c:60:4b:4c:a9:14:
-                    e2:dd:15:58:46:37:33:26:b7:e7:2e:5d:ed:42:e4:
-                    c5:4d
+                    00:c3:18:69:6b:c9:47:29:98:8e:b1:56:c2:2e:fa:
+                    0e:5e:bc:23:80:b3:07:62:24:d2:42:5b:f1:4a:bf:
+                    a9:c8:21:75:c8:e3:e6:2c:1f:87:3c:6e:7c:1b:ed:
+                    39:32:95:b7:40:b2:60:48:c3:9a:16:08:fe:6d:67:
+                    88:34:3b:77:77:70:1c:70:5a:d1:1f:5f:04:21:54:
+                    b9:0c:e3:41:85:1d:58:ee:2f:ed:f3:0e:ef:d8:23:
+                    a1:fa:73:fb:4c:28:e0:e5:e6:4d:0b:02:52:49:86:
+                    c7:be:7e:bd:e6:56:76:8b:70:8e:0a:8f:06:33:20:
+                    1d:7b:5b:aa:d0:c5:1b:ab:9b:cc:54:09:3c:bf:e4:
+                    40:66:f1:fb:d6:f7:16:9d:c4:19:d4:c3:f2:ff:07:
+                    bc:6f:5a:9e:25:1b:02:4a:a5:ec:42:96:3a:70:d2:
+                    6c:99:2b:ce:be:e8:d2:01:ef:d5:ba:b0:cf:94:3e:
+                    82:d0:01:d6:4b:71:80:03:0a:12:45:86:79:81:d8:
+                    4b:d2:e8:b5:b7:2c:6c:9a:4c:8a:10:10:e4:e4:f5:
+                    df:ce:84:91:ca:d1:46:e0:84:73:17:66:db:69:43:
+                    78:80:83:be:14:4d:f1:3e:1a:d6:6c:f5:de:45:f3:
+                    39:af:91:d5:3d:54:44:bf:41:cc:73:68:1a:fc:24:
+                    db:91
                 Exponent: 65537 (0x10001)
         X509v3 extensions:
             X509v3 Subject Key Identifier: 
-                BC:DD:62:D9:76:DA:1B:D2:54:6B:CF:E0:66:9B:1E:1E:7B:56:0C:0B
+                9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48
             X509v3 Authority Key Identifier: 
-                keyid:BC:DD:62:D9:76:DA:1B:D2:54:6B:CF:E0:66:9B:1E:1E:7B:56:0C:0B
+                keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48
 
             X509v3 Basic Constraints: 
                 CA:TRUE
     Signature Algorithm: sha1WithRSAEncryption
-         7d:0a:f5:cb:8d:d3:5d:bd:99:8e:f8:2b:0f:ba:eb:c2:d9:a6:
-         27:4f:2e:7b:2f:0e:64:d8:1c:35:50:4e:ee:fc:90:b9:8d:6d:
-         a8:c5:c6:06:b0:af:f3:2d:bf:3b:b8:42:07:dd:18:7d:6d:95:
-         54:57:85:18:60:47:2f:eb:78:1b:f9:e8:17:fd:5a:0d:87:17:
-         28:ac:4c:6a:e6:bc:29:f4:f4:55:70:29:42:de:85:ea:ab:6c:
-         23:06:64:30:75:02:8e:53:bc:5e:01:33:37:cc:1e:cd:b8:a4:
-         fd:ca:e4:5f:65:3b:83:1c:86:f1:55:02:a0:3a:8f:db:91:b7:
-         40:14:b4:e7:8d:d2:ee:73:ba:e3:e5:34:2d:bc:94:6f:4e:24:
-         06:f7:5f:8b:0e:a7:8e:6b:de:5e:75:f4:32:9a:50:b1:44:33:
-         9a:d0:05:e2:78:82:ff:db:da:8a:63:eb:a9:dd:d1:bf:a0:61:
-         ad:e3:9e:8a:24:5d:62:0e:e7:4c:91:7f:ef:df:34:36:3b:2f:
-         5d:f5:84:b2:2f:c4:6d:93:96:1a:6f:30:28:f1:da:12:9a:64:
-         b4:40:33:1d:bd:de:2b:53:a8:ea:be:d6:bc:4e:96:f5:44:fb:
-         32:18:ae:d5:1f:f6:69:af:b6:4e:7b:1d:58:ec:3b:a9:53:a3:
-         5e:58:c8:9e
+         10:25:c8:dc:0c:55:5c:cb:83:6e:79:ef:77:ec:0d:8e:0c:06:
+         c1:4b:0c:d6:f7:75:52:21:b8:17:4a:38:88:9d:b3:78:c4:42:
+         fb:b8:7c:14:38:10:fb:ac:da:11:00:5b:42:87:5e:45:9f:6d:
+         4e:42:a4:9a:18:06:39:0f:45:a6:96:89:32:d6:59:b3:d3:8e:
+         e3:95:b6:c4:a2:4b:74:2f:67:c1:fb:bb:f9:72:6f:37:4a:e7:
+         f4:48:33:71:df:b8:f5:e6:41:3f:d5:d5:2f:26:09:f8:0e:92:
+         ff:70:ea:f6:ab:58:fb:90:04:d6:43:2e:8f:b1:fb:06:ab:69:
+         d0:dc:a8:f8:5b:07:f2:d4:66:1f:63:f8:5d:c1:9e:41:44:bb:
+         c9:e8:7d:e0:46:e4:a7:c8:32:5f:31:62:e5:1c:5c:89:dd:b7:
+         a2:4f:9e:0d:13:b8:5f:b1:84:53:4c:1f:ce:19:e1:01:00:5e:
+         bf:41:55:94:a9:a5:13:db:f2:59:f3:d6:4e:b9:9d:9d:b9:0a:
+         d9:b2:18:6d:7c:b1:f7:96:aa:bd:f6:f9:95:0f:4a:6e:3c:7c:
+         46:5b:df:d4:78:ec:9a:dc:e2:e3:01:e6:88:77:39:93:9c:ba:
+         2a:63:f9:25:4b:4f:ac:08:79:39:c6:7b:df:07:35:ba:c0:c2:
+         50:bf:5a:81
 -----BEGIN CERTIFICATE-----
-MIIDbTCCAlWgAwIBAgIJALCSZLHy2iHQMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
+MIIDbTCCAlWgAwIBAgIJAILtv0HIgJGbMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
 BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW
-MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xMzAxMDQxOTQ3MDdaFw0yMzAxMDIx
-OTQ3MDdaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
+MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yODAxMTcx
+OTA5MDZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg
 Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI
-hvcNAQEBBQADggEPADCCAQoCggEBAOfe6eMMnwC2of0rW5bSb8zgvoa5IF7sA3pV
-q+qk6flJhdJm1e3HeupWji2P50LiYiipn9Ybjuu1tJyfFKvf5pSLdh0+bSRh7Qy/
-AIphDN9cyDZzFgDNR7ptpKR0iIMjChn8Cac8SkvT5x0t5OpMVCHzJtuJNxjUArtA
-Ml+k/y0c99S77I7PXIKs5nwIbEiFYQd/JeBc4Lw0X+C5BEd1yEcLjbzWyGhfM4Ni
-0iBENbGtgRqKzbw1sFyLR9YY6ZwYl8wBPCnM6B7k5MG43ufCERiHWpM02KYl9xRx
-6+QhotIPLi7UYgA109bvXGBLTKkU4t0VWEY3Mya35y5d7ULkxU0CAwEAAaNQME4w
-HQYDVR0OBBYEFLzdYtl22hvSVGvP4GabHh57VgwLMB8GA1UdIwQYMBaAFLzdYtl2
-2hvSVGvP4GabHh57VgwLMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
-AH0K9cuN0129mY74Kw+668LZpidPLnsvDmTYHDVQTu78kLmNbajFxgawr/Mtvzu4
-QgfdGH1tlVRXhRhgRy/reBv56Bf9Wg2HFyisTGrmvCn09FVwKULeheqrbCMGZDB1
-Ao5TvF4BMzfMHs24pP3K5F9lO4MchvFVAqA6j9uRt0AUtOeN0u5zuuPlNC28lG9O
-JAb3X4sOp45r3l519DKaULFEM5rQBeJ4gv/b2opj66nd0b+gYa3jnookXWIO50yR
-f+/fNDY7L131hLIvxG2TlhpvMCjx2hKaZLRAMx293itTqOq+1rxOlvVE+zIYrtUf
-9mmvtk57HVjsO6lTo15YyJ4=
+hvcNAQEBBQADggEPADCCAQoCggEBAMMYaWvJRymYjrFWwi76Dl68I4CzB2Ik0kJb
+8Uq/qcghdcjj5iwfhzxufBvtOTKVt0CyYEjDmhYI/m1niDQ7d3dwHHBa0R9fBCFU
+uQzjQYUdWO4v7fMO79gjofpz+0wo4OXmTQsCUkmGx75+veZWdotwjgqPBjMgHXtb
+qtDFG6ubzFQJPL/kQGbx+9b3Fp3EGdTD8v8HvG9aniUbAkql7EKWOnDSbJkrzr7o
+0gHv1bqwz5Q+gtAB1ktxgAMKEkWGeYHYS9LotbcsbJpMihAQ5OT1386EkcrRRuCE
+cxdm22lDeICDvhRN8T4a1mz13kXzOa+R1T1URL9BzHNoGvwk25ECAwEAAaNQME4w
+HQYDVR0OBBYEFJrPz27rcT3bPPGuiGtWcgPLCKdIMB8GA1UdIwQYMBaAFJrPz27r
+cT3bPPGuiGtWcgPLCKdIMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB
+ABAlyNwMVVzLg25573fsDY4MBsFLDNb3dVIhuBdKOIids3jEQvu4fBQ4EPus2hEA
+W0KHXkWfbU5CpJoYBjkPRaaWiTLWWbPTjuOVtsSiS3QvZ8H7u/lybzdK5/RIM3Hf
+uPXmQT/V1S8mCfgOkv9w6varWPuQBNZDLo+x+waradDcqPhbB/LUZh9j+F3BnkFE
+u8nofeBG5KfIMl8xYuUcXIndt6JPng0TuF+xhFNMH84Z4QEAXr9BVZSppRPb8lnz
+1k65nZ25CtmyGG18sfeWqr32+ZUPSm48fEZb39R47Jrc4uMB5oh3OZOcuipj+SVL
+T6wIeTnGe98HNbrAwlC/WoE=
 -----END CERTIFICATE-----
diff --git a/Lib/test/pycakey.pem b/Lib/test/pycakey.pem
index fc6effefb21..16b75879d09 100644
--- a/Lib/test/pycakey.pem
+++ b/Lib/test/pycakey.pem
@@ -1,28 +1,28 @@
 -----BEGIN PRIVATE KEY-----
-MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDn3unjDJ8AtqH9
-K1uW0m/M4L6GuSBe7AN6VavqpOn5SYXSZtXtx3rqVo4tj+dC4mIoqZ/WG47rtbSc
-nxSr3+aUi3YdPm0kYe0MvwCKYQzfXMg2cxYAzUe6baSkdIiDIwoZ/AmnPEpL0+cd
-LeTqTFQh8ybbiTcY1AK7QDJfpP8tHPfUu+yOz1yCrOZ8CGxIhWEHfyXgXOC8NF/g
-uQRHdchHC4281shoXzODYtIgRDWxrYEais28NbBci0fWGOmcGJfMATwpzOge5OTB
-uN7nwhEYh1qTNNimJfcUcevkIaLSDy4u1GIANdPW71xgS0ypFOLdFVhGNzMmt+cu
-Xe1C5MVNAgMBAAECggEBAJPM7QuUrPn4cLN/Ysd15lwTWn9oHDFFgkYFvCs66gXE
-ju/6Kx2BjWE4wTJby09AHM/MqB0DvguT7Mf1Q2j3tPQ1HZowg8OwRDleuwp6KIls
-jBbhL0Jdl/5HC67ktWvZ9wNvO/wFG1rQfT6FVajf9LUbWEaSZbOG2SLhHfsHorzu
-xjTJaI3bQ/0+79B1exwk5ruwhzFRd/XpY8hls7D/RfPIuHDlBghkW3N59KFWrf5h
-6bNEh2THm0+IyGcGqs0FD+QCOXyvsjwSUswqrr2ctLREOeDcd5ReUjSxYgjcJRrm
-J7ceIY/+uwDJxw/OlnmBvF6pQMkKwYW2gFztu+g2t4UCgYEA/9yo01Exz4crxXsy
-tAlnDJM++nZcm07rtFjTKHUfKY/cCgNTa8udM0svnfwlid/dpgLsI38gx04HHC1i
-EZ4acz+ToIWedLxM0nq73//xeRWEazOvCz1mMTZaMldahTWAyzN8qVK2B/625Yy4
-wNYWyweBBwEB8MzaCs73spksXOsCgYEA5/7wvhiofYGFAfMuANeJIwDL2OtBnoOv
-mVNfCmi3GC38fzwyi5ZpskWDiS2woJ+LQfs9Qu4EcZbUFLd7gbeOvb5gmFUtYope
-LitUUKunIR18MkQ+mQDBpQPQPhk4QJP5reCbWkrfTu7b5o/iS41s6fBTFmuzhLcT
-C71vFdCyeKcCgYAiCCqYeOtELDmBOeLDmaCQRqGQ1N96dOPbCBmF/xYXBCCDYG/f
-HaUaJnz96YTgstsbcrYP/p/Qgqtlbw/lQf9IpwMuzbcG1ejt8g89OyDWNyt2ytgU
-iaUnFJCos3/Byh0Iah/BsdOueo2/OJl2ZMOBW80orlSgv86cs2y037TL4wKBgQDm
-OOyW+MlbowhnIvfoBfwlLEkefnej4nKD6WRLZBcue5Qyf355X06Mhsc9foXlH+6G
-D9h/bswiHNdhp6N82rdgPGiHQx/CxiUoE/+b/nvgNO5mw6qLE2EXbG1e8pAMJcyE
-bHw+YkawggDfELI036fRj5gki8SeUz8nS1nNgElbyQKBgCRDX9Jh+MwSLu4QBWdt
-/fi+lv3K6kun/fI7EOV1vCV/j871tICu7pu5BrOLxAHqoVfU9AUX299/2KjCb5pv
-kjogiUK6qWCWBlfuqDNWGCoUGt1rhznUva0nNjSMy5rinBhhjpROZC2pw48lOluP
-UuvXsaPph7GTqPuy4Kab12YC
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDDGGlryUcpmI6x
+VsIu+g5evCOAswdiJNJCW/FKv6nIIXXI4+YsH4c8bnwb7TkylbdAsmBIw5oWCP5t
+Z4g0O3d3cBxwWtEfXwQhVLkM40GFHVjuL+3zDu/YI6H6c/tMKODl5k0LAlJJhse+
+fr3mVnaLcI4KjwYzIB17W6rQxRurm8xUCTy/5EBm8fvW9xadxBnUw/L/B7xvWp4l
+GwJKpexCljpw0myZK86+6NIB79W6sM+UPoLQAdZLcYADChJFhnmB2EvS6LW3LGya
+TIoQEOTk9d/OhJHK0UbghHMXZttpQ3iAg74UTfE+GtZs9d5F8zmvkdU9VES/Qcxz
+aBr8JNuRAgMBAAECggEAZHgv4hg3k45C/cSmH7caq2LMDb0kskAwH4hlzI7DipLg
+q2Hh6Rsbc92aAG+8IvbC9ohl2VMSCQL8s667j9qH/XQ40QuT4kn2QIv2+FIYLcsd
+Pxxjt+YbUf2XrvkHkwMCPqLJTkAVzFOijdGLThF83vZJz9oz4SRKyno8j2LSix68
+WEfnjdyWqYb0eS0luKrLHw+IL7bD5vfc/P0q6u31zJ9h8zEyN5EBCj5OxM/hD0VO
+nObrp6r9Bs+xx+yRx+8J5Db6LPXggl5nBqsqrDKVDe6uTysYVgstqkfaDv1L78Vu
+3BNdKPAdJ+ucPJrQufzFHBDIIN+Xwckf/09gdQagGQKBgQDnvFaOjZfqc6wL/kNK
+tszQtedbdwP20L+EWdNEVsVWK1TOw36Pmkrp2AYLXMd7W1QQu0KukM89EFb84wKo
+s4C9V/ch162mUhEAveaLioi7bMwMPIib2V6pHmYGG8nQVRvgkZVYx6ZtPEvWye1v
+wmCzzxxK0gC6PQGxp8MSv9yXDwKBgQDXhe57ufc52pgJ+Agyl4PLkllIbG2DKQHG
+LwY06v73jllirTpWBOBvN0NvEsI2Pj4aK/BXRNYN1PS7xi/3C6MVWxnOpBtbq3H5
+DwFb5mpfgJmhV6DZ6jMw7h3Yvy35ViKoiI9UK3eTmhkerH3DsILEje7jE9dGmIOJ
+4oLa50JjXwKBgQDdTfyveMNasIrejTzATmC89Or0a22KuQIdKBddjSw5xXnhV8s2
+4temCJqFIV6UDLz0mZDt2vc+zqr0KOtyJrLMoAQv+qQoUPlR5wkTvAImU5luGiUw
+CN+gzJoMPV93KMBNr1qcBVaHvWyDvCWXdF8beLABOBpfwUEr4xWlgzrruwKBgCvf
+cr2zDJW1Xu/gkuKhn02ofA5XLC/gACF03yGUmNSSILYKp25tTba2HD8XJXvfTcsM
+GL/bHmvwZuV2obr7nnYxdl5vX7ZYfzoBCPjJPew1BJEognD50PPr9R1zRYuVMjb2
+nZ63vn7IhsaMvIlCfExAzFljZ5ZSY6yE9LhVDVmnAoGBALOwMwpkm1drx5UNSJO7
+70Q8kYzg0oQhCo/7b6DWbAglDPSWQS5IA4rHYOwL3sE+69G2Exe+1454rVDisojW
+XdSyA3svI/YQeom8R2LIM/ayCPxCc3/Dxy9+aQQT4lW3F0XQIxod/QsQJxpZIOnF
+jOSPclypgV2X6dDOwDkd2Tgh
 -----END PRIVATE KEY-----
diff --git a/Lib/test/revocation.crl b/Lib/test/revocation.crl
index 6d89b08ebea..53cb4b3721d 100644
--- a/Lib/test/revocation.crl
+++ b/Lib/test/revocation.crl
@@ -1,11 +1,11 @@
 -----BEGIN X509 CRL-----
 MIIBpjCBjwIBATANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJYWTEmMCQGA1UE
 CgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNVBAMMDW91ci1j
-YS1zZXJ2ZXIXDTEzMTEyMTE3MDg0N1oXDTIzMDkzMDE3MDg0N1qgDjAMMAoGA1Ud
-FAQDAgEAMA0GCSqGSIb3DQEBBQUAA4IBAQCNJXC2mVKauEeN3LlQ3ZtM5gkH3ExH
-+i4bmJjtJn497WwvvoIeUdrmVXgJQR93RtV37hZwN0SXMLlNmUZPH4rHhihayw4m
-unCzVj/OhCCY7/TPjKuJ1O/0XhaLBpBVjQN7R/1ujoRKbSia/CD3vcn7Fqxzw7LK
-fSRCKRGTj1CZiuxrphtFchwALXSiFDy9mr2ZKhImcyq1PydfgEzU78APpOkMQsIC
-UNJ/cf3c9emzf+dUtcMEcejQ3mynBo4eIGg1EW42bz4q4hSjzQlKcBV0muw5qXhc
-HOxH2iTFhQ7SrvVuK/dM14rYM4B5mSX3nRC1kNmXpS9j3wJDhuwmjHed
+YS1zZXJ2ZXIXDTE4MDExOTE5MDkwNloXDTI3MTEyODE5MDkwNlqgDjAMMAoGA1Ud
+FAQDAgEAMA0GCSqGSIb3DQEBBQUAA4IBAQBYVzH8n2LdyJJ/t8CpWjz652hZJ0sY
+DeNYcwuTPzR9CbSwEwNbf0kY9bgWXGfoRD2SnnCnuNNJXO2MuXtxf6qYx2ZjhJm8
+qgxXs0Bz4agRswNMbumjHCmqIv1t88vbrO0+ItEZDK7RJVIMBtVJ0XYOHvD/IG/i
+zqa1Fl3uCTvQbTJ2TrqzJeP/Vl40hOD+VdBBZK3j0r4AkCKU3tAiHYTGmHKhPxy1
+f8Yet+4SRMGp1BdDezTI1bICpSZhRJ4geW0UzuCZnXPW8IZzioUmdUBAmAMHPWFr
+B0sTTc/ntD4jHG1/T5b0oiDMbXIbh5Iz9iQNcY0IbotkCw39h+K90wY6
 -----END X509 CRL-----
diff --git a/Lib/test/secp384r1.pem b/Lib/test/secp384r1.pem
new file mode 100644
index 00000000000..eef7117af7a
--- /dev/null
+++ b/Lib/test/secp384r1.pem
@@ -0,0 +1,7 @@
+$ openssl genpkey -genparam -algorithm EC -pkeyopt ec_paramgen_curve:secp384r1 -pkeyopt ec_param_enc:named_curve -text
+-----BEGIN EC PARAMETERS-----
+BgUrgQQAIg==
+-----END EC PARAMETERS-----
+ECDSA-Parameters: (384 bit)
+ASN1 OID: secp384r1
+NIST CURVE: P-384
diff --git a/Lib/test/ssl_cert.pem b/Lib/test/ssl_cert.pem
index 47a7d7e37e8..b1dd3f387f7 100644
--- a/Lib/test/ssl_cert.pem
+++ b/Lib/test/ssl_cert.pem
@@ -1,15 +1,20 @@
 -----BEGIN CERTIFICATE-----
-MIICVDCCAb2gAwIBAgIJANfHOBkZr8JOMA0GCSqGSIb3DQEBBQUAMF8xCzAJBgNV
-BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u
-IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMDEw
-MDgyMzAxNTZaFw0yMDEwMDUyMzAxNTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH
-Ew5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZvdW5k
-YXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAw
-gYkCgYEA21vT5isq7F68amYuuNpSFlKDPrMUCa4YWYqZRt2OZ+/3NKaZ2xAiSwr7
-6MrQF70t5nLbSPpqE5+5VrS58SY+g/sXLiFd6AplH1wJZwh78DofbFYXUggktFMt
-pTyiX8jtP66bkcPkDADA089RI1TQR6Ca+n7HFa7c1fabVV6i3zkCAwEAAaMYMBYw
-FAYDVR0RBA0wC4IJbG9jYWxob3N0MA0GCSqGSIb3DQEBBQUAA4GBAHPctQBEQ4wd
-BJ6+JcpIraopLn8BGhbjNWj40mmRqWB/NAWF6M5ne7KpGAu7tLeG4hb1zLaldK8G
-lxy2GPSRF6LFS48dpEj2HbMv2nvv6xxalDMJ9+DicWgAKTQ6bcX2j3GUkCR0g/T1
-CRlNBAAlvhKzO7Clpf9l0YKBEfraJByX
+MIIDWTCCAkGgAwIBAgIJAPm6B21bar2bMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV
+BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u
+IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODAx
+MTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH
+DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k
+YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEP
+ADCCAQoCggEBAKvvsX2gEti4shve3iYMc+jE4Se7WHs1Bol2f21H8qNboDOFdeb1
+RKHjmq3exHpajywOUEgne9nKHJY/3f2phR4Y5klqG6liLgiSpVyRlcBGbeT2qEAj
+9oLiLFUXLGfGDds2mTwivQDLJBWi51j7ff5k2Pr58fN5ugYMn24T9FNyn0moT+qj
+SFoBNm58l9jrdkJSlgWfqPlbiMa+mqDn/SFtrwLF2Trbfzu42Sd9UdIzMaSSrzbN
+sGm53pNhCh8KndWUQ8GPP2IsLPoUU4qAtmZuTxCx2S1cXrN9EkmT69tlOH84YfSn
+96Ih9bWRc7M5y5bfVdEVM+fKQl3hBRf05qMCAwEAAaMYMBYwFAYDVR0RBA0wC4IJ
+bG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQAtQ8f37cCEk7/rAcbYR53ce3iK
+Vpihb0U2ni1QjG9Tg9UIExkIGkwTiCm7kwQL+GEStBu9AG/QVrOjeTriRiddhWkk
+ze8kRaI3AC/63t6Vh9Q1x6PESgeE4OtAO9JpJCf4GILglA789Y/b/GF8zJZQxR13
+qpB4ZwWw7gCBhdEW59u6CFeBmfDa58hM8lWvuVoRrTi7bjUeC6PAn5HVMzZSykhu
+4HaUfBp6bKFjuym2+h/VvM1n8C3chjVSmutsLb6ELdD8IK0vPV/yf5+LN256zSsS
+dyUZYd8XwQaioEMKdbhLvnehyzHiWfQIUR3BdhONxoIJhHv/EAo8eCkHHYIF
 -----END CERTIFICATE-----
diff --git a/Lib/test/ssl_key.passwd.pem b/Lib/test/ssl_key.passwd.pem
index 2524672e70d..669c7ce3c81 100644
--- a/Lib/test/ssl_key.passwd.pem
+++ b/Lib/test/ssl_key.passwd.pem
@@ -1,18 +1,30 @@
 -----BEGIN RSA PRIVATE KEY-----
 Proc-Type: 4,ENCRYPTED
-DEK-Info: DES-EDE3-CBC,1A8D9D2A02EC698A
+DEK-Info: DES-EDE3-CBC,2D5DD30B9D440DBB
 
-kJYbfZ8L0sfe9Oty3gw0aloNnY5E8fegRfQLZlNoxTl6jNt0nIwI8kDJ36CZgR9c
-u3FDJm/KqrfUoz8vW+qEnWhSG7QPX2wWGPHd4K94Yz/FgrRzZ0DoK7XxXq9gOtVA
-AVGQhnz32p+6WhfGsCr9ArXEwRZrTk/FvzEPaU5fHcoSkrNVAGX8IpSVkSDwEDQr
-Gv17+cfk99UV1OCza6yKHoFkTtrC+PZU71LomBabivS2Oc4B9hYuSR2hF01wTHP+
-YlWNagZOOVtNz4oKK9x9eNQpmfQXQvPPTfusexKIbKfZrMvJoxcm1gfcZ0H/wK6P
-6wmXSG35qMOOztCZNtperjs1wzEBXznyK8QmLcAJBjkfarABJX9vBEzZV0OUKhy+
-noORFwHTllphbmydLhu6ehLUZMHPhzAS5UN7srtpSN81eerDMy0RMUAwA7/PofX1
-94Me85Q8jP0PC9ETdsJcPqLzAPETEYu0ELewKRcrdyWi+tlLFrpE5KT/s5ecbl9l
-7B61U4Kfd1PIXc/siINhU3A3bYK+845YyUArUOnKf1kEox7p1RpD7yFqVT04lRTo
-cibNKATBusXSuBrp2G6GNuhWEOSafWCKJQAzgCYIp6ZTV2khhMUGppc/2H3CF6cO
-zX0KtlPVZC7hLkB6HT8SxYUwF1zqWY7+/XPPdc37MeEZ87Q3UuZwqORLY+Z0hpgt
-L5JXBCoklZhCAaN2GqwFLXtGiRSRFGY7xXIhbDTlE65Wv1WGGgDLMKGE1gOz3yAo
-2jjG1+yAHJUdE69XTFHSqSkvaloA1W03LdMXZ9VuQJ/ySXCie6ABAQ==
+01gIwpy3XxPGsY0PoK59vAxdLhkVj3odO0Z1ULamUzIte6ThKL1HqnZiUlXpYKfK
+XqHVVeQ1xouxiDRNFLJ4CqBG4HbRtqTkl+sfaNTVveL18lOOMAZy6W3dCGAnWOTZ
+Z0RJyZlQxxjNQLuko4tIvFkrShXIgdiVFjwAhRU0KTUb7UQ2xfFA9R0Kfde30pzz
+zSjb/OmYqAIhkdvafGXvJxzZAorQkU9akDh+uJ6cht5B/RGZsbKACYDSv2WSV5yW
+r+fKVYcTup33r0Jj8hAD6fVY15K8BJknpkF9HfSlZnmmr2WDaffLokOOnCV/I1ie
+WD7ENA7K//48km5D3Ogh2b2/0Iwuzjq8Mvd8aR39N9nINbGR+HNT85pawoo1S0W9
+pQTU4XTmxfXjtR2287C6XZyQ/tBwvNDMFPVhlxsGOdLYwoV5e/L1t1qIfkTlbuvd
+JaMzOhSSLjiC156IFoH7PTPe+g75hw2b32XJURFGlaYknHF7P4BmCiwXOQYo5CCo
+MQGGlw5qBCqODrIsc03wpL2jUzgvyPqLyaw395ITuSoGX+WO7vUQaGW0Tz/sOoTs
+3pK+bTi2QmqZMe7xBOj07CYMMOo4QPrM6NpbObt+Jja2UXaxvKa9BwqCEQzA4pQZ
+8ZHHfEWIaDffKTGkAlqm+S8qCtsrEZJhIn3aI/ikzK8v+YkWw6w+8t/tR1V8ET/s
+CoYGIR7I8WhdfKAwgx2QT5bt1jkYKJyKPm4Iacp2mNh9gNFVq+JSKF318e7BrR3+
+wyqMkDxRYnov3ybtf6kiICxPREDqa6UG1xRq3SbWz6NnIF/1hoHs79YlSYbMfXNU
+ffIxBaXNCcH6jM9duP2YRnO29jLwfLM/mmokTBBjyOBaKZia9GPa4naXoATXW3z+
+Xx4EKIUkKdb53kiV6NtEKMPialAnkeoHTEjyLPgaV8mCHLvGQbnxbYwvpPJH0e2f
+CWgiw6ci4ROOzcZ7HJHIDUprwK0xRKn43hoI44fivlSHOFX6da6o3wIqhEUqMKwL
+JQDS1GORRk1ndRXP+7Ub1dO+Vo/DqO1VcTr2o5RwZ1LWPnzLqbCG50mvTLH4djB+
++hf6vlmnFC30N3yUFXWE5vS10nJHYP88dD9CB2RsaWzpxD9Zxl+PKcRsppen6HyO
+u3b71a/TBOkJcI+lkOatEFvbuqzBAqhMceMctO+Dl55RFsbxfIw/IXZjdP0PYZ0C
+t20DrIdBsvl9F/mfYpmkV4DF7yci78DqnRBcxylVNF2vwX7o+2fq/TsEwsHn3KnT
+kvcF5Cq8Vr5C8ugWX8JfveNym0BjLu6Lr58qS4a6qCNGEGPFKyB+xkm4KEScbarQ
+aLbEbfulMM7q9//sEOOLexIx7mNoLd29Xzn5hsLCAZLWX6wMq6JVJ/zbBOAHDbBT
+yhi03yd5Kvw3swSt4QZj+uR3qTFwxkXUFiVvrSfxRZoyKsxsLr9Z7D8aoH9Rkb2L
+6KjZ31nt9Drh7NJfh6ReANBW6INdDW0Y2mbzoDozLszAYjVfuUUEE76iJqXY0N4W
+kNr0OQQTUtDpVk0AZZZvy17xV+rkqGgwlOqTvHbwFYEQvgwVz4EKUw==
 -----END RSA PRIVATE KEY-----
diff --git a/Lib/test/ssl_key.pem b/Lib/test/ssl_key.pem
index 3fd3bbd54a3..b63f38bc5cf 100644
--- a/Lib/test/ssl_key.pem
+++ b/Lib/test/ssl_key.pem
@@ -1,16 +1,28 @@
 -----BEGIN PRIVATE KEY-----
-MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANtb0+YrKuxevGpm
-LrjaUhZSgz6zFAmuGFmKmUbdjmfv9zSmmdsQIksK++jK0Be9LeZy20j6ahOfuVa0
-ufEmPoP7Fy4hXegKZR9cCWcIe/A6H2xWF1IIJLRTLaU8ol/I7T+um5HD5AwAwNPP
-USNU0Eegmvp+xxWu3NX2m1Veot85AgMBAAECgYA3ZdZ673X0oexFlq7AAmrutkHt
-CL7LvwrpOiaBjhyTxTeSNWzvtQBkIU8DOI0bIazA4UreAFffwtvEuPmonDb3F+Iq
-SMAu42XcGyVZEl+gHlTPU9XRX7nTOXVt+MlRRRxL6t9GkGfUAXI3XxJDXW3c0vBK
-UL9xqD8cORXOfE06rQJBAP8mEX1ERkR64Ptsoe4281vjTlNfIbs7NMPkUnrn9N/Y
-BLhjNIfQ3HFZG8BTMLfX7kCS9D593DW5tV4Z9BP/c6cCQQDcFzCcVArNh2JSywOQ
-ZfTfRbJg/Z5Lt9Fkngv1meeGNPgIMLN8Sg679pAOOWmzdMO3V706rNPzSVMME7E5
-oPIfAkEA8pDddarP5tCvTTgUpmTFbakm0KoTZm2+FzHcnA4jRh+XNTjTOv98Y6Ik
-eO5d1ZnKXseWvkZncQgxfdnMqqpj5wJAcNq/RVne1DbYlwWchT2Si65MYmmJ8t+F
-0mcsULqjOnEMwf5e+ptq5LzwbyrHZYq5FNk7ocufPv/ZQrcSSC+cFwJBAKvOJByS
-x56qyGeZLOQlWS2JS3KJo59XuLFGqcbgN9Om9xFa41Yb4N9NvplFivsvZdw3m1Q/
-SPIXQuT8RMPDVNQ=
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCr77F9oBLYuLIb
+3t4mDHPoxOEnu1h7NQaJdn9tR/KjW6AzhXXm9USh45qt3sR6Wo8sDlBIJ3vZyhyW
+P939qYUeGOZJahupYi4IkqVckZXARm3k9qhAI/aC4ixVFyxnxg3bNpk8Ir0AyyQV
+oudY+33+ZNj6+fHzeboGDJ9uE/RTcp9JqE/qo0haATZufJfY63ZCUpYFn6j5W4jG
+vpqg5/0hba8Cxdk62387uNknfVHSMzGkkq82zbBpud6TYQofCp3VlEPBjz9iLCz6
+FFOKgLZmbk8QsdktXF6zfRJJk+vbZTh/OGH0p/eiIfW1kXOzOcuW31XRFTPnykJd
+4QUX9OajAgMBAAECggEAHppmXDbuw9Z0FVPg9KLIysioTtsgz6VLiZIm8juZK4x2
+glUh/D7xvWL2uDXrgN+3lh7iGUW13LkFx5SMncbbo9TIwI57Z/XKvcnkVwquve+L
+RfLFVc1Q5lD9lROv2rS86KTaN4LzYz3FKXi6dvMkpPAsUtfEQhMLkmISypQQq/1z
+EJaqo7r85OjN7e0wKazlKZpOzJEa5FQLMVRjTRFhLFNbHXX/tAet2jw+umATKbw8
+hYgiuZ44TwSEd9JeIV/oSYWfI/3HetuYW0ru3caiztRF2NySNu8lcsWgNC7fIku9
+mcHjtSNzs91QN1Qlu7GQvvhpt6OWDirNDCW+49WGaQKBgQDg9SDhfF0jRYslgYbH
+cqO4ggaFdHjrAAYpwnAgvanhFZL/zEqm5G1E7l/e2fCkJ9VOSFO0A208chvwMcr+
+dCjHE2tVdE81aQ2v/Eo83VdS1RcOV4Y75yPH48rMhxPaHvxWD/FFDbf0/P2mtPB7
+SZ3kIeZMkE1wxdaO3AKUbQoozwKBgQDDqYgg7kVtygyICE1mB8Hwp6nUxFTczG7y
+4XcsDqMIrKmw+PbQluvkoHoStxeVrsTloDhkTjIrpmYLyAiazg+PUJdkd6xrfLSj
+VV6X93W0S/1egEb1F1CGFxtk8v/PWH4K76EPL2vxXdxjywz3GWlrL9yDYaB2szzS
+DqgwVMqx7QKBgDCD7UF0Bsoyl13RX3XoPXLvZ+SkR+e2q52Z94C4JskKVBeiwX7Y
+yNAS8M4pBoMArDoj0xmBm69rlKbqtjLGbnzwrTdSzDpim7cWnBQgUFLm7gAD1Elb
+AhZ8BCK0Bw4FnLoa2hfga4oEfdfUMgEE0W5/+SEOBgWKRUmuHUhRc911AoGAY2EN
+YmSDYSM5wDIvVb5k9B3EtevOiqNPSw/XnsoEZtiEC/44JnQxdltIBY93bDBrk5IQ
+cmoBM4h91kgQjshQwOMXMhFSwvmBKmCm/hrTbvMVytTutXfVD3ZXFKwT4DW7N0TF
+ElhsxBh/YzRz7mG62JVjtFt2zDN3ld2Z8YpvtXUCgYEA4EJ4ObS5YyvcXAKHJFo6
+Fxmavyrf8LSm3MFA65uSnFvWukMVqqRMReQc5jvpxHKCis+XvnHzyOfL0gW9ZTi7
+tWGGbBi0TRJCa8BkvgngUZxOxUlMfg/7cVxOIB0TPoUSgxFd/+qVz4GZMvr0dPu7
+eAF7J/8ECVvb0wSPTUI1N3c=
 -----END PRIVATE KEY-----
diff --git a/Lib/test/test_asyncio/utils.py b/Lib/test/test_asyncio/utils.py
index a78e01988d8..6c809770b4a 100644
--- a/Lib/test/test_asyncio/utils.py
+++ b/Lib/test/test_asyncio/utils.py
@@ -50,17 +50,23 @@ def data_file(filename):
 ONLYKEY = data_file('ssl_key.pem')
 SIGNED_CERTFILE = data_file('keycert3.pem')
 SIGNING_CA = data_file('pycacert.pem')
-PEERCERT = {'serialNumber': 'B09264B1F2DA21D1',
-            'version': 1,
-            'subject': ((('countryName', 'XY'),),
-                    (('localityName', 'Castle Anthrax'),),
-                    (('organizationName', 'Python Software Foundation'),),
-                    (('commonName', 'localhost'),)),
-            'issuer': ((('countryName', 'XY'),),
-                    (('organizationName', 'Python Software Foundation CA'),),
-                    (('commonName', 'our-ca-server'),)),
-            'notAfter': 'Nov 13 19:47:07 2022 GMT',
-            'notBefore': 'Jan  4 19:47:07 2013 GMT'}
+PEERCERT = {
+    'OCSP': ('http://testca.pythontest.net/testca/ocsp/',),
+    'caIssuers': ('http://testca.pythontest.net/testca/pycacert.cer',),
+    'crlDistributionPoints': ('http://testca.pythontest.net/testca/revocation.crl',),
+    'issuer': ((('countryName', 'XY'),),
+            (('organizationName', 'Python Software Foundation CA'),),
+            (('commonName', 'our-ca-server'),)),
+    'notAfter': 'Nov 28 19:09:06 2027 GMT',
+    'notBefore': 'Jan 19 19:09:06 2018 GMT',
+    'serialNumber': '82EDBF41C880919C',
+    'subject': ((('countryName', 'XY'),),
+             (('localityName', 'Castle Anthrax'),),
+             (('organizationName', 'Python Software Foundation'),),
+             (('commonName', 'localhost'),)),
+    'subjectAltName': (('DNS', 'localhost'),),
+    'version': 3
+}
 
 
 def simple_server_sslcontext():
diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index 98d3e571b73..e3fa4233ed6 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -54,6 +54,21 @@ def data_file(*name):
 CAFILE_NEURONIO = data_file("capath", "4e1295a3.0")
 CAFILE_CACERT = data_file("capath", "5ed36f99.0")
 
+CERTFILE_INFO = {
+    'issuer': ((('countryName', 'XY'),),
+               (('localityName', 'Castle Anthrax'),),
+               (('organizationName', 'Python Software Foundation'),),
+               (('commonName', 'localhost'),)),
+    'notAfter': 'Jan 17 19:09:06 2028 GMT',
+    'notBefore': 'Jan 19 19:09:06 2018 GMT',
+    'serialNumber': 'F9BA076D5B6ABD9B',
+    'subject': ((('countryName', 'XY'),),
+             (('localityName', 'Castle Anthrax'),),
+             (('organizationName', 'Python Software Foundation'),),
+             (('commonName', 'localhost'),)),
+    'subjectAltName': (('DNS', 'localhost'),),
+    'version': 3
+}
 
 # empty CRL
 CRLFILE = data_file("revocation.crl")
@@ -61,8 +76,30 @@ def data_file(*name):
 # Two keys and certs signed by the same CA (for SNI tests)
 SIGNED_CERTFILE = data_file("keycert3.pem")
 SIGNED_CERTFILE_HOSTNAME = 'localhost'
+
+SIGNED_CERTFILE_INFO = {
+    'OCSP': ('http://testca.pythontest.net/testca/ocsp/',),
+    'caIssuers': ('http://testca.pythontest.net/testca/pycacert.cer',),
+    'crlDistributionPoints': ('http://testca.pythontest.net/testca/revocation.crl',),
+    'issuer': ((('countryName', 'XY'),),
+            (('organizationName', 'Python Software Foundation CA'),),
+            (('commonName', 'our-ca-server'),)),
+    'notAfter': 'Nov 28 19:09:06 2027 GMT',
+    'notBefore': 'Jan 19 19:09:06 2018 GMT',
+    'serialNumber': '82EDBF41C880919C',
+    'subject': ((('countryName', 'XY'),),
+             (('localityName', 'Castle Anthrax'),),
+             (('organizationName', 'Python Software Foundation'),),
+             (('commonName', 'localhost'),)),
+    'subjectAltName': (('DNS', 'localhost'),),
+    'version': 3
+}
+
 SIGNED_CERTFILE2 = data_file("keycert4.pem")
 SIGNED_CERTFILE2_HOSTNAME = 'fakehostname'
+SIGNED_CERTFILE_ECC = data_file("keycertecc.pem")
+SIGNED_CERTFILE_ECC_HOSTNAME = 'localhost-ecc'
+
 # Same certificate as pycacert.pem, but without extra text in file
 SIGNING_CA = data_file("capath", "ceff1710.0")
 # cert with all kinds of subject alt names
@@ -276,26 +313,15 @@ def test_parse_cert(self):
         # note that this uses an 'unofficial' function in _ssl.c,
         # provided solely for this test, to exercise the certificate
         # parsing code
-        p = ssl._ssl._test_decode_cert(CERTFILE)
-        if support.verbose:
-            sys.stdout.write("\n" + pprint.pformat(p) + "\n")
-        self.assertEqual(p['issuer'],
-                         ((('countryName', 'XY'),),
-                          (('localityName', 'Castle Anthrax'),),
-                          (('organizationName', 'Python Software Foundation'),),
-                          (('commonName', 'localhost'),))
-                        )
-        # Note the next three asserts will fail if the keys are regenerated
-        self.assertEqual(p['notAfter'], asn1time('Oct  5 23:01:56 2020 GMT'))
-        self.assertEqual(p['notBefore'], asn1time('Oct  8 23:01:56 2010 GMT'))
-        self.assertEqual(p['serialNumber'], 'D7C7381919AFC24E')
-        self.assertEqual(p['subject'],
-                         ((('countryName', 'XY'),),
-                          (('localityName', 'Castle Anthrax'),),
-                          (('organizationName', 'Python Software Foundation'),),
-                          (('commonName', 'localhost'),))
-                        )
-        self.assertEqual(p['subjectAltName'], (('DNS', 'localhost'),))
+        self.assertEqual(
+            ssl._ssl._test_decode_cert(CERTFILE),
+            CERTFILE_INFO
+        )
+        self.assertEqual(
+            ssl._ssl._test_decode_cert(SIGNED_CERTFILE),
+            SIGNED_CERTFILE_INFO
+        )
+
         # Issue #13034: the subjectAltName in some certificates
         # (notably projects.developer.nokia.com:443) wasn't parsed
         p = ssl._ssl._test_decode_cert(NOKIACERT)
@@ -2337,8 +2363,8 @@ def try_protocol_combo(server_protocol, client_protocol, expect_success,
 
     for ctx in (client_context, server_context):
         ctx.verify_mode = certsreqs
-        ctx.load_cert_chain(CERTFILE)
-        ctx.load_verify_locations(CERTFILE)
+        ctx.load_cert_chain(SIGNED_CERTFILE)
+        ctx.load_verify_locations(SIGNING_CA)
     try:
         stats = server_params_test(client_context, server_context,
                                    chatty=False, connectionchatty=False)
@@ -2522,6 +2548,50 @@ def test_check_hostname(self):
                                             "check_hostname requires server_hostname"):
                     client_context.wrap_socket(s)
 
+    def test_ecc_cert(self):
+        client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+        client_context.load_verify_locations(SIGNING_CA)
+        client_context.set_ciphers('ECDHE:ECDSA:!NULL:!aRSA')
+        hostname = SIGNED_CERTFILE_ECC_HOSTNAME
+
+        server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+        # load ECC cert
+        server_context.load_cert_chain(SIGNED_CERTFILE_ECC)
+
+        # correct hostname should verify
+        server = ThreadedEchoServer(context=server_context, chatty=True)
+        with server:
+            with client_context.wrap_socket(socket.socket(),
+                                            server_hostname=hostname) as s:
+                s.connect((HOST, server.port))
+                cert = s.getpeercert()
+                self.assertTrue(cert, "Can't get peer certificate.")
+                cipher = s.cipher()[0].split('-')
+                self.assertTrue(cipher[:2], ('ECDHE', 'ECDSA'))
+
+    def test_dual_rsa_ecc(self):
+        client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+        client_context.load_verify_locations(SIGNING_CA)
+        # only ECDSA certs
+        client_context.set_ciphers('ECDHE:ECDSA:!NULL:!aRSA')
+        hostname = SIGNED_CERTFILE_ECC_HOSTNAME
+
+        server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+        # load ECC and RSA key/cert pairs
+        server_context.load_cert_chain(SIGNED_CERTFILE_ECC)
+        server_context.load_cert_chain(SIGNED_CERTFILE)
+
+        # correct hostname should verify
+        server = ThreadedEchoServer(context=server_context, chatty=True)
+        with server:
+            with client_context.wrap_socket(socket.socket(),
+                                            server_hostname=hostname) as s:
+                s.connect((HOST, server.port))
+                cert = s.getpeercert()
+                self.assertTrue(cert, "Can't get peer certificate.")
+                cipher = s.cipher()[0].split('-')
+                self.assertTrue(cipher[:2], ('ECDHE', 'ECDSA'))
+
     def test_wrong_cert(self):
         """Connecting when the server rejects the client's certificate
 
@@ -2819,7 +2889,7 @@ def test_starttls(self):
 
     def test_socketserver(self):
         """Using socketserver to create and manage SSL connections."""
-        server = make_https_server(self, certfile=CERTFILE)
+        server = make_https_server(self, certfile=SIGNED_CERTFILE)
         # try to connect
         if support.verbose:
             sys.stdout.write('\n')
@@ -2829,7 +2899,7 @@ def test_socketserver(self):
         # now fetch the same data from the HTTPS server
         url = 'https://localhost:%d/%s' % (
             server.port, os.path.split(CERTFILE)[1])
-        context = ssl.create_default_context(cafile=CERTFILE)
+        context = ssl.create_default_context(cafile=SIGNING_CA)
         f = urllib.request.urlopen(url, context=context)
         try:
             dlen = f.info().get("content-length")
@@ -3112,8 +3182,8 @@ def test_server_accept(self):
         # SSLContext.wrap_socket().
         context = ssl.SSLContext(ssl.PROTOCOL_TLS)
         context.verify_mode = ssl.CERT_REQUIRED
-        context.load_verify_locations(CERTFILE)
-        context.load_cert_chain(CERTFILE)
+        context.load_verify_locations(SIGNING_CA)
+        context.load_cert_chain(SIGNED_CERTFILE)
         server = socket.socket(socket.AF_INET)
         host = "127.0.0.1"
         port = support.bind_port(server)
@@ -3562,8 +3632,8 @@ def test_sendfile(self):
         self.addCleanup(support.unlink, support.TESTFN)
         context = ssl.SSLContext(ssl.PROTOCOL_TLS)
         context.verify_mode = ssl.CERT_REQUIRED
-        context.load_verify_locations(CERTFILE)
-        context.load_cert_chain(CERTFILE)
+        context.load_verify_locations(SIGNING_CA)
+        context.load_cert_chain(SIGNED_CERTFILE)
         server = ThreadedEchoServer(context=context, chatty=False)
         with server:
             with context.wrap_socket(socket.socket()) as s:
diff --git a/Misc/NEWS.d/next/Tests/2018-01-19-20-47-11.bpo-32602.dz41pq.rst b/Misc/NEWS.d/next/Tests/2018-01-19-20-47-11.bpo-32602.dz41pq.rst
new file mode 100644
index 00000000000..dde812ecacf
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-01-19-20-47-11.bpo-32602.dz41pq.rst
@@ -0,0 +1 @@
+Add test certs and test for ECDSA cert and EC/RSA dual mode.


From webhook-mailer at python.org  Sat Jan 20 09:42:47 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sat, 20 Jan 2018 14:42:47 -0000
Subject: [Python-checkins] bpo-32503: Avoid creating too small frames in
 pickles. (#5127)
Message-ID: <mailman.69.1516459370.2623.python-checkins@python.org>

https://github.com/python/cpython/commit/1211c9a9897a174b7261ca258cabf289815a40d8
commit: 1211c9a9897a174b7261ca258cabf289815a40d8
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-20T16:42:44+02:00
summary:

bpo-32503: Avoid creating too small frames in pickles. (#5127)

files:
A Misc/NEWS.d/next/Library/2018-01-07-09-22-26.bpo-32503.ViMxpD.rst
M Lib/pickle.py
M Lib/test/pickletester.py
M Modules/_pickle.c

diff --git a/Lib/pickle.py b/Lib/pickle.py
index 301e8cf5589..e6d003787ba 100644
--- a/Lib/pickle.py
+++ b/Lib/pickle.py
@@ -183,6 +183,7 @@ def __init__(self, value):
 
 class _Framer:
 
+    _FRAME_SIZE_MIN = 4
     _FRAME_SIZE_TARGET = 64 * 1024
 
     def __init__(self, file_write):
@@ -203,11 +204,12 @@ def commit_frame(self, force=False):
             if f.tell() >= self._FRAME_SIZE_TARGET or force:
                 data = f.getbuffer()
                 write = self.file_write
-                # Issue a single call to the write method of the underlying
-                # file object for the frame opcode with the size of the
-                # frame. The concatenation is expected to be less expensive
-                # than issuing an additional call to write.
-                write(FRAME + pack("<Q", len(data)))
+                if len(data) >= self._FRAME_SIZE_MIN:
+                    # Issue a single call to the write method of the underlying
+                    # file object for the frame opcode with the size of the
+                    # frame. The concatenation is expected to be less expensive
+                    # than issuing an additional call to write.
+                    write(FRAME + pack("<Q", len(data)))
 
                 # Issue a separate call to write to append the frame
                 # contents without concatenation to the above to avoid a
diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
index 062ec5ae72b..b84b87861d0 100644
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -2037,6 +2037,7 @@ def test_setitems_on_non_dicts(self):
 
     # Exercise framing (proto >= 4) for significant workloads
 
+    FRAME_SIZE_MIN = 4
     FRAME_SIZE_TARGET = 64 * 1024
 
     def check_frame_opcodes(self, pickled):
@@ -2047,36 +2048,43 @@ def check_frame_opcodes(self, pickled):
         framed by default and are therefore considered a frame by themselves in
         the following consistency check.
         """
-        last_arg = last_pos = last_frame_opcode_size = None
-        frameless_opcode_sizes = {
-            'BINBYTES': 5,
-            'BINUNICODE': 5,
-            'BINBYTES8': 9,
-            'BINUNICODE8': 9,
-        }
+        frame_end = frameless_start = None
+        frameless_opcodes = {'BINBYTES', 'BINUNICODE', 'BINBYTES8', 'BINUNICODE8'}
         for op, arg, pos in pickletools.genops(pickled):
-            if op.name in frameless_opcode_sizes:
-                if len(arg) > self.FRAME_SIZE_TARGET:
-                    frame_opcode_size = frameless_opcode_sizes[op.name]
-                    arg = len(arg)
-                else:
-                    continue
-            elif op.name == 'FRAME':
-                frame_opcode_size = 9
-            else:
-                continue
-
-            if last_pos is not None:
-                # The previous frame's size should be equal to the number
-                # of bytes up to the current frame.
-                frame_size = pos - last_pos - last_frame_opcode_size
-                self.assertEqual(frame_size, last_arg)
-            last_arg, last_pos = arg, pos
-            last_frame_opcode_size = frame_opcode_size
-        # The last frame's size should be equal to the number of bytes up
-        # to the pickle's end.
-        frame_size = len(pickled) - last_pos - last_frame_opcode_size
-        self.assertEqual(frame_size, last_arg)
+            if frame_end is not None:
+                self.assertLessEqual(pos, frame_end)
+                if pos == frame_end:
+                    frame_end = None
+
+            if frame_end is not None:  # framed
+                self.assertNotEqual(op.name, 'FRAME')
+                if op.name in frameless_opcodes:
+                    # Only short bytes and str objects should be written
+                    # in a frame
+                    self.assertLessEqual(len(arg), self.FRAME_SIZE_TARGET)
+
+            else:  # not framed
+                if (op.name == 'FRAME' or
+                    (op.name in frameless_opcodes and
+                     len(arg) > self.FRAME_SIZE_TARGET)):
+                    # Frame or large bytes or str object
+                    if frameless_start is not None:
+                        # Only short data should be written outside of a frame
+                        self.assertLess(pos - frameless_start,
+                                        self.FRAME_SIZE_MIN)
+                        frameless_start = None
+                elif frameless_start is None and op.name != 'PROTO':
+                    frameless_start = pos
+
+            if op.name == 'FRAME':
+                self.assertGreaterEqual(arg, self.FRAME_SIZE_MIN)
+                frame_end = pos + 9 + arg
+
+        pos = len(pickled)
+        if frame_end is not None:
+            self.assertEqual(frame_end, pos)
+        elif frameless_start is not None:
+            self.assertLess(pos - frameless_start, self.FRAME_SIZE_MIN)
 
     def test_framing_many_objects(self):
         obj = list(range(10**5))
@@ -2095,7 +2103,8 @@ def test_framing_many_objects(self):
 
     def test_framing_large_objects(self):
         N = 1024 * 1024
-        obj = [b'x' * N, b'y' * N, 'z' * N]
+        small_items = [[i] for i in range(10)]
+        obj = [b'x' * N, *small_items, b'y' * N, 'z' * N]
         for proto in range(4, pickle.HIGHEST_PROTOCOL + 1):
             for fast in [False, True]:
                 with self.subTest(proto=proto, fast=fast):
@@ -2119,12 +2128,9 @@ def test_framing_large_objects(self):
                     # Perform full equality check if the lengths match.
                     self.assertEqual(obj, unpickled)
                     n_frames = count_opcode(pickle.FRAME, pickled)
-                    if not fast:
-                        # One frame per memoize for each large object.
-                        self.assertGreaterEqual(n_frames, len(obj))
-                    else:
-                        # One frame at the beginning and one at the end.
-                        self.assertGreaterEqual(n_frames, 2)
+                    # A single frame for small objects between
+                    # first two large objects.
+                    self.assertEqual(n_frames, 1)
                     self.check_frame_opcodes(pickled)
 
     def test_optional_frames(self):
@@ -2152,7 +2158,9 @@ def remove_frames(pickled, keep_frame=None):
 
         frame_size = self.FRAME_SIZE_TARGET
         num_frames = 20
-        obj = [bytes([i]) * frame_size for i in range(num_frames)]
+        # Large byte objects (dict values) intermitted with small objects
+        # (dict keys)
+        obj = {i: bytes([i]) * frame_size for i in range(num_frames)}
 
         for proto in range(4, pickle.HIGHEST_PROTOCOL + 1):
             pickled = self.dumps(obj, proto)
diff --git a/Misc/NEWS.d/next/Library/2018-01-07-09-22-26.bpo-32503.ViMxpD.rst b/Misc/NEWS.d/next/Library/2018-01-07-09-22-26.bpo-32503.ViMxpD.rst
new file mode 100644
index 00000000000..609c59308e6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-07-09-22-26.bpo-32503.ViMxpD.rst
@@ -0,0 +1 @@
+Pickling with protocol 4 no longer creates too small frames.
diff --git a/Modules/_pickle.c b/Modules/_pickle.c
index f8cbea12e80..f06a87d04f2 100644
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -119,8 +119,8 @@ enum {
     /* Prefetch size when unpickling (disabled on unpeekable streams) */
     PREFETCH = 8192 * 16,
 
+    FRAME_SIZE_MIN = 4,
     FRAME_SIZE_TARGET = 64 * 1024,
-
     FRAME_HEADER_SIZE = 9
 };
 
@@ -949,13 +949,6 @@ _write_size64(char *out, size_t value)
     }
 }
 
-static void
-_Pickler_WriteFrameHeader(PicklerObject *self, char *qdata, size_t frame_len)
-{
-    qdata[0] = FRAME;
-    _write_size64(qdata + 1, frame_len);
-}
-
 static int
 _Pickler_CommitFrame(PicklerObject *self)
 {
@@ -966,7 +959,14 @@ _Pickler_CommitFrame(PicklerObject *self)
         return 0;
     frame_len = self->output_len - self->frame_start - FRAME_HEADER_SIZE;
     qdata = PyBytes_AS_STRING(self->output_buffer) + self->frame_start;
-    _Pickler_WriteFrameHeader(self, qdata, frame_len);
+    if (frame_len >= FRAME_SIZE_MIN) {
+        qdata[0] = FRAME;
+        _write_size64(qdata + 1, frame_len);
+    }
+    else {
+        memmove(qdata, qdata + FRAME_HEADER_SIZE, frame_len);
+        self->output_len -= FRAME_HEADER_SIZE;
+    }
     self->frame_start = -1;
     return 0;
 }


From solipsis at pitrou.net  Sun Jan 21 04:28:02 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 21 Jan 2018 09:28:02 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=3
Message-ID: <20180121092802.1.53C6D331DC138EE8@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [7, -7, 1] memory blocks, sum=1
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [0, -2, 2] memory blocks, sum=0
test_multiprocessing_spawn leaked [-2, 2, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloghlFKMh', '--timeout', '7200']

From webhook-mailer at python.org  Sun Jan 21 09:44:10 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sun, 21 Jan 2018 14:44:10 -0000
Subject: [Python-checkins] bpo-32591: Add native coroutine origin tracking
 (#5250)
Message-ID: <mailman.0.1516545850.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/fc2f407829d9817ddacccae6944dd0879cfaca24
commit: fc2f407829d9817ddacccae6944dd0879cfaca24
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-21T09:44:07-05:00
summary:

bpo-32591: Add native coroutine origin tracking (#5250)

* Add coro.cr_origin and sys.set_coroutine_origin_tracking_depth
* Use coroutine origin information in the unawaited coroutine warning
* Stop using set_coroutine_wrapper in asyncio debug mode
* In BaseEventLoop.set_debug, enable debugging in the correct thread

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-20-00-50-33.bpo-32591.666kl6.rst
A Python/clinic/sysmodule.c.h
M Doc/library/inspect.rst
M Doc/library/sys.rst
M Doc/whatsnew/3.7.rst
M Include/ceval.h
M Include/genobject.h
M Include/pystate.h
M Include/warnings.h
M Lib/asyncio/base_events.py
M Lib/asyncio/coroutines.py
M Lib/test/test_asyncio/test_pep492.py
M Lib/test/test_coroutines.py
M Lib/warnings.py
M Misc/ACKS
M Objects/genobject.c
M Python/_warnings.c
M Python/ceval.c
M Python/pystate.c
M Python/sysmodule.c

diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
index 6be28a2b31c..147e802cac4 100644
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -34,6 +34,9 @@ provided as convenient choices for the second argument to :func:`getmembers`.
 They also help you determine when you can expect to find the following special
 attributes:
 
+.. this function name is too big to fit in the ascii-art table below
+.. |coroutine-origin-link| replace:: :func:`sys.set_coroutine_origin_tracking_depth`
+
 +-----------+-------------------+---------------------------+
 | Type      | Attribute         | Description               |
 +===========+===================+===========================+
@@ -215,6 +218,10 @@ attributes:
 +-----------+-------------------+---------------------------+
 |           | cr_code           | code                      |
 +-----------+-------------------+---------------------------+
+|           | cr_origin         | where coroutine was       |
+|           |                   | created, or ``None``. See |
+|           |                   | |coroutine-origin-link|   |
++-----------+-------------------+---------------------------+
 | builtin   | __doc__           | documentation string      |
 +-----------+-------------------+---------------------------+
 |           | __name__          | original name of this     |
@@ -234,6 +241,9 @@ attributes:
    The ``__name__`` attribute of generators is now set from the function
    name, instead of the code name, and it can now be modified.
 
+.. versionchanged:: 3.7
+
+   Add ``cr_origin`` attribute to coroutines.
 
 .. function:: getmembers(object[, predicate])
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 957d02b2a30..54281a3f9a0 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -675,6 +675,18 @@ always available.
       for details.)
 
 
+.. function:: get_coroutine_origin_tracking_depth()
+
+   Get the current coroutine origin tracking depth, as set by
+   func:`set_coroutine_origin_tracking_depth`.
+
+   .. versionadded:: 3.7
+
+   .. note::
+      This function has been added on a provisional basis (see :pep:`411`
+      for details.)  Use it only for debugging purposes.
+
+
 .. function:: get_coroutine_wrapper()
 
    Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`.
@@ -686,6 +698,10 @@ always available.
       This function has been added on a provisional basis (see :pep:`411`
       for details.)  Use it only for debugging purposes.
 
+   .. deprecated:: 3.7
+      The coroutine wrapper functionality has been deprecated, and
+      will be removed in 3.8. See :issue:`32591` for details.
+
 
 .. data:: hash_info
 
@@ -1212,6 +1228,26 @@ always available.
       This function has been added on a provisional basis (see :pep:`411`
       for details.)
 
+.. function:: set_coroutine_origin_tracking_depth(depth)
+
+   Allows enabling or disabling coroutine origin tracking. When
+   enabled, the ``cr_origin`` attribute on coroutine objects will
+   contain a tuple of (filename, line number, function name) tuples
+   describing the traceback where the coroutine object was created,
+   with the most recent call first. When disabled, ``cr_origin`` will
+   be None.
+
+   To enable, pass a *depth* value greater than zero; this sets the
+   number of frames whose information will be captured. To disable,
+   pass set *depth* to zero.
+
+   This setting is thread-specific.
+
+   .. versionadded:: 3.7
+
+   .. note::
+      This function has been added on a provisional basis (see :pep:`411`
+      for details.)  Use it only for debugging purposes.
 
 .. function:: set_coroutine_wrapper(wrapper)
 
@@ -1252,6 +1288,10 @@ always available.
       This function has been added on a provisional basis (see :pep:`411`
       for details.)  Use it only for debugging purposes.
 
+   .. deprecated:: 3.7
+      The coroutine wrapper functionality has been deprecated, and
+      will be removed in 3.8. See :issue:`32591` for details.
+
 .. function:: _enablelegacywindowsfsencoding()
 
    Changes the default filesystem encoding and errors mode to 'mbcs' and
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 009df38d8ec..54a3f14b0bb 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -510,6 +510,9 @@ sys
 
 Added :attr:`sys.flags.dev_mode` flag for the new development mode.
 
+Deprecated :func:`sys.set_coroutine_wrapper` and
+:func:`sys.get_coroutine_wrapper`.
+
 time
 ----
 
diff --git a/Include/ceval.h b/Include/ceval.h
index 70306b8bbd8..bce8a0beed8 100644
--- a/Include/ceval.h
+++ b/Include/ceval.h
@@ -31,6 +31,8 @@ PyAPI_FUNC(PyObject *) PyEval_CallMethod(PyObject *obj,
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(void) PyEval_SetProfile(Py_tracefunc, PyObject *);
 PyAPI_FUNC(void) PyEval_SetTrace(Py_tracefunc, PyObject *);
+PyAPI_FUNC(void) _PyEval_SetCoroutineOriginTrackingDepth(int new_depth);
+PyAPI_FUNC(int) _PyEval_GetCoroutineOriginTrackingDepth(void);
 PyAPI_FUNC(void) _PyEval_SetCoroutineWrapper(PyObject *);
 PyAPI_FUNC(PyObject *) _PyEval_GetCoroutineWrapper(void);
 PyAPI_FUNC(void) _PyEval_SetAsyncGenFirstiter(PyObject *);
diff --git a/Include/genobject.h b/Include/genobject.h
index 87fbe17d4ab..16b983339cc 100644
--- a/Include/genobject.h
+++ b/Include/genobject.h
@@ -51,6 +51,7 @@ PyAPI_FUNC(void) _PyGen_Finalize(PyObject *self);
 #ifndef Py_LIMITED_API
 typedef struct {
     _PyGenObject_HEAD(cr)
+    PyObject *cr_origin;
 } PyCoroObject;
 
 PyAPI_DATA(PyTypeObject) PyCoro_Type;
diff --git a/Include/pystate.h b/Include/pystate.h
index a4815286a74..5a69e1471a0 100644
--- a/Include/pystate.h
+++ b/Include/pystate.h
@@ -262,6 +262,8 @@ typedef struct _ts {
     void (*on_delete)(void *);
     void *on_delete_data;
 
+    int coroutine_origin_tracking_depth;
+
     PyObject *coroutine_wrapper;
     int in_coroutine_wrapper;
 
diff --git a/Include/warnings.h b/Include/warnings.h
index a3f83ff6967..a675bb5dfcb 100644
--- a/Include/warnings.h
+++ b/Include/warnings.h
@@ -56,6 +56,10 @@ PyErr_WarnExplicitFormat(PyObject *category,
 #define PyErr_Warn(category, msg) PyErr_WarnEx(category, msg, 1)
 #endif
 
+#ifndef Py_LIMITED_API
+void _PyErr_WarnUnawaitedCoroutine(PyObject *coro);
+#endif
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index 00c84a835c8..a10e706d504 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -34,6 +34,7 @@
 except ImportError:  # pragma: no cover
     ssl = None
 
+from . import constants
 from . import coroutines
 from . import events
 from . import futures
@@ -224,7 +225,8 @@ def __init__(self):
         self.slow_callback_duration = 0.1
         self._current_handle = None
         self._task_factory = None
-        self._coroutine_wrapper_set = False
+        self._coroutine_origin_tracking_enabled = False
+        self._coroutine_origin_tracking_saved_depth = None
 
         if hasattr(sys, 'get_asyncgen_hooks'):
             # Python >= 3.6
@@ -382,7 +384,7 @@ def run_forever(self):
         if events._get_running_loop() is not None:
             raise RuntimeError(
                 'Cannot run the event loop while another loop is running')
-        self._set_coroutine_wrapper(self._debug)
+        self._set_coroutine_origin_tracking(self._debug)
         self._thread_id = threading.get_ident()
         if self._asyncgens is not None:
             old_agen_hooks = sys.get_asyncgen_hooks()
@@ -398,7 +400,7 @@ def run_forever(self):
             self._stopping = False
             self._thread_id = None
             events._set_running_loop(None)
-            self._set_coroutine_wrapper(False)
+            self._set_coroutine_origin_tracking(False)
             if self._asyncgens is not None:
                 sys.set_asyncgen_hooks(*old_agen_hooks)
 
@@ -1531,39 +1533,20 @@ def _run_once(self):
                 handle._run()
         handle = None  # Needed to break cycles when an exception occurs.
 
-    def _set_coroutine_wrapper(self, enabled):
-        try:
-            set_wrapper = sys.set_coroutine_wrapper
-            get_wrapper = sys.get_coroutine_wrapper
-        except AttributeError:
-            return
-
-        enabled = bool(enabled)
-        if self._coroutine_wrapper_set == enabled:
+    def _set_coroutine_origin_tracking(self, enabled):
+        if bool(enabled) == bool(self._coroutine_origin_tracking_enabled):
             return
 
-        wrapper = coroutines.debug_wrapper
-        current_wrapper = get_wrapper()
-
         if enabled:
-            if current_wrapper not in (None, wrapper):
-                warnings.warn(
-                    f"loop.set_debug(True): cannot set debug coroutine "
-                    f"wrapper; another wrapper is already set "
-                    f"{current_wrapper!r}",
-                    RuntimeWarning)
-            else:
-                set_wrapper(wrapper)
-                self._coroutine_wrapper_set = True
+            self._coroutine_origin_tracking_saved_depth = (
+                sys.get_coroutine_origin_tracking_depth())
+            sys.set_coroutine_origin_tracking_depth(
+                constants.DEBUG_STACK_DEPTH)
         else:
-            if current_wrapper not in (None, wrapper):
-                warnings.warn(
-                    f"loop.set_debug(False): cannot unset debug coroutine "
-                    f"wrapper; another wrapper was set {current_wrapper!r}",
-                    RuntimeWarning)
-            else:
-                set_wrapper(None)
-                self._coroutine_wrapper_set = False
+            sys.set_coroutine_origin_tracking_depth(
+                self._coroutine_origin_tracking_saved_depth)
+
+        self._coroutine_origin_tracking_enabled = enabled
 
     def get_debug(self):
         return self._debug
@@ -1572,4 +1555,4 @@ def set_debug(self, enabled):
         self._debug = enabled
 
         if self.is_running():
-            self._set_coroutine_wrapper(enabled)
+            self.call_soon_threadsafe(self._set_coroutine_origin_tracking, enabled)
diff --git a/Lib/asyncio/coroutines.py b/Lib/asyncio/coroutines.py
index 9c860a452b5..5a29100321f 100644
--- a/Lib/asyncio/coroutines.py
+++ b/Lib/asyncio/coroutines.py
@@ -32,14 +32,6 @@ def _is_debug_mode():
 _DEBUG = _is_debug_mode()
 
 
-def debug_wrapper(gen):
-    # This function is called from 'sys.set_coroutine_wrapper'.
-    # We only wrap here coroutines defined via 'async def' syntax.
-    # Generator-based coroutines are wrapped in @coroutine
-    # decorator.
-    return CoroWrapper(gen, None)
-
-
 class CoroWrapper:
     # Wrapper for coroutine object in _DEBUG mode.
 
@@ -87,39 +79,16 @@ def gi_code(self):
         return self.gen.gi_code
 
     def __await__(self):
-        cr_await = getattr(self.gen, 'cr_await', None)
-        if cr_await is not None:
-            raise RuntimeError(
-                f"Cannot await on coroutine {self.gen!r} while it's "
-                f"awaiting for {cr_await!r}")
         return self
 
     @property
     def gi_yieldfrom(self):
         return self.gen.gi_yieldfrom
 
-    @property
-    def cr_await(self):
-        return self.gen.cr_await
-
-    @property
-    def cr_running(self):
-        return self.gen.cr_running
-
-    @property
-    def cr_code(self):
-        return self.gen.cr_code
-
-    @property
-    def cr_frame(self):
-        return self.gen.cr_frame
-
     def __del__(self):
         # Be careful accessing self.gen.frame -- self.gen might not exist.
         gen = getattr(self, 'gen', None)
         frame = getattr(gen, 'gi_frame', None)
-        if frame is None:
-            frame = getattr(gen, 'cr_frame', None)
         if frame is not None and frame.f_lasti == -1:
             msg = f'{self!r} was never yielded from'
             tb = getattr(self, '_source_traceback', ())
@@ -141,8 +110,6 @@ def coroutine(func):
     if inspect.iscoroutinefunction(func):
         # In Python 3.5 that's all we need to do for coroutines
         # defined with "async def".
-        # Wrapping in CoroWrapper will happen via
-        # 'sys.set_coroutine_wrapper' function.
         return func
 
     if inspect.isgeneratorfunction(func):
diff --git a/Lib/test/test_asyncio/test_pep492.py b/Lib/test/test_asyncio/test_pep492.py
index 289f7e59db7..f2d588f5444 100644
--- a/Lib/test/test_asyncio/test_pep492.py
+++ b/Lib/test/test_asyncio/test_pep492.py
@@ -1,5 +1,6 @@
 """Tests support for new syntax introduced by PEP 492."""
 
+import sys
 import types
 import unittest
 
@@ -148,35 +149,14 @@ def test_async_def_coroutines(self):
         data = self.loop.run_until_complete(foo())
         self.assertEqual(data, 'spam')
 
-    @mock.patch('asyncio.coroutines.logger')
-    def test_async_def_wrapped(self, m_log):
-        async def foo():
-            pass
+    def test_debug_mode_manages_coroutine_origin_tracking(self):
         async def start():
-            foo_coro = foo()
-            self.assertRegex(
-                repr(foo_coro),
-                r'<CoroWrapper .*\.foo\(\) running at .*pep492.*>')
-
-            with support.check_warnings((r'.*foo.*was never',
-                                         RuntimeWarning)):
-                foo_coro = None
-                support.gc_collect()
-                self.assertTrue(m_log.error.called)
-                message = m_log.error.call_args[0][0]
-                self.assertRegex(message,
-                                 r'CoroWrapper.*foo.*was never')
+            self.assertTrue(sys.get_coroutine_origin_tracking_depth() > 0)
 
+        self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 0)
         self.loop.set_debug(True)
         self.loop.run_until_complete(start())
-
-        async def start():
-            foo_coro = foo()
-            task = asyncio.ensure_future(foo_coro, loop=self.loop)
-            self.assertRegex(repr(task), r'Task.*foo.*running')
-
-        self.loop.run_until_complete(start())
-
+        self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 0)
 
     def test_types_coroutine(self):
         def gen():
@@ -226,9 +206,9 @@ def test_double_await(self):
                 t.cancel()
 
         self.loop.set_debug(True)
-        with self.assertRaisesRegex(
-            RuntimeError,
-            r'Cannot await.*test_double_await.*\bafunc\b.*while.*\bsleep\b'):
+        with self.assertRaises(
+                RuntimeError,
+                msg='coroutine is being awaited already'):
 
             self.loop.run_until_complete(runner())
 
diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py
index d7d38a3bf9d..8a531b88892 100644
--- a/Lib/test/test_coroutines.py
+++ b/Lib/test/test_coroutines.py
@@ -2,6 +2,7 @@
 import copy
 import inspect
 import pickle
+import re
 import sys
 import types
 import unittest
@@ -1974,9 +1975,11 @@ def wrap(gen):
             wrapped = gen
             return gen
 
-        self.assertIsNone(sys.get_coroutine_wrapper())
+        with self.assertWarns(DeprecationWarning):
+            self.assertIsNone(sys.get_coroutine_wrapper())
 
-        sys.set_coroutine_wrapper(wrap)
+        with self.assertWarns(DeprecationWarning):
+            sys.set_coroutine_wrapper(wrap)
         self.assertIs(sys.get_coroutine_wrapper(), wrap)
         try:
             f = foo()
@@ -2041,6 +2044,130 @@ def wrap(gen):
             sys.set_coroutine_wrapper(None)
 
 
+class OriginTrackingTest(unittest.TestCase):
+    def here(self):
+        info = inspect.getframeinfo(inspect.currentframe().f_back)
+        return (info.filename, info.lineno)
+
+    def test_origin_tracking(self):
+        orig_depth = sys.get_coroutine_origin_tracking_depth()
+        try:
+            async def corofn():
+                pass
+
+            sys.set_coroutine_origin_tracking_depth(0)
+            self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 0)
+
+            with contextlib.closing(corofn()) as coro:
+                self.assertIsNone(coro.cr_origin)
+
+            sys.set_coroutine_origin_tracking_depth(1)
+            self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 1)
+
+            fname, lineno = self.here()
+            with contextlib.closing(corofn()) as coro:
+                self.assertEqual(coro.cr_origin,
+                                 ((fname, lineno + 1, "test_origin_tracking"),))
+
+            sys.set_coroutine_origin_tracking_depth(2)
+            self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 2)
+
+            def nested():
+                return (self.here(), corofn())
+            fname, lineno = self.here()
+            ((nested_fname, nested_lineno), coro) = nested()
+            with contextlib.closing(coro):
+                self.assertEqual(coro.cr_origin,
+                                 ((nested_fname, nested_lineno, "nested"),
+                                  (fname, lineno + 1, "test_origin_tracking")))
+
+            # Check we handle running out of frames correctly
+            sys.set_coroutine_origin_tracking_depth(1000)
+            with contextlib.closing(corofn()) as coro:
+                self.assertTrue(2 < len(coro.cr_origin) < 1000)
+
+            # We can't set depth negative
+            with self.assertRaises(ValueError):
+                sys.set_coroutine_origin_tracking_depth(-1)
+            # And trying leaves it unchanged
+            self.assertEqual(sys.get_coroutine_origin_tracking_depth(), 1000)
+
+        finally:
+            sys.set_coroutine_origin_tracking_depth(orig_depth)
+
+    def test_origin_tracking_warning(self):
+        async def corofn():
+            pass
+
+        a1_filename, a1_lineno = self.here()
+        def a1():
+            return corofn()  # comment in a1
+        a1_lineno += 2
+
+        a2_filename, a2_lineno = self.here()
+        def a2():
+            return a1()  # comment in a2
+        a2_lineno += 2
+
+        def check(depth, msg):
+            sys.set_coroutine_origin_tracking_depth(depth)
+            with warnings.catch_warnings(record=True) as wlist:
+                a2()
+                support.gc_collect()
+            # This might be fragile if other warnings somehow get triggered
+            # inside our 'with' block... let's worry about that if/when it
+            # happens.
+            self.assertTrue(len(wlist) == 1)
+            self.assertIs(wlist[0].category, RuntimeWarning)
+            self.assertEqual(msg, str(wlist[0].message))
+
+        orig_depth = sys.get_coroutine_origin_tracking_depth()
+        try:
+            msg = check(0, f"coroutine '{corofn.__qualname__}' was never awaited")
+            check(1, "".join([
+                f"coroutine '{corofn.__qualname__}' was never awaited\n",
+                "Coroutine created at (most recent call last)\n",
+                f'  File "{a1_filename}", line {a1_lineno}, in a1\n',
+                f'    return corofn()  # comment in a1',
+            ]))
+            check(2, "".join([
+                f"coroutine '{corofn.__qualname__}' was never awaited\n",
+                "Coroutine created at (most recent call last)\n",
+                f'  File "{a2_filename}", line {a2_lineno}, in a2\n',
+                f'    return a1()  # comment in a2\n',
+                f'  File "{a1_filename}", line {a1_lineno}, in a1\n',
+                f'    return corofn()  # comment in a1',
+            ]))
+
+        finally:
+            sys.set_coroutine_origin_tracking_depth(orig_depth)
+
+    def test_unawaited_warning_when_module_broken(self):
+        # Make sure we don't blow up too bad if
+        # warnings._warn_unawaited_coroutine is broken somehow (e.g. because
+        # of shutdown problems)
+        async def corofn():
+            pass
+
+        orig_wuc = warnings._warn_unawaited_coroutine
+        try:
+            warnings._warn_unawaited_coroutine = lambda coro: 1/0
+            with support.captured_stderr() as stream:
+                corofn()
+                support.gc_collect()
+            self.assertIn("Exception ignored in", stream.getvalue())
+            self.assertIn("ZeroDivisionError", stream.getvalue())
+            self.assertIn("was never awaited", stream.getvalue())
+
+            del warnings._warn_unawaited_coroutine
+            with support.captured_stderr() as stream:
+                corofn()
+                support.gc_collect()
+            self.assertIn("was never awaited", stream.getvalue())
+
+        finally:
+            warnings._warn_unawaited_coroutine = orig_wuc
+
 @support.cpython_only
 class CAPITest(unittest.TestCase):
 
diff --git a/Lib/warnings.py b/Lib/warnings.py
index 76ad4dac018..81f98647786 100644
--- a/Lib/warnings.py
+++ b/Lib/warnings.py
@@ -488,6 +488,29 @@ def __exit__(self, *exc_info):
         self._module._showwarnmsg_impl = self._showwarnmsg_impl
 
 
+# Private utility function called by _PyErr_WarnUnawaitedCoroutine
+def _warn_unawaited_coroutine(coro):
+    msg_lines = [
+        f"coroutine '{coro.__qualname__}' was never awaited\n"
+    ]
+    if coro.cr_origin is not None:
+        import linecache, traceback
+        def extract():
+            for filename, lineno, funcname in reversed(coro.cr_origin):
+                line = linecache.getline(filename, lineno)
+                yield (filename, lineno, funcname, line)
+        msg_lines.append("Coroutine created at (most recent call last)\n")
+        msg_lines += traceback.format_list(list(extract()))
+    msg = "".join(msg_lines).rstrip("\n")
+    # Passing source= here means that if the user happens to have tracemalloc
+    # enabled and tracking where the coroutine was created, the warning will
+    # contain that traceback. This does mean that if they have *both*
+    # coroutine origin tracking *and* tracemalloc enabled, they'll get two
+    # partially-redundant tracebacks. If we wanted to be clever we could
+    # probably detect this case and avoid it, but for now we don't bother.
+    warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
+
+
 # filters contains a sequence of filter 5-tuples
 # The components of the 5-tuple are:
 # - an action: error, ignore, always, default, module, or once
diff --git a/Misc/ACKS b/Misc/ACKS
index 009b072d680..900604ca4c6 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1486,6 +1486,7 @@ Christopher Smith
 Eric V. Smith
 Gregory P. Smith
 Mark Smith
+Nathaniel J. Smith
 Roy Smith
 Ryan Smith-Roberts
 Rafal Smotrzyk
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-20-00-50-33.bpo-32591.666kl6.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-20-00-50-33.bpo-32591.666kl6.rst
new file mode 100644
index 00000000000..e3f3d59aca7
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-20-00-50-33.bpo-32591.666kl6.rst	
@@ -0,0 +1,4 @@
+Added built-in support for tracking the origin of coroutine objects; see
+sys.set_coroutine_origin_tracking_depth and CoroutineType.cr_origin. This
+replaces the asyncio debug mode's use of coroutine wrapping for native
+coroutine objects.
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 00a882379fc..0a34c1f6a97 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -32,6 +32,8 @@ gen_traverse(PyGenObject *gen, visitproc visit, void *arg)
     Py_VISIT(gen->gi_code);
     Py_VISIT(gen->gi_name);
     Py_VISIT(gen->gi_qualname);
+    /* No need to visit cr_origin, because it's just tuples/str/int, so can't
+       participate in a reference cycle. */
     return exc_state_traverse(&gen->gi_exc_state, visit, arg);
 }
 
@@ -75,9 +77,7 @@ _PyGen_Finalize(PyObject *self)
         ((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE &&
         gen->gi_frame->f_lasti == -1) {
         if (!error_value) {
-            PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
-                             "coroutine '%.50S' was never awaited",
-                             gen->gi_qualname);
+            _PyErr_WarnUnawaitedCoroutine((PyObject *)gen);
         }
     }
     else {
@@ -137,6 +137,9 @@ gen_dealloc(PyGenObject *gen)
         gen->gi_frame->f_gen = NULL;
         Py_CLEAR(gen->gi_frame);
     }
+    if (((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE) {
+        Py_CLEAR(((PyCoroObject *)gen)->cr_origin);
+    }
     Py_CLEAR(gen->gi_code);
     Py_CLEAR(gen->gi_name);
     Py_CLEAR(gen->gi_qualname);
@@ -990,6 +993,7 @@ static PyMemberDef coro_memberlist[] = {
     {"cr_frame",     T_OBJECT, offsetof(PyCoroObject, cr_frame),    READONLY},
     {"cr_running",   T_BOOL,   offsetof(PyCoroObject, cr_running),  READONLY},
     {"cr_code",      T_OBJECT, offsetof(PyCoroObject, cr_code),     READONLY},
+    {"cr_origin",    T_OBJECT, offsetof(PyCoroObject, cr_origin),   READONLY},
     {NULL}      /* Sentinel */
 };
 
@@ -1158,10 +1162,59 @@ PyTypeObject _PyCoroWrapper_Type = {
     0,                                          /* tp_free */
 };
 
+static PyObject *
+compute_cr_origin(int origin_depth)
+{
+    PyFrameObject *frame = PyEval_GetFrame();
+    /* First count how many frames we have */
+    int frame_count = 0;
+    for (; frame && frame_count < origin_depth; ++frame_count) {
+        frame = frame->f_back;
+    }
+
+    /* Now collect them */
+    PyObject *cr_origin = PyTuple_New(frame_count);
+    frame = PyEval_GetFrame();
+    for (int i = 0; i < frame_count; ++i) {
+        PyObject *frameinfo = Py_BuildValue(
+            "OiO",
+            frame->f_code->co_filename,
+            PyFrame_GetLineNumber(frame),
+            frame->f_code->co_name);
+        if (!frameinfo) {
+            Py_DECREF(cr_origin);
+            return NULL;
+        }
+        PyTuple_SET_ITEM(cr_origin, i, frameinfo);
+        frame = frame->f_back;
+    }
+
+    return cr_origin;
+}
+
 PyObject *
 PyCoro_New(PyFrameObject *f, PyObject *name, PyObject *qualname)
 {
-    return gen_new_with_qualname(&PyCoro_Type, f, name, qualname);
+    PyObject *coro = gen_new_with_qualname(&PyCoro_Type, f, name, qualname);
+    if (!coro) {
+        return NULL;
+    }
+
+    PyThreadState *tstate = PyThreadState_GET();
+    int origin_depth = tstate->coroutine_origin_tracking_depth;
+
+    if (origin_depth == 0) {
+        ((PyCoroObject *)coro)->cr_origin = NULL;
+    } else {
+        PyObject *cr_origin = compute_cr_origin(origin_depth);
+        if (!cr_origin) {
+            Py_DECREF(coro);
+            return NULL;
+        }
+        ((PyCoroObject *)coro)->cr_origin = cr_origin;
+    }
+
+    return coro;
 }
 
 
diff --git a/Python/_warnings.c b/Python/_warnings.c
index c286364dda0..c3417cccb17 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -1153,6 +1153,53 @@ PyErr_WarnExplicitFormat(PyObject *category,
     return ret;
 }
 
+void
+_PyErr_WarnUnawaitedCoroutine(PyObject *coro)
+{
+    /* First, we attempt to funnel the warning through
+       warnings._warn_unawaited_coroutine.
+
+       This could raise an exception, due to:
+       - a bug
+       - some kind of shutdown-related brokenness
+       - succeeding, but with an "error" warning filter installed, so the
+         warning is converted into a RuntimeWarning exception
+
+       In the first two cases, we want to print the error (so we know what it
+       is!), and then print a warning directly as a fallback. In the last
+       case, we want to print the error (since it's the warning!), but *not*
+       do a fallback. And after we print the error we can't check for what
+       type of error it was (because PyErr_WriteUnraisable clears it), so we
+       need a flag to keep track.
+
+       Since this is called from __del__ context, it's careful to never raise
+       an exception.
+    */
+    _Py_IDENTIFIER(_warn_unawaited_coroutine);
+    int warned = 0;
+    PyObject *fn = get_warnings_attr(&PyId__warn_unawaited_coroutine, 1);
+    if (fn) {
+        PyObject *res = PyObject_CallFunctionObjArgs(fn, coro, NULL);
+        Py_DECREF(fn);
+        if (res || PyErr_ExceptionMatches(PyExc_RuntimeWarning)) {
+            warned = 1;
+        }
+        Py_XDECREF(res);
+    }
+
+    if (PyErr_Occurred()) {
+        PyErr_WriteUnraisable(coro);
+    }
+    if (!warned) {
+        PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
+                         "coroutine '%.50S' was never awaited",
+                         ((PyCoroObject *)coro)->cr_qualname);
+        /* Maybe *that* got converted into an exception */
+        if (PyErr_Occurred()) {
+            PyErr_WriteUnraisable(coro);
+        }
+    }
+}
 
 PyDoc_STRVAR(warn_explicit_doc,
 "Low-level inferface to warnings functionality.");
diff --git a/Python/ceval.c b/Python/ceval.c
index 9276755f0d1..2b7c0c80242 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -4387,6 +4387,21 @@ PyEval_SetTrace(Py_tracefunc func, PyObject *arg)
                            || (tstate->c_profilefunc != NULL));
 }
 
+void
+_PyEval_SetCoroutineOriginTrackingDepth(int new_depth)
+{
+    assert(new_depth >= 0);
+    PyThreadState *tstate = PyThreadState_GET();
+    tstate->coroutine_origin_tracking_depth = new_depth;
+}
+
+int
+_PyEval_GetCoroutineOriginTrackingDepth(void)
+{
+    PyThreadState *tstate = PyThreadState_GET();
+    return tstate->coroutine_origin_tracking_depth;
+}
+
 void
 _PyEval_SetCoroutineWrapper(PyObject *wrapper)
 {
diff --git a/Python/clinic/sysmodule.c.h b/Python/clinic/sysmodule.c.h
new file mode 100644
index 00000000000..3e1480513f6
--- /dev/null
+++ b/Python/clinic/sysmodule.c.h
@@ -0,0 +1,66 @@
+/*[clinic input]
+preserve
+[clinic start generated code]*/
+
+PyDoc_STRVAR(sys_set_coroutine_origin_tracking_depth__doc__,
+"set_coroutine_origin_tracking_depth($module, /, depth)\n"
+"--\n"
+"\n"
+"Enable or disable origin tracking for coroutine objects in this thread.\n"
+"\n"
+"Coroutine objects will track \'depth\' frames of traceback information about\n"
+"where they came from, available in their cr_origin attribute. Set depth of 0\n"
+"to disable.");
+
+#define SYS_SET_COROUTINE_ORIGIN_TRACKING_DEPTH_METHODDEF    \
+    {"set_coroutine_origin_tracking_depth", (PyCFunction)sys_set_coroutine_origin_tracking_depth, METH_FASTCALL|METH_KEYWORDS, sys_set_coroutine_origin_tracking_depth__doc__},
+
+static PyObject *
+sys_set_coroutine_origin_tracking_depth_impl(PyObject *module, int depth);
+
+static PyObject *
+sys_set_coroutine_origin_tracking_depth(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"depth", NULL};
+    static _PyArg_Parser _parser = {"i:set_coroutine_origin_tracking_depth", _keywords, 0};
+    int depth;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &depth)) {
+        goto exit;
+    }
+    return_value = sys_set_coroutine_origin_tracking_depth_impl(module, depth);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(sys_get_coroutine_origin_tracking_depth__doc__,
+"get_coroutine_origin_tracking_depth($module, /)\n"
+"--\n"
+"\n"
+"Check status of origin tracking for coroutine objects in this thread.");
+
+#define SYS_GET_COROUTINE_ORIGIN_TRACKING_DEPTH_METHODDEF    \
+    {"get_coroutine_origin_tracking_depth", (PyCFunction)sys_get_coroutine_origin_tracking_depth, METH_NOARGS, sys_get_coroutine_origin_tracking_depth__doc__},
+
+static int
+sys_get_coroutine_origin_tracking_depth_impl(PyObject *module);
+
+static PyObject *
+sys_get_coroutine_origin_tracking_depth(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    int _return_value;
+
+    _return_value = sys_get_coroutine_origin_tracking_depth_impl(module);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyLong_FromLong((long)_return_value);
+
+exit:
+    return return_value;
+}
+/*[clinic end generated code: output=4a3ac42b97d710ff input=a9049054013a1b77]*/
diff --git a/Python/pystate.c b/Python/pystate.c
index 028292e42fb..9c25a26460e 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -305,6 +305,8 @@ new_threadstate(PyInterpreterState *interp, int init)
         tstate->on_delete = NULL;
         tstate->on_delete_data = NULL;
 
+        tstate->coroutine_origin_tracking_depth = 0;
+
         tstate->coroutine_wrapper = NULL;
         tstate->in_coroutine_wrapper = 0;
 
diff --git a/Python/sysmodule.c b/Python/sysmodule.c
index c0542436fdf..873657f4a4f 100644
--- a/Python/sysmodule.c
+++ b/Python/sysmodule.c
@@ -34,6 +34,13 @@ extern void *PyWin_DLLhModule;
 extern const char *PyWin_DLLVersionString;
 #endif
 
+/*[clinic input]
+module sys
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=3726b388feee8cea]*/
+
+#include "clinic/sysmodule.c.h"
+
 _Py_IDENTIFIER(_);
 _Py_IDENTIFIER(__sizeof__);
 _Py_IDENTIFIER(_xoptions);
@@ -710,9 +717,51 @@ sys_setrecursionlimit(PyObject *self, PyObject *args)
     Py_RETURN_NONE;
 }
 
+/*[clinic input]
+sys.set_coroutine_origin_tracking_depth
+
+  depth: int
+
+Enable or disable origin tracking for coroutine objects in this thread.
+
+Coroutine objects will track 'depth' frames of traceback information about
+where they came from, available in their cr_origin attribute. Set depth of 0
+to disable.
+[clinic start generated code]*/
+
+static PyObject *
+sys_set_coroutine_origin_tracking_depth_impl(PyObject *module, int depth)
+/*[clinic end generated code: output=0a2123c1cc6759c5 input=9083112cccc1bdcb]*/
+{
+    if (depth < 0) {
+        PyErr_SetString(PyExc_ValueError, "depth must be >= 0");
+        return NULL;
+    }
+    _PyEval_SetCoroutineOriginTrackingDepth(depth);
+    Py_RETURN_NONE;
+}
+
+/*[clinic input]
+sys.get_coroutine_origin_tracking_depth -> int
+
+Check status of origin tracking for coroutine objects in this thread.
+[clinic start generated code]*/
+
+static int
+sys_get_coroutine_origin_tracking_depth_impl(PyObject *module)
+/*[clinic end generated code: output=3699f7be95a3afb8 input=335266a71205b61a]*/
+{
+    return _PyEval_GetCoroutineOriginTrackingDepth();
+}
+
 static PyObject *
 sys_set_coroutine_wrapper(PyObject *self, PyObject *wrapper)
 {
+    if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                     "set_coroutine_wrapper is deprecated", 1) < 0) {
+        return NULL;
+    }
+
     if (wrapper != Py_None) {
         if (!PyCallable_Check(wrapper)) {
             PyErr_Format(PyExc_TypeError,
@@ -737,6 +786,10 @@ Set a wrapper for coroutine objects."
 static PyObject *
 sys_get_coroutine_wrapper(PyObject *self, PyObject *args)
 {
+    if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                     "get_coroutine_wrapper is deprecated", 1) < 0) {
+        return NULL;
+    }
     PyObject *wrapper = _PyEval_GetCoroutineWrapper();
     if (wrapper == NULL) {
         wrapper = Py_None;
@@ -1512,6 +1565,8 @@ static PyMethodDef sys_methods[] = {
     {"call_tracing", sys_call_tracing, METH_VARARGS, call_tracing_doc},
     {"_debugmallocstats", sys_debugmallocstats, METH_NOARGS,
      debugmallocstats_doc},
+    SYS_SET_COROUTINE_ORIGIN_TRACKING_DEPTH_METHODDEF
+    SYS_GET_COROUTINE_ORIGIN_TRACKING_DEPTH_METHODDEF
     {"set_coroutine_wrapper", sys_set_coroutine_wrapper, METH_O,
      set_coroutine_wrapper_doc},
     {"get_coroutine_wrapper", sys_get_coroutine_wrapper, METH_NOARGS,


From webhook-mailer at python.org  Sun Jan 21 14:57:02 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sun, 21 Jan 2018 19:57:02 -0000
Subject: [Python-checkins] bpo-32314: Fix asyncio.run() to cancel runinng
 tasks on shutdown (#5262)
Message-ID: <mailman.1.1516564623.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/a4afcdfa55ddffa4b9ae3b0cf101628c7bff4102
commit: a4afcdfa55ddffa4b9ae3b0cf101628c7bff4102
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-21T14:56:59-05:00
summary:

bpo-32314: Fix asyncio.run() to cancel runinng tasks on shutdown (#5262)

files:
M Lib/asyncio/base_events.py
M Lib/asyncio/runners.py
M Lib/test/test_asyncio/test_runners.py
M Lib/test/test_asyncio/utils.py

diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index a10e706d504..ca9eee765e3 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -228,14 +228,9 @@ def __init__(self):
         self._coroutine_origin_tracking_enabled = False
         self._coroutine_origin_tracking_saved_depth = None
 
-        if hasattr(sys, 'get_asyncgen_hooks'):
-            # Python >= 3.6
-            # A weak set of all asynchronous generators that are
-            # being iterated by the loop.
-            self._asyncgens = weakref.WeakSet()
-        else:
-            self._asyncgens = None
-
+        # A weak set of all asynchronous generators that are
+        # being iterated by the loop.
+        self._asyncgens = weakref.WeakSet()
         # Set to True when `loop.shutdown_asyncgens` is called.
         self._asyncgens_shutdown_called = False
 
@@ -354,7 +349,7 @@ def _asyncgen_firstiter_hook(self, agen):
         """Shutdown all active asynchronous generators."""
         self._asyncgens_shutdown_called = True
 
-        if self._asyncgens is None or not len(self._asyncgens):
+        if not len(self._asyncgens):
             # If Python version is <3.6 or we don't have any asynchronous
             # generators alive.
             return
@@ -386,10 +381,10 @@ def run_forever(self):
                 'Cannot run the event loop while another loop is running')
         self._set_coroutine_origin_tracking(self._debug)
         self._thread_id = threading.get_ident()
-        if self._asyncgens is not None:
-            old_agen_hooks = sys.get_asyncgen_hooks()
-            sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook,
-                                   finalizer=self._asyncgen_finalizer_hook)
+
+        old_agen_hooks = sys.get_asyncgen_hooks()
+        sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook,
+                               finalizer=self._asyncgen_finalizer_hook)
         try:
             events._set_running_loop(self)
             while True:
@@ -401,8 +396,7 @@ def run_forever(self):
             self._thread_id = None
             events._set_running_loop(None)
             self._set_coroutine_origin_tracking(False)
-            if self._asyncgens is not None:
-                sys.set_asyncgen_hooks(*old_agen_hooks)
+            sys.set_asyncgen_hooks(*old_agen_hooks)
 
     def run_until_complete(self, future):
         """Run until the Future is done.
@@ -1374,6 +1368,7 @@ def call_exception_handler(self, context):
         - 'message': Error message;
         - 'exception' (optional): Exception object;
         - 'future' (optional): Future instance;
+        - 'task' (optional): Task instance;
         - 'handle' (optional): Handle instance;
         - 'protocol' (optional): Protocol instance;
         - 'transport' (optional): Transport instance;
diff --git a/Lib/asyncio/runners.py b/Lib/asyncio/runners.py
index 94d94097ab9..bb54b725278 100644
--- a/Lib/asyncio/runners.py
+++ b/Lib/asyncio/runners.py
@@ -2,6 +2,7 @@
 
 from . import coroutines
 from . import events
+from . import tasks
 
 
 def run(main, *, debug=False):
@@ -42,7 +43,31 @@ def run(main, *, debug=False):
         return loop.run_until_complete(main)
     finally:
         try:
+            _cancel_all_tasks(loop)
             loop.run_until_complete(loop.shutdown_asyncgens())
         finally:
             events.set_event_loop(None)
             loop.close()
+
+
+def _cancel_all_tasks(loop):
+    to_cancel = [task for task in tasks.all_tasks(loop)
+                 if not task.done()]
+    if not to_cancel:
+        return
+
+    for task in to_cancel:
+        task.cancel()
+
+    loop.run_until_complete(
+        tasks.gather(*to_cancel, loop=loop, return_exceptions=True))
+
+    for task in to_cancel:
+        if task.cancelled():
+            continue
+        if task.exception() is not None:
+            loop.call_exception_handler({
+                'message': 'unhandled exception during asyncio.run() shutdown',
+                'exception': task.exception(),
+                'task': task,
+            })
diff --git a/Lib/test/test_asyncio/test_runners.py b/Lib/test/test_asyncio/test_runners.py
index c52bd9443ea..3b58ddee443 100644
--- a/Lib/test/test_asyncio/test_runners.py
+++ b/Lib/test/test_asyncio/test_runners.py
@@ -2,6 +2,7 @@
 import unittest
 
 from unittest import mock
+from . import utils as test_utils
 
 
 class TestPolicy(asyncio.AbstractEventLoopPolicy):
@@ -98,3 +99,81 @@ def test_asyncio_run_from_running_loop(self):
         with self.assertRaisesRegex(RuntimeError,
                                     'cannot be called from a running'):
             asyncio.run(main())
+
+    def test_asyncio_run_cancels_hanging_tasks(self):
+        lo_task = None
+
+        async def leftover():
+            await asyncio.sleep(0.1)
+
+        async def main():
+            nonlocal lo_task
+            lo_task = asyncio.create_task(leftover())
+            return 123
+
+        self.assertEqual(asyncio.run(main()), 123)
+        self.assertTrue(lo_task.done())
+
+    def test_asyncio_run_reports_hanging_tasks_errors(self):
+        lo_task = None
+        call_exc_handler_mock = mock.Mock()
+
+        async def leftover():
+            try:
+                await asyncio.sleep(0.1)
+            except asyncio.CancelledError:
+                1 / 0
+
+        async def main():
+            loop = asyncio.get_running_loop()
+            loop.call_exception_handler = call_exc_handler_mock
+
+            nonlocal lo_task
+            lo_task = asyncio.create_task(leftover())
+            return 123
+
+        self.assertEqual(asyncio.run(main()), 123)
+        self.assertTrue(lo_task.done())
+
+        call_exc_handler_mock.assert_called_with({
+            'message': test_utils.MockPattern(r'asyncio.run.*shutdown'),
+            'task': lo_task,
+            'exception': test_utils.MockInstanceOf(ZeroDivisionError)
+        })
+
+    def test_asyncio_run_closes_gens_after_hanging_tasks_errors(self):
+        spinner = None
+        lazyboy = None
+
+        class FancyExit(Exception):
+            pass
+
+        async def fidget():
+            while True:
+                yield 1
+                await asyncio.sleep(1)
+
+        async def spin():
+            nonlocal spinner
+            spinner = fidget()
+            try:
+                async for the_meaning_of_life in spinner:  # NoQA
+                    pass
+            except asyncio.CancelledError:
+                1 / 0
+
+        async def main():
+            loop = asyncio.get_running_loop()
+            loop.call_exception_handler = mock.Mock()
+
+            nonlocal lazyboy
+            lazyboy = asyncio.create_task(spin())
+            raise FancyExit
+
+        with self.assertRaises(FancyExit):
+            asyncio.run(main())
+
+        self.assertTrue(lazyboy.done())
+
+        self.assertIsNone(spinner.ag_frame)
+        self.assertFalse(spinner.ag_running)
diff --git a/Lib/test/test_asyncio/utils.py b/Lib/test/test_asyncio/utils.py
index 6c809770b4a..f756ec9016f 100644
--- a/Lib/test/test_asyncio/utils.py
+++ b/Lib/test/test_asyncio/utils.py
@@ -485,6 +485,14 @@ def __eq__(self, other):
         return bool(re.search(str(self), other, re.S))
 
 
+class MockInstanceOf:
+    def __init__(self, type):
+        self._type = type
+
+    def __eq__(self, other):
+        return isinstance(other, self._type)
+
+
 def get_function_source(func):
     source = format_helpers._get_function_source(func)
     if source is None:


From webhook-mailer at python.org  Sun Jan 21 20:47:11 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 22 Jan 2018 01:47:11 -0000
Subject: [Python-checkins] bpo-32591: Fix PyExc_WarnFormat call (follow-up
 commit) (#5263)
Message-ID: <mailman.2.1516585633.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/3510334361d6d39aad89e4e0d9bccd0695668583
commit: 3510334361d6d39aad89e4e0d9bccd0695668583
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-21T20:47:04-05:00
summary:

bpo-32591: Fix PyExc_WarnFormat call (follow-up commit) (#5263)

The previous version was correct in terms of behaviour, but
checking the return value of PyErr_WarnFormat allows to
avoid calling PyErr_Occurred and silences the coverity alarm.

files:
M Python/_warnings.c

diff --git a/Python/_warnings.c b/Python/_warnings.c
index c3417cccb17..20ba7a16616 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -1191,11 +1191,10 @@ _PyErr_WarnUnawaitedCoroutine(PyObject *coro)
         PyErr_WriteUnraisable(coro);
     }
     if (!warned) {
-        PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
-                         "coroutine '%.50S' was never awaited",
-                         ((PyCoroObject *)coro)->cr_qualname);
-        /* Maybe *that* got converted into an exception */
-        if (PyErr_Occurred()) {
+        if (PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
+                             "coroutine '%.50S' was never awaited",
+                             ((PyCoroObject *)coro)->cr_qualname) < 0)
+        {
             PyErr_WriteUnraisable(coro);
         }
     }


From webhook-mailer at python.org  Mon Jan 22 02:45:34 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Mon, 22 Jan 2018 07:45:34 -0000
Subject: [Python-checkins] urllib.request: Remove unused import (GH-5268)
Message-ID: <mailman.3.1516607135.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/579e0b80b953b8a47385bc50844dbaac45d6f437
commit: 579e0b80b953b8a47385bc50844dbaac45d6f437
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-22T16:45:31+09:00
summary:

urllib.request: Remove unused import (GH-5268)

files:
M Lib/urllib/request.py

diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py
index a192d527d8b..2b769421c56 100644
--- a/Lib/urllib/request.py
+++ b/Lib/urllib/request.py
@@ -1792,7 +1792,6 @@ def retrieve(self, url, filename=None, reporthook=None, data=None):
             if filename:
                 tfp = open(filename, 'wb')
             else:
-                import tempfile
                 garbage, path = splittype(url)
                 garbage, path = splithost(path or "")
                 path, garbage = splitquery(path or "")


From solipsis at pitrou.net  Mon Jan 22 04:22:00 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 22 Jan 2018 09:22:00 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=7
Message-ID: <20180122092200.1.B5CD9BA4D6112563@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [-7, 1, 7] memory blocks, sum=1
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_spawn leaked [0, 2, 0] memory blocks, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogx71l5Q', '--timeout', '7200']

From webhook-mailer at python.org  Mon Jan 22 10:56:40 2018
From: webhook-mailer at python.org (Ethan Furman)
Date: Mon, 22 Jan 2018 15:56:40 -0000
Subject: [Python-checkins] bpo-31801: Enum: add _ignore_ as class option
 (#5237)
Message-ID: <mailman.4.1516636601.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/a4b1bb4801f7a941ff9e86b96da539be1c288833
commit: a4b1bb4801f7a941ff9e86b96da539be1c288833
branch: master
author: Ethan Furman <ethan at stoneleaf.us>
committer: GitHub <noreply at github.com>
date: 2018-01-22T07:56:37-08:00
summary:

bpo-31801:  Enum:  add _ignore_ as class option (#5237)

* bpo-31801:  Enum:  add _ignore_ as class option

_ignore_ is a list, or white-space seperated str, of names that will not
be candidates for members; these names, and _ignore_ itself, are removed
from the final class.

* bpo-31801:  Enum:  add documentation for _ignore_

* bpo-31801: Enum: remove trailing whitespace

* bpo-31801: Enum: fix bulleted list format

* bpo-31801: add version added for _ignore_

files:
A Misc/NEWS.d/next/Library/2018-01-18-13-47-40.bpo-31801.3UGH1h.rst
M Doc/library/enum.rst
M Lib/enum.py
M Lib/test/test_enum.py

diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst
index 5c1b226efc7..fc65a3d078f 100644
--- a/Doc/library/enum.rst
+++ b/Doc/library/enum.rst
@@ -379,7 +379,8 @@ The rules for what is allowed are as follows: names that start and end with
 a single underscore are reserved by enum and cannot be used; all other
 attributes defined within an enumeration will become members of this
 enumeration, with the exception of special methods (:meth:`__str__`,
-:meth:`__add__`, etc.) and descriptors (methods are also descriptors).
+:meth:`__add__`, etc.), descriptors (methods are also descriptors), and
+variable names listed in :attr:`_ignore_`.
 
 Note:  if your enumeration defines :meth:`__new__` and/or :meth:`__init__` then
 whatever value(s) were given to the enum member will be passed into those
@@ -943,6 +944,25 @@ will be passed to those methods::
     9.802652743337129
 
 
+TimePeriod
+^^^^^^^^^^
+
+An example to show the :attr:`_ignore_` attribute in use::
+
+    >>> from datetime import timedelta
+    >>> class Period(timedelta, Enum):
+    ...     "different lengths of time"
+    ...     _ignore_ = 'Period i'
+    ...     Period = vars()
+    ...     for i in range(367):
+    ...         Period['day_%d' % i] = i
+    ...
+    >>> list(Period)[:2]
+    [<Period.day_0: datetime.timedelta(0)>, <Period.day_1: datetime.timedelta(days=1)>]
+    >>> list(Period)[-2:]
+    [<Period.day_365: datetime.timedelta(days=365)>, <Period.day_366: datetime.timedelta(days=366)>]
+
+
 How are Enums different?
 ------------------------
 
@@ -994,6 +1014,9 @@ Supported ``_sunder_`` names
 
 - ``_missing_`` -- a lookup function used when a value is not found; may be
   overridden
+- ``_ignore_`` -- a list of names, either as a :func:`list` or a :func:`str`,
+  that will not be transformed into members, and will be removed from the final
+  class
 - ``_order_`` -- used in Python 2/3 code to ensure member order is consistent
   (class attribute, removed during class creation)
 - ``_generate_next_value_`` -- used by the `Functional API`_ and by
@@ -1001,6 +1024,7 @@ Supported ``_sunder_`` names
   overridden
 
 .. versionadded:: 3.6 ``_missing_``, ``_order_``, ``_generate_next_value_``
+.. versionadded:: 3.7 ``_ignore_``
 
 To help keep Python 2 / Python 3 code in sync an :attr:`_order_` attribute can
 be provided.  It will be checked against the actual order of the enumeration
diff --git a/Lib/enum.py b/Lib/enum.py
index fe7cb20fc06..e5fe6f3b94a 100644
--- a/Lib/enum.py
+++ b/Lib/enum.py
@@ -64,6 +64,7 @@ def __init__(self):
         super().__init__()
         self._member_names = []
         self._last_values = []
+        self._ignore = []
 
     def __setitem__(self, key, value):
         """Changes anything not dundered or not a descriptor.
@@ -77,17 +78,28 @@ def __setitem__(self, key, value):
         if _is_sunder(key):
             if key not in (
                     '_order_', '_create_pseudo_member_',
-                    '_generate_next_value_', '_missing_',
+                    '_generate_next_value_', '_missing_', '_ignore_',
                     ):
                 raise ValueError('_names_ are reserved for future Enum use')
             if key == '_generate_next_value_':
                 setattr(self, '_generate_next_value', value)
+            elif key == '_ignore_':
+                if isinstance(value, str):
+                    value = value.replace(',',' ').split()
+                else:
+                    value = list(value)
+                self._ignore = value
+                already = set(value) & set(self._member_names)
+                if already:
+                    raise ValueError('_ignore_ cannot specify already set names: %r' % (already, ))
         elif _is_dunder(key):
             if key == '__order__':
                 key = '_order_'
         elif key in self._member_names:
             # descriptor overwriting an enum?
             raise TypeError('Attempted to reuse key: %r' % key)
+        elif key in self._ignore:
+            pass
         elif not _is_descriptor(value):
             if key in self:
                 # enum overwriting a descriptor?
@@ -124,6 +136,12 @@ def __new__(metacls, cls, bases, classdict):
         # cannot be mixed with other types (int, float, etc.) if it has an
         # inherited __new__ unless a new __new__ is defined (or the resulting
         # class will fail).
+        #
+        # remove any keys listed in _ignore_
+        classdict.setdefault('_ignore_', []).append('_ignore_')
+        ignore = classdict['_ignore_']
+        for key in ignore:
+            classdict.pop(key, None)
         member_type, first_enum = metacls._get_mixins_(bases)
         __new__, save_new, use_args = metacls._find_new_(classdict, member_type,
                                                         first_enum)
diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py
index e6324d4d8c8..97559712b1d 100644
--- a/Lib/test/test_enum.py
+++ b/Lib/test/test_enum.py
@@ -8,7 +8,12 @@
 from io import StringIO
 from pickle import dumps, loads, PicklingError, HIGHEST_PROTOCOL
 from test import support
+from datetime import timedelta
 
+try:
+    import threading
+except ImportError:
+    threading = None
 
 # for pickle tests
 try:
@@ -1547,6 +1552,34 @@ def surface_gravity(self):
         self.assertEqual(round(Planet.EARTH.surface_gravity, 2), 9.80)
         self.assertEqual(Planet.EARTH.value, (5.976e+24, 6.37814e6))
 
+    def test_ignore(self):
+        class Period(timedelta, Enum):
+            '''
+            different lengths of time
+            '''
+            def __new__(cls, value, period):
+                obj = timedelta.__new__(cls, value)
+                obj._value_ = value
+                obj.period = period
+                return obj
+            _ignore_ = 'Period i'
+            Period = vars()
+            for i in range(13):
+                Period['month_%d' % i] = i*30, 'month'
+            for i in range(53):
+                Period['week_%d' % i] = i*7, 'week'
+            for i in range(32):
+                Period['day_%d' % i] = i, 'day'
+            OneDay = day_1
+            OneWeek = week_1
+            OneMonth = month_1
+        self.assertFalse(hasattr(Period, '_ignore_'))
+        self.assertFalse(hasattr(Period, 'Period'))
+        self.assertFalse(hasattr(Period, 'i'))
+        self.assertTrue(isinstance(Period.day_1, timedelta))
+        self.assertTrue(Period.month_1 is Period.day_30)
+        self.assertTrue(Period.week_4 is Period.day_28)
+
     def test_nonhash_value(self):
         class AutoNumberInAList(Enum):
             def __new__(cls):
diff --git a/Misc/NEWS.d/next/Library/2018-01-18-13-47-40.bpo-31801.3UGH1h.rst b/Misc/NEWS.d/next/Library/2018-01-18-13-47-40.bpo-31801.3UGH1h.rst
new file mode 100644
index 00000000000..48043c046fa
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-18-13-47-40.bpo-31801.3UGH1h.rst
@@ -0,0 +1,2 @@
+Add ``_ignore_`` to ``Enum`` so temporary variables can be used during class
+construction without being turned into members.


From webhook-mailer at python.org  Mon Jan 22 11:54:46 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 22 Jan 2018 16:54:46 -0000
Subject: [Python-checkins] bpo-31179: Make dict.copy() up to 5.5 times
 faster. (#3067)
Message-ID: <mailman.5.1516640087.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/b0a7a037b8fde56b62f886d5188bced7776777b4
commit: b0a7a037b8fde56b62f886d5188bced7776777b4
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-22T11:54:41-05:00
summary:

bpo-31179: Make dict.copy() up to 5.5 times faster. (#3067)

files:
A Misc/NEWS.d/next/Core and Builtins/2017-08-10-17-32-48.bpo-31179.XcgLYI.rst
M Lib/test/test_dict.py
M Objects/dictobject.c

diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py
index 4386eda3ae4..aa149d31eb0 100644
--- a/Lib/test/test_dict.py
+++ b/Lib/test/test_dict.py
@@ -271,11 +271,57 @@ def __new__(cls):
         self.assertEqual(baddict3.fromkeys({"a", "b", "c"}), res)
 
     def test_copy(self):
-        d = {1:1, 2:2, 3:3}
-        self.assertEqual(d.copy(), {1:1, 2:2, 3:3})
+        d = {1: 1, 2: 2, 3: 3}
+        self.assertIsNot(d.copy(), d)
+        self.assertEqual(d.copy(), d)
+        self.assertEqual(d.copy(), {1: 1, 2: 2, 3: 3})
+
+        copy = d.copy()
+        d[4] = 4
+        self.assertNotEqual(copy, d)
+
         self.assertEqual({}.copy(), {})
         self.assertRaises(TypeError, d.copy, None)
 
+    def test_copy_fuzz(self):
+        for dict_size in [10, 100, 1000, 10000, 100000]:
+            dict_size = random.randrange(
+                dict_size // 2, dict_size + dict_size // 2)
+            with self.subTest(dict_size=dict_size):
+                d = {}
+                for i in range(dict_size):
+                    d[i] = i
+
+                d2 = d.copy()
+                self.assertIsNot(d2, d)
+                self.assertEqual(d, d2)
+                d2['key'] = 'value'
+                self.assertNotEqual(d, d2)
+                self.assertEqual(len(d2), len(d) + 1)
+
+    def test_copy_maintains_tracking(self):
+        class A:
+            pass
+
+        key = A()
+
+        for d in ({}, {'a': 1}, {key: 'val'}):
+            d2 = d.copy()
+            self.assertEqual(gc.is_tracked(d), gc.is_tracked(d2))
+
+    def test_copy_noncompact(self):
+        # Dicts don't compact themselves on del/pop operations.
+        # Copy will use a slow merging strategy that produces
+        # a compacted copy when roughly 33% of dict is a non-used
+        # keys-space (to optimize memory footprint).
+        # In this test we want to hit the slow/compacting
+        # branch of dict.copy() and make sure it works OK.
+        d = {k: k for k in range(1000)}
+        for k in range(950):
+            del d[k]
+        d2 = d.copy()
+        self.assertEqual(d2, d)
+
     def test_get(self):
         d = {}
         self.assertIs(d.get('c'), None)
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-08-10-17-32-48.bpo-31179.XcgLYI.rst b/Misc/NEWS.d/next/Core and Builtins/2017-08-10-17-32-48.bpo-31179.XcgLYI.rst
new file mode 100644
index 00000000000..12ebc8bfc79
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-08-10-17-32-48.bpo-31179.XcgLYI.rst	
@@ -0,0 +1 @@
+Make dict.copy() up to 5.5 times faster.
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index b20b85c909e..259fa255653 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -615,6 +615,52 @@ new_dict_with_shared_keys(PyDictKeysObject *keys)
     return new_dict(keys, values);
 }
 
+
+static PyObject *
+clone_combined_dict(PyDictObject *orig)
+{
+    assert(PyDict_CheckExact(orig));
+    assert(orig->ma_values == NULL);
+    assert(orig->ma_keys->dk_refcnt == 1);
+
+    Py_ssize_t keys_size = _PyDict_KeysSize(orig->ma_keys);
+    PyDictKeysObject *keys = PyObject_Malloc(keys_size);
+    if (keys == NULL) {
+        PyErr_NoMemory();
+        return NULL;
+    }
+
+    memcpy(keys, orig->ma_keys, keys_size);
+
+    /* After copying key/value pairs, we need to incref all
+       keys and values and they are about to be co-owned by a
+       new dict object. */
+    PyDictKeyEntry *ep0 = DK_ENTRIES(keys);
+    Py_ssize_t n = keys->dk_nentries;
+    for (Py_ssize_t i = 0; i < n; i++) {
+        PyDictKeyEntry *entry = &ep0[i];
+        PyObject *value = entry->me_value;
+        if (value != NULL) {
+            Py_INCREF(value);
+            Py_INCREF(entry->me_key);
+        }
+    }
+
+    PyDictObject *new = (PyDictObject *)new_dict(keys, NULL);
+    if (new == NULL) {
+        /* In case of an error, `new_dict()` takes care of
+           cleaning up `keys`. */
+        return NULL;
+    }
+    new->ma_used = orig->ma_used;
+    assert(_PyDict_CheckConsistency(new));
+    if (_PyObject_GC_IS_TRACKED(orig)) {
+        /* Maintain tracking. */
+        _PyObject_GC_TRACK(new);
+    }
+    return (PyObject *)new;
+}
+
 PyObject *
 PyDict_New(void)
 {
@@ -2484,7 +2530,13 @@ PyDict_Copy(PyObject *o)
         PyErr_BadInternalCall();
         return NULL;
     }
+
     mp = (PyDictObject *)o;
+    if (mp->ma_used == 0) {
+        /* The dict is empty; just return a new dict. */
+        return PyDict_New();
+    }
+
     if (_PyDict_HasSplitTable(mp)) {
         PyDictObject *split_copy;
         Py_ssize_t size = USABLE_FRACTION(DK_SIZE(mp->ma_keys));
@@ -2510,6 +2562,27 @@ PyDict_Copy(PyObject *o)
             _PyObject_GC_TRACK(split_copy);
         return (PyObject *)split_copy;
     }
+
+    if (PyDict_CheckExact(mp) && mp->ma_values == NULL &&
+            (mp->ma_used >= (mp->ma_keys->dk_nentries * 2) / 3))
+    {
+        /* Use fast-copy if:
+
+           (1) 'mp' is an instance of a subclassed dict; and
+
+           (2) 'mp' is not a split-dict; and
+
+           (3) if 'mp' is non-compact ('del' operation does not resize dicts),
+               do fast-copy only if it has at most 1/3 non-used keys.
+
+           The last condition (3) is important to guard against a pathalogical
+           case when a large dict is almost emptied with multiple del/pop
+           operations and copied after that.  In cases like this, we defer to
+           PyDict_Merge, which produces a compacted copy.
+        */
+        return clone_combined_dict(mp);
+    }
+
     copy = PyDict_New();
     if (copy == NULL)
         return NULL;


From webhook-mailer at python.org  Mon Jan 22 12:32:55 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 22 Jan 2018 17:32:55 -0000
Subject: [Python-checkins] bpo-32593: Drop FreeBSD 9 and older support
 (#5232)
Message-ID: <mailman.6.1516642377.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/13ff24582c99dfb439b1af7295b401415e7eb05b
commit: 13ff24582c99dfb439b1af7295b401415e7eb05b
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-22T18:32:50+01:00
summary:

bpo-32593: Drop FreeBSD 9 and older support (#5232)

Drop support of FreeBSD 9 and older.

files:
A Misc/NEWS.d/next/Build/2018-01-18-11-10-52.bpo-32593.XIrf3v.rst
M Doc/library/time.rst
M Doc/whatsnew/3.7.rst
M Include/py_curses.h
M Include/pyport.h
M Lib/test/test_asyncio/test_events.py
M Lib/test/test_io.py
M Lib/test/test_logging.py
M Lib/test/test_os.py
M Lib/test/test_posix.py
M Lib/test/test_resource.py
M Lib/test/test_signal.py
M Lib/test/test_socket.py
M Lib/test/test_threading.py
M Lib/unittest/test/test_break.py
M Python/thread_pthread.h
M configure.ac
M setup.py

diff --git a/Doc/library/time.rst b/Doc/library/time.rst
index ccbb3f37877..3eddc3f1ad3 100644
--- a/Doc/library/time.rst
+++ b/Doc/library/time.rst
@@ -792,7 +792,7 @@ These constants are used as parameters for :func:`clock_getres` and
 
    High-resolution per-process timer from the CPU.
 
-   Availability: FreeBSD 3 or later, NetBSD 7 or later, OpenBSD.
+   Availability: FreeBSD, NetBSD 7 or later, OpenBSD.
 
    .. versionadded:: 3.7
 
@@ -812,7 +812,7 @@ These constants are used as parameters for :func:`clock_getres` and
    suspended, providing accurate uptime measurement, both absolute and
    interval.
 
-   Availability: FreeBSD 7 or later, OpenBSD 5.5 or later.
+   Availability: FreeBSD, OpenBSD 5.5 or later.
 
    .. versionadded:: 3.7
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 54a3f14b0bb..0418aa6d7ad 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -806,6 +806,11 @@ Windows Only
 Removed
 =======
 
+Platform Support Removals
+-------------------------
+
+* FreeBSD 9 and older are no longer supported.
+
 API and Feature Removals
 ------------------------
 
diff --git a/Include/py_curses.h b/Include/py_curses.h
index 597f419eaca..0eebc362a15 100644
--- a/Include/py_curses.h
+++ b/Include/py_curses.h
@@ -12,31 +12,15 @@
 #endif
 #endif /* __APPLE__ */
 
-#ifdef __FreeBSD__
-/*
-** On FreeBSD, [n]curses.h and stdlib.h/wchar.h use different guards
-** against multiple definition of wchar_t and wint_t.
-*/
-#ifdef _XOPEN_SOURCE_EXTENDED
-#ifndef __FreeBSD_version
-#include <osreldate.h>
-#endif
-#if __FreeBSD_version >= 500000
-#ifndef __wchar_t
-#define __wchar_t
-#endif
-#ifndef __wint_t
-#define __wint_t
-#endif
-#else
-#ifndef _WCHAR_T
-#define _WCHAR_T
-#endif
-#ifndef _WINT_T
-#define _WINT_T
-#endif
-#endif
-#endif
+/* On FreeBSD, [n]curses.h and stdlib.h/wchar.h use different guards
+   against multiple definition of wchar_t and wint_t. */
+#if defined(__FreeBSD__) && defined(_XOPEN_SOURCE_EXTENDED)
+# ifndef __wchar_t
+#   define __wchar_t
+# endif
+# ifndef __wint_t
+#   define __wint_t
+# endif
 #endif
 
 #if !defined(HAVE_CURSES_IS_PAD) && defined(WINDOW_HAS_FLAGS)
diff --git a/Include/pyport.h b/Include/pyport.h
index f2e247a374e..c1f4c7fbb52 100644
--- a/Include/pyport.h
+++ b/Include/pyport.h
@@ -564,18 +564,8 @@ extern char * _getpty(int *, int, mode_t, int);
  * workaround was provided by Tim Robbins of FreeBSD project.
  */
 
-#ifdef __FreeBSD__
-#include <osreldate.h>
-#if (__FreeBSD_version >= 500040 && __FreeBSD_version < 602113) || \
-    (__FreeBSD_version >= 700000 && __FreeBSD_version < 700054) || \
-    (__FreeBSD_version >= 800000 && __FreeBSD_version < 800001)
-# define _PY_PORT_CTYPE_UTF8_ISSUE
-#endif
-#endif
-
-
 #if defined(__APPLE__)
-# define _PY_PORT_CTYPE_UTF8_ISSUE
+#  define _PY_PORT_CTYPE_UTF8_ISSUE
 #endif
 
 #ifdef _PY_PORT_CTYPE_UTF8_ISSUE
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index 4140f03967e..e4b053681b4 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -1471,8 +1471,6 @@ def test_unclosed_pipe_transport(self):
     @unittest.skipUnless(sys.platform != 'win32',
                          "Don't support pipes for Windows")
     @unittest.skipIf(sys.platform == 'darwin', 'test hangs on MacOS')
-    # Issue #20495: The test hangs on FreeBSD 7.2 but pass on FreeBSD 9
-    @support.requires_freebsd_version(8)
     def test_read_pty_output(self):
         proto = MyReadPipeProto(loop=self.loop)
 
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
index 3aee5f1083c..c6345e9199c 100644
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -4107,8 +4107,6 @@ def test_interrupted_write_unbuffered(self):
     def test_interrupted_write_buffered(self):
         self.check_interrupted_write(b"xy", b"xy", mode="wb")
 
-    # Issue #22331: The test hangs on FreeBSD 7.2
-    @support.requires_freebsd_version(8)
     def test_interrupted_write_text(self):
         self.check_interrupted_write("xy", b"xy", mode="w", encoding="ascii")
 
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
index 54049d2fb36..eee2ed0c19e 100644
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -773,14 +773,7 @@ def serve_forever(self, poll_interval):
                               :func:`select` or :func:`poll` call by
                               :func:`asyncore.loop`.
         """
-        try:
-            asyncore.loop(poll_interval, map=self._map)
-        except OSError:
-            # On FreeBSD 8, closing the server repeatably
-            # raises this error. We swallow it if the
-            # server has been closed.
-            if self.connected or self.accepting:
-                raise
+        asyncore.loop(poll_interval, map=self._map)
 
     def stop(self, timeout=None):
         """
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
index 83e214d91ea..02611d223b7 100644
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -782,9 +782,7 @@ def test_environb(self):
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
-    # On FreeBSD < 7 and OS X < 10.6, unsetenv() doesn't return a value (issue
-    # #13415).
-    @support.requires_freebsd_version(7)
+    # On OS X < 10.6, unsetenv() doesn't return a value (bpo-13415).
     @support.requires_mac_ver(10, 6)
     def test_unset_error(self):
         if sys.platform == "win32":
diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
index 44b8d6a7ad0..6c504063179 100644
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -559,6 +559,7 @@ def test_makedev(self):
         self.assertRaises(TypeError, posix.minor)
         self.assertRaises((ValueError, OverflowError), posix.minor, -1)
 
+        # FIXME: reenable these tests on FreeBSD with the kernel fix
         if sys.platform.startswith('freebsd') and dev >= 0x1_0000_0000:
             self.skipTest("bpo-31044: on FreeBSD CURRENT, minor() truncates "
                           "64-bit dev to 32-bit")
diff --git a/Lib/test/test_resource.py b/Lib/test/test_resource.py
index cc9c57024de..4b852789be7 100644
--- a/Lib/test/test_resource.py
+++ b/Lib/test/test_resource.py
@@ -138,7 +138,6 @@ def test_linux_constants(self):
             with contextlib.suppress(AttributeError):
                 self.assertIsInstance(getattr(resource, 'RLIMIT_' + attr), int)
 
-    @support.requires_freebsd_version(9)
     def test_freebsd_contants(self):
         for attr in ['SWAP', 'SBSIZE', 'NPTS']:
             with contextlib.suppress(AttributeError):
diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py
index f17123c3bb5..48b7a392e50 100644
--- a/Lib/test/test_signal.py
+++ b/Lib/test/test_signal.py
@@ -56,9 +56,6 @@ def test_getsignal(self):
         self.assertEqual(signal.getsignal(signal.SIGHUP), hup)
 
     # Issue 3864, unknown if this affects earlier versions of freebsd also
-    @unittest.skipIf(sys.platform=='freebsd6',
-        'inter process signals not reliable (do not mix well with threading) '
-        'on freebsd6')
     def test_interprocess_signal(self):
         dirname = os.path.dirname(__file__)
         script = os.path.join(dirname, 'signalinterproctester.py')
@@ -651,7 +648,7 @@ def test_itimer_real(self):
         self.assertEqual(self.hndl_called, True)
 
     # Issue 3864, unknown if this affects earlier versions of freebsd also
-    @unittest.skipIf(sys.platform in ('freebsd6', 'netbsd5'),
+    @unittest.skipIf(sys.platform in ('netbsd5',),
         'itimer not reliable (does not mix well with threading) on some BSDs.')
     def test_itimer_virtual(self):
         self.itimer = signal.ITIMER_VIRTUAL
@@ -673,9 +670,6 @@ def test_itimer_virtual(self):
         # and the handler should have been called
         self.assertEqual(self.hndl_called, True)
 
-    # Issue 3864, unknown if this affects earlier versions of freebsd also
-    @unittest.skipIf(sys.platform=='freebsd6',
-        'itimer not reliable (does not mix well with threading) on freebsd6')
     def test_itimer_prof(self):
         self.itimer = signal.ITIMER_PROF
         signal.signal(signal.SIGPROF, self.sig_prof)
@@ -762,16 +756,6 @@ def handler(signum, frame):
 
             signal.signal(signum, handler)
 
-            if sys.platform == 'freebsd6':
-                # Issue #12392 and #12469: send a signal to the main thread
-                # doesn't work before the creation of the first thread on
-                # FreeBSD 6
-                def noop():
-                    pass
-                thread = threading.Thread(target=noop)
-                thread.start()
-                thread.join()
-
             tid = threading.get_ident()
             try:
                 signal.pthread_kill(tid, signum)
@@ -1010,9 +994,6 @@ def read_sigmask():
         """
         assert_python_ok('-c', code)
 
-    @unittest.skipIf(sys.platform == 'freebsd6',
-        "issue #12392: send a signal to the main thread doesn't work "
-        "before the creation of the first thread on FreeBSD 6")
     @unittest.skipUnless(hasattr(signal, 'pthread_kill'),
                          'need signal.pthread_kill()')
     def test_pthread_kill_main_thread(self):
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 43688eacf06..22d142c8f80 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -2598,9 +2598,6 @@ def testRecvmsgShorter(self):
     def _testRecvmsgShorter(self):
         self.sendToServer(MSG)
 
-    # FreeBSD < 8 doesn't always set the MSG_TRUNC flag when a truncated
-    # datagram is received (issue #13001).
-    @support.requires_freebsd_version(8)
     def testRecvmsgTrunc(self):
         # Receive part of message, check for truncation indicators.
         msg, ancdata, flags, addr = self.doRecvmsg(self.serv_sock,
@@ -2610,7 +2607,6 @@ def testRecvmsgTrunc(self):
         self.assertEqual(ancdata, [])
         self.checkFlags(flags, eor=False)
 
-    @support.requires_freebsd_version(8)
     def _testRecvmsgTrunc(self):
         self.sendToServer(MSG)
 
diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py
index 007581d7213..db70dfa95df 100644
--- a/Lib/test/test_threading.py
+++ b/Lib/test/test_threading.py
@@ -25,8 +25,7 @@
 # #12316 and #11870), and fork() from a worker thread is known to trigger
 # problems with some operating systems (issue #3863): skip problematic tests
 # on platforms known to behave badly.
-platforms_to_skip = ('freebsd4', 'freebsd5', 'freebsd6', 'netbsd5',
-                     'hp-ux11')
+platforms_to_skip = ('netbsd5', 'hp-ux11')
 
 
 # A trivial mutable counter.
diff --git a/Lib/unittest/test/test_break.py b/Lib/unittest/test/test_break.py
index 2c7501952c7..aa2c69eea01 100644
--- a/Lib/unittest/test/test_break.py
+++ b/Lib/unittest/test/test_break.py
@@ -10,8 +10,6 @@
 
 @unittest.skipUnless(hasattr(os, 'kill'), "Test requires os.kill")
 @unittest.skipIf(sys.platform =="win32", "Test cannot run on Windows")
- at unittest.skipIf(sys.platform == 'freebsd6', "Test kills regrtest on freebsd6 "
-    "if threads have been used")
 class TestBreak(unittest.TestCase):
     int_handler = None
 
@@ -267,22 +265,16 @@ def test():
 
 @unittest.skipUnless(hasattr(os, 'kill'), "Test requires os.kill")
 @unittest.skipIf(sys.platform =="win32", "Test cannot run on Windows")
- at unittest.skipIf(sys.platform == 'freebsd6', "Test kills regrtest on freebsd6 "
-    "if threads have been used")
 class TestBreakDefaultIntHandler(TestBreak):
     int_handler = signal.default_int_handler
 
 @unittest.skipUnless(hasattr(os, 'kill'), "Test requires os.kill")
 @unittest.skipIf(sys.platform =="win32", "Test cannot run on Windows")
- at unittest.skipIf(sys.platform == 'freebsd6', "Test kills regrtest on freebsd6 "
-    "if threads have been used")
 class TestBreakSignalIgnored(TestBreak):
     int_handler = signal.SIG_IGN
 
 @unittest.skipUnless(hasattr(os, 'kill'), "Test requires os.kill")
 @unittest.skipIf(sys.platform =="win32", "Test cannot run on Windows")
- at unittest.skipIf(sys.platform == 'freebsd6', "Test kills regrtest on freebsd6 "
-    "if threads have been used")
 class TestBreakSignalDefault(TestBreak):
     int_handler = signal.SIG_DFL
 
diff --git a/Misc/NEWS.d/next/Build/2018-01-18-11-10-52.bpo-32593.XIrf3v.rst b/Misc/NEWS.d/next/Build/2018-01-18-11-10-52.bpo-32593.XIrf3v.rst
new file mode 100644
index 00000000000..e033cf2cedf
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2018-01-18-11-10-52.bpo-32593.XIrf3v.rst
@@ -0,0 +1 @@
+Drop support of FreeBSD 9 and older.
diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h
index b7463c0ca67..697140558fd 100644
--- a/Python/thread_pthread.h
+++ b/Python/thread_pthread.h
@@ -56,16 +56,6 @@
 #endif
 #endif
 
-/* Before FreeBSD 5.4, system scope threads was very limited resource
-   in default setting.  So the process scope is preferred to get
-   enough number of threads to work. */
-#ifdef __FreeBSD__
-#include <osreldate.h>
-#if __FreeBSD_version >= 500000 && __FreeBSD_version < 504101
-#undef PTHREAD_SYSTEM_SCHED_SUPPORTED
-#endif
-#endif
-
 #if !defined(pthread_attr_default)
 #  define pthread_attr_default ((pthread_attr_t *)NULL)
 #endif
diff --git a/configure.ac b/configure.ac
index 39e2e8e769e..03b0f501aff 100644
--- a/configure.ac
+++ b/configure.ac
@@ -481,10 +481,6 @@ case $ac_sys_system/$ac_sys_release in
   # but used in struct sockaddr.sa_family. Reported by Tim Rice.
   SCO_SV/3.2)
     define_xopen_source=no;;
-  # On FreeBSD 4, the math functions C89 does not cover are never defined
-  # with _XOPEN_SOURCE and __BSD_VISIBLE does not re-enable them.
-  FreeBSD/4.*)
-    define_xopen_source=no;;
   # On MacOS X 10.2, a bug in ncurses.h means that it craps out if
   # _XOPEN_EXTENDED_SOURCE is defined. Apparently, this is fixed in 10.3, which
   # identifies itself as Darwin/7.*
diff --git a/setup.py b/setup.py
index c23628a2a94..1da40a426a2 100644
--- a/setup.py
+++ b/setup.py
@@ -1571,12 +1571,6 @@ class db_found(Exception): pass
             macros = dict()
             libraries = []
 
-        elif host_platform in ('freebsd4', 'freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
-            # FreeBSD's P1003.1b semaphore support is very experimental
-            # and has many known problems. (as of June 2008)
-            macros = dict()
-            libraries = []
-
         elif host_platform.startswith('openbsd'):
             macros = dict()
             libraries = []


From webhook-mailer at python.org  Mon Jan 22 13:07:37 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 22 Jan 2018 18:07:37 -0000
Subject: [Python-checkins] bpo-29240: PyUnicode_DecodeLocale() uses UTF-8 on
 Android (#5272)
Message-ID: <mailman.7.1516644457.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/9089a265918754d95e105a7c4c409ac9352c87bb
commit: 9089a265918754d95e105a7c4c409ac9352c87bb
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-22T19:07:32+01:00
summary:

bpo-29240: PyUnicode_DecodeLocale() uses UTF-8 on Android (#5272)

PyUnicode_DecodeLocaleAndSize(), PyUnicode_DecodeLocale() and
PyUnicode_EncodeLocale() now use always use the UTF-8 encoding on
Android, instead of the current locale encoding.

On Android API 19, mbstowcs() and wcstombs() are broken and cannot be
used.

files:
M Doc/c-api/unicode.rst
M Python/fileutils.c

diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 3f6c0559907..92e22b16a4e 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -760,7 +760,8 @@ system.
                                                         Py_ssize_t len, \
                                                         const char *errors)
 
-   Decode a string from the current locale encoding. The supported
+   Decode a string from UTF-8 on Android, or from the current locale encoding
+   on other platforms. The supported
    error handlers are ``"strict"`` and ``"surrogateescape"``
    (:pep:`383`). The decoder uses ``"strict"`` error handler if
    *errors* is ``NULL``.  *str* must end with a null character but
@@ -780,7 +781,7 @@ system.
 
    .. versionchanged:: 3.7
       The function now also uses the current locale encoding for the
-      ``surrogateescape`` error handler. Previously, :c:func:`Py_DecodeLocale`
+      ``surrogateescape`` error handler, except on Android. Previously, :c:func:`Py_DecodeLocale`
       was used for the ``surrogateescape``, and the current locale encoding was
       used for ``strict``.
 
@@ -795,7 +796,8 @@ system.
 
 .. c:function:: PyObject* PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
 
-   Encode a Unicode object to the current locale encoding. The
+   Encode a Unicode object to UTF-8 on Android, or to the current locale
+   encoding on other platforms. The
    supported error handlers are ``"strict"`` and ``"surrogateescape"``
    (:pep:`383`). The encoder uses ``"strict"`` error handler if
    *errors* is ``NULL``. Return a :class:`bytes` object. *unicode* cannot
@@ -815,7 +817,8 @@ system.
 
    .. versionchanged:: 3.7
       The function now also uses the current locale encoding for the
-      ``surrogateescape`` error handler. Previously, :c:func:`Py_EncodeLocale`
+      ``surrogateescape`` error handler, except on Android. Previously,
+      :c:func:`Py_EncodeLocale`
       was used for the ``surrogateescape``, and the current locale encoding was
       used for ``strict``.
 
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 9a1435cfb32..d610639688e 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -449,7 +449,12 @@ _Py_DecodeLocaleEx(const char* arg, wchar_t **wstr, size_t *wlen,
                    int current_locale, int surrogateescape)
 {
     if (current_locale) {
+#ifdef __ANDROID__
+        return _Py_DecodeUTF8Ex(arg, strlen(arg), wstr, wlen, reason,
+                                surrogateescape);
+#else
         return decode_current_locale(arg, wstr, wlen, reason, surrogateescape);
+#endif
     }
 
 #if defined(__APPLE__) || defined(__ANDROID__)
@@ -605,8 +610,13 @@ encode_locale_ex(const wchar_t *text, char **str, size_t *error_pos,
                  int raw_malloc, int current_locale, int surrogateescape)
 {
     if (current_locale) {
+#ifdef __ANDROID__
+        return _Py_EncodeUTF8Ex(text, str, error_pos, reason,
+                                raw_malloc, surrogateescape);
+#else
         return encode_current_locale(text, str, error_pos, reason,
                                      raw_malloc, surrogateescape);
+#endif
     }
 
 #if defined(__APPLE__) || defined(__ANDROID__)


From webhook-mailer at python.org  Mon Jan 22 19:11:22 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Tue, 23 Jan 2018 00:11:22 -0000
Subject: [Python-checkins] bpo-32436: Implement PEP 567 (#5027)
Message-ID: <mailman.8.1516666283.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/f23746a934177c48eff754411aba54c31d6be2f0
commit: f23746a934177c48eff754411aba54c31d6be2f0
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-22T19:11:18-05:00
summary:

bpo-32436: Implement PEP 567 (#5027)

files:
A Include/context.h
A Include/internal/context.h
A Include/internal/hamt.h
A Lib/contextvars.py
A Lib/test/test_context.py
A Misc/NEWS.d/next/Core and Builtins/2017-12-28-00-20-42.bpo-32436.H159Jv.rst
A Modules/_contextvarsmodule.c
A Modules/clinic/_contextvarsmodule.c.h
A PCbuild/_contextvars.vcxproj
A PCbuild/_contextvars.vcxproj.filters
A Python/clinic/context.c.h
A Python/context.c
A Python/hamt.c
M Include/Python.h
M Include/pystate.h
M Lib/asyncio/base_events.py
M Lib/asyncio/base_futures.py
M Lib/asyncio/events.py
M Lib/asyncio/futures.py
M Lib/asyncio/selector_events.py
M Lib/asyncio/tasks.py
M Lib/asyncio/unix_events.py
M Lib/test/test_asyncio/test_base_events.py
M Lib/test/test_asyncio/test_futures.py
M Lib/test/test_asyncio/test_tasks.py
M Lib/test/test_asyncio/utils.py
M Makefile.pre.in
M Modules/Setup.dist
M Modules/_asynciomodule.c
M Modules/_testcapimodule.c
M Modules/clinic/_asynciomodule.c.h
M Modules/gcmodule.c
M Objects/object.c
M PCbuild/_decimal.vcxproj
M PCbuild/pcbuild.proj
M PCbuild/pythoncore.vcxproj
M PCbuild/pythoncore.vcxproj.filters
M Python/pylifecycle.c
M Python/pystate.c
M Tools/msi/lib/lib_files.wxs
M setup.py

diff --git a/Include/Python.h b/Include/Python.h
index 20051e7ff08..dd595ea5e4c 100644
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -109,6 +109,7 @@
 #include "pyerrors.h"
 
 #include "pystate.h"
+#include "context.h"
 
 #include "pyarena.h"
 #include "modsupport.h"
diff --git a/Include/context.h b/Include/context.h
new file mode 100644
index 00000000000..f872dceee0c
--- /dev/null
+++ b/Include/context.h
@@ -0,0 +1,86 @@
+#ifndef Py_CONTEXT_H
+#define Py_CONTEXT_H
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef Py_LIMITED_API
+
+
+PyAPI_DATA(PyTypeObject) PyContext_Type;
+typedef struct _pycontextobject PyContext;
+
+PyAPI_DATA(PyTypeObject) PyContextVar_Type;
+typedef struct _pycontextvarobject PyContextVar;
+
+PyAPI_DATA(PyTypeObject) PyContextToken_Type;
+typedef struct _pycontexttokenobject PyContextToken;
+
+
+#define PyContext_CheckExact(o) (Py_TYPE(o) == &PyContext_Type)
+#define PyContextVar_CheckExact(o) (Py_TYPE(o) == &PyContextVar_Type)
+#define PyContextToken_CheckExact(o) (Py_TYPE(o) == &PyContextToken_Type)
+
+
+PyAPI_FUNC(PyContext *) PyContext_New(void);
+PyAPI_FUNC(PyContext *) PyContext_Copy(PyContext *);
+PyAPI_FUNC(PyContext *) PyContext_CopyCurrent(void);
+
+PyAPI_FUNC(int) PyContext_Enter(PyContext *);
+PyAPI_FUNC(int) PyContext_Exit(PyContext *);
+
+
+/* Create a new context variable.
+
+   default_value can be NULL.
+*/
+PyAPI_FUNC(PyContextVar *) PyContextVar_New(
+    const char *name, PyObject *default_value);
+
+
+/* Get a value for the variable.
+
+   Returns -1 if an error occurred during lookup.
+
+   Returns 0 if value either was or was not found.
+
+   If value was found, *value will point to it.
+   If not, it will point to:
+
+   - default_value, if not NULL;
+   - the default value of "var", if not NULL;
+   - NULL.
+
+   '*value' will be a new ref, if not NULL.
+*/
+PyAPI_FUNC(int) PyContextVar_Get(
+    PyContextVar *var, PyObject *default_value, PyObject **value);
+
+
+/* Set a new value for the variable.
+   Returns NULL if an error occurs.
+*/
+PyAPI_FUNC(PyContextToken *) PyContextVar_Set(
+    PyContextVar *var, PyObject *value);
+
+
+/* Reset a variable to its previous value.
+   Returns 0 on sucess, -1 on error.
+*/
+PyAPI_FUNC(int) PyContextVar_Reset(
+    PyContextVar *var, PyContextToken *token);
+
+
+/* This method is exposed only for CPython tests. Don not use it. */
+PyAPI_FUNC(PyObject *) _PyContext_NewHamtForTests(void);
+
+
+PyAPI_FUNC(int) PyContext_ClearFreeList(void);
+
+
+#endif /* !Py_LIMITED_API */
+
+#ifdef __cplusplus
+}
+#endif
+#endif /* !Py_CONTEXT_H */
diff --git a/Include/internal/context.h b/Include/internal/context.h
new file mode 100644
index 00000000000..59f88f2614e
--- /dev/null
+++ b/Include/internal/context.h
@@ -0,0 +1,41 @@
+#ifndef Py_INTERNAL_CONTEXT_H
+#define Py_INTERNAL_CONTEXT_H
+
+
+#include "internal/hamt.h"
+
+
+struct _pycontextobject {
+    PyObject_HEAD
+    PyContext *ctx_prev;
+    PyHamtObject *ctx_vars;
+    PyObject *ctx_weakreflist;
+    int ctx_entered;
+};
+
+
+struct _pycontextvarobject {
+    PyObject_HEAD
+    PyObject *var_name;
+    PyObject *var_default;
+    PyObject *var_cached;
+    uint64_t var_cached_tsid;
+    uint64_t var_cached_tsver;
+    Py_hash_t var_hash;
+};
+
+
+struct _pycontexttokenobject {
+    PyObject_HEAD
+    PyContext *tok_ctx;
+    PyContextVar *tok_var;
+    PyObject *tok_oldval;
+    int tok_used;
+};
+
+
+int _PyContext_Init(void);
+void _PyContext_Fini(void);
+
+
+#endif /* !Py_INTERNAL_CONTEXT_H */
diff --git a/Include/internal/hamt.h b/Include/internal/hamt.h
new file mode 100644
index 00000000000..52488d0858d
--- /dev/null
+++ b/Include/internal/hamt.h
@@ -0,0 +1,113 @@
+#ifndef Py_INTERNAL_HAMT_H
+#define Py_INTERNAL_HAMT_H
+
+
+#define _Py_HAMT_MAX_TREE_DEPTH 7
+
+
+#define PyHamt_Check(o) (Py_TYPE(o) == &_PyHamt_Type)
+
+
+/* Abstract tree node. */
+typedef struct {
+    PyObject_HEAD
+} PyHamtNode;
+
+
+/* An HAMT immutable mapping collection. */
+typedef struct {
+    PyObject_HEAD
+    PyHamtNode *h_root;
+    PyObject *h_weakreflist;
+    Py_ssize_t h_count;
+} PyHamtObject;
+
+
+/* A struct to hold the state of depth-first traverse of the tree.
+
+   HAMT is an immutable collection.  Iterators will hold a strong reference
+   to it, and every node in the HAMT has strong references to its children.
+
+   So for iterators, we can implement zero allocations and zero reference
+   inc/dec depth-first iteration.
+
+   - i_nodes: an array of seven pointers to tree nodes
+   - i_level: the current node in i_nodes
+   - i_pos: an array of positions within nodes in i_nodes.
+*/
+typedef struct {
+    PyHamtNode *i_nodes[_Py_HAMT_MAX_TREE_DEPTH];
+    Py_ssize_t i_pos[_Py_HAMT_MAX_TREE_DEPTH];
+    int8_t i_level;
+} PyHamtIteratorState;
+
+
+/* Base iterator object.
+
+   Contains the iteration state, a pointer to the HAMT tree,
+   and a pointer to the 'yield function'.  The latter is a simple
+   function that returns a key/value tuple for the 'Items' iterator,
+   just a key for the 'Keys' iterator, and a value for the 'Values'
+   iterator.
+*/
+typedef struct {
+    PyObject_HEAD
+    PyHamtObject *hi_obj;
+    PyHamtIteratorState hi_iter;
+    binaryfunc hi_yield;
+} PyHamtIterator;
+
+
+PyAPI_DATA(PyTypeObject) _PyHamt_Type;
+PyAPI_DATA(PyTypeObject) _PyHamt_ArrayNode_Type;
+PyAPI_DATA(PyTypeObject) _PyHamt_BitmapNode_Type;
+PyAPI_DATA(PyTypeObject) _PyHamt_CollisionNode_Type;
+PyAPI_DATA(PyTypeObject) _PyHamtKeys_Type;
+PyAPI_DATA(PyTypeObject) _PyHamtValues_Type;
+PyAPI_DATA(PyTypeObject) _PyHamtItems_Type;
+
+
+/* Create a new HAMT immutable mapping. */
+PyHamtObject * _PyHamt_New(void);
+
+/* Return a new collection based on "o", but with an additional
+   key/val pair. */
+PyHamtObject * _PyHamt_Assoc(PyHamtObject *o, PyObject *key, PyObject *val);
+
+/* Return a new collection based on "o", but without "key". */
+PyHamtObject * _PyHamt_Without(PyHamtObject *o, PyObject *key);
+
+/* Find "key" in the "o" collection.
+
+   Return:
+   - -1: An error ocurred.
+   - 0: "key" wasn't found in "o".
+   - 1: "key" is in "o"; "*val" is set to its value (a borrowed ref).
+*/
+int _PyHamt_Find(PyHamtObject *o, PyObject *key, PyObject **val);
+
+/* Check if "v" is equal to "w".
+
+   Return:
+   - 0: v != w
+   - 1: v == w
+   - -1: An error occurred.
+*/
+int _PyHamt_Eq(PyHamtObject *v, PyHamtObject *w);
+
+/* Return the size of "o"; equivalent of "len(o)". */
+Py_ssize_t _PyHamt_Len(PyHamtObject *o);
+
+/* Return a Keys iterator over "o". */
+PyObject * _PyHamt_NewIterKeys(PyHamtObject *o);
+
+/* Return a Values iterator over "o". */
+PyObject * _PyHamt_NewIterValues(PyHamtObject *o);
+
+/* Return a Items iterator over "o". */
+PyObject * _PyHamt_NewIterItems(PyHamtObject *o);
+
+int _PyHamt_Init(void);
+void _PyHamt_Fini(void);
+
+#endif /* !Py_INTERNAL_HAMT_H */
diff --git a/Include/pystate.h b/Include/pystate.h
index 5a69e1471a0..d004be5e906 100644
--- a/Include/pystate.h
+++ b/Include/pystate.h
@@ -143,6 +143,8 @@ typedef struct _is {
     /* AtExit module */
     void (*pyexitfunc)(PyObject *);
     PyObject *pyexitmodule;
+
+    uint64_t tstate_next_unique_id;
 } PyInterpreterState;
 #endif   /* !Py_LIMITED_API */
 
@@ -270,6 +272,12 @@ typedef struct _ts {
     PyObject *async_gen_firstiter;
     PyObject *async_gen_finalizer;
 
+    PyObject *context;
+    uint64_t context_ver;
+
+    /* Unique thread state id. */
+    uint64_t id;
+
     /* XXX signal handlers should also be here */
 
 } PyThreadState;
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index ca9eee765e3..e722cf26b51 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -489,7 +489,7 @@ def time(self):
         """
         return time.monotonic()
 
-    def call_later(self, delay, callback, *args):
+    def call_later(self, delay, callback, *args, context=None):
         """Arrange for a callback to be called at a given time.
 
         Return a Handle: an opaque object with a cancel() method that
@@ -505,12 +505,13 @@ def call_later(self, delay, callback, *args):
         Any positional arguments after the callback will be passed to
         the callback when it is called.
         """
-        timer = self.call_at(self.time() + delay, callback, *args)
+        timer = self.call_at(self.time() + delay, callback, *args,
+                             context=context)
         if timer._source_traceback:
             del timer._source_traceback[-1]
         return timer
 
-    def call_at(self, when, callback, *args):
+    def call_at(self, when, callback, *args, context=None):
         """Like call_later(), but uses an absolute time.
 
         Absolute time corresponds to the event loop's time() method.
@@ -519,14 +520,14 @@ def call_at(self, when, callback, *args):
         if self._debug:
             self._check_thread()
             self._check_callback(callback, 'call_at')
-        timer = events.TimerHandle(when, callback, args, self)
+        timer = events.TimerHandle(when, callback, args, self, context)
         if timer._source_traceback:
             del timer._source_traceback[-1]
         heapq.heappush(self._scheduled, timer)
         timer._scheduled = True
         return timer
 
-    def call_soon(self, callback, *args):
+    def call_soon(self, callback, *args, context=None):
         """Arrange for a callback to be called as soon as possible.
 
         This operates as a FIFO queue: callbacks are called in the
@@ -540,7 +541,7 @@ def call_soon(self, callback, *args):
         if self._debug:
             self._check_thread()
             self._check_callback(callback, 'call_soon')
-        handle = self._call_soon(callback, args)
+        handle = self._call_soon(callback, args, context)
         if handle._source_traceback:
             del handle._source_traceback[-1]
         return handle
@@ -555,8 +556,8 @@ def _check_callback(self, callback, method):
                 f'a callable object was expected by {method}(), '
                 f'got {callback!r}')
 
-    def _call_soon(self, callback, args):
-        handle = events.Handle(callback, args, self)
+    def _call_soon(self, callback, args, context):
+        handle = events.Handle(callback, args, self, context)
         if handle._source_traceback:
             del handle._source_traceback[-1]
         self._ready.append(handle)
@@ -579,12 +580,12 @@ def _check_thread(self):
                 "Non-thread-safe operation invoked on an event loop other "
                 "than the current one")
 
-    def call_soon_threadsafe(self, callback, *args):
+    def call_soon_threadsafe(self, callback, *args, context=None):
         """Like call_soon(), but thread-safe."""
         self._check_closed()
         if self._debug:
             self._check_callback(callback, 'call_soon_threadsafe')
-        handle = self._call_soon(callback, args)
+        handle = self._call_soon(callback, args, context)
         if handle._source_traceback:
             del handle._source_traceback[-1]
         self._write_to_self()
diff --git a/Lib/asyncio/base_futures.py b/Lib/asyncio/base_futures.py
index 008812eda91..5182884e16d 100644
--- a/Lib/asyncio/base_futures.py
+++ b/Lib/asyncio/base_futures.py
@@ -41,13 +41,13 @@ def format_cb(callback):
         return format_helpers._format_callback_source(callback, ())
 
     if size == 1:
-        cb = format_cb(cb[0])
+        cb = format_cb(cb[0][0])
     elif size == 2:
-        cb = '{}, {}'.format(format_cb(cb[0]), format_cb(cb[1]))
+        cb = '{}, {}'.format(format_cb(cb[0][0]), format_cb(cb[1][0]))
     elif size > 2:
-        cb = '{}, <{} more>, {}'.format(format_cb(cb[0]),
+        cb = '{}, <{} more>, {}'.format(format_cb(cb[0][0]),
                                         size - 2,
-                                        format_cb(cb[-1]))
+                                        format_cb(cb[-1][0]))
     return f'cb=[{cb}]'
 
 
diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index d5365dc480d..5c68d4cb97d 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -11,6 +11,7 @@
     '_get_running_loop',
 )
 
+import contextvars
 import os
 import socket
 import subprocess
@@ -32,9 +33,13 @@ class Handle:
     """Object returned by callback registration methods."""
 
     __slots__ = ('_callback', '_args', '_cancelled', '_loop',
-                 '_source_traceback', '_repr', '__weakref__')
+                 '_source_traceback', '_repr', '__weakref__',
+                 '_context')
 
-    def __init__(self, callback, args, loop):
+    def __init__(self, callback, args, loop, context=None):
+        if context is None:
+            context = contextvars.copy_context()
+        self._context = context
         self._loop = loop
         self._callback = callback
         self._args = args
@@ -80,7 +85,7 @@ def cancelled(self):
 
     def _run(self):
         try:
-            self._callback(*self._args)
+            self._context.run(self._callback, *self._args)
         except Exception as exc:
             cb = format_helpers._format_callback_source(
                 self._callback, self._args)
@@ -101,9 +106,9 @@ class TimerHandle(Handle):
 
     __slots__ = ['_scheduled', '_when']
 
-    def __init__(self, when, callback, args, loop):
+    def __init__(self, when, callback, args, loop, context=None):
         assert when is not None
-        super().__init__(callback, args, loop)
+        super().__init__(callback, args, loop, context)
         if self._source_traceback:
             del self._source_traceback[-1]
         self._when = when
diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py
index 1c05b2231c1..59621ffb6e7 100644
--- a/Lib/asyncio/futures.py
+++ b/Lib/asyncio/futures.py
@@ -6,6 +6,7 @@
 )
 
 import concurrent.futures
+import contextvars
 import logging
 import sys
 
@@ -144,8 +145,8 @@ def _schedule_callbacks(self):
             return
 
         self._callbacks[:] = []
-        for callback in callbacks:
-            self._loop.call_soon(callback, self)
+        for callback, ctx in callbacks:
+            self._loop.call_soon(callback, self, context=ctx)
 
     def cancelled(self):
         """Return True if the future was cancelled."""
@@ -192,7 +193,7 @@ def exception(self):
         self.__log_traceback = False
         return self._exception
 
-    def add_done_callback(self, fn):
+    def add_done_callback(self, fn, *, context=None):
         """Add a callback to be run when the future becomes done.
 
         The callback is called with a single argument - the future object. If
@@ -200,9 +201,11 @@ def add_done_callback(self, fn):
         scheduled with call_soon.
         """
         if self._state != _PENDING:
-            self._loop.call_soon(fn, self)
+            self._loop.call_soon(fn, self, context=context)
         else:
-            self._callbacks.append(fn)
+            if context is None:
+                context = contextvars.copy_context()
+            self._callbacks.append((fn, context))
 
     # New method not in PEP 3148.
 
@@ -211,7 +214,9 @@ def remove_done_callback(self, fn):
 
         Returns the number of callbacks removed.
         """
-        filtered_callbacks = [f for f in self._callbacks if f != fn]
+        filtered_callbacks = [(f, ctx)
+                              for (f, ctx) in self._callbacks
+                              if f != fn]
         removed_count = len(self._callbacks) - len(filtered_callbacks)
         if removed_count:
             self._callbacks[:] = filtered_callbacks
diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
index 5692e38486a..9446ae6a3bc 100644
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -256,7 +256,7 @@ def _ensure_fd_no_transport(self, fd):
 
     def _add_reader(self, fd, callback, *args):
         self._check_closed()
-        handle = events.Handle(callback, args, self)
+        handle = events.Handle(callback, args, self, None)
         try:
             key = self._selector.get_key(fd)
         except KeyError:
@@ -292,7 +292,7 @@ def _remove_reader(self, fd):
 
     def _add_writer(self, fd, callback, *args):
         self._check_closed()
-        handle = events.Handle(callback, args, self)
+        handle = events.Handle(callback, args, self, None)
         try:
             key = self._selector.get_key(fd)
         except KeyError:
diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py
index b11808853e2..609b8e8a048 100644
--- a/Lib/asyncio/tasks.py
+++ b/Lib/asyncio/tasks.py
@@ -10,6 +10,7 @@
 )
 
 import concurrent.futures
+import contextvars
 import functools
 import inspect
 import types
@@ -96,8 +97,9 @@ def __init__(self, coro, *, loop=None):
         self._must_cancel = False
         self._fut_waiter = None
         self._coro = coro
+        self._context = contextvars.copy_context()
 
-        self._loop.call_soon(self._step)
+        self._loop.call_soon(self._step, context=self._context)
         _register_task(self)
 
     def __del__(self):
@@ -229,15 +231,18 @@ def _step(self, exc=None):
                     new_exc = RuntimeError(
                         f'Task {self!r} got Future '
                         f'{result!r} attached to a different loop')
-                    self._loop.call_soon(self._step, new_exc)
+                    self._loop.call_soon(
+                        self._step, new_exc, context=self._context)
                 elif blocking:
                     if result is self:
                         new_exc = RuntimeError(
                             f'Task cannot await on itself: {self!r}')
-                        self._loop.call_soon(self._step, new_exc)
+                        self._loop.call_soon(
+                            self._step, new_exc, context=self._context)
                     else:
                         result._asyncio_future_blocking = False
-                        result.add_done_callback(self._wakeup)
+                        result.add_done_callback(
+                            self._wakeup, context=self._context)
                         self._fut_waiter = result
                         if self._must_cancel:
                             if self._fut_waiter.cancel():
@@ -246,21 +251,24 @@ def _step(self, exc=None):
                     new_exc = RuntimeError(
                         f'yield was used instead of yield from '
                         f'in task {self!r} with {result!r}')
-                    self._loop.call_soon(self._step, new_exc)
+                    self._loop.call_soon(
+                        self._step, new_exc, context=self._context)
 
             elif result is None:
                 # Bare yield relinquishes control for one event loop iteration.
-                self._loop.call_soon(self._step)
+                self._loop.call_soon(self._step, context=self._context)
             elif inspect.isgenerator(result):
                 # Yielding a generator is just wrong.
                 new_exc = RuntimeError(
                     f'yield was used instead of yield from for '
                     f'generator in task {self!r} with {result}')
-                self._loop.call_soon(self._step, new_exc)
+                self._loop.call_soon(
+                    self._step, new_exc, context=self._context)
             else:
                 # Yielding something else is an error.
                 new_exc = RuntimeError(f'Task got bad yield: {result!r}')
-                self._loop.call_soon(self._step, new_exc)
+                self._loop.call_soon(
+                    self._step, new_exc, context=self._context)
         finally:
             _leave_task(self._loop, self)
             self = None  # Needed to break cycles when an exception occurs.
diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
index 028a0ca8f83..9b9d0043b50 100644
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -92,7 +92,7 @@ def add_signal_handler(self, sig, callback, *args):
         except (ValueError, OSError) as exc:
             raise RuntimeError(str(exc))
 
-        handle = events.Handle(callback, args, self)
+        handle = events.Handle(callback, args, self, None)
         self._signal_handlers[sig] = handle
 
         try:
diff --git a/Lib/contextvars.py b/Lib/contextvars.py
new file mode 100644
index 00000000000..d78c80dfe6f
--- /dev/null
+++ b/Lib/contextvars.py
@@ -0,0 +1,4 @@
+from _contextvars import Context, ContextVar, Token, copy_context
+
+
+__all__ = ('Context', 'ContextVar', 'Token', 'copy_context')
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index fc3b81096da..8d72df6a72e 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -192,14 +192,14 @@ def test_close(self):
         self.assertRaises(RuntimeError, self.loop.run_until_complete, f)
 
     def test__add_callback_handle(self):
-        h = asyncio.Handle(lambda: False, (), self.loop)
+        h = asyncio.Handle(lambda: False, (), self.loop, None)
 
         self.loop._add_callback(h)
         self.assertFalse(self.loop._scheduled)
         self.assertIn(h, self.loop._ready)
 
     def test__add_callback_cancelled_handle(self):
-        h = asyncio.Handle(lambda: False, (), self.loop)
+        h = asyncio.Handle(lambda: False, (), self.loop, None)
         h.cancel()
 
         self.loop._add_callback(h)
@@ -333,9 +333,9 @@ def test_thread(loop, debug, create_loop=False):
 
     def test__run_once(self):
         h1 = asyncio.TimerHandle(time.monotonic() + 5.0, lambda: True, (),
-                                 self.loop)
+                                 self.loop, None)
         h2 = asyncio.TimerHandle(time.monotonic() + 10.0, lambda: True, (),
-                                 self.loop)
+                                 self.loop, None)
 
         h1.cancel()
 
@@ -390,7 +390,7 @@ def cb(loop):
             handle = loop.call_soon(lambda: True)
 
         h = asyncio.TimerHandle(time.monotonic() - 1, cb, (self.loop,),
-                                self.loop)
+                                self.loop, None)
 
         self.loop._process_events = mock.Mock()
         self.loop._scheduled.append(h)
diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py
index ab45ee39ab9..37f4c6562fe 100644
--- a/Lib/test/test_asyncio/test_futures.py
+++ b/Lib/test/test_asyncio/test_futures.py
@@ -565,16 +565,22 @@ def __del__(self):
 @unittest.skipUnless(hasattr(futures, '_CFuture'),
                      'requires the C _asyncio module')
 class CFutureTests(BaseFutureTests, test_utils.TestCase):
-    cls = futures._CFuture
+    try:
+        cls = futures._CFuture
+    except AttributeError:
+        cls = None
 
 
 @unittest.skipUnless(hasattr(futures, '_CFuture'),
                      'requires the C _asyncio module')
 class CSubFutureTests(BaseFutureTests, test_utils.TestCase):
-    class CSubFuture(futures._CFuture):
-        pass
+    try:
+        class CSubFuture(futures._CFuture):
+            pass
 
-    cls = CSubFuture
+        cls = CSubFuture
+    except AttributeError:
+        cls = None
 
 
 class PyFutureTests(BaseFutureTests, test_utils.TestCase):
diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index 26e4f643d1a..96d2658cb4c 100644
--- a/Lib/test/test_asyncio/test_tasks.py
+++ b/Lib/test/test_asyncio/test_tasks.py
@@ -2,10 +2,11 @@
 
 import collections
 import contextlib
+import contextvars
 import functools
 import gc
 import io
-import os
+import random
 import re
 import sys
 import types
@@ -1377,9 +1378,9 @@ def __init__(self, *args, **kwds):
                 self.cb_added = False
                 super().__init__(*args, **kwds)
 
-            def add_done_callback(self, fn):
+            def add_done_callback(self, *args, **kwargs):
                 self.cb_added = True
-                super().add_done_callback(fn)
+                super().add_done_callback(*args, **kwargs)
 
         fut = Fut(loop=self.loop)
         result = None
@@ -2091,7 +2092,7 @@ def main():
 
     @mock.patch('asyncio.base_events.logger')
     def test_error_in_call_soon(self, m_log):
-        def call_soon(callback, *args):
+        def call_soon(callback, *args, **kwargs):
             raise ValueError
         self.loop.call_soon = call_soon
 
@@ -2176,6 +2177,91 @@ def test_bare_create_task(self):
 
         self.loop.run_until_complete(coro())
 
+    def test_context_1(self):
+        cvar = contextvars.ContextVar('cvar', default='nope')
+
+        async def sub():
+            await asyncio.sleep(0.01, loop=loop)
+            self.assertEqual(cvar.get(), 'nope')
+            cvar.set('something else')
+
+        async def main():
+            self.assertEqual(cvar.get(), 'nope')
+            subtask = self.new_task(loop, sub())
+            cvar.set('yes')
+            self.assertEqual(cvar.get(), 'yes')
+            await subtask
+            self.assertEqual(cvar.get(), 'yes')
+
+        loop = asyncio.new_event_loop()
+        try:
+            task = self.new_task(loop, main())
+            loop.run_until_complete(task)
+        finally:
+            loop.close()
+
+    def test_context_2(self):
+        cvar = contextvars.ContextVar('cvar', default='nope')
+
+        async def main():
+            def fut_on_done(fut):
+                # This change must not pollute the context
+                # of the "main()" task.
+                cvar.set('something else')
+
+            self.assertEqual(cvar.get(), 'nope')
+
+            for j in range(2):
+                fut = self.new_future(loop)
+                fut.add_done_callback(fut_on_done)
+                cvar.set(f'yes{j}')
+                loop.call_soon(fut.set_result, None)
+                await fut
+                self.assertEqual(cvar.get(), f'yes{j}')
+
+                for i in range(3):
+                    # Test that task passed its context to add_done_callback:
+                    cvar.set(f'yes{i}-{j}')
+                    await asyncio.sleep(0.001, loop=loop)
+                    self.assertEqual(cvar.get(), f'yes{i}-{j}')
+
+        loop = asyncio.new_event_loop()
+        try:
+            task = self.new_task(loop, main())
+            loop.run_until_complete(task)
+        finally:
+            loop.close()
+
+        self.assertEqual(cvar.get(), 'nope')
+
+    def test_context_3(self):
+        # Run 100 Tasks in parallel, each modifying cvar.
+
+        cvar = contextvars.ContextVar('cvar', default=-1)
+
+        async def sub(num):
+            for i in range(10):
+                cvar.set(num + i)
+                await asyncio.sleep(
+                    random.uniform(0.001, 0.05), loop=loop)
+                self.assertEqual(cvar.get(), num + i)
+
+        async def main():
+            tasks = []
+            for i in range(100):
+                task = loop.create_task(sub(random.randint(0, 10)))
+                tasks.append(task)
+
+            await asyncio.gather(*tasks, loop=loop)
+
+        loop = asyncio.new_event_loop()
+        try:
+            loop.run_until_complete(main())
+        finally:
+            loop.close()
+
+        self.assertEqual(cvar.get(), -1)
+
 
 def add_subclass_tests(cls):
     BaseTask = cls.Task
@@ -2193,9 +2279,9 @@ def _schedule_callbacks(self):
             self.calls['_schedule_callbacks'] += 1
             return super()._schedule_callbacks()
 
-        def add_done_callback(self, *args):
+        def add_done_callback(self, *args, **kwargs):
             self.calls['add_done_callback'] += 1
-            return super().add_done_callback(*args)
+            return super().add_done_callback(*args, **kwargs)
 
     class Task(CommonFuture, BaseTask):
         def _step(self, *args):
@@ -2486,10 +2572,13 @@ class PyIntrospectionTests(unittest.TestCase, BaseTaskIntrospectionTests):
 @unittest.skipUnless(hasattr(tasks, '_c_register_task'),
                      'requires the C _asyncio module')
 class CIntrospectionTests(unittest.TestCase, BaseTaskIntrospectionTests):
-    _register_task = staticmethod(tasks._c_register_task)
-    _unregister_task = staticmethod(tasks._c_unregister_task)
-    _enter_task = staticmethod(tasks._c_enter_task)
-    _leave_task = staticmethod(tasks._c_leave_task)
+    if hasattr(tasks, '_c_register_task'):
+        _register_task = staticmethod(tasks._c_register_task)
+        _unregister_task = staticmethod(tasks._c_unregister_task)
+        _enter_task = staticmethod(tasks._c_enter_task)
+        _leave_task = staticmethod(tasks._c_leave_task)
+    else:
+        _register_task = _unregister_task = _enter_task = _leave_task = None
 
 
 class BaseCurrentLoopTests:
diff --git a/Lib/test/test_asyncio/utils.py b/Lib/test/test_asyncio/utils.py
index f756ec9016f..96dfe2f85b4 100644
--- a/Lib/test/test_asyncio/utils.py
+++ b/Lib/test/test_asyncio/utils.py
@@ -365,7 +365,7 @@ def close(self):
                 raise AssertionError("Time generator is not finished")
 
     def _add_reader(self, fd, callback, *args):
-        self.readers[fd] = events.Handle(callback, args, self)
+        self.readers[fd] = events.Handle(callback, args, self, None)
 
     def _remove_reader(self, fd):
         self.remove_reader_count[fd] += 1
@@ -391,7 +391,7 @@ def assert_no_reader(self, fd):
             raise AssertionError(f'fd {fd} is registered')
 
     def _add_writer(self, fd, callback, *args):
-        self.writers[fd] = events.Handle(callback, args, self)
+        self.writers[fd] = events.Handle(callback, args, self, None)
 
     def _remove_writer(self, fd):
         self.remove_writer_count[fd] += 1
@@ -457,9 +457,9 @@ def _run_once(self):
             self.advance_time(advance)
         self._timers = []
 
-    def call_at(self, when, callback, *args):
+    def call_at(self, when, callback, *args, context=None):
         self._timers.append(when)
-        return super().call_at(when, callback, *args)
+        return super().call_at(when, callback, *args, context=context)
 
     def _process_events(self, event_list):
         return
diff --git a/Lib/test/test_context.py b/Lib/test/test_context.py
new file mode 100644
index 00000000000..74d05fc5c41
--- /dev/null
+++ b/Lib/test/test_context.py
@@ -0,0 +1,1064 @@
+import concurrent.futures
+import contextvars
+import functools
+import gc
+import random
+import time
+import unittest
+import weakref
+
+try:
+    from _testcapi import hamt
+except ImportError:
+    hamt = None
+
+
+def isolated_context(func):
+    """Needed to make reftracking test mode work."""
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        ctx = contextvars.Context()
+        return ctx.run(func, *args, **kwargs)
+    return wrapper
+
+
+class ContextTest(unittest.TestCase):
+    def test_context_var_new_1(self):
+        with self.assertRaisesRegex(TypeError, 'takes exactly 1'):
+            contextvars.ContextVar()
+
+        with self.assertRaisesRegex(TypeError, 'must be a str'):
+            contextvars.ContextVar(1)
+
+        c = contextvars.ContextVar('a')
+        self.assertNotEqual(hash(c), hash('a'))
+
+    def test_context_var_new_2(self):
+        self.assertIsNone(contextvars.ContextVar[int])
+
+    @isolated_context
+    def test_context_var_repr_1(self):
+        c = contextvars.ContextVar('a')
+        self.assertIn('a', repr(c))
+
+        c = contextvars.ContextVar('a', default=123)
+        self.assertIn('123', repr(c))
+
+        lst = []
+        c = contextvars.ContextVar('a', default=lst)
+        lst.append(c)
+        self.assertIn('...', repr(c))
+        self.assertIn('...', repr(lst))
+
+        t = c.set(1)
+        self.assertIn(repr(c), repr(t))
+        self.assertNotIn(' used ', repr(t))
+        c.reset(t)
+        self.assertIn(' used ', repr(t))
+
+    def test_context_subclassing_1(self):
+        with self.assertRaisesRegex(TypeError, 'not an acceptable base type'):
+            class MyContextVar(contextvars.ContextVar):
+                # Potentially we might want ContextVars to be subclassable.
+                pass
+
+        with self.assertRaisesRegex(TypeError, 'not an acceptable base type'):
+            class MyContext(contextvars.Context):
+                pass
+
+        with self.assertRaisesRegex(TypeError, 'not an acceptable base type'):
+            class MyToken(contextvars.Token):
+                pass
+
+    def test_context_new_1(self):
+        with self.assertRaisesRegex(TypeError, 'any arguments'):
+            contextvars.Context(1)
+        with self.assertRaisesRegex(TypeError, 'any arguments'):
+            contextvars.Context(1, a=1)
+        with self.assertRaisesRegex(TypeError, 'any arguments'):
+            contextvars.Context(a=1)
+        contextvars.Context(**{})
+
+    def test_context_typerrors_1(self):
+        ctx = contextvars.Context()
+
+        with self.assertRaisesRegex(TypeError, 'ContextVar key was expected'):
+            ctx[1]
+        with self.assertRaisesRegex(TypeError, 'ContextVar key was expected'):
+            1 in ctx
+        with self.assertRaisesRegex(TypeError, 'ContextVar key was expected'):
+            ctx.get(1)
+
+    def test_context_get_context_1(self):
+        ctx = contextvars.copy_context()
+        self.assertIsInstance(ctx, contextvars.Context)
+
+    def test_context_run_1(self):
+        ctx = contextvars.Context()
+
+        with self.assertRaisesRegex(TypeError, 'missing 1 required'):
+            ctx.run()
+
+    def test_context_run_2(self):
+        ctx = contextvars.Context()
+
+        def func(*args, **kwargs):
+            kwargs['spam'] = 'foo'
+            args += ('bar',)
+            return args, kwargs
+
+        for f in (func, functools.partial(func)):
+            # partial doesn't support FASTCALL
+
+            self.assertEqual(ctx.run(f), (('bar',), {'spam': 'foo'}))
+            self.assertEqual(ctx.run(f, 1), ((1, 'bar'), {'spam': 'foo'}))
+
+            self.assertEqual(
+                ctx.run(f, a=2),
+                (('bar',), {'a': 2, 'spam': 'foo'}))
+
+            self.assertEqual(
+                ctx.run(f, 11, a=2),
+                ((11, 'bar'), {'a': 2, 'spam': 'foo'}))
+
+            a = {}
+            self.assertEqual(
+                ctx.run(f, 11, **a),
+                ((11, 'bar'), {'spam': 'foo'}))
+            self.assertEqual(a, {})
+
+    def test_context_run_3(self):
+        ctx = contextvars.Context()
+
+        def func(*args, **kwargs):
+            1 / 0
+
+        with self.assertRaises(ZeroDivisionError):
+            ctx.run(func)
+        with self.assertRaises(ZeroDivisionError):
+            ctx.run(func, 1, 2)
+        with self.assertRaises(ZeroDivisionError):
+            ctx.run(func, 1, 2, a=123)
+
+    @isolated_context
+    def test_context_run_4(self):
+        ctx1 = contextvars.Context()
+        ctx2 = contextvars.Context()
+        var = contextvars.ContextVar('var')
+
+        def func2():
+            self.assertIsNone(var.get(None))
+
+        def func1():
+            self.assertIsNone(var.get(None))
+            var.set('spam')
+            ctx2.run(func2)
+            self.assertEqual(var.get(None), 'spam')
+
+            cur = contextvars.copy_context()
+            self.assertEqual(len(cur), 1)
+            self.assertEqual(cur[var], 'spam')
+            return cur
+
+        returned_ctx = ctx1.run(func1)
+        self.assertEqual(ctx1, returned_ctx)
+        self.assertEqual(returned_ctx[var], 'spam')
+        self.assertIn(var, returned_ctx)
+
+    def test_context_run_5(self):
+        ctx = contextvars.Context()
+        var = contextvars.ContextVar('var')
+
+        def func():
+            self.assertIsNone(var.get(None))
+            var.set('spam')
+            1 / 0
+
+        with self.assertRaises(ZeroDivisionError):
+            ctx.run(func)
+
+        self.assertIsNone(var.get(None))
+
+    def test_context_run_6(self):
+        ctx = contextvars.Context()
+        c = contextvars.ContextVar('a', default=0)
+
+        def fun():
+            self.assertEqual(c.get(), 0)
+            self.assertIsNone(ctx.get(c))
+
+            c.set(42)
+            self.assertEqual(c.get(), 42)
+            self.assertEqual(ctx.get(c), 42)
+
+        ctx.run(fun)
+
+    def test_context_run_7(self):
+        ctx = contextvars.Context()
+
+        def fun():
+            with self.assertRaisesRegex(RuntimeError, 'is already entered'):
+                ctx.run(fun)
+
+        ctx.run(fun)
+
+    @isolated_context
+    def test_context_getset_1(self):
+        c = contextvars.ContextVar('c')
+        with self.assertRaises(LookupError):
+            c.get()
+
+        self.assertIsNone(c.get(None))
+
+        t0 = c.set(42)
+        self.assertEqual(c.get(), 42)
+        self.assertEqual(c.get(None), 42)
+        self.assertIs(t0.old_value, t0.MISSING)
+        self.assertIs(t0.old_value, contextvars.Token.MISSING)
+        self.assertIs(t0.var, c)
+
+        t = c.set('spam')
+        self.assertEqual(c.get(), 'spam')
+        self.assertEqual(c.get(None), 'spam')
+        self.assertEqual(t.old_value, 42)
+        c.reset(t)
+
+        self.assertEqual(c.get(), 42)
+        self.assertEqual(c.get(None), 42)
+
+        c.set('spam2')
+        with self.assertRaisesRegex(RuntimeError, 'has already been used'):
+            c.reset(t)
+        self.assertEqual(c.get(), 'spam2')
+
+        ctx1 = contextvars.copy_context()
+        self.assertIn(c, ctx1)
+
+        c.reset(t0)
+        with self.assertRaisesRegex(RuntimeError, 'has already been used'):
+            c.reset(t0)
+        self.assertIsNone(c.get(None))
+
+        self.assertIn(c, ctx1)
+        self.assertEqual(ctx1[c], 'spam2')
+        self.assertEqual(ctx1.get(c, 'aa'), 'spam2')
+        self.assertEqual(len(ctx1), 1)
+        self.assertEqual(list(ctx1.items()), [(c, 'spam2')])
+        self.assertEqual(list(ctx1.values()), ['spam2'])
+        self.assertEqual(list(ctx1.keys()), [c])
+        self.assertEqual(list(ctx1), [c])
+
+        ctx2 = contextvars.copy_context()
+        self.assertNotIn(c, ctx2)
+        with self.assertRaises(KeyError):
+            ctx2[c]
+        self.assertEqual(ctx2.get(c, 'aa'), 'aa')
+        self.assertEqual(len(ctx2), 0)
+        self.assertEqual(list(ctx2), [])
+
+    @isolated_context
+    def test_context_getset_2(self):
+        v1 = contextvars.ContextVar('v1')
+        v2 = contextvars.ContextVar('v2')
+
+        t1 = v1.set(42)
+        with self.assertRaisesRegex(ValueError, 'by a different'):
+            v2.reset(t1)
+
+    @isolated_context
+    def test_context_getset_3(self):
+        c = contextvars.ContextVar('c', default=42)
+        ctx = contextvars.Context()
+
+        def fun():
+            self.assertEqual(c.get(), 42)
+            with self.assertRaises(KeyError):
+                ctx[c]
+            self.assertIsNone(ctx.get(c))
+            self.assertEqual(ctx.get(c, 'spam'), 'spam')
+            self.assertNotIn(c, ctx)
+            self.assertEqual(list(ctx.keys()), [])
+
+            t = c.set(1)
+            self.assertEqual(list(ctx.keys()), [c])
+            self.assertEqual(ctx[c], 1)
+
+            c.reset(t)
+            self.assertEqual(list(ctx.keys()), [])
+            with self.assertRaises(KeyError):
+                ctx[c]
+
+        ctx.run(fun)
+
+    @isolated_context
+    def test_context_getset_4(self):
+        c = contextvars.ContextVar('c', default=42)
+        ctx = contextvars.Context()
+
+        tok = ctx.run(c.set, 1)
+
+        with self.assertRaisesRegex(ValueError, 'different Context'):
+            c.reset(tok)
+
+    @isolated_context
+    def test_context_getset_5(self):
+        c = contextvars.ContextVar('c', default=42)
+        c.set([])
+
+        def fun():
+            c.set([])
+            c.get().append(42)
+            self.assertEqual(c.get(), [42])
+
+        contextvars.copy_context().run(fun)
+        self.assertEqual(c.get(), [])
+
+    def test_context_copy_1(self):
+        ctx1 = contextvars.Context()
+        c = contextvars.ContextVar('c', default=42)
+
+        def ctx1_fun():
+            c.set(10)
+
+            ctx2 = ctx1.copy()
+            self.assertEqual(ctx2[c], 10)
+
+            c.set(20)
+            self.assertEqual(ctx1[c], 20)
+            self.assertEqual(ctx2[c], 10)
+
+            ctx2.run(ctx2_fun)
+            self.assertEqual(ctx1[c], 20)
+            self.assertEqual(ctx2[c], 30)
+
+        def ctx2_fun():
+            self.assertEqual(c.get(), 10)
+            c.set(30)
+            self.assertEqual(c.get(), 30)
+
+        ctx1.run(ctx1_fun)
+
+    @isolated_context
+    def test_context_threads_1(self):
+        cvar = contextvars.ContextVar('cvar')
+
+        def sub(num):
+            for i in range(10):
+                cvar.set(num + i)
+                time.sleep(random.uniform(0.001, 0.05))
+                self.assertEqual(cvar.get(), num + i)
+            return num
+
+        tp = concurrent.futures.ThreadPoolExecutor(max_workers=10)
+        try:
+            results = list(tp.map(sub, range(10)))
+        finally:
+            tp.shutdown()
+        self.assertEqual(results, list(range(10)))
+
+
+# HAMT Tests
+
+
+class HashKey:
+    _crasher = None
+
+    def __init__(self, hash, name, *, error_on_eq_to=None):
+        assert hash != -1
+        self.name = name
+        self.hash = hash
+        self.error_on_eq_to = error_on_eq_to
+
+    def __repr__(self):
+        return f'<Key name:{self.name} hash:{self.hash}>'
+
+    def __hash__(self):
+        if self._crasher is not None and self._crasher.error_on_hash:
+            raise HashingError
+
+        return self.hash
+
+    def __eq__(self, other):
+        if not isinstance(other, HashKey):
+            return NotImplemented
+
+        if self._crasher is not None and self._crasher.error_on_eq:
+            raise EqError
+
+        if self.error_on_eq_to is not None and self.error_on_eq_to is other:
+            raise ValueError(f'cannot compare {self!r} to {other!r}')
+        if other.error_on_eq_to is not None and other.error_on_eq_to is self:
+            raise ValueError(f'cannot compare {other!r} to {self!r}')
+
+        return (self.name, self.hash) == (other.name, other.hash)
+
+
+class KeyStr(str):
+    def __hash__(self):
+        if HashKey._crasher is not None and HashKey._crasher.error_on_hash:
+            raise HashingError
+        return super().__hash__()
+
+    def __eq__(self, other):
+        if HashKey._crasher is not None and HashKey._crasher.error_on_eq:
+            raise EqError
+        return super().__eq__(other)
+
+
+class HaskKeyCrasher:
+    def __init__(self, *, error_on_hash=False, error_on_eq=False):
+        self.error_on_hash = error_on_hash
+        self.error_on_eq = error_on_eq
+
+    def __enter__(self):
+        if HashKey._crasher is not None:
+            raise RuntimeError('cannot nest crashers')
+        HashKey._crasher = self
+
+    def __exit__(self, *exc):
+        HashKey._crasher = None
+
+
+class HashingError(Exception):
+    pass
+
+
+class EqError(Exception):
+    pass
+
+
+ at unittest.skipIf(hamt is None, '_testcapi lacks "hamt()" function')
+class HamtTest(unittest.TestCase):
+
+    def test_hashkey_helper_1(self):
+        k1 = HashKey(10, 'aaa')
+        k2 = HashKey(10, 'bbb')
+
+        self.assertNotEqual(k1, k2)
+        self.assertEqual(hash(k1), hash(k2))
+
+        d = dict()
+        d[k1] = 'a'
+        d[k2] = 'b'
+
+        self.assertEqual(d[k1], 'a')
+        self.assertEqual(d[k2], 'b')
+
+    def test_hamt_basics_1(self):
+        h = hamt()
+        h = None  # NoQA
+
+    def test_hamt_basics_2(self):
+        h = hamt()
+        self.assertEqual(len(h), 0)
+
+        h2 = h.set('a', 'b')
+        self.assertIsNot(h, h2)
+        self.assertEqual(len(h), 0)
+        self.assertEqual(len(h2), 1)
+
+        self.assertIsNone(h.get('a'))
+        self.assertEqual(h.get('a', 42), 42)
+
+        self.assertEqual(h2.get('a'), 'b')
+
+        h3 = h2.set('b', 10)
+        self.assertIsNot(h2, h3)
+        self.assertEqual(len(h), 0)
+        self.assertEqual(len(h2), 1)
+        self.assertEqual(len(h3), 2)
+        self.assertEqual(h3.get('a'), 'b')
+        self.assertEqual(h3.get('b'), 10)
+
+        self.assertIsNone(h.get('b'))
+        self.assertIsNone(h2.get('b'))
+
+        self.assertIsNone(h.get('a'))
+        self.assertEqual(h2.get('a'), 'b')
+
+        h = h2 = h3 = None
+
+    def test_hamt_basics_3(self):
+        h = hamt()
+        o = object()
+        h1 = h.set('1', o)
+        h2 = h1.set('1', o)
+        self.assertIs(h1, h2)
+
+    def test_hamt_basics_4(self):
+        h = hamt()
+        h1 = h.set('key', [])
+        h2 = h1.set('key', [])
+        self.assertIsNot(h1, h2)
+        self.assertEqual(len(h1), 1)
+        self.assertEqual(len(h2), 1)
+        self.assertIsNot(h1.get('key'), h2.get('key'))
+
+    def test_hamt_collision_1(self):
+        k1 = HashKey(10, 'aaa')
+        k2 = HashKey(10, 'bbb')
+        k3 = HashKey(10, 'ccc')
+
+        h = hamt()
+        h2 = h.set(k1, 'a')
+        h3 = h2.set(k2, 'b')
+
+        self.assertEqual(h.get(k1), None)
+        self.assertEqual(h.get(k2), None)
+
+        self.assertEqual(h2.get(k1), 'a')
+        self.assertEqual(h2.get(k2), None)
+
+        self.assertEqual(h3.get(k1), 'a')
+        self.assertEqual(h3.get(k2), 'b')
+
+        h4 = h3.set(k2, 'cc')
+        h5 = h4.set(k3, 'aa')
+
+        self.assertEqual(h3.get(k1), 'a')
+        self.assertEqual(h3.get(k2), 'b')
+        self.assertEqual(h4.get(k1), 'a')
+        self.assertEqual(h4.get(k2), 'cc')
+        self.assertEqual(h4.get(k3), None)
+        self.assertEqual(h5.get(k1), 'a')
+        self.assertEqual(h5.get(k2), 'cc')
+        self.assertEqual(h5.get(k2), 'cc')
+        self.assertEqual(h5.get(k3), 'aa')
+
+        self.assertEqual(len(h), 0)
+        self.assertEqual(len(h2), 1)
+        self.assertEqual(len(h3), 2)
+        self.assertEqual(len(h4), 2)
+        self.assertEqual(len(h5), 3)
+
+    def test_hamt_stress(self):
+        COLLECTION_SIZE = 7000
+        TEST_ITERS_EVERY = 647
+        CRASH_HASH_EVERY = 97
+        CRASH_EQ_EVERY = 11
+        RUN_XTIMES = 3
+
+        for _ in range(RUN_XTIMES):
+            h = hamt()
+            d = dict()
+
+            for i in range(COLLECTION_SIZE):
+                key = KeyStr(i)
+
+                if not (i % CRASH_HASH_EVERY):
+                    with HaskKeyCrasher(error_on_hash=True):
+                        with self.assertRaises(HashingError):
+                            h.set(key, i)
+
+                h = h.set(key, i)
+
+                if not (i % CRASH_EQ_EVERY):
+                    with HaskKeyCrasher(error_on_eq=True):
+                        with self.assertRaises(EqError):
+                            h.get(KeyStr(i))  # really trigger __eq__
+
+                d[key] = i
+                self.assertEqual(len(d), len(h))
+
+                if not (i % TEST_ITERS_EVERY):
+                    self.assertEqual(set(h.items()), set(d.items()))
+                    self.assertEqual(len(h.items()), len(d.items()))
+
+            self.assertEqual(len(h), COLLECTION_SIZE)
+
+            for key in range(COLLECTION_SIZE):
+                self.assertEqual(h.get(KeyStr(key), 'not found'), key)
+
+            keys_to_delete = list(range(COLLECTION_SIZE))
+            random.shuffle(keys_to_delete)
+            for iter_i, i in enumerate(keys_to_delete):
+                key = KeyStr(i)
+
+                if not (iter_i % CRASH_HASH_EVERY):
+                    with HaskKeyCrasher(error_on_hash=True):
+                        with self.assertRaises(HashingError):
+                            h.delete(key)
+
+                if not (iter_i % CRASH_EQ_EVERY):
+                    with HaskKeyCrasher(error_on_eq=True):
+                        with self.assertRaises(EqError):
+                            h.delete(KeyStr(i))
+
+                h = h.delete(key)
+                self.assertEqual(h.get(key, 'not found'), 'not found')
+                del d[key]
+                self.assertEqual(len(d), len(h))
+
+                if iter_i == COLLECTION_SIZE // 2:
+                    hm = h
+                    dm = d.copy()
+
+                if not (iter_i % TEST_ITERS_EVERY):
+                    self.assertEqual(set(h.keys()), set(d.keys()))
+                    self.assertEqual(len(h.keys()), len(d.keys()))
+
+            self.assertEqual(len(d), 0)
+            self.assertEqual(len(h), 0)
+
+            # ============
+
+            for key in dm:
+                self.assertEqual(hm.get(str(key)), dm[key])
+            self.assertEqual(len(dm), len(hm))
+
+            for i, key in enumerate(keys_to_delete):
+                hm = hm.delete(str(key))
+                self.assertEqual(hm.get(str(key), 'not found'), 'not found')
+                dm.pop(str(key), None)
+                self.assertEqual(len(d), len(h))
+
+                if not (i % TEST_ITERS_EVERY):
+                    self.assertEqual(set(h.values()), set(d.values()))
+                    self.assertEqual(len(h.values()), len(d.values()))
+
+            self.assertEqual(len(d), 0)
+            self.assertEqual(len(h), 0)
+            self.assertEqual(list(h.items()), [])
+
+    def test_hamt_delete_1(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+        C = HashKey(102, 'C')
+        D = HashKey(103, 'D')
+        E = HashKey(104, 'E')
+        Z = HashKey(-100, 'Z')
+
+        Er = HashKey(103, 'Er', error_on_eq_to=D)
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+
+        orig_len = len(h)
+
+        # BitmapNode(size=10 bitmap=0b111110000 id=0x10eadc618):
+        #     <Key name:A hash:100>: 'a'
+        #     <Key name:B hash:101>: 'b'
+        #     <Key name:C hash:102>: 'c'
+        #     <Key name:D hash:103>: 'd'
+        #     <Key name:E hash:104>: 'e'
+
+        h = h.delete(C)
+        self.assertEqual(len(h), orig_len - 1)
+
+        with self.assertRaisesRegex(ValueError, 'cannot compare'):
+            h.delete(Er)
+
+        h = h.delete(D)
+        self.assertEqual(len(h), orig_len - 2)
+
+        h2 = h.delete(Z)
+        self.assertIs(h2, h)
+
+        h = h.delete(A)
+        self.assertEqual(len(h), orig_len - 3)
+
+        self.assertEqual(h.get(A, 42), 42)
+        self.assertEqual(h.get(B), 'b')
+        self.assertEqual(h.get(E), 'e')
+
+    def test_hamt_delete_2(self):
+        A = HashKey(100, 'A')
+        B = HashKey(201001, 'B')
+        C = HashKey(101001, 'C')
+        D = HashKey(103, 'D')
+        E = HashKey(104, 'E')
+        Z = HashKey(-100, 'Z')
+
+        Er = HashKey(201001, 'Er', error_on_eq_to=B)
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+
+        orig_len = len(h)
+
+        # BitmapNode(size=8 bitmap=0b1110010000):
+        #     <Key name:A hash:100>: 'a'
+        #     <Key name:D hash:103>: 'd'
+        #     <Key name:E hash:104>: 'e'
+        #     NULL:
+        #         BitmapNode(size=4 bitmap=0b100000000001000000000):
+        #             <Key name:B hash:201001>: 'b'
+        #             <Key name:C hash:101001>: 'c'
+
+        with self.assertRaisesRegex(ValueError, 'cannot compare'):
+            h.delete(Er)
+
+        h = h.delete(Z)
+        self.assertEqual(len(h), orig_len)
+
+        h = h.delete(C)
+        self.assertEqual(len(h), orig_len - 1)
+
+        h = h.delete(B)
+        self.assertEqual(len(h), orig_len - 2)
+
+        h = h.delete(A)
+        self.assertEqual(len(h), orig_len - 3)
+
+        self.assertEqual(h.get(D), 'd')
+        self.assertEqual(h.get(E), 'e')
+
+        h = h.delete(A)
+        h = h.delete(B)
+        h = h.delete(D)
+        h = h.delete(E)
+        self.assertEqual(len(h), 0)
+
+    def test_hamt_delete_3(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+        C = HashKey(100100, 'C')
+        D = HashKey(100100, 'D')
+        E = HashKey(104, 'E')
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+
+        orig_len = len(h)
+
+        # BitmapNode(size=6 bitmap=0b100110000):
+        #     NULL:
+        #         BitmapNode(size=4 bitmap=0b1000000000000000000001000):
+        #             <Key name:A hash:100>: 'a'
+        #             NULL:
+        #                 CollisionNode(size=4 id=0x108572410):
+        #                     <Key name:C hash:100100>: 'c'
+        #                     <Key name:D hash:100100>: 'd'
+        #     <Key name:B hash:101>: 'b'
+        #     <Key name:E hash:104>: 'e'
+
+        h = h.delete(A)
+        self.assertEqual(len(h), orig_len - 1)
+
+        h = h.delete(E)
+        self.assertEqual(len(h), orig_len - 2)
+
+        self.assertEqual(h.get(C), 'c')
+        self.assertEqual(h.get(B), 'b')
+
+    def test_hamt_delete_4(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+        C = HashKey(100100, 'C')
+        D = HashKey(100100, 'D')
+        E = HashKey(100100, 'E')
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+
+        orig_len = len(h)
+
+        # BitmapNode(size=4 bitmap=0b110000):
+        #     NULL:
+        #         BitmapNode(size=4 bitmap=0b1000000000000000000001000):
+        #             <Key name:A hash:100>: 'a'
+        #             NULL:
+        #                 CollisionNode(size=6 id=0x10515ef30):
+        #                     <Key name:C hash:100100>: 'c'
+        #                     <Key name:D hash:100100>: 'd'
+        #                     <Key name:E hash:100100>: 'e'
+        #     <Key name:B hash:101>: 'b'
+
+        h = h.delete(D)
+        self.assertEqual(len(h), orig_len - 1)
+
+        h = h.delete(E)
+        self.assertEqual(len(h), orig_len - 2)
+
+        h = h.delete(C)
+        self.assertEqual(len(h), orig_len - 3)
+
+        h = h.delete(A)
+        self.assertEqual(len(h), orig_len - 4)
+
+        h = h.delete(B)
+        self.assertEqual(len(h), 0)
+
+    def test_hamt_delete_5(self):
+        h = hamt()
+
+        keys = []
+        for i in range(17):
+            key = HashKey(i, str(i))
+            keys.append(key)
+            h = h.set(key, f'val-{i}')
+
+        collision_key16 = HashKey(16, '18')
+        h = h.set(collision_key16, 'collision')
+
+        # ArrayNode(id=0x10f8b9318):
+        #     0::
+        #     BitmapNode(size=2 count=1 bitmap=0b1):
+        #         <Key name:0 hash:0>: 'val-0'
+        #
+        # ... 14 more BitmapNodes ...
+        #
+        #     15::
+        #     BitmapNode(size=2 count=1 bitmap=0b1):
+        #         <Key name:15 hash:15>: 'val-15'
+        #
+        #     16::
+        #     BitmapNode(size=2 count=1 bitmap=0b1):
+        #         NULL:
+        #             CollisionNode(size=4 id=0x10f2f5af8):
+        #                 <Key name:16 hash:16>: 'val-16'
+        #                 <Key name:18 hash:16>: 'collision'
+
+        self.assertEqual(len(h), 18)
+
+        h = h.delete(keys[2])
+        self.assertEqual(len(h), 17)
+
+        h = h.delete(collision_key16)
+        self.assertEqual(len(h), 16)
+        h = h.delete(keys[16])
+        self.assertEqual(len(h), 15)
+
+        h = h.delete(keys[1])
+        self.assertEqual(len(h), 14)
+        h = h.delete(keys[1])
+        self.assertEqual(len(h), 14)
+
+        for key in keys:
+            h = h.delete(key)
+        self.assertEqual(len(h), 0)
+
+    def test_hamt_items_1(self):
+        A = HashKey(100, 'A')
+        B = HashKey(201001, 'B')
+        C = HashKey(101001, 'C')
+        D = HashKey(103, 'D')
+        E = HashKey(104, 'E')
+        F = HashKey(110, 'F')
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+        h = h.set(F, 'f')
+
+        it = h.items()
+        self.assertEqual(
+            set(list(it)),
+            {(A, 'a'), (B, 'b'), (C, 'c'), (D, 'd'), (E, 'e'), (F, 'f')})
+
+    def test_hamt_items_2(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+        C = HashKey(100100, 'C')
+        D = HashKey(100100, 'D')
+        E = HashKey(100100, 'E')
+        F = HashKey(110, 'F')
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+        h = h.set(F, 'f')
+
+        it = h.items()
+        self.assertEqual(
+            set(list(it)),
+            {(A, 'a'), (B, 'b'), (C, 'c'), (D, 'd'), (E, 'e'), (F, 'f')})
+
+    def test_hamt_keys_1(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+        C = HashKey(100100, 'C')
+        D = HashKey(100100, 'D')
+        E = HashKey(100100, 'E')
+        F = HashKey(110, 'F')
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(B, 'b')
+        h = h.set(C, 'c')
+        h = h.set(D, 'd')
+        h = h.set(E, 'e')
+        h = h.set(F, 'f')
+
+        self.assertEqual(set(list(h.keys())), {A, B, C, D, E, F})
+        self.assertEqual(set(list(h)), {A, B, C, D, E, F})
+
+    def test_hamt_items_3(self):
+        h = hamt()
+        self.assertEqual(len(h.items()), 0)
+        self.assertEqual(list(h.items()), [])
+
+    def test_hamt_eq_1(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+        C = HashKey(100100, 'C')
+        D = HashKey(100100, 'D')
+        E = HashKey(120, 'E')
+
+        h1 = hamt()
+        h1 = h1.set(A, 'a')
+        h1 = h1.set(B, 'b')
+        h1 = h1.set(C, 'c')
+        h1 = h1.set(D, 'd')
+
+        h2 = hamt()
+        h2 = h2.set(A, 'a')
+
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+        h2 = h2.set(B, 'b')
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+        h2 = h2.set(C, 'c')
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+        h2 = h2.set(D, 'd2')
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+        h2 = h2.set(D, 'd')
+        self.assertTrue(h1 == h2)
+        self.assertFalse(h1 != h2)
+
+        h2 = h2.set(E, 'e')
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+        h2 = h2.delete(D)
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+        h2 = h2.set(E, 'd')
+        self.assertFalse(h1 == h2)
+        self.assertTrue(h1 != h2)
+
+    def test_hamt_eq_2(self):
+        A = HashKey(100, 'A')
+        Er = HashKey(100, 'Er', error_on_eq_to=A)
+
+        h1 = hamt()
+        h1 = h1.set(A, 'a')
+
+        h2 = hamt()
+        h2 = h2.set(Er, 'a')
+
+        with self.assertRaisesRegex(ValueError, 'cannot compare'):
+            h1 == h2
+
+        with self.assertRaisesRegex(ValueError, 'cannot compare'):
+            h1 != h2
+
+    def test_hamt_gc_1(self):
+        A = HashKey(100, 'A')
+
+        h = hamt()
+        h = h.set(0, 0)  # empty HAMT node is memoized in hamt.c
+        ref = weakref.ref(h)
+
+        a = []
+        a.append(a)
+        a.append(h)
+        b = []
+        a.append(b)
+        b.append(a)
+        h = h.set(A, b)
+
+        del h, a, b
+
+        gc.collect()
+        gc.collect()
+        gc.collect()
+
+        self.assertIsNone(ref())
+
+    def test_hamt_gc_2(self):
+        A = HashKey(100, 'A')
+        B = HashKey(101, 'B')
+
+        h = hamt()
+        h = h.set(A, 'a')
+        h = h.set(A, h)
+
+        ref = weakref.ref(h)
+        hi = h.items()
+        next(hi)
+
+        del h, hi
+
+        gc.collect()
+        gc.collect()
+        gc.collect()
+
+        self.assertIsNone(ref())
+
+    def test_hamt_in_1(self):
+        A = HashKey(100, 'A')
+        AA = HashKey(100, 'A')
+
+        B = HashKey(101, 'B')
+
+        h = hamt()
+        h = h.set(A, 1)
+
+        self.assertTrue(A in h)
+        self.assertFalse(B in h)
+
+        with self.assertRaises(EqError):
+            with HaskKeyCrasher(error_on_eq=True):
+                AA in h
+
+        with self.assertRaises(HashingError):
+            with HaskKeyCrasher(error_on_hash=True):
+                AA in h
+
+    def test_hamt_getitem_1(self):
+        A = HashKey(100, 'A')
+        AA = HashKey(100, 'A')
+
+        B = HashKey(101, 'B')
+
+        h = hamt()
+        h = h.set(A, 1)
+
+        self.assertEqual(h[A], 1)
+        self.assertEqual(h[AA], 1)
+
+        with self.assertRaises(KeyError):
+            h[B]
+
+        with self.assertRaises(EqError):
+            with HaskKeyCrasher(error_on_eq=True):
+                h[AA]
+
+        with self.assertRaises(HashingError):
+            with HaskKeyCrasher(error_on_hash=True):
+                h[AA]
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/Makefile.pre.in b/Makefile.pre.in
index d1512671735..162006a2f9c 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -354,6 +354,8 @@ PYTHON_OBJS=	\
 		Python/pylifecycle.o \
 		Python/pymath.o \
 		Python/pystate.o \
+		Python/context.o \
+		Python/hamt.o \
 		Python/pythonrun.o \
 		Python/pytime.o \
 		Python/bootstrap_hash.o \
@@ -996,6 +998,7 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/pymem.h \
 		$(srcdir)/Include/pyport.h \
 		$(srcdir)/Include/pystate.h \
+		$(srcdir)/Include/context.h \
 		$(srcdir)/Include/pystrcmp.h \
 		$(srcdir)/Include/pystrtod.h \
 		$(srcdir)/Include/pystrhex.h \
@@ -1023,6 +1026,7 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/internal/mem.h \
 		$(srcdir)/Include/internal/pygetopt.h \
 		$(srcdir)/Include/internal/pystate.h \
+		$(srcdir)/Include/internal/context.h \
 		$(srcdir)/Include/internal/warnings.h \
 		$(DTRACE_HEADERS)
 
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-28-00-20-42.bpo-32436.H159Jv.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-28-00-20-42.bpo-32436.H159Jv.rst
new file mode 100644
index 00000000000..8586d771781
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-28-00-20-42.bpo-32436.H159Jv.rst	
@@ -0,0 +1 @@
+Implement PEP 567
diff --git a/Modules/Setup.dist b/Modules/Setup.dist
index 1f2d56c0655..239550e26b5 100644
--- a/Modules/Setup.dist
+++ b/Modules/Setup.dist
@@ -176,6 +176,7 @@ _symtable symtablemodule.c
 #array arraymodule.c	# array objects
 #cmath cmathmodule.c _math.c # -lm # complex math library functions
 #math mathmodule.c _math.c # -lm # math library functions, e.g. sin()
+#_contextvars _contextvarsmodule.c  # Context Variables
 #_struct _struct.c	# binary structure packing/unpacking
 #_weakref _weakref.c	# basic weak reference support
 #_testcapi _testcapimodule.c    # Python C API test module
diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c
index 22ce32c593c..f77ec999b09 100644
--- a/Modules/_asynciomodule.c
+++ b/Modules/_asynciomodule.c
@@ -36,6 +36,7 @@ static PyObject *asyncio_task_print_stack_func;
 static PyObject *asyncio_task_repr_info_func;
 static PyObject *asyncio_InvalidStateError;
 static PyObject *asyncio_CancelledError;
+static PyObject *context_kwname;
 
 
 /* WeakSet containing all alive tasks. */
@@ -59,6 +60,7 @@ typedef enum {
     PyObject_HEAD                                                           \
     PyObject *prefix##_loop;                                                \
     PyObject *prefix##_callback0;                                           \
+    PyContext *prefix##_context0;                                           \
     PyObject *prefix##_callbacks;                                           \
     PyObject *prefix##_exception;                                           \
     PyObject *prefix##_result;                                              \
@@ -77,6 +79,7 @@ typedef struct {
     FutureObj_HEAD(task)
     PyObject *task_fut_waiter;
     PyObject *task_coro;
+    PyContext *task_context;
     int task_must_cancel;
     int task_log_destroy_pending;
 } TaskObj;
@@ -336,11 +339,38 @@ get_event_loop(void)
 
 
 static int
-call_soon(PyObject *loop, PyObject *func, PyObject *arg)
+call_soon(PyObject *loop, PyObject *func, PyObject *arg, PyContext *ctx)
 {
     PyObject *handle;
-    handle = _PyObject_CallMethodIdObjArgs(
-        loop, &PyId_call_soon, func, arg, NULL);
+    PyObject *stack[3];
+    Py_ssize_t nargs;
+
+    if (ctx == NULL) {
+        handle = _PyObject_CallMethodIdObjArgs(
+            loop, &PyId_call_soon, func, arg, NULL);
+    }
+    else {
+        /* Use FASTCALL to pass a keyword-only argument to call_soon */
+
+        PyObject *callable = _PyObject_GetAttrId(loop, &PyId_call_soon);
+        if (callable == NULL) {
+            return -1;
+        }
+
+        /* All refs in 'stack' are borrowed. */
+        nargs = 1;
+        stack[0] = func;
+        if (arg != NULL) {
+            stack[1] = arg;
+            nargs++;
+        }
+        stack[nargs] = (PyObject *)ctx;
+
+        handle = _PyObject_FastCallKeywords(
+            callable, stack, nargs, context_kwname);
+        Py_DECREF(callable);
+    }
+
     if (handle == NULL) {
         return -1;
     }
@@ -387,8 +417,11 @@ future_schedule_callbacks(FutureObj *fut)
         /* There's a 1st callback */
 
         int ret = call_soon(
-            fut->fut_loop, fut->fut_callback0, (PyObject *)fut);
+            fut->fut_loop, fut->fut_callback0,
+            (PyObject *)fut, fut->fut_context0);
+
         Py_CLEAR(fut->fut_callback0);
+        Py_CLEAR(fut->fut_context0);
         if (ret) {
             /* If an error occurs in pure-Python implementation,
                all callbacks are cleared. */
@@ -413,9 +446,11 @@ future_schedule_callbacks(FutureObj *fut)
     }
 
     for (i = 0; i < len; i++) {
-        PyObject *cb = PyList_GET_ITEM(fut->fut_callbacks, i);
+        PyObject *cb_tup = PyList_GET_ITEM(fut->fut_callbacks, i);
+        PyObject *cb = PyTuple_GET_ITEM(cb_tup, 0);
+        PyObject *ctx = PyTuple_GET_ITEM(cb_tup, 1);
 
-        if (call_soon(fut->fut_loop, cb, (PyObject *)fut)) {
+        if (call_soon(fut->fut_loop, cb, (PyObject *)fut, (PyContext *)ctx)) {
             /* If an error occurs in pure-Python implementation,
                all callbacks are cleared. */
             Py_CLEAR(fut->fut_callbacks);
@@ -462,6 +497,7 @@ future_init(FutureObj *fut, PyObject *loop)
     }
 
     fut->fut_callback0 = NULL;
+    fut->fut_context0 = NULL;
     fut->fut_callbacks = NULL;
 
     return 0;
@@ -566,7 +602,7 @@ future_get_result(FutureObj *fut, PyObject **result)
 }
 
 static PyObject *
-future_add_done_callback(FutureObj *fut, PyObject *arg)
+future_add_done_callback(FutureObj *fut, PyObject *arg, PyContext *ctx)
 {
     if (!future_is_alive(fut)) {
         PyErr_SetString(PyExc_RuntimeError, "uninitialized Future object");
@@ -576,7 +612,7 @@ future_add_done_callback(FutureObj *fut, PyObject *arg)
     if (fut->fut_state != STATE_PENDING) {
         /* The future is done/cancelled, so schedule the callback
            right away. */
-        if (call_soon(fut->fut_loop, arg, (PyObject*) fut)) {
+        if (call_soon(fut->fut_loop, arg, (PyObject*) fut, ctx)) {
             return NULL;
         }
     }
@@ -602,24 +638,38 @@ future_add_done_callback(FutureObj *fut, PyObject *arg)
                 with a new list and add the new callback to it.
         */
 
-        if (fut->fut_callbacks != NULL) {
-            int err = PyList_Append(fut->fut_callbacks, arg);
-            if (err != 0) {
-                return NULL;
-            }
-        }
-        else if (fut->fut_callback0 == NULL) {
+        if (fut->fut_callbacks == NULL && fut->fut_callback0 == NULL) {
             Py_INCREF(arg);
             fut->fut_callback0 = arg;
+            Py_INCREF(ctx);
+            fut->fut_context0 = ctx;
         }
         else {
-            fut->fut_callbacks = PyList_New(1);
-            if (fut->fut_callbacks == NULL) {
+            PyObject *tup = PyTuple_New(2);
+            if (tup == NULL) {
                 return NULL;
             }
-
             Py_INCREF(arg);
-            PyList_SET_ITEM(fut->fut_callbacks, 0, arg);
+            PyTuple_SET_ITEM(tup, 0, arg);
+            Py_INCREF(ctx);
+            PyTuple_SET_ITEM(tup, 1, (PyObject *)ctx);
+
+            if (fut->fut_callbacks != NULL) {
+                int err = PyList_Append(fut->fut_callbacks, tup);
+                if (err) {
+                    Py_DECREF(tup);
+                    return NULL;
+                }
+                Py_DECREF(tup);
+            }
+            else {
+                fut->fut_callbacks = PyList_New(1);
+                if (fut->fut_callbacks == NULL) {
+                    return NULL;
+                }
+
+                PyList_SET_ITEM(fut->fut_callbacks, 0, tup);  /* borrow */
+            }
         }
     }
 
@@ -676,6 +726,7 @@ FutureObj_clear(FutureObj *fut)
 {
     Py_CLEAR(fut->fut_loop);
     Py_CLEAR(fut->fut_callback0);
+    Py_CLEAR(fut->fut_context0);
     Py_CLEAR(fut->fut_callbacks);
     Py_CLEAR(fut->fut_result);
     Py_CLEAR(fut->fut_exception);
@@ -689,6 +740,7 @@ FutureObj_traverse(FutureObj *fut, visitproc visit, void *arg)
 {
     Py_VISIT(fut->fut_loop);
     Py_VISIT(fut->fut_callback0);
+    Py_VISIT(fut->fut_context0);
     Py_VISIT(fut->fut_callbacks);
     Py_VISIT(fut->fut_result);
     Py_VISIT(fut->fut_exception);
@@ -821,6 +873,8 @@ _asyncio.Future.add_done_callback
 
     fn: object
     /
+    *
+    context: object = NULL
 
 Add a callback to be run when the future becomes done.
 
@@ -830,10 +884,21 @@ scheduled with call_soon.
 [clinic start generated code]*/
 
 static PyObject *
-_asyncio_Future_add_done_callback(FutureObj *self, PyObject *fn)
-/*[clinic end generated code: output=819e09629b2ec2b5 input=8f818b39990b027d]*/
+_asyncio_Future_add_done_callback_impl(FutureObj *self, PyObject *fn,
+                                       PyObject *context)
+/*[clinic end generated code: output=7ce635bbc9554c1e input=15ab0693a96e9533]*/
 {
-    return future_add_done_callback(self, fn);
+    if (context == NULL) {
+        context = (PyObject *)PyContext_CopyCurrent();
+        if (context == NULL) {
+            return NULL;
+        }
+        PyObject *res = future_add_done_callback(
+            self, fn, (PyContext *)context);
+        Py_DECREF(context);
+        return res;
+    }
+    return future_add_done_callback(self, fn, (PyContext *)context);
 }
 
 /*[clinic input]
@@ -865,6 +930,7 @@ _asyncio_Future_remove_done_callback(FutureObj *self, PyObject *fn)
         if (cmp == 1) {
             /* callback0 == fn */
             Py_CLEAR(self->fut_callback0);
+            Py_CLEAR(self->fut_context0);
             cleared_callback0 = 1;
         }
     }
@@ -880,8 +946,9 @@ _asyncio_Future_remove_done_callback(FutureObj *self, PyObject *fn)
     }
 
     if (len == 1) {
+        PyObject *cb_tup = PyList_GET_ITEM(self->fut_callbacks, 0);
         int cmp = PyObject_RichCompareBool(
-            fn, PyList_GET_ITEM(self->fut_callbacks, 0), Py_EQ);
+            fn, PyTuple_GET_ITEM(cb_tup, 0), Py_EQ);
         if (cmp == -1) {
             return NULL;
         }
@@ -903,7 +970,7 @@ _asyncio_Future_remove_done_callback(FutureObj *self, PyObject *fn)
         int ret;
         PyObject *item = PyList_GET_ITEM(self->fut_callbacks, i);
         Py_INCREF(item);
-        ret = PyObject_RichCompareBool(fn, item, Py_EQ);
+        ret = PyObject_RichCompareBool(fn, PyTuple_GET_ITEM(item, 0), Py_EQ);
         if (ret == 0) {
             if (j < len) {
                 PyList_SET_ITEM(newlist, j, item);
@@ -1081,47 +1148,49 @@ static PyObject *
 FutureObj_get_callbacks(FutureObj *fut)
 {
     Py_ssize_t i;
-    Py_ssize_t len;
-    PyObject *new_list;
 
     ENSURE_FUTURE_ALIVE(fut)
 
-    if (fut->fut_callbacks == NULL) {
-        if (fut->fut_callback0 == NULL) {
+    if (fut->fut_callback0 == NULL) {
+        if (fut->fut_callbacks == NULL) {
             Py_RETURN_NONE;
         }
-        else {
-            new_list = PyList_New(1);
-            if (new_list == NULL) {
-                return NULL;
-            }
-            Py_INCREF(fut->fut_callback0);
-            PyList_SET_ITEM(new_list, 0, fut->fut_callback0);
-            return new_list;
-        }
-    }
 
-    assert(fut->fut_callbacks != NULL);
-
-    if (fut->fut_callback0 == NULL) {
         Py_INCREF(fut->fut_callbacks);
         return fut->fut_callbacks;
     }
 
-    assert(fut->fut_callback0 != NULL);
+    Py_ssize_t len = 1;
+    if (fut->fut_callbacks != NULL) {
+        len += PyList_GET_SIZE(fut->fut_callbacks);
+    }
 
-    len = PyList_GET_SIZE(fut->fut_callbacks);
-    new_list = PyList_New(len + 1);
+
+    PyObject *new_list = PyList_New(len);
     if (new_list == NULL) {
         return NULL;
     }
 
+    PyObject *tup0 = PyTuple_New(2);
+    if (tup0 == NULL) {
+        Py_DECREF(new_list);
+        return NULL;
+    }
+
     Py_INCREF(fut->fut_callback0);
-    PyList_SET_ITEM(new_list, 0, fut->fut_callback0);
-    for (i = 0; i < len; i++) {
-        PyObject *cb = PyList_GET_ITEM(fut->fut_callbacks, i);
-        Py_INCREF(cb);
-        PyList_SET_ITEM(new_list, i + 1, cb);
+    PyTuple_SET_ITEM(tup0, 0, fut->fut_callback0);
+    assert(fut->fut_context0 != NULL);
+    Py_INCREF(fut->fut_context0);
+    PyTuple_SET_ITEM(tup0, 1, (PyObject *)fut->fut_context0);
+
+    PyList_SET_ITEM(new_list, 0, tup0);
+
+    if (fut->fut_callbacks != NULL) {
+        for (i = 0; i < PyList_GET_SIZE(fut->fut_callbacks); i++) {
+            PyObject *cb = PyList_GET_ITEM(fut->fut_callbacks, i);
+            Py_INCREF(cb);
+            PyList_SET_ITEM(new_list, i + 1, cb);
+        }
     }
 
     return new_list;
@@ -1912,6 +1981,11 @@ _asyncio_Task___init___impl(TaskObj *self, PyObject *coro, PyObject *loop)
         return -1;
     }
 
+    self->task_context = PyContext_CopyCurrent();
+    if (self->task_context == NULL) {
+        return -1;
+    }
+
     self->task_fut_waiter = NULL;
     self->task_must_cancel = 0;
     self->task_log_destroy_pending = 1;
@@ -1928,6 +2002,7 @@ static int
 TaskObj_clear(TaskObj *task)
 {
     (void)FutureObj_clear((FutureObj*) task);
+    Py_CLEAR(task->task_context);
     Py_CLEAR(task->task_coro);
     Py_CLEAR(task->task_fut_waiter);
     return 0;
@@ -1936,6 +2011,7 @@ TaskObj_clear(TaskObj *task)
 static int
 TaskObj_traverse(TaskObj *task, visitproc visit, void *arg)
 {
+    Py_VISIT(task->task_context);
     Py_VISIT(task->task_coro);
     Py_VISIT(task->task_fut_waiter);
     (void)FutureObj_traverse((FutureObj*) task, visit, arg);
@@ -2451,7 +2527,7 @@ task_call_step_soon(TaskObj *task, PyObject *arg)
         return -1;
     }
 
-    int ret = call_soon(task->task_loop, cb, NULL);
+    int ret = call_soon(task->task_loop, cb, NULL, task->task_context);
     Py_DECREF(cb);
     return ret;
 }
@@ -2650,7 +2726,8 @@ task_step_impl(TaskObj *task, PyObject *exc)
             if (wrapper == NULL) {
                 goto fail;
             }
-            res = future_add_done_callback((FutureObj*)result, wrapper);
+            res = future_add_done_callback(
+                (FutureObj*)result, wrapper, task->task_context);
             Py_DECREF(wrapper);
             if (res == NULL) {
                 goto fail;
@@ -2724,14 +2801,23 @@ task_step_impl(TaskObj *task, PyObject *exc)
                     goto fail;
                 }
 
-                /* result.add_done_callback(task._wakeup) */
                 wrapper = TaskWakeupMethWrapper_new(task);
                 if (wrapper == NULL) {
                     goto fail;
                 }
-                res = _PyObject_CallMethodIdObjArgs(result,
-                                                    &PyId_add_done_callback,
-                                                    wrapper, NULL);
+
+                /* result.add_done_callback(task._wakeup) */
+                PyObject *add_cb = _PyObject_GetAttrId(
+                    result, &PyId_add_done_callback);
+                if (add_cb == NULL) {
+                    goto fail;
+                }
+                PyObject *stack[2];
+                stack[0] = wrapper;
+                stack[1] = (PyObject *)task->task_context;
+                res = _PyObject_FastCallKeywords(
+                    add_cb, stack, 1, context_kwname);
+                Py_DECREF(add_cb);
                 Py_DECREF(wrapper);
                 if (res == NULL) {
                     goto fail;
@@ -3141,6 +3227,8 @@ module_free(void *m)
     Py_CLEAR(current_tasks);
     Py_CLEAR(iscoroutine_typecache);
 
+    Py_CLEAR(context_kwname);
+
     module_free_freelists();
 }
 
@@ -3164,6 +3252,17 @@ module_init(void)
         goto fail;
     }
 
+
+    context_kwname = PyTuple_New(1);
+    if (context_kwname == NULL) {
+        goto fail;
+    }
+    PyObject *context_str = PyUnicode_FromString("context");
+    if (context_str == NULL) {
+        goto fail;
+    }
+    PyTuple_SET_ITEM(context_kwname, 0, context_str);
+
 #define WITH_MOD(NAME) \
     Py_CLEAR(module); \
     module = PyImport_ImportModule(NAME); \
diff --git a/Modules/_contextvarsmodule.c b/Modules/_contextvarsmodule.c
new file mode 100644
index 00000000000..b7d112dd601
--- /dev/null
+++ b/Modules/_contextvarsmodule.c
@@ -0,0 +1,75 @@
+#include "Python.h"
+
+#include "clinic/_contextvarsmodule.c.h"
+
+/*[clinic input]
+module _contextvars
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=a0955718c8b8cea6]*/
+
+
+/*[clinic input]
+_contextvars.copy_context
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_copy_context_impl(PyObject *module)
+/*[clinic end generated code: output=1fcd5da7225c4fa9 input=89bb9ae485888440]*/
+{
+    return (PyObject *)PyContext_CopyCurrent();
+}
+
+
+PyDoc_STRVAR(module_doc, "Context Variables");
+
+static PyMethodDef _contextvars_methods[] = {
+    _CONTEXTVARS_COPY_CONTEXT_METHODDEF
+    {NULL, NULL}
+};
+
+static struct PyModuleDef _contextvarsmodule = {
+    PyModuleDef_HEAD_INIT,      /* m_base */
+    "_contextvars",             /* m_name */
+    module_doc,                 /* m_doc */
+    -1,                         /* m_size */
+    _contextvars_methods,       /* m_methods */
+    NULL,                       /* m_slots */
+    NULL,                       /* m_traverse */
+    NULL,                       /* m_clear */
+    NULL,                       /* m_free */
+};
+
+PyMODINIT_FUNC
+PyInit__contextvars(void)
+{
+    PyObject *m = PyModule_Create(&_contextvarsmodule);
+    if (m == NULL) {
+        return NULL;
+    }
+
+    Py_INCREF(&PyContext_Type);
+    if (PyModule_AddObject(m, "Context",
+                           (PyObject *)&PyContext_Type) < 0)
+    {
+        Py_DECREF(&PyContext_Type);
+        return NULL;
+    }
+
+    Py_INCREF(&PyContextVar_Type);
+    if (PyModule_AddObject(m, "ContextVar",
+                           (PyObject *)&PyContextVar_Type) < 0)
+    {
+        Py_DECREF(&PyContextVar_Type);
+        return NULL;
+    }
+
+    Py_INCREF(&PyContextToken_Type);
+    if (PyModule_AddObject(m, "Token",
+                           (PyObject *)&PyContextToken_Type) < 0)
+    {
+        Py_DECREF(&PyContextToken_Type);
+        return NULL;
+    }
+
+    return m;
+}
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 0d6bf45032d..e3be7d3d829 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -4438,6 +4438,13 @@ test_pythread_tss_key_state(PyObject *self, PyObject *args)
 }
 
 
+static PyObject*
+new_hamt(PyObject *self, PyObject *args)
+{
+    return _PyContext_NewHamtForTests();
+}
+
+
 static PyMethodDef TestMethods[] = {
     {"raise_exception",         raise_exception,                 METH_VARARGS},
     {"raise_memoryerror",   (PyCFunction)raise_memoryerror,  METH_NOARGS},
@@ -4655,6 +4662,7 @@ static PyMethodDef TestMethods[] = {
     {"get_mapping_values", get_mapping_values, METH_O},
     {"get_mapping_items", get_mapping_items, METH_O},
     {"test_pythread_tss_key_state", test_pythread_tss_key_state, METH_VARARGS},
+    {"hamt", new_hamt, METH_NOARGS},
     {NULL, NULL} /* sentinel */
 };
 
diff --git a/Modules/clinic/_asynciomodule.c.h b/Modules/clinic/_asynciomodule.c.h
index f2e0f405ada..9fc9d6b17c4 100644
--- a/Modules/clinic/_asynciomodule.c.h
+++ b/Modules/clinic/_asynciomodule.c.h
@@ -110,7 +110,7 @@ PyDoc_STRVAR(_asyncio_Future_set_exception__doc__,
     {"set_exception", (PyCFunction)_asyncio_Future_set_exception, METH_O, _asyncio_Future_set_exception__doc__},
 
 PyDoc_STRVAR(_asyncio_Future_add_done_callback__doc__,
-"add_done_callback($self, fn, /)\n"
+"add_done_callback($self, fn, /, *, context=None)\n"
 "--\n"
 "\n"
 "Add a callback to be run when the future becomes done.\n"
@@ -120,7 +120,30 @@ PyDoc_STRVAR(_asyncio_Future_add_done_callback__doc__,
 "scheduled with call_soon.");
 
 #define _ASYNCIO_FUTURE_ADD_DONE_CALLBACK_METHODDEF    \
-    {"add_done_callback", (PyCFunction)_asyncio_Future_add_done_callback, METH_O, _asyncio_Future_add_done_callback__doc__},
+    {"add_done_callback", (PyCFunction)_asyncio_Future_add_done_callback, METH_FASTCALL|METH_KEYWORDS, _asyncio_Future_add_done_callback__doc__},
+
+static PyObject *
+_asyncio_Future_add_done_callback_impl(FutureObj *self, PyObject *fn,
+                                       PyObject *context);
+
+static PyObject *
+_asyncio_Future_add_done_callback(FutureObj *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"", "context", NULL};
+    static _PyArg_Parser _parser = {"O|$O:add_done_callback", _keywords, 0};
+    PyObject *fn;
+    PyObject *context = NULL;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &fn, &context)) {
+        goto exit;
+    }
+    return_value = _asyncio_Future_add_done_callback_impl(self, fn, context);
+
+exit:
+    return return_value;
+}
 
 PyDoc_STRVAR(_asyncio_Future_remove_done_callback__doc__,
 "remove_done_callback($self, fn, /)\n"
@@ -763,4 +786,4 @@ _asyncio__leave_task(PyObject *module, PyObject *const *args, Py_ssize_t nargs,
 exit:
     return return_value;
 }
-/*[clinic end generated code: output=616e814431893dcc input=a9049054013a1b77]*/
+/*[clinic end generated code: output=bcbaf1b2480f4aa9 input=a9049054013a1b77]*/
diff --git a/Modules/clinic/_contextvarsmodule.c.h b/Modules/clinic/_contextvarsmodule.c.h
new file mode 100644
index 00000000000..b1885e41c35
--- /dev/null
+++ b/Modules/clinic/_contextvarsmodule.c.h
@@ -0,0 +1,21 @@
+/*[clinic input]
+preserve
+[clinic start generated code]*/
+
+PyDoc_STRVAR(_contextvars_copy_context__doc__,
+"copy_context($module, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_COPY_CONTEXT_METHODDEF    \
+    {"copy_context", (PyCFunction)_contextvars_copy_context, METH_NOARGS, _contextvars_copy_context__doc__},
+
+static PyObject *
+_contextvars_copy_context_impl(PyObject *module);
+
+static PyObject *
+_contextvars_copy_context(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    return _contextvars_copy_context_impl(module);
+}
+/*[clinic end generated code: output=26e07024451baf52 input=a9049054013a1b77]*/
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index ea3c294dcff..8ba1093c029 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -24,6 +24,7 @@
 */
 
 #include "Python.h"
+#include "internal/context.h"
 #include "internal/mem.h"
 #include "internal/pystate.h"
 #include "frameobject.h"        /* for PyFrame_ClearFreeList */
@@ -790,6 +791,7 @@ clear_freelists(void)
     (void)PyDict_ClearFreeList();
     (void)PySet_ClearFreeList();
     (void)PyAsyncGen_ClearFreeLists();
+    (void)PyContext_ClearFreeList();
 }
 
 /* This is the main function.  Read this to understand how the
diff --git a/Objects/object.c b/Objects/object.c
index 62d7fbebf40..8cec6e2f122 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -3,6 +3,7 @@
 
 #include "Python.h"
 #include "internal/pystate.h"
+#include "internal/context.h"
 #include "frameobject.h"
 
 #ifdef __cplusplus
diff --git a/PCbuild/_contextvars.vcxproj b/PCbuild/_contextvars.vcxproj
new file mode 100644
index 00000000000..7418e86570a
--- /dev/null
+++ b/PCbuild/_contextvars.vcxproj
@@ -0,0 +1,77 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup Label="ProjectConfigurations">
+    <ProjectConfiguration Include="Debug|Win32">
+      <Configuration>Debug</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Debug|x64">
+      <Configuration>Debug</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|Win32">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGInstrument|x64">
+      <Configuration>PGInstrument</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|Win32">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="PGUpdate|x64">
+      <Configuration>PGUpdate</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|Win32">
+      <Configuration>Release</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|x64">
+      <Configuration>Release</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+  </ItemGroup>
+  <PropertyGroup Label="Globals">
+    <ProjectGuid>{B8BF1D81-09DC-42D4-B406-4F868B33A89E}</ProjectGuid>
+    <RootNamespace>_contextvars</RootNamespace>
+    <Keyword>Win32Proj</Keyword>
+  </PropertyGroup>
+  <Import Project="python.props" />
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>DynamicLibrary</ConfigurationType>
+    <CharacterSet>NotSet</CharacterSet>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+  <PropertyGroup>
+    <TargetExt>.pyd</TargetExt>
+  </PropertyGroup>
+  <ImportGroup Label="ExtensionSettings">
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="pyproject.props" />
+  </ImportGroup>
+  <PropertyGroup Label="UserMacros" />
+  <PropertyGroup>
+    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
+  </PropertyGroup>
+  <ItemGroup>
+    <ClCompile Include="..\Modules\_contextvarsmodule.c" />
+  </ItemGroup>
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <ProjectReference Include="pythoncore.vcxproj">
+      <Project>{cf7ac3d1-e2df-41d2-bea6-1e2556cdea26}</Project>
+      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
+    </ProjectReference>
+  </ItemGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+  <ImportGroup Label="ExtensionTargets">
+  </ImportGroup>
+</Project>
diff --git a/PCbuild/_contextvars.vcxproj.filters b/PCbuild/_contextvars.vcxproj.filters
new file mode 100644
index 00000000000..b3002b7ff67
--- /dev/null
+++ b/PCbuild/_contextvars.vcxproj.filters
@@ -0,0 +1,16 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <ResourceCompile Include="..\PC\python_nt.rc" />
+  </ItemGroup>
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{7CBD8910-233D-4E9A-9164-9BA66C1F0E6D}</UniqueIdentifier>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="..\Modules\_contextvarsmodule.c">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
diff --git a/PCbuild/_decimal.vcxproj b/PCbuild/_decimal.vcxproj
index b14f31093ff..df9f600cdaf 100644
--- a/PCbuild/_decimal.vcxproj
+++ b/PCbuild/_decimal.vcxproj
@@ -121,4 +121,4 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-</Project>
\ No newline at end of file
+</Project>
diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj
index 848b3b2877f..5e341959bd3 100644
--- a/PCbuild/pcbuild.proj
+++ b/PCbuild/pcbuild.proj
@@ -49,7 +49,7 @@
     <!-- pyshellext.dll -->
     <Projects Include="pyshellext.vcxproj" />
     <!-- Extension modules -->
-    <ExtensionModules Include="_asyncio;_ctypes;_decimal;_distutils_findvs;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;_queue;select;unicodedata;winsound" />
+    <ExtensionModules Include="_asyncio;_contextvars;_ctypes;_decimal;_distutils_findvs;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;_queue;select;unicodedata;winsound" />
     <!-- Extension modules that require external sources -->
     <ExternalModules Include="_bz2;_lzma;_sqlite3" />
     <!-- _ssl will build _socket as well, which may cause conflicts in parallel builds -->
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
index bf2ce669f08..fbcd0512139 100644
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -94,6 +94,7 @@
     <ClInclude Include="..\Include\codecs.h" />
     <ClInclude Include="..\Include\compile.h" />
     <ClInclude Include="..\Include\complexobject.h" />
+    <ClInclude Include="..\Include\context.h" />
     <ClInclude Include="..\Include\datetime.h" />
     <ClInclude Include="..\Include\descrobject.h" />
     <ClInclude Include="..\Include\dictobject.h" />
@@ -112,7 +113,9 @@
     <ClInclude Include="..\Include\import.h" />
     <ClInclude Include="..\Include\internal\ceval.h" />
     <ClInclude Include="..\Include\internal\condvar.h" />
+    <ClInclude Include="..\Include\internal\context.h" />
     <ClInclude Include="..\Include\internal\gil.h" />
+    <ClInclude Include="..\Include\internal\hamt.h" />
     <ClInclude Include="..\Include\internal\mem.h" />
     <ClInclude Include="..\Include\internal\pystate.h" />
     <ClInclude Include="..\Include\internal\warnings.h" />
@@ -232,6 +235,7 @@
     <ClCompile Include="..\Modules\_blake2\blake2s_impl.c" />
     <ClCompile Include="..\Modules\_codecsmodule.c" />
     <ClCompile Include="..\Modules\_collectionsmodule.c" />
+    <ClCompile Include="..\Modules\_contextvarsmodule.c" />
     <ClCompile Include="..\Modules\_csv.c" />
     <ClCompile Include="..\Modules\_functoolsmodule.c" />
     <ClCompile Include="..\Modules\_heapqmodule.c" />
@@ -359,6 +363,7 @@
     <ClCompile Include="..\Python\ceval.c" />
     <ClCompile Include="..\Python\codecs.c" />
     <ClCompile Include="..\Python\compile.c" />
+    <ClCompile Include="..\Python\context.c" />
     <ClCompile Include="..\Python\dynamic_annotations.c" />
     <ClCompile Include="..\Python\dynload_win.c" />
     <ClCompile Include="..\Python\errors.c" />
@@ -373,6 +378,7 @@
     <ClCompile Include="..\Python\getplatform.c" />
     <ClCompile Include="..\Python\getversion.c" />
     <ClCompile Include="..\Python\graminit.c" />
+    <ClCompile Include="..\Python\hamt.c" />
     <ClCompile Include="..\Python\import.c" />
     <ClCompile Include="..\Python\importdl.c" />
     <ClCompile Include="..\Python\marshal.c" />
diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters
index 13600cb5c9f..a10686c194b 100644
--- a/PCbuild/pythoncore.vcxproj.filters
+++ b/PCbuild/pythoncore.vcxproj.filters
@@ -81,6 +81,9 @@
     <ClInclude Include="..\Include\complexobject.h">
       <Filter>Include</Filter>
     </ClInclude>
+    <ClInclude Include="..\Include\context.h">
+      <Filter>Include</Filter>
+    </ClInclude>
     <ClInclude Include="..\Include\datetime.h">
       <Filter>Include</Filter>
     </ClInclude>
@@ -135,9 +138,15 @@
     <ClInclude Include="..\Include\internal\condvar.h">
       <Filter>Include</Filter>
     </ClInclude>
+    <ClInclude Include="..\Include\internal\context.h">
+      <Filter>Include</Filter>
+    </ClInclude>
     <ClInclude Include="..\Include\internal\gil.h">
       <Filter>Include</Filter>
     </ClInclude>
+    <ClInclude Include="..\Include\internal\hamt.h">
+      <Filter>Include</Filter>
+    </ClInclude>
     <ClInclude Include="..\Include\internal\mem.h">
       <Filter>Include</Filter>
     </ClInclude>
@@ -842,6 +851,9 @@
     <ClCompile Include="..\Python\compile.c">
       <Filter>Python</Filter>
     </ClCompile>
+    <ClCompile Include="..\Python\context.h">
+      <Filter>Python</Filter>
+    </ClCompile>
     <ClCompile Include="..\Python\dynamic_annotations.c">
       <Filter>Python</Filter>
     </ClCompile>
@@ -884,6 +896,9 @@
     <ClCompile Include="..\Python\graminit.c">
       <Filter>Python</Filter>
     </ClCompile>
+    <ClCompile Include="..\Python\hamt.h">
+      <Filter>Python</Filter>
+    </ClCompile>
     <ClCompile Include="..\Python\import.c">
       <Filter>Python</Filter>
     </ClCompile>
@@ -998,6 +1013,9 @@
     <ClCompile Include="..\Modules\_asynciomodule.c">
       <Filter>Modules</Filter>
     </ClCompile>
+    <ClCompile Include="..\Modules\_contextvarsmodule.c">
+      <Filter>Modules</Filter>
+    </ClCompile>
     <ClCompile Include="$(zlibDir)\adler32.c">
       <Filter>Modules\zlib</Filter>
     </ClCompile>
diff --git a/Python/clinic/context.c.h b/Python/clinic/context.c.h
new file mode 100644
index 00000000000..dcf4c214e1b
--- /dev/null
+++ b/Python/clinic/context.c.h
@@ -0,0 +1,146 @@
+/*[clinic input]
+preserve
+[clinic start generated code]*/
+
+PyDoc_STRVAR(_contextvars_Context_get__doc__,
+"get($self, key, default=None, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXT_GET_METHODDEF    \
+    {"get", (PyCFunction)_contextvars_Context_get, METH_FASTCALL, _contextvars_Context_get__doc__},
+
+static PyObject *
+_contextvars_Context_get_impl(PyContext *self, PyObject *key,
+                              PyObject *default_value);
+
+static PyObject *
+_contextvars_Context_get(PyContext *self, PyObject *const *args, Py_ssize_t nargs)
+{
+    PyObject *return_value = NULL;
+    PyObject *key;
+    PyObject *default_value = Py_None;
+
+    if (!_PyArg_UnpackStack(args, nargs, "get",
+        1, 2,
+        &key, &default_value)) {
+        goto exit;
+    }
+    return_value = _contextvars_Context_get_impl(self, key, default_value);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_contextvars_Context_items__doc__,
+"items($self, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXT_ITEMS_METHODDEF    \
+    {"items", (PyCFunction)_contextvars_Context_items, METH_NOARGS, _contextvars_Context_items__doc__},
+
+static PyObject *
+_contextvars_Context_items_impl(PyContext *self);
+
+static PyObject *
+_contextvars_Context_items(PyContext *self, PyObject *Py_UNUSED(ignored))
+{
+    return _contextvars_Context_items_impl(self);
+}
+
+PyDoc_STRVAR(_contextvars_Context_keys__doc__,
+"keys($self, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXT_KEYS_METHODDEF    \
+    {"keys", (PyCFunction)_contextvars_Context_keys, METH_NOARGS, _contextvars_Context_keys__doc__},
+
+static PyObject *
+_contextvars_Context_keys_impl(PyContext *self);
+
+static PyObject *
+_contextvars_Context_keys(PyContext *self, PyObject *Py_UNUSED(ignored))
+{
+    return _contextvars_Context_keys_impl(self);
+}
+
+PyDoc_STRVAR(_contextvars_Context_values__doc__,
+"values($self, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXT_VALUES_METHODDEF    \
+    {"values", (PyCFunction)_contextvars_Context_values, METH_NOARGS, _contextvars_Context_values__doc__},
+
+static PyObject *
+_contextvars_Context_values_impl(PyContext *self);
+
+static PyObject *
+_contextvars_Context_values(PyContext *self, PyObject *Py_UNUSED(ignored))
+{
+    return _contextvars_Context_values_impl(self);
+}
+
+PyDoc_STRVAR(_contextvars_Context_copy__doc__,
+"copy($self, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXT_COPY_METHODDEF    \
+    {"copy", (PyCFunction)_contextvars_Context_copy, METH_NOARGS, _contextvars_Context_copy__doc__},
+
+static PyObject *
+_contextvars_Context_copy_impl(PyContext *self);
+
+static PyObject *
+_contextvars_Context_copy(PyContext *self, PyObject *Py_UNUSED(ignored))
+{
+    return _contextvars_Context_copy_impl(self);
+}
+
+PyDoc_STRVAR(_contextvars_ContextVar_get__doc__,
+"get($self, default=None, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXTVAR_GET_METHODDEF    \
+    {"get", (PyCFunction)_contextvars_ContextVar_get, METH_FASTCALL, _contextvars_ContextVar_get__doc__},
+
+static PyObject *
+_contextvars_ContextVar_get_impl(PyContextVar *self, PyObject *default_value);
+
+static PyObject *
+_contextvars_ContextVar_get(PyContextVar *self, PyObject *const *args, Py_ssize_t nargs)
+{
+    PyObject *return_value = NULL;
+    PyObject *default_value = NULL;
+
+    if (!_PyArg_UnpackStack(args, nargs, "get",
+        0, 1,
+        &default_value)) {
+        goto exit;
+    }
+    return_value = _contextvars_ContextVar_get_impl(self, default_value);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_contextvars_ContextVar_set__doc__,
+"set($self, value, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXTVAR_SET_METHODDEF    \
+    {"set", (PyCFunction)_contextvars_ContextVar_set, METH_O, _contextvars_ContextVar_set__doc__},
+
+PyDoc_STRVAR(_contextvars_ContextVar_reset__doc__,
+"reset($self, token, /)\n"
+"--\n"
+"\n");
+
+#define _CONTEXTVARS_CONTEXTVAR_RESET_METHODDEF    \
+    {"reset", (PyCFunction)_contextvars_ContextVar_reset, METH_O, _contextvars_ContextVar_reset__doc__},
+/*[clinic end generated code: output=d9a675e3a52a14fc input=a9049054013a1b77]*/
diff --git a/Python/context.c b/Python/context.c
new file mode 100644
index 00000000000..2f1d0f5c342
--- /dev/null
+++ b/Python/context.c
@@ -0,0 +1,1220 @@
+#include "Python.h"
+
+#include "structmember.h"
+#include "internal/pystate.h"
+#include "internal/context.h"
+#include "internal/hamt.h"
+
+
+#define CONTEXT_FREELIST_MAXLEN 255
+static PyContext *ctx_freelist = NULL;
+static Py_ssize_t ctx_freelist_len = 0;
+
+
+#include "clinic/context.c.h"
+/*[clinic input]
+module _contextvars
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=a0955718c8b8cea6]*/
+
+
+/////////////////////////// Context API
+
+
+static PyContext *
+context_new_empty(void);
+
+static PyContext *
+context_new_from_vars(PyHamtObject *vars);
+
+static inline PyContext *
+context_get(void);
+
+static PyContextToken *
+token_new(PyContext *ctx, PyContextVar *var, PyObject *val);
+
+static PyContextVar *
+contextvar_new(PyObject *name, PyObject *def);
+
+static int
+contextvar_set(PyContextVar *var, PyObject *val);
+
+static int
+contextvar_del(PyContextVar *var);
+
+
+PyObject *
+_PyContext_NewHamtForTests(void)
+{
+    return (PyObject *)_PyHamt_New();
+}
+
+
+PyContext *
+PyContext_New(void)
+{
+    return context_new_empty();
+}
+
+
+PyContext *
+PyContext_Copy(PyContext * ctx)
+{
+    return context_new_from_vars(ctx->ctx_vars);
+}
+
+
+PyContext *
+PyContext_CopyCurrent(void)
+{
+    PyContext *ctx = context_get();
+    if (ctx == NULL) {
+        return NULL;
+    }
+
+    return context_new_from_vars(ctx->ctx_vars);
+}
+
+
+int
+PyContext_Enter(PyContext *ctx)
+{
+    if (ctx->ctx_entered) {
+        PyErr_Format(PyExc_RuntimeError,
+                     "cannot enter context: %R is already entered", ctx);
+        return -1;
+    }
+
+    PyThreadState *ts = PyThreadState_Get();
+
+    ctx->ctx_prev = (PyContext *)ts->context;  /* borrow */
+    ctx->ctx_entered = 1;
+
+    Py_INCREF(ctx);
+    ts->context = (PyObject *)ctx;
+    ts->context_ver++;
+
+    return 0;
+}
+
+
+int
+PyContext_Exit(PyContext *ctx)
+{
+    if (!ctx->ctx_entered) {
+        PyErr_Format(PyExc_RuntimeError,
+                     "cannot exit context: %R has not been entered", ctx);
+        return -1;
+    }
+
+    PyThreadState *ts = PyThreadState_Get();
+
+    if (ts->context != (PyObject *)ctx) {
+        /* Can only happen if someone misuses the C API */
+        PyErr_SetString(PyExc_RuntimeError,
+                        "cannot exit context: thread state references "
+                        "a different context object");
+        return -1;
+    }
+
+    Py_SETREF(ts->context, (PyObject *)ctx->ctx_prev);
+    ts->context_ver++;
+
+    ctx->ctx_prev = NULL;
+    ctx->ctx_entered = 0;
+
+    return 0;
+}
+
+
+PyContextVar *
+PyContextVar_New(const char *name, PyObject *def)
+{
+    PyObject *pyname = PyUnicode_FromString(name);
+    if (pyname == NULL) {
+        return NULL;
+    }
+    return contextvar_new(pyname, def);
+}
+
+
+int
+PyContextVar_Get(PyContextVar *var, PyObject *def, PyObject **val)
+{
+    assert(PyContextVar_CheckExact(var));
+
+    PyThreadState *ts = PyThreadState_Get();
+    if (ts->context == NULL) {
+        goto not_found;
+    }
+
+    if (var->var_cached != NULL &&
+            var->var_cached_tsid == ts->id &&
+            var->var_cached_tsver == ts->context_ver)
+    {
+        *val = var->var_cached;
+        goto found;
+    }
+
+    assert(PyContext_CheckExact(ts->context));
+    PyHamtObject *vars = ((PyContext *)ts->context)->ctx_vars;
+
+    PyObject *found = NULL;
+    int res = _PyHamt_Find(vars, (PyObject*)var, &found);
+    if (res < 0) {
+        goto error;
+    }
+    if (res == 1) {
+        assert(found != NULL);
+        var->var_cached = found;  /* borrow */
+        var->var_cached_tsid = ts->id;
+        var->var_cached_tsver = ts->context_ver;
+
+        *val = found;
+        goto found;
+    }
+
+not_found:
+    if (def == NULL) {
+        if (var->var_default != NULL) {
+            *val = var->var_default;
+            goto found;
+        }
+
+        *val = NULL;
+        goto found;
+    }
+    else {
+        *val = def;
+        goto found;
+   }
+
+found:
+    Py_XINCREF(*val);
+    return 0;
+
+error:
+    *val = NULL;
+    return -1;
+}
+
+
+PyContextToken *
+PyContextVar_Set(PyContextVar *var, PyObject *val)
+{
+    if (!PyContextVar_CheckExact(var)) {
+        PyErr_SetString(
+            PyExc_TypeError, "an instance of ContextVar was expected");
+        return NULL;
+    }
+
+    PyContext *ctx = context_get();
+    if (ctx == NULL) {
+        return NULL;
+    }
+
+    PyObject *old_val = NULL;
+    int found = _PyHamt_Find(ctx->ctx_vars, (PyObject *)var, &old_val);
+    if (found < 0) {
+        return NULL;
+    }
+
+    Py_XINCREF(old_val);
+    PyContextToken *tok = token_new(ctx, var, old_val);
+    Py_XDECREF(old_val);
+
+    if (contextvar_set(var, val)) {
+        Py_DECREF(tok);
+        return NULL;
+    }
+
+    return tok;
+}
+
+
+int
+PyContextVar_Reset(PyContextVar *var, PyContextToken *tok)
+{
+    if (tok->tok_used) {
+        PyErr_Format(PyExc_RuntimeError,
+                     "%R has already been used once", tok);
+        return -1;
+    }
+
+    if (var != tok->tok_var) {
+        PyErr_Format(PyExc_ValueError,
+                     "%R was created by a different ContextVar", tok);
+        return -1;
+    }
+
+    PyContext *ctx = context_get();
+    if (ctx != tok->tok_ctx) {
+        PyErr_Format(PyExc_ValueError,
+                     "%R was created in a different Context", tok);
+        return -1;
+    }
+
+    tok->tok_used = 1;
+
+    if (tok->tok_oldval == NULL) {
+        return contextvar_del(var);
+    }
+    else {
+        return contextvar_set(var, tok->tok_oldval);
+    }
+}
+
+
+/////////////////////////// PyContext
+
+/*[clinic input]
+class _contextvars.Context "PyContext *" "&PyContext_Type"
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=bdf87f8e0cb580e8]*/
+
+
+static inline PyContext *
+_context_alloc(void)
+{
+    PyContext *ctx;
+    if (ctx_freelist_len) {
+        ctx_freelist_len--;
+        ctx = ctx_freelist;
+        ctx_freelist = (PyContext *)ctx->ctx_weakreflist;
+        ctx->ctx_weakreflist = NULL;
+        _Py_NewReference((PyObject *)ctx);
+    }
+    else {
+        ctx = PyObject_GC_New(PyContext, &PyContext_Type);
+        if (ctx == NULL) {
+            return NULL;
+        }
+    }
+
+    ctx->ctx_vars = NULL;
+    ctx->ctx_prev = NULL;
+    ctx->ctx_entered = 0;
+    ctx->ctx_weakreflist = NULL;
+
+    return ctx;
+}
+
+
+static PyContext *
+context_new_empty(void)
+{
+    PyContext *ctx = _context_alloc();
+    if (ctx == NULL) {
+        return NULL;
+    }
+
+    ctx->ctx_vars = _PyHamt_New();
+    if (ctx->ctx_vars == NULL) {
+        Py_DECREF(ctx);
+        return NULL;
+    }
+
+    _PyObject_GC_TRACK(ctx);
+    return ctx;
+}
+
+
+static PyContext *
+context_new_from_vars(PyHamtObject *vars)
+{
+    PyContext *ctx = _context_alloc();
+    if (ctx == NULL) {
+        return NULL;
+    }
+
+    Py_INCREF(vars);
+    ctx->ctx_vars = vars;
+
+    _PyObject_GC_TRACK(ctx);
+    return ctx;
+}
+
+
+static inline PyContext *
+context_get(void)
+{
+    PyThreadState *ts = PyThreadState_Get();
+    PyContext *current_ctx = (PyContext *)ts->context;
+    if (current_ctx == NULL) {
+        current_ctx = context_new_empty();
+        if (current_ctx == NULL) {
+            return NULL;
+        }
+        ts->context = (PyObject *)current_ctx;
+    }
+    return current_ctx;
+}
+
+static int
+context_check_key_type(PyObject *key)
+{
+    if (!PyContextVar_CheckExact(key)) {
+        // abort();
+        PyErr_Format(PyExc_TypeError,
+                     "a ContextVar key was expected, got %R", key);
+        return -1;
+    }
+    return 0;
+}
+
+static PyObject *
+context_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
+{
+    if (PyTuple_Size(args) || (kwds != NULL && PyDict_Size(kwds))) {
+        PyErr_SetString(
+            PyExc_TypeError, "Context() does not accept any arguments");
+        return NULL;
+    }
+    return (PyObject *)PyContext_New();
+}
+
+static int
+context_tp_clear(PyContext *self)
+{
+    Py_CLEAR(self->ctx_prev);
+    Py_CLEAR(self->ctx_vars);
+    return 0;
+}
+
+static int
+context_tp_traverse(PyContext *self, visitproc visit, void *arg)
+{
+    Py_VISIT(self->ctx_prev);
+    Py_VISIT(self->ctx_vars);
+    return 0;
+}
+
+static void
+context_tp_dealloc(PyContext *self)
+{
+    _PyObject_GC_UNTRACK(self);
+
+    if (self->ctx_weakreflist != NULL) {
+        PyObject_ClearWeakRefs((PyObject*)self);
+    }
+    (void)context_tp_clear(self);
+
+    if (ctx_freelist_len < CONTEXT_FREELIST_MAXLEN) {
+        ctx_freelist_len++;
+        self->ctx_weakreflist = (PyObject *)ctx_freelist;
+        ctx_freelist = self;
+    }
+    else {
+        Py_TYPE(self)->tp_free(self);
+    }
+}
+
+static PyObject *
+context_tp_iter(PyContext *self)
+{
+    return _PyHamt_NewIterKeys(self->ctx_vars);
+}
+
+static PyObject *
+context_tp_richcompare(PyObject *v, PyObject *w, int op)
+{
+    if (!PyContext_CheckExact(v) || !PyContext_CheckExact(w) ||
+            (op != Py_EQ && op != Py_NE))
+    {
+        Py_RETURN_NOTIMPLEMENTED;
+    }
+
+    int res = _PyHamt_Eq(
+        ((PyContext *)v)->ctx_vars, ((PyContext *)w)->ctx_vars);
+    if (res < 0) {
+        return NULL;
+    }
+
+    if (op == Py_NE) {
+        res = !res;
+    }
+
+    if (res) {
+        Py_RETURN_TRUE;
+    }
+    else {
+        Py_RETURN_FALSE;
+    }
+}
+
+static Py_ssize_t
+context_tp_len(PyContext *self)
+{
+    return _PyHamt_Len(self->ctx_vars);
+}
+
+static PyObject *
+context_tp_subscript(PyContext *self, PyObject *key)
+{
+    if (context_check_key_type(key)) {
+        return NULL;
+    }
+    PyObject *val = NULL;
+    int found = _PyHamt_Find(self->ctx_vars, key, &val);
+    if (found < 0) {
+        return NULL;
+    }
+    if (found == 0) {
+        PyErr_SetObject(PyExc_KeyError, key);
+        return NULL;
+    }
+    Py_INCREF(val);
+    return val;
+}
+
+static int
+context_tp_contains(PyContext *self, PyObject *key)
+{
+    if (context_check_key_type(key)) {
+        return -1;
+    }
+    PyObject *val = NULL;
+    return _PyHamt_Find(self->ctx_vars, key, &val);
+}
+
+
+/*[clinic input]
+_contextvars.Context.get
+    key: object
+    default: object = None
+    /
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_Context_get_impl(PyContext *self, PyObject *key,
+                              PyObject *default_value)
+/*[clinic end generated code: output=0c54aa7664268189 input=8d4c33c8ecd6d769]*/
+{
+    if (context_check_key_type(key)) {
+        return NULL;
+    }
+
+    PyObject *val = NULL;
+    int found = _PyHamt_Find(self->ctx_vars, key, &val);
+    if (found < 0) {
+        return NULL;
+    }
+    if (found == 0) {
+        Py_INCREF(default_value);
+        return default_value;
+    }
+    Py_INCREF(val);
+    return val;
+}
+
+
+/*[clinic input]
+_contextvars.Context.items
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_Context_items_impl(PyContext *self)
+/*[clinic end generated code: output=fa1655c8a08502af input=2d570d1455004979]*/
+{
+    return _PyHamt_NewIterItems(self->ctx_vars);
+}
+
+
+/*[clinic input]
+_contextvars.Context.keys
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_Context_keys_impl(PyContext *self)
+/*[clinic end generated code: output=177227c6b63ec0e2 input=13005e142fbbf37d]*/
+{
+    return _PyHamt_NewIterKeys(self->ctx_vars);
+}
+
+
+/*[clinic input]
+_contextvars.Context.values
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_Context_values_impl(PyContext *self)
+/*[clinic end generated code: output=d286dabfc8db6dde input=c2cbc40a4470e905]*/
+{
+    return _PyHamt_NewIterValues(self->ctx_vars);
+}
+
+
+/*[clinic input]
+_contextvars.Context.copy
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_Context_copy_impl(PyContext *self)
+/*[clinic end generated code: output=30ba8896c4707a15 input=3e3fd72d598653ab]*/
+{
+    return (PyObject *)context_new_from_vars(self->ctx_vars);
+}
+
+
+static PyObject *
+context_run(PyContext *self, PyObject *const *args,
+            Py_ssize_t nargs, PyObject *kwnames)
+{
+    if (nargs < 1) {
+        PyErr_SetString(PyExc_TypeError,
+                        "run() missing 1 required positional argument");
+        return NULL;
+    }
+
+    if (PyContext_Enter(self)) {
+        return NULL;
+    }
+
+    PyObject *call_result = _PyObject_FastCallKeywords(
+        args[0], args + 1, nargs - 1, kwnames);
+
+    if (PyContext_Exit(self)) {
+        return NULL;
+    }
+
+    return call_result;
+}
+
+
+static PyMethodDef PyContext_methods[] = {
+    _CONTEXTVARS_CONTEXT_GET_METHODDEF
+    _CONTEXTVARS_CONTEXT_ITEMS_METHODDEF
+    _CONTEXTVARS_CONTEXT_KEYS_METHODDEF
+    _CONTEXTVARS_CONTEXT_VALUES_METHODDEF
+    _CONTEXTVARS_CONTEXT_COPY_METHODDEF
+    {"run", (PyCFunction)context_run, METH_FASTCALL | METH_KEYWORDS, NULL},
+    {NULL, NULL}
+};
+
+static PySequenceMethods PyContext_as_sequence = {
+    0,                                   /* sq_length */
+    0,                                   /* sq_concat */
+    0,                                   /* sq_repeat */
+    0,                                   /* sq_item */
+    0,                                   /* sq_slice */
+    0,                                   /* sq_ass_item */
+    0,                                   /* sq_ass_slice */
+    (objobjproc)context_tp_contains,     /* sq_contains */
+    0,                                   /* sq_inplace_concat */
+    0,                                   /* sq_inplace_repeat */
+};
+
+static PyMappingMethods PyContext_as_mapping = {
+    (lenfunc)context_tp_len,             /* mp_length */
+    (binaryfunc)context_tp_subscript,    /* mp_subscript */
+};
+
+PyTypeObject PyContext_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "Context",
+    sizeof(PyContext),
+    .tp_methods = PyContext_methods,
+    .tp_as_mapping = &PyContext_as_mapping,
+    .tp_as_sequence = &PyContext_as_sequence,
+    .tp_iter = (getiterfunc)context_tp_iter,
+    .tp_dealloc = (destructor)context_tp_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_richcompare = context_tp_richcompare,
+    .tp_traverse = (traverseproc)context_tp_traverse,
+    .tp_clear = (inquiry)context_tp_clear,
+    .tp_new = context_tp_new,
+    .tp_weaklistoffset = offsetof(PyContext, ctx_weakreflist),
+    .tp_hash = PyObject_HashNotImplemented,
+};
+
+
+/////////////////////////// ContextVar
+
+
+static int
+contextvar_set(PyContextVar *var, PyObject *val)
+{
+    var->var_cached = NULL;
+    PyThreadState *ts = PyThreadState_Get();
+
+    PyContext *ctx = context_get();
+    if (ctx == NULL) {
+        return -1;
+    }
+
+    PyHamtObject *new_vars = _PyHamt_Assoc(
+        ctx->ctx_vars, (PyObject *)var, val);
+    if (new_vars == NULL) {
+        return -1;
+    }
+
+    Py_SETREF(ctx->ctx_vars, new_vars);
+
+    var->var_cached = val;  /* borrow */
+    var->var_cached_tsid = ts->id;
+    var->var_cached_tsver = ts->context_ver;
+    return 0;
+}
+
+static int
+contextvar_del(PyContextVar *var)
+{
+    var->var_cached = NULL;
+
+    PyContext *ctx = context_get();
+    if (ctx == NULL) {
+        return -1;
+    }
+
+    PyHamtObject *vars = ctx->ctx_vars;
+    PyHamtObject *new_vars = _PyHamt_Without(vars, (PyObject *)var);
+    if (new_vars == NULL) {
+        return -1;
+    }
+
+    if (vars == new_vars) {
+        Py_DECREF(new_vars);
+        PyErr_SetObject(PyExc_LookupError, (PyObject *)var);
+        return -1;
+    }
+
+    Py_SETREF(ctx->ctx_vars, new_vars);
+    return 0;
+}
+
+static Py_hash_t
+contextvar_generate_hash(void *addr, PyObject *name)
+{
+    /* Take hash of `name` and XOR it with the object's addr.
+
+       The structure of the tree is encoded in objects' hashes, which
+       means that sufficiently similar hashes would result in tall trees
+       with many Collision nodes.  Which would, in turn, result in slower
+       get and set operations.
+
+       The XORing helps to ensure that:
+
+       (1) sequentially allocated ContextVar objects have
+           different hashes;
+
+       (2) context variables with equal names have
+           different hashes.
+    */
+
+    Py_hash_t name_hash = PyObject_Hash(name);
+    if (name_hash == -1) {
+        return -1;
+    }
+
+    Py_hash_t res = _Py_HashPointer(addr) ^ name_hash;
+    return res == -1 ? -2 : res;
+}
+
+static PyContextVar *
+contextvar_new(PyObject *name, PyObject *def)
+{
+    if (!PyUnicode_Check(name)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "context variable name must be a str");
+        return NULL;
+    }
+
+    PyContextVar *var = PyObject_GC_New(PyContextVar, &PyContextVar_Type);
+    if (var == NULL) {
+        return NULL;
+    }
+
+    var->var_hash = contextvar_generate_hash(var, name);
+    if (var->var_hash == -1) {
+        Py_DECREF(var);
+        return NULL;
+    }
+
+    Py_INCREF(name);
+    var->var_name = name;
+
+    Py_XINCREF(def);
+    var->var_default = def;
+
+    var->var_cached = NULL;
+    var->var_cached_tsid = 0;
+    var->var_cached_tsver = 0;
+
+    if (_PyObject_GC_IS_TRACKED(name) ||
+            (def != NULL && _PyObject_GC_IS_TRACKED(def)))
+    {
+        PyObject_GC_Track(var);
+    }
+    return var;
+}
+
+
+/*[clinic input]
+class _contextvars.ContextVar "PyContextVar *" "&PyContextVar_Type"
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=445da935fa8883c3]*/
+
+
+static PyObject *
+contextvar_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
+{
+    static char *kwlist[] = {"", "default", NULL};
+    PyObject *name;
+    PyObject *def = NULL;
+
+    if (!PyArg_ParseTupleAndKeywords(
+            args, kwds, "O|$O:ContextVar", kwlist, &name, &def))
+    {
+        return NULL;
+    }
+
+    return (PyObject *)contextvar_new(name, def);
+}
+
+static int
+contextvar_tp_clear(PyContextVar *self)
+{
+    Py_CLEAR(self->var_name);
+    Py_CLEAR(self->var_default);
+    self->var_cached = NULL;
+    self->var_cached_tsid = 0;
+    self->var_cached_tsver = 0;
+    return 0;
+}
+
+static int
+contextvar_tp_traverse(PyContextVar *self, visitproc visit, void *arg)
+{
+    Py_VISIT(self->var_name);
+    Py_VISIT(self->var_default);
+    return 0;
+}
+
+static void
+contextvar_tp_dealloc(PyContextVar *self)
+{
+    PyObject_GC_UnTrack(self);
+    (void)contextvar_tp_clear(self);
+    Py_TYPE(self)->tp_free(self);
+}
+
+static Py_hash_t
+contextvar_tp_hash(PyContextVar *self)
+{
+    return self->var_hash;
+}
+
+static PyObject *
+contextvar_tp_repr(PyContextVar *self)
+{
+    _PyUnicodeWriter writer;
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_PyUnicodeWriter_WriteASCIIString(
+            &writer, "<ContextVar name=", 17) < 0)
+    {
+        goto error;
+    }
+
+    PyObject *name = PyObject_Repr(self->var_name);
+    if (name == NULL) {
+        goto error;
+    }
+    if (_PyUnicodeWriter_WriteStr(&writer, name) < 0) {
+        Py_DECREF(name);
+        goto error;
+    }
+    Py_DECREF(name);
+
+    if (self->var_default != NULL) {
+        if (_PyUnicodeWriter_WriteASCIIString(&writer, " default=", 9) < 0) {
+            goto error;
+        }
+
+        PyObject *def = PyObject_Repr(self->var_default);
+        if (def == NULL) {
+            goto error;
+        }
+        if (_PyUnicodeWriter_WriteStr(&writer, def) < 0) {
+            Py_DECREF(def);
+            goto error;
+        }
+        Py_DECREF(def);
+    }
+
+    PyObject *addr = PyUnicode_FromFormat(" at %p>", self);
+    if (addr == NULL) {
+        goto error;
+    }
+    if (_PyUnicodeWriter_WriteStr(&writer, addr) < 0) {
+        Py_DECREF(addr);
+        goto error;
+    }
+    Py_DECREF(addr);
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+
+
+/*[clinic input]
+_contextvars.ContextVar.get
+    default: object = NULL
+    /
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_ContextVar_get_impl(PyContextVar *self, PyObject *default_value)
+/*[clinic end generated code: output=0746bd0aa2ced7bf input=8d002b02eebbb247]*/
+{
+    if (!PyContextVar_CheckExact(self)) {
+        PyErr_SetString(
+            PyExc_TypeError, "an instance of ContextVar was expected");
+        return NULL;
+    }
+
+    PyObject *val;
+    if (PyContextVar_Get(self, default_value, &val) < 0) {
+        return NULL;
+    }
+
+    if (val == NULL) {
+        PyErr_SetObject(PyExc_LookupError, (PyObject *)self);
+        return NULL;
+    }
+
+    return val;
+}
+
+/*[clinic input]
+_contextvars.ContextVar.set
+    value: object
+    /
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_ContextVar_set(PyContextVar *self, PyObject *value)
+/*[clinic end generated code: output=446ed5e820d6d60b input=a2d88f57c6d86f7c]*/
+{
+    return (PyObject *)PyContextVar_Set(self, value);
+}
+
+/*[clinic input]
+_contextvars.ContextVar.reset
+    token: object
+    /
+[clinic start generated code]*/
+
+static PyObject *
+_contextvars_ContextVar_reset(PyContextVar *self, PyObject *token)
+/*[clinic end generated code: output=d4ee34d0742d62ee input=4c871b6f1f31a65f]*/
+{
+    if (!PyContextToken_CheckExact(token)) {
+        PyErr_Format(PyExc_TypeError,
+                     "expected an instance of Token, got %R", token);
+        return NULL;
+    }
+
+    if (PyContextVar_Reset(self, (PyContextToken *)token)) {
+        return NULL;
+    }
+
+    Py_RETURN_NONE;
+}
+
+
+static PyObject *
+contextvar_cls_getitem(PyObject *self, PyObject *args)
+{
+    Py_RETURN_NONE;
+}
+
+
+static PyMethodDef PyContextVar_methods[] = {
+    _CONTEXTVARS_CONTEXTVAR_GET_METHODDEF
+    _CONTEXTVARS_CONTEXTVAR_SET_METHODDEF
+    _CONTEXTVARS_CONTEXTVAR_RESET_METHODDEF
+    {"__class_getitem__", contextvar_cls_getitem,
+        METH_VARARGS | METH_STATIC, NULL},
+    {NULL, NULL}
+};
+
+PyTypeObject PyContextVar_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "ContextVar",
+    sizeof(PyContextVar),
+    .tp_methods = PyContextVar_methods,
+    .tp_dealloc = (destructor)contextvar_tp_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_traverse = (traverseproc)contextvar_tp_traverse,
+    .tp_clear = (inquiry)contextvar_tp_clear,
+    .tp_new = contextvar_tp_new,
+    .tp_free = PyObject_GC_Del,
+    .tp_hash = (hashfunc)contextvar_tp_hash,
+    .tp_repr = (reprfunc)contextvar_tp_repr,
+};
+
+
+/////////////////////////// Token
+
+static PyObject * get_token_missing(void);
+
+
+/*[clinic input]
+class _contextvars.Token "PyContextToken *" "&PyContextToken_Type"
+[clinic start generated code]*/
+/*[clinic end generated code: output=da39a3ee5e6b4b0d input=338a5e2db13d3f5b]*/
+
+
+static PyObject *
+token_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
+{
+    PyErr_SetString(PyExc_RuntimeError,
+                    "Tokens can only be created by ContextVars");
+    return NULL;
+}
+
+static int
+token_tp_clear(PyContextToken *self)
+{
+    Py_CLEAR(self->tok_ctx);
+    Py_CLEAR(self->tok_var);
+    Py_CLEAR(self->tok_oldval);
+    return 0;
+}
+
+static int
+token_tp_traverse(PyContextToken *self, visitproc visit, void *arg)
+{
+    Py_VISIT(self->tok_ctx);
+    Py_VISIT(self->tok_var);
+    Py_VISIT(self->tok_oldval);
+    return 0;
+}
+
+static void
+token_tp_dealloc(PyContextToken *self)
+{
+    PyObject_GC_UnTrack(self);
+    (void)token_tp_clear(self);
+    Py_TYPE(self)->tp_free(self);
+}
+
+static PyObject *
+token_tp_repr(PyContextToken *self)
+{
+    _PyUnicodeWriter writer;
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_PyUnicodeWriter_WriteASCIIString(&writer, "<Token", 6) < 0) {
+        goto error;
+    }
+
+    if (self->tok_used) {
+        if (_PyUnicodeWriter_WriteASCIIString(&writer, " used", 5) < 0) {
+            goto error;
+        }
+    }
+
+    if (_PyUnicodeWriter_WriteASCIIString(&writer, " var=", 5) < 0) {
+        goto error;
+    }
+
+    PyObject *var = PyObject_Repr((PyObject *)self->tok_var);
+    if (var == NULL) {
+        goto error;
+    }
+    if (_PyUnicodeWriter_WriteStr(&writer, var) < 0) {
+        Py_DECREF(var);
+        goto error;
+    }
+    Py_DECREF(var);
+
+    PyObject *addr = PyUnicode_FromFormat(" at %p>", self);
+    if (addr == NULL) {
+        goto error;
+    }
+    if (_PyUnicodeWriter_WriteStr(&writer, addr) < 0) {
+        Py_DECREF(addr);
+        goto error;
+    }
+    Py_DECREF(addr);
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+
+static PyObject *
+token_get_var(PyContextToken *self)
+{
+    Py_INCREF(self->tok_var);
+    return (PyObject *)self->tok_var;
+}
+
+static PyObject *
+token_get_old_value(PyContextToken *self)
+{
+    if (self->tok_oldval == NULL) {
+        return get_token_missing();
+    }
+
+    Py_INCREF(self->tok_oldval);
+    return self->tok_oldval;
+}
+
+static PyGetSetDef PyContextTokenType_getsetlist[] = {
+    {"var", (getter)token_get_var, NULL, NULL},
+    {"old_value", (getter)token_get_old_value, NULL, NULL},
+    {NULL}
+};
+
+PyTypeObject PyContextToken_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "Token",
+    sizeof(PyContextToken),
+    .tp_getset = PyContextTokenType_getsetlist,
+    .tp_dealloc = (destructor)token_tp_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_traverse = (traverseproc)token_tp_traverse,
+    .tp_clear = (inquiry)token_tp_clear,
+    .tp_new = token_tp_new,
+    .tp_free = PyObject_GC_Del,
+    .tp_hash = PyObject_HashNotImplemented,
+    .tp_repr = (reprfunc)token_tp_repr,
+};
+
+static PyContextToken *
+token_new(PyContext *ctx, PyContextVar *var, PyObject *val)
+{
+    PyContextToken *tok = PyObject_GC_New(PyContextToken, &PyContextToken_Type);
+    if (tok == NULL) {
+        return NULL;
+    }
+
+    Py_INCREF(ctx);
+    tok->tok_ctx = ctx;
+
+    Py_INCREF(var);
+    tok->tok_var = var;
+
+    Py_XINCREF(val);
+    tok->tok_oldval = val;
+
+    tok->tok_used = 0;
+
+    PyObject_GC_Track(tok);
+    return tok;
+}
+
+
+/////////////////////////// Token.MISSING
+
+
+static PyObject *_token_missing;
+
+
+typedef struct {
+    PyObject_HEAD
+} PyContextTokenMissing;
+
+
+static PyObject *
+context_token_missing_tp_repr(PyObject *self)
+{
+    return PyUnicode_FromString("<Token.MISSING>");
+}
+
+
+PyTypeObject PyContextTokenMissing_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "Token.MISSING",
+    sizeof(PyContextTokenMissing),
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT,
+    .tp_repr = context_token_missing_tp_repr,
+};
+
+
+static PyObject *
+get_token_missing(void)
+{
+    if (_token_missing != NULL) {
+        Py_INCREF(_token_missing);
+        return _token_missing;
+    }
+
+    _token_missing = (PyObject *)PyObject_New(
+        PyContextTokenMissing, &PyContextTokenMissing_Type);
+    if (_token_missing == NULL) {
+        return NULL;
+    }
+
+    Py_INCREF(_token_missing);
+    return _token_missing;
+}
+
+
+///////////////////////////
+
+
+int
+PyContext_ClearFreeList(void)
+{
+    int size = ctx_freelist_len;
+    while (ctx_freelist_len) {
+        PyContext *ctx = ctx_freelist;
+        ctx_freelist = (PyContext *)ctx->ctx_weakreflist;
+        ctx->ctx_weakreflist = NULL;
+        PyObject_GC_Del(ctx);
+        ctx_freelist_len--;
+    }
+    return size;
+}
+
+
+void
+_PyContext_Fini(void)
+{
+    Py_CLEAR(_token_missing);
+    (void)PyContext_ClearFreeList();
+    (void)_PyHamt_Fini();
+}
+
+
+int
+_PyContext_Init(void)
+{
+    if (!_PyHamt_Init()) {
+        return 0;
+    }
+
+    if ((PyType_Ready(&PyContext_Type) < 0) ||
+        (PyType_Ready(&PyContextVar_Type) < 0) ||
+        (PyType_Ready(&PyContextToken_Type) < 0) ||
+        (PyType_Ready(&PyContextTokenMissing_Type) < 0))
+    {
+        return 0;
+    }
+
+    PyObject *missing = get_token_missing();
+    if (PyDict_SetItemString(
+        PyContextToken_Type.tp_dict, "MISSING", missing))
+    {
+        Py_DECREF(missing);
+        return 0;
+    }
+    Py_DECREF(missing);
+
+    return 1;
+}
diff --git a/Python/hamt.c b/Python/hamt.c
new file mode 100644
index 00000000000..8ba5082026f
--- /dev/null
+++ b/Python/hamt.c
@@ -0,0 +1,2982 @@
+#include "Python.h"
+
+#include "structmember.h"
+#include "internal/pystate.h"
+#include "internal/hamt.h"
+
+/* popcnt support in Visual Studio */
+#ifdef _MSC_VER
+#include <intrin.h>
+#endif
+
+/*
+This file provides an implemention of an immutable mapping using the
+Hash Array Mapped Trie (or HAMT) datastructure.
+
+This design allows to have:
+
+1. Efficient copy: immutable mappings can be copied by reference,
+   making it an O(1) operation.
+
+2. Efficient mutations: due to structural sharing, only a portion of
+   the trie needs to be copied when the collection is mutated.  The
+   cost of set/delete operations is O(log N).
+
+3. Efficient lookups: O(log N).
+
+(where N is number of key/value items in the immutable mapping.)
+
+
+HAMT
+====
+
+The core idea of HAMT is that the shape of the trie is encoded into the
+hashes of keys.
+
+Say we want to store a K/V pair in our mapping.  First, we calculate the
+hash of K, let's say it's 19830128, or in binary:
+
+    0b1001011101001010101110000 = 19830128
+
+Now let's partition this bit representation of the hash into blocks of
+5 bits each:
+
+    0b00_00000_10010_11101_00101_01011_10000 = 19830128
+          (6)   (5)   (4)   (3)   (2)   (1)
+
+Each block of 5 bits represents a number betwen 0 and 31.  So if we have
+a tree that consists of nodes, each of which is an array of 32 pointers,
+those 5-bit blocks will encode a position on a single tree level.
+
+For example, storing the key K with hash 19830128, results in the following
+tree structure:
+
+                     (array of 32 pointers)
+                     +---+ -- +----+----+----+ -- +----+
+  root node          | 0 | .. | 15 | 16 | 17 | .. | 31 |   0b10000 = 16 (1)
+  (level 1)          +---+ -- +----+----+----+ -- +----+
+                                      |
+                     +---+ -- +----+----+----+ -- +----+
+  a 2nd level node   | 0 | .. | 10 | 11 | 12 | .. | 31 |   0b01011 = 11 (2)
+                     +---+ -- +----+----+----+ -- +----+
+                                      |
+                     +---+ -- +----+----+----+ -- +----+
+  a 3rd level node   | 0 | .. | 04 | 05 | 06 | .. | 31 |   0b01011 = 5  (3)
+                     +---+ -- +----+----+----+ -- +----+
+                                      |
+                     +---+ -- +----+----+----+----+
+  a 4th level node   | 0 | .. | 04 | 29 | 30 | 31 |        0b11101 = 29 (4)
+                     +---+ -- +----+----+----+----+
+                                      |
+                     +---+ -- +----+----+----+ -- +----+
+  a 5th level node   | 0 | .. | 17 | 18 | 19 | .. | 31 |   0b10010 = 18 (5)
+                     +---+ -- +----+----+----+ -- +----+
+                                      |
+                       +--------------+
+                       |
+                     +---+ -- +----+----+----+ -- +----+
+  a 6th level node   | 0 | .. | 15 | 16 | 17 | .. | 31 |   0b00000 = 0  (6)
+                     +---+ -- +----+----+----+ -- +----+
+                       |
+                       V -- our value (or collision)
+
+To rehash: for a K/V pair, the hash of K encodes where in the tree V will
+be stored.
+
+To optimize memory footprint and handle hash collisions, our implementation
+uses three different types of nodes:
+
+ * A Bitmap node;
+ * An Array node;
+ * A Collision node.
+
+Because we implement an immutable dictionary, our nodes are also
+immutable.  Therefore, when we need to modify a node, we copy it, and
+do that modification to the copy.
+
+
+Array Nodes
+-----------
+
+These nodes are very simple.  Essentially they are arrays of 32 pointers
+we used to illustrate the high-level idea in the previous section.
+
+We use Array nodes only when we need to store more than 16 pointers
+in a single node.
+
+Array nodes do not store key objects or value objects.  They are used
+only as an indirection level - their pointers point to other nodes in
+the tree.
+
+
+Bitmap Node
+-----------
+
+Allocating a new 32-pointers array for every node of our tree would be
+very expensive.  Unless we store millions of keys, most of tree nodes would
+be very sparse.
+
+When we have less than 16 elements in a node, we don't want to use the
+Array node, that would mean that we waste a lot of memory.  Instead,
+we can use bitmap compression and can have just as many pointers
+as we need!
+
+Bitmap nodes consist of two fields:
+
+1. An array of pointers.  If a Bitmap node holds N elements, the
+   array will be of N pointers.
+
+2. A 32bit integer -- a bitmap field.  If an N-th bit is set in the
+   bitmap, it means that the node has an N-th element.
+
+For example, say we need to store a 3 elements sparse array:
+
+   +---+  --  +---+  --  +----+  --  +----+
+   | 0 |  ..  | 4 |  ..  | 11 |  ..  | 17 |
+   +---+  --  +---+  --  +----+  --  +----+
+                |          |           |
+                o1         o2          o3
+
+We allocate a three-pointer Bitmap node.  Its bitmap field will be
+then set to:
+
+   0b_00100_00010_00000_10000 == (1 << 17) | (1 << 11) | (1 << 4)
+
+To check if our Bitmap node has an I-th element we can do:
+
+   bitmap & (1 << I)
+
+
+And here's a formula to calculate a position in our pointer array
+which would correspond to an I-th element:
+
+   popcount(bitmap & ((1 << I) - 1))
+
+
+Let's break it down:
+
+ * `popcount` is a function that returns a number of bits set to 1;
+
+ * `((1 << I) - 1)` is a mask to filter the bitmask to contain bits
+   set to the *right* of our bit.
+
+
+So for our 17, 11, and 4 indexes:
+
+ * bitmap & ((1 << 17) - 1) == 0b100000010000 => 2 bits are set => index is 2.
+
+ * bitmap & ((1 << 11) - 1) == 0b10000 => 1 bit is set => index is 1.
+
+ * bitmap & ((1 << 4) - 1) == 0b0 => 0 bits are set => index is 0.
+
+
+To conclude: Bitmap nodes are just like Array nodes -- they can store
+a number of pointers, but use bitmap compression to eliminate unused
+pointers.
+
+
+Bitmap nodes have two pointers for each item:
+
+  +----+----+----+----+  --  +----+----+
+  | k1 | v1 | k2 | v2 |  ..  | kN | vN |
+  +----+----+----+----+  --  +----+----+
+
+When kI == NULL, vI points to another tree level.
+
+When kI != NULL, the actual key object is stored in kI, and its
+value is stored in vI.
+
+
+Collision Nodes
+---------------
+
+Collision nodes are simple arrays of pointers -- two pointers per
+key/value.  When there's a hash collision, say for k1/v1 and k2/v2
+we have `hash(k1)==hash(k2)`.  Then our collision node will be:
+
+  +----+----+----+----+
+  | k1 | v1 | k2 | v2 |
+  +----+----+----+----+
+
+
+Tree Structure
+--------------
+
+All nodes are PyObjects.
+
+The `PyHamtObject` object has a pointer to the root node (h_root),
+and has a length field (h_count).
+
+High-level functions accept a PyHamtObject object and dispatch to
+lower-level functions depending on what kind of node h_root points to.
+
+
+Operations
+==========
+
+There are three fundamental operations on an immutable dictionary:
+
+1. "o.assoc(k, v)" will return a new immutable dictionary, that will be
+   a copy of "o", but with the "k/v" item set.
+
+   Functions in this file:
+
+        hamt_node_assoc, hamt_node_bitmap_assoc,
+        hamt_node_array_assoc, hamt_node_collision_assoc
+
+   `hamt_node_assoc` function accepts a node object, and calls
+   other functions depending on its actual type.
+
+2. "o.find(k)" will lookup key "k" in "o".
+
+   Functions:
+
+        hamt_node_find, hamt_node_bitmap_find,
+        hamt_node_array_find, hamt_node_collision_find
+
+3. "o.without(k)" will return a new immutable dictionary, that will be
+   a copy of "o", buth without the "k" key.
+
+   Functions:
+
+        hamt_node_without, hamt_node_bitmap_without,
+        hamt_node_array_without, hamt_node_collision_without
+
+
+Further Reading
+===============
+
+1. http://blog.higher-order.net/2009/09/08/understanding-clojures-persistenthashmap-deftwice.html
+
+2. http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashmap-part-ii.html
+
+3. Clojure's PersistentHashMap implementation:
+   https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentHashMap.java
+
+
+Debug
+=====
+
+The HAMT datatype is accessible for testing purposes under the
+`_testcapi` module:
+
+    >>> from _testcapi import hamt
+    >>> h = hamt()
+    >>> h2 = h.set('a', 2)
+    >>> h3 = h2.set('b', 3)
+    >>> list(h3)
+    ['a', 'b']
+
+When CPython is built in debug mode, a '__dump__()' method is available
+to introspect the tree:
+
+    >>> print(h3.__dump__())
+    HAMT(len=2):
+        BitmapNode(size=4 count=2 bitmap=0b110 id=0x10eb9d9e8):
+            'a': 2
+            'b': 3
+*/
+
+
+#define IS_ARRAY_NODE(node)     (Py_TYPE(node) == &_PyHamt_ArrayNode_Type)
+#define IS_BITMAP_NODE(node)    (Py_TYPE(node) == &_PyHamt_BitmapNode_Type)
+#define IS_COLLISION_NODE(node) (Py_TYPE(node) == &_PyHamt_CollisionNode_Type)
+
+
+/* Return type for 'find' (lookup a key) functions.
+
+   * F_ERROR - an error occurred;
+   * F_NOT_FOUND - the key was not found;
+   * F_FOUND - the key was found.
+*/
+typedef enum {F_ERROR, F_NOT_FOUND, F_FOUND} hamt_find_t;
+
+
+/* Return type for 'without' (delete a key) functions.
+
+   * W_ERROR - an error occurred;
+   * W_NOT_FOUND - the key was not found: there's nothing to delete;
+   * W_EMPTY - the key was found: the node/tree would be empty
+     if the key is deleted;
+   * W_NEWNODE - the key was found: a new node/tree is returned
+     without that key.
+*/
+typedef enum {W_ERROR, W_NOT_FOUND, W_EMPTY, W_NEWNODE} hamt_without_t;
+
+
+/* Low-level iterator protocol type.
+
+   * I_ITEM - a new item has been yielded;
+   * I_END - the whole tree was visited (similar to StopIteration).
+*/
+typedef enum {I_ITEM, I_END} hamt_iter_t;
+
+
+#define HAMT_ARRAY_NODE_SIZE 32
+
+
+typedef struct {
+    PyObject_HEAD
+    PyHamtNode *a_array[HAMT_ARRAY_NODE_SIZE];
+    Py_ssize_t a_count;
+} PyHamtNode_Array;
+
+
+typedef struct {
+    PyObject_VAR_HEAD
+    uint32_t b_bitmap;
+    PyObject *b_array[1];
+} PyHamtNode_Bitmap;
+
+
+typedef struct {
+    PyObject_VAR_HEAD
+    int32_t c_hash;
+    PyObject *c_array[1];
+} PyHamtNode_Collision;
+
+
+static PyHamtNode_Bitmap *_empty_bitmap_node;
+static PyHamtObject *_empty_hamt;
+
+
+static PyHamtObject *
+hamt_alloc(void);
+
+static PyHamtNode *
+hamt_node_assoc(PyHamtNode *node,
+                uint32_t shift, int32_t hash,
+                PyObject *key, PyObject *val, int* added_leaf);
+
+static hamt_without_t
+hamt_node_without(PyHamtNode *node,
+                  uint32_t shift, int32_t hash,
+                  PyObject *key,
+                  PyHamtNode **new_node);
+
+static hamt_find_t
+hamt_node_find(PyHamtNode *node,
+               uint32_t shift, int32_t hash,
+               PyObject *key, PyObject **val);
+
+#ifdef Py_DEBUG
+static int
+hamt_node_dump(PyHamtNode *node,
+               _PyUnicodeWriter *writer, int level);
+#endif
+
+static PyHamtNode *
+hamt_node_array_new(Py_ssize_t);
+
+static PyHamtNode *
+hamt_node_collision_new(int32_t hash, Py_ssize_t size);
+
+static inline Py_ssize_t
+hamt_node_collision_count(PyHamtNode_Collision *node);
+
+
+#ifdef Py_DEBUG
+static void
+_hamt_node_array_validate(void *o)
+{
+    assert(IS_ARRAY_NODE(o));
+    PyHamtNode_Array *node = (PyHamtNode_Array*)(o);
+    Py_ssize_t i = 0, count = 0;
+    for (; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        if (node->a_array[i] != NULL) {
+            count++;
+        }
+    }
+    assert(count == node->a_count);
+}
+
+#define VALIDATE_ARRAY_NODE(NODE) \
+    do { _hamt_node_array_validate(NODE); } while (0);
+#else
+#define VALIDATE_ARRAY_NODE(NODE)
+#endif
+
+
+/* Returns -1 on error */
+static inline int32_t
+hamt_hash(PyObject *o)
+{
+    Py_hash_t hash = PyObject_Hash(o);
+
+#if SIZEOF_PY_HASH_T <= 4
+    return hash;
+#else
+    if (hash == -1) {
+        /* exception */
+        return -1;
+    }
+
+    /* While it's suboptimal to reduce Python's 64 bit hash to
+       32 bits via XOR, it seems that the resulting hash function
+       is good enough (this is also how Long type is hashed in Java.)
+       Storing 10, 100, 1000 Python strings results in a relatively
+       shallow and uniform tree structure.
+
+       Please don't change this hashing algorithm, as there are many
+       tests that test some exact tree shape to cover all code paths.
+    */
+    int32_t xored = (int32_t)(hash & 0xffffffffl) ^ (int32_t)(hash >> 32);
+    return xored == -1 ? -2 : xored;
+#endif
+}
+
+static inline uint32_t
+hamt_mask(int32_t hash, uint32_t shift)
+{
+    return (((uint32_t)hash >> shift) & 0x01f);
+}
+
+static inline uint32_t
+hamt_bitpos(int32_t hash, uint32_t shift)
+{
+    return (uint32_t)1 << hamt_mask(hash, shift);
+}
+
+static inline uint32_t
+hamt_bitcount(uint32_t i)
+{
+#if defined(__GNUC__) && (__GNUC__ > 4)
+    return (uint32_t)__builtin_popcountl(i);
+#elif defined(__clang__) && (__clang_major__ > 3)
+    return (uint32_t)__builtin_popcountl(i);
+#elif defined(_MSC_VER)
+    return (uint32_t)__popcnt(i);
+#else
+    /* https://graphics.stanford.edu/~seander/bithacks.html */
+    i = i - ((i >> 1) & 0x55555555);
+    i = (i & 0x33333333) + ((i >> 2) & 0x33333333);
+    return ((i + (i >> 4) & 0xF0F0F0F) * 0x1010101) >> 24;
+#endif
+}
+
+static inline uint32_t
+hamt_bitindex(uint32_t bitmap, uint32_t bit)
+{
+    return hamt_bitcount(bitmap & (bit - 1));
+}
+
+
+/////////////////////////////////// Dump Helpers
+#ifdef Py_DEBUG
+
+static int
+_hamt_dump_ident(_PyUnicodeWriter *writer, int level)
+{
+    /* Write `'    ' * level` to the `writer` */
+    PyObject *str = NULL;
+    PyObject *num = NULL;
+    PyObject *res = NULL;
+    int ret = -1;
+
+    str = PyUnicode_FromString("    ");
+    if (str == NULL) {
+        goto error;
+    }
+
+    num = PyLong_FromLong((long)level);
+    if (num == NULL) {
+        goto error;
+    }
+
+    res = PyNumber_Multiply(str, num);
+    if (res == NULL) {
+        goto error;
+    }
+
+    ret = _PyUnicodeWriter_WriteStr(writer, res);
+
+error:
+    Py_XDECREF(res);
+    Py_XDECREF(str);
+    Py_XDECREF(num);
+    return ret;
+}
+
+static int
+_hamt_dump_format(_PyUnicodeWriter *writer, const char *format, ...)
+{
+    /* A convenient helper combining _PyUnicodeWriter_WriteStr and
+       PyUnicode_FromFormatV.
+    */
+    PyObject* msg;
+    int ret;
+
+    va_list vargs;
+#ifdef HAVE_STDARG_PROTOTYPES
+    va_start(vargs, format);
+#else
+    va_start(vargs);
+#endif
+    msg = PyUnicode_FromFormatV(format, vargs);
+    va_end(vargs);
+
+    if (msg == NULL) {
+        return -1;
+    }
+
+    ret = _PyUnicodeWriter_WriteStr(writer, msg);
+    Py_DECREF(msg);
+    return ret;
+}
+
+#endif  /* Py_DEBUG */
+/////////////////////////////////// Bitmap Node
+
+
+static PyHamtNode *
+hamt_node_bitmap_new(Py_ssize_t size)
+{
+    /* Create a new bitmap node of size 'size' */
+
+    PyHamtNode_Bitmap *node;
+    Py_ssize_t i;
+
+    assert(size >= 0);
+    assert(size % 2 == 0);
+
+    if (size == 0 && _empty_bitmap_node != NULL) {
+        Py_INCREF(_empty_bitmap_node);
+        return (PyHamtNode *)_empty_bitmap_node;
+    }
+
+    /* No freelist; allocate a new bitmap node */
+    node = PyObject_GC_NewVar(
+        PyHamtNode_Bitmap, &_PyHamt_BitmapNode_Type, size);
+    if (node == NULL) {
+        return NULL;
+    }
+
+    Py_SIZE(node) = size;
+
+    for (i = 0; i < size; i++) {
+        node->b_array[i] = NULL;
+    }
+
+    node->b_bitmap = 0;
+
+    _PyObject_GC_TRACK(node);
+
+    if (size == 0 && _empty_bitmap_node == NULL) {
+        /* Since bitmap nodes are immutable, we can cache the instance
+           for size=0 and reuse it whenever we need an empty bitmap node.
+        */
+        _empty_bitmap_node = node;
+        Py_INCREF(_empty_bitmap_node);
+    }
+
+    return (PyHamtNode *)node;
+}
+
+static inline Py_ssize_t
+hamt_node_bitmap_count(PyHamtNode_Bitmap *node)
+{
+    return Py_SIZE(node) / 2;
+}
+
+static PyHamtNode_Bitmap *
+hamt_node_bitmap_clone(PyHamtNode_Bitmap *node)
+{
+    /* Clone a bitmap node; return a new one with the same child notes. */
+
+    PyHamtNode_Bitmap *clone;
+    Py_ssize_t i;
+
+    clone = (PyHamtNode_Bitmap *)hamt_node_bitmap_new(Py_SIZE(node));
+    if (clone == NULL) {
+        return NULL;
+    }
+
+    for (i = 0; i < Py_SIZE(node); i++) {
+        Py_XINCREF(node->b_array[i]);
+        clone->b_array[i] = node->b_array[i];
+    }
+
+    clone->b_bitmap = node->b_bitmap;
+    return clone;
+}
+
+static PyHamtNode_Bitmap *
+hamt_node_bitmap_clone_without(PyHamtNode_Bitmap *o, uint32_t bit)
+{
+    assert(bit & o->b_bitmap);
+    assert(hamt_node_bitmap_count(o) > 1);
+
+    PyHamtNode_Bitmap *new = (PyHamtNode_Bitmap *)hamt_node_bitmap_new(
+        Py_SIZE(o) - 2);
+    if (new == NULL) {
+        return NULL;
+    }
+
+    uint32_t idx = hamt_bitindex(o->b_bitmap, bit);
+    uint32_t key_idx = 2 * idx;
+    uint32_t val_idx = key_idx + 1;
+    uint32_t i;
+
+    for (i = 0; i < key_idx; i++) {
+        Py_XINCREF(o->b_array[i]);
+        new->b_array[i] = o->b_array[i];
+    }
+
+    for (i = val_idx + 1; i < Py_SIZE(o); i++) {
+        Py_XINCREF(o->b_array[i]);
+        new->b_array[i - 2] = o->b_array[i];
+    }
+
+    new->b_bitmap = o->b_bitmap & ~bit;
+    return new;
+}
+
+static PyHamtNode *
+hamt_node_new_bitmap_or_collision(uint32_t shift,
+                                  PyObject *key1, PyObject *val1,
+                                  int32_t key2_hash,
+                                  PyObject *key2, PyObject *val2)
+{
+    /* Helper method.  Creates a new node for key1/val and key2/val2
+       pairs.
+
+       If key1 hash is equal to the hash of key2, a Collision node
+       will be created.  If they are not equal, a Bitmap node is
+       created.
+    */
+
+    int32_t key1_hash = hamt_hash(key1);
+    if (key1_hash == -1) {
+        return NULL;
+    }
+
+    if (key1_hash == key2_hash) {
+        PyHamtNode_Collision *n;
+        n = (PyHamtNode_Collision *)hamt_node_collision_new(key1_hash, 4);
+        if (n == NULL) {
+            return NULL;
+        }
+
+        Py_INCREF(key1);
+        n->c_array[0] = key1;
+        Py_INCREF(val1);
+        n->c_array[1] = val1;
+
+        Py_INCREF(key2);
+        n->c_array[2] = key2;
+        Py_INCREF(val2);
+        n->c_array[3] = val2;
+
+        return (PyHamtNode *)n;
+    }
+    else {
+        int added_leaf = 0;
+        PyHamtNode *n = hamt_node_bitmap_new(0);
+        if (n == NULL) {
+            return NULL;
+        }
+
+        PyHamtNode *n2 = hamt_node_assoc(
+            n, shift, key1_hash, key1, val1, &added_leaf);
+        Py_DECREF(n);
+        if (n2 == NULL) {
+            return NULL;
+        }
+
+        n = hamt_node_assoc(n2, shift, key2_hash, key2, val2, &added_leaf);
+        Py_DECREF(n2);
+        if (n == NULL) {
+            return NULL;
+        }
+
+        return n;
+    }
+}
+
+static PyHamtNode *
+hamt_node_bitmap_assoc(PyHamtNode_Bitmap *self,
+                       uint32_t shift, int32_t hash,
+                       PyObject *key, PyObject *val, int* added_leaf)
+{
+    /* assoc operation for bitmap nodes.
+
+       Return: a new node, or self if key/val already is in the
+       collection.
+
+       'added_leaf' is later used in '_PyHamt_Assoc' to determine if
+       `hamt.set(key, val)` increased the size of the collection.
+    */
+
+    uint32_t bit = hamt_bitpos(hash, shift);
+    uint32_t idx = hamt_bitindex(self->b_bitmap, bit);
+
+    /* Bitmap node layout:
+
+    +------+------+------+------+  ---  +------+------+
+    | key1 | val1 | key2 | val2 |  ...  | keyN | valN |
+    +------+------+------+------+  ---  +------+------+
+    where `N < Py_SIZE(node)`.
+
+    The `node->b_bitmap` field is a bitmap.  For a given
+    `(shift, hash)` pair we can determine:
+
+     - If this node has the corresponding key/val slots.
+     - The index of key/val slots.
+    */
+
+    if (self->b_bitmap & bit) {
+        /* The key is set in this node */
+
+        uint32_t key_idx = 2 * idx;
+        uint32_t val_idx = key_idx + 1;
+
+        assert(val_idx < Py_SIZE(self));
+
+        PyObject *key_or_null = self->b_array[key_idx];
+        PyObject *val_or_node = self->b_array[val_idx];
+
+        if (key_or_null == NULL) {
+            /* key is NULL.  This means that we have a few keys
+               that have the same (hash, shift) pair. */
+
+            assert(val_or_node != NULL);
+
+            PyHamtNode *sub_node = hamt_node_assoc(
+                (PyHamtNode *)val_or_node,
+                shift + 5, hash, key, val, added_leaf);
+            if (sub_node == NULL) {
+                return NULL;
+            }
+
+            if (val_or_node == (PyObject *)sub_node) {
+                Py_DECREF(sub_node);
+                Py_INCREF(self);
+                return (PyHamtNode *)self;
+            }
+
+            PyHamtNode_Bitmap *ret = hamt_node_bitmap_clone(self);
+            if (ret == NULL) {
+                return NULL;
+            }
+            Py_SETREF(ret->b_array[val_idx], (PyObject*)sub_node);
+            return (PyHamtNode *)ret;
+        }
+
+        assert(key != NULL);
+        /* key is not NULL.  This means that we have only one other
+           key in this collection that matches our hash for this shift. */
+
+        int comp_err = PyObject_RichCompareBool(key, key_or_null, Py_EQ);
+        if (comp_err < 0) {  /* exception in __eq__ */
+            return NULL;
+        }
+        if (comp_err == 1) {  /* key == key_or_null */
+            if (val == val_or_node) {
+                /* we already have the same key/val pair; return self. */
+                Py_INCREF(self);
+                return (PyHamtNode *)self;
+            }
+
+            /* We're setting a new value for the key we had before.
+               Make a new bitmap node with a replaced value, and return it. */
+            PyHamtNode_Bitmap *ret = hamt_node_bitmap_clone(self);
+            if (ret == NULL) {
+                return NULL;
+            }
+            Py_INCREF(val);
+            Py_SETREF(ret->b_array[val_idx], val);
+            return (PyHamtNode *)ret;
+        }
+
+        /* It's a new key, and it has the same index as *one* another key.
+           We have a collision.  We need to create a new node which will
+           combine the existing key and the key we're adding.
+
+           `hamt_node_new_bitmap_or_collision` will either create a new
+           Collision node if the keys have identical hashes, or
+           a new Bitmap node.
+        */
+        PyHamtNode *sub_node = hamt_node_new_bitmap_or_collision(
+            shift + 5,
+            key_or_null, val_or_node,  /* existing key/val */
+            hash,
+            key, val  /* new key/val */
+        );
+        if (sub_node == NULL) {
+            return NULL;
+        }
+
+        PyHamtNode_Bitmap *ret = hamt_node_bitmap_clone(self);
+        if (ret == NULL) {
+            Py_DECREF(sub_node);
+            return NULL;
+        }
+        Py_SETREF(ret->b_array[key_idx], NULL);
+        Py_SETREF(ret->b_array[val_idx], (PyObject *)sub_node);
+
+        *added_leaf = 1;
+        return (PyHamtNode *)ret;
+    }
+    else {
+        /* There was no key before with the same (shift,hash). */
+
+        uint32_t n = hamt_bitcount(self->b_bitmap);
+
+        if (n >= 16) {
+            /* When we have a situation where we want to store more
+               than 16 nodes at one level of the tree, we no longer
+               want to use the Bitmap node with bitmap encoding.
+
+               Instead we start using an Array node, which has
+               simpler (faster) implementation at the expense of
+               having prealocated 32 pointers for its keys/values
+               pairs.
+
+               Small hamt objects (<30 keys) usually don't have any
+               Array nodes at all.  Betwen ~30 and ~400 keys hamt
+               objects usually have one Array node, and usually it's
+               a root node.
+            */
+
+            uint32_t jdx = hamt_mask(hash, shift);
+            /* 'jdx' is the index of where the new key should be added
+               in the new Array node we're about to create. */
+
+            PyHamtNode *empty = NULL;
+            PyHamtNode_Array *new_node = NULL;
+            PyHamtNode *res = NULL;
+
+            /* Create a new Array node. */
+            new_node = (PyHamtNode_Array *)hamt_node_array_new(n + 1);
+            if (new_node == NULL) {
+                goto fin;
+            }
+
+            /* Create an empty bitmap node for the next
+               hamt_node_assoc call. */
+            empty = hamt_node_bitmap_new(0);
+            if (empty == NULL) {
+                goto fin;
+            }
+
+            /* Make a new bitmap node for the key/val we're adding.
+               Set that bitmap node to new-array-node[jdx]. */
+            new_node->a_array[jdx] = hamt_node_assoc(
+                empty, shift + 5, hash, key, val, added_leaf);
+            if (new_node->a_array[jdx] == NULL) {
+                goto fin;
+            }
+
+            /* Copy existing key/value pairs from the current Bitmap
+               node to the new Array node we've just created. */
+            Py_ssize_t i, j;
+            for (i = 0, j = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+                if (((self->b_bitmap >> i) & 1) != 0) {
+                    /* Ensure we don't accidentally override `jdx` element
+                       we set few lines above.
+                    */
+                    assert(new_node->a_array[i] == NULL);
+
+                    if (self->b_array[j] == NULL) {
+                        new_node->a_array[i] =
+                            (PyHamtNode *)self->b_array[j + 1];
+                        Py_INCREF(new_node->a_array[i]);
+                    }
+                    else {
+                        int32_t rehash = hamt_hash(self->b_array[j]);
+                        if (rehash == -1) {
+                            goto fin;
+                        }
+
+                        new_node->a_array[i] = hamt_node_assoc(
+                            empty, shift + 5,
+                            rehash,
+                            self->b_array[j],
+                            self->b_array[j + 1],
+                            added_leaf);
+
+                        if (new_node->a_array[i] == NULL) {
+                            goto fin;
+                        }
+                    }
+                    j += 2;
+                }
+            }
+
+            VALIDATE_ARRAY_NODE(new_node)
+
+            /* That's it! */
+            res = (PyHamtNode *)new_node;
+
+        fin:
+            Py_XDECREF(empty);
+            if (res == NULL) {
+                Py_XDECREF(new_node);
+            }
+            return res;
+        }
+        else {
+            /* We have less than 16 keys at this level; let's just
+               create a new bitmap node out of this node with the
+               new key/val pair added. */
+
+            uint32_t key_idx = 2 * idx;
+            uint32_t val_idx = key_idx + 1;
+            Py_ssize_t i;
+
+            *added_leaf = 1;
+
+            /* Allocate new Bitmap node which can have one more key/val
+               pair in addition to what we have already. */
+            PyHamtNode_Bitmap *new_node =
+                (PyHamtNode_Bitmap *)hamt_node_bitmap_new(2 * (n + 1));
+            if (new_node == NULL) {
+                return NULL;
+            }
+
+            /* Copy all keys/values that will be before the new key/value
+               we are adding. */
+            for (i = 0; i < key_idx; i++) {
+                Py_XINCREF(self->b_array[i]);
+                new_node->b_array[i] = self->b_array[i];
+            }
+
+            /* Set the new key/value to the new Bitmap node. */
+            Py_INCREF(key);
+            new_node->b_array[key_idx] = key;
+            Py_INCREF(val);
+            new_node->b_array[val_idx] = val;
+
+            /* Copy all keys/values that will be after the new key/value
+               we are adding. */
+            for (i = key_idx; i < Py_SIZE(self); i++) {
+                Py_XINCREF(self->b_array[i]);
+                new_node->b_array[i + 2] = self->b_array[i];
+            }
+
+            new_node->b_bitmap = self->b_bitmap | bit;
+            return (PyHamtNode *)new_node;
+        }
+    }
+}
+
+static hamt_without_t
+hamt_node_bitmap_without(PyHamtNode_Bitmap *self,
+                         uint32_t shift, int32_t hash,
+                         PyObject *key,
+                         PyHamtNode **new_node)
+{
+    uint32_t bit = hamt_bitpos(hash, shift);
+    if ((self->b_bitmap & bit) == 0) {
+        return W_NOT_FOUND;
+    }
+
+    uint32_t idx = hamt_bitindex(self->b_bitmap, bit);
+
+    uint32_t key_idx = 2 * idx;
+    uint32_t val_idx = key_idx + 1;
+
+    PyObject *key_or_null = self->b_array[key_idx];
+    PyObject *val_or_node = self->b_array[val_idx];
+
+    if (key_or_null == NULL) {
+        /* key == NULL means that 'value' is another tree node. */
+
+        PyHamtNode *sub_node = NULL;
+
+        hamt_without_t res = hamt_node_without(
+            (PyHamtNode *)val_or_node,
+            shift + 5, hash, key, &sub_node);
+
+        switch (res) {
+            case W_EMPTY:
+                /* It's impossible for us to receive a W_EMPTY here:
+
+                    - Array nodes are converted to Bitmap nodes when
+                      we delete 16th item from them;
+
+                    - Collision nodes are converted to Bitmap when
+                      there is one item in them;
+
+                    - Bitmap node's without() inlines single-item
+                      sub-nodes.
+
+                   So in no situation we can have a single-item
+                   Bitmap child of another Bitmap node.
+                */
+                Py_UNREACHABLE();
+
+            case W_NEWNODE: {
+                assert(sub_node != NULL);
+
+                if (IS_BITMAP_NODE(sub_node)) {
+                    PyHamtNode_Bitmap *sub_tree = (PyHamtNode_Bitmap *)sub_node;
+                    if (hamt_node_bitmap_count(sub_tree) == 1 &&
+                            sub_tree->b_array[0] != NULL)
+                    {
+                        /* A bitmap node with one key/value pair.  Just
+                           merge it into this node.
+
+                           Note that we don't inline Bitmap nodes that
+                           have a NULL key -- those nodes point to another
+                           tree level, and we cannot simply move tree levels
+                           up or down.
+                        */
+
+                        PyHamtNode_Bitmap *clone = hamt_node_bitmap_clone(self);
+                        if (clone == NULL) {
+                            Py_DECREF(sub_node);
+                            return W_ERROR;
+                        }
+
+                        PyObject *key = sub_tree->b_array[0];
+                        PyObject *val = sub_tree->b_array[1];
+
+                        Py_INCREF(key);
+                        Py_XSETREF(clone->b_array[key_idx], key);
+                        Py_INCREF(val);
+                        Py_SETREF(clone->b_array[val_idx], val);
+
+                        Py_DECREF(sub_tree);
+
+                        *new_node = (PyHamtNode *)clone;
+                        return W_NEWNODE;
+                    }
+                }
+
+#ifdef Py_DEBUG
+                /* Ensure that Collision.without implementation
+                   converts to Bitmap nodes itself.
+                */
+                if (IS_COLLISION_NODE(sub_node)) {
+                    assert(hamt_node_collision_count(
+                            (PyHamtNode_Collision*)sub_node) > 1);
+                }
+#endif
+
+                PyHamtNode_Bitmap *clone = hamt_node_bitmap_clone(self);
+                Py_SETREF(clone->b_array[val_idx],
+                          (PyObject *)sub_node);  /* borrow */
+
+                *new_node = (PyHamtNode *)clone;
+                return W_NEWNODE;
+            }
+
+            case W_ERROR:
+            case W_NOT_FOUND:
+                assert(sub_node == NULL);
+                return res;
+
+            default:
+                Py_UNREACHABLE();
+        }
+    }
+    else {
+        /* We have a regular key/value pair */
+
+        int cmp = PyObject_RichCompareBool(key_or_null, key, Py_EQ);
+        if (cmp < 0) {
+            return W_ERROR;
+        }
+        if (cmp == 0) {
+            return W_NOT_FOUND;
+        }
+
+        if (hamt_node_bitmap_count(self) == 1) {
+            return W_EMPTY;
+        }
+
+        *new_node = (PyHamtNode *)
+            hamt_node_bitmap_clone_without(self, bit);
+        if (*new_node == NULL) {
+            return W_ERROR;
+        }
+
+        return W_NEWNODE;
+    }
+}
+
+static hamt_find_t
+hamt_node_bitmap_find(PyHamtNode_Bitmap *self,
+                      uint32_t shift, int32_t hash,
+                      PyObject *key, PyObject **val)
+{
+    /* Lookup a key in a Bitmap node. */
+
+    uint32_t bit = hamt_bitpos(hash, shift);
+    uint32_t idx;
+    uint32_t key_idx;
+    uint32_t val_idx;
+    PyObject *key_or_null;
+    PyObject *val_or_node;
+    int comp_err;
+
+    if ((self->b_bitmap & bit) == 0) {
+        return F_NOT_FOUND;
+    }
+
+    idx = hamt_bitindex(self->b_bitmap, bit);
+    assert(idx >= 0);
+    key_idx = idx * 2;
+    val_idx = key_idx + 1;
+
+    assert(val_idx < Py_SIZE(self));
+
+    key_or_null = self->b_array[key_idx];
+    val_or_node = self->b_array[val_idx];
+
+    if (key_or_null == NULL) {
+        /* There are a few keys that have the same hash at the current shift
+           that match our key.  Dispatch the lookup further down the tree. */
+        assert(val_or_node != NULL);
+        return hamt_node_find((PyHamtNode *)val_or_node,
+                              shift + 5, hash, key, val);
+    }
+
+    /* We have only one key -- a potential match.  Let's compare if the
+       key we are looking at is equal to the key we are looking for. */
+    assert(key != NULL);
+    comp_err = PyObject_RichCompareBool(key, key_or_null, Py_EQ);
+    if (comp_err < 0) {  /* exception in __eq__ */
+        return F_ERROR;
+    }
+    if (comp_err == 1) {  /* key == key_or_null */
+        *val = val_or_node;
+        return F_FOUND;
+    }
+
+    return F_NOT_FOUND;
+}
+
+static int
+hamt_node_bitmap_traverse(PyHamtNode_Bitmap *self, visitproc visit, void *arg)
+{
+    /* Bitmap's tp_traverse */
+
+    Py_ssize_t i;
+
+    for (i = Py_SIZE(self); --i >= 0; ) {
+        Py_VISIT(self->b_array[i]);
+    }
+
+    return 0;
+}
+
+static void
+hamt_node_bitmap_dealloc(PyHamtNode_Bitmap *self)
+{
+    /* Bitmap's tp_dealloc */
+
+    Py_ssize_t len = Py_SIZE(self);
+    Py_ssize_t i;
+
+    PyObject_GC_UnTrack(self);
+    Py_TRASHCAN_SAFE_BEGIN(self)
+
+    if (len > 0) {
+        i = len;
+        while (--i >= 0) {
+            Py_XDECREF(self->b_array[i]);
+        }
+    }
+
+    Py_TYPE(self)->tp_free((PyObject *)self);
+    Py_TRASHCAN_SAFE_END(self)
+}
+
+#ifdef Py_DEBUG
+static int
+hamt_node_bitmap_dump(PyHamtNode_Bitmap *node,
+                      _PyUnicodeWriter *writer, int level)
+{
+    /* Debug build: __dump__() method implementation for Bitmap nodes. */
+
+    Py_ssize_t i;
+    PyObject *tmp1;
+    PyObject *tmp2;
+
+    if (_hamt_dump_ident(writer, level + 1)) {
+        goto error;
+    }
+
+    if (_hamt_dump_format(writer, "BitmapNode(size=%zd count=%zd ",
+                          Py_SIZE(node), Py_SIZE(node) / 2))
+    {
+        goto error;
+    }
+
+    tmp1 = PyLong_FromUnsignedLong(node->b_bitmap);
+    if (tmp1 == NULL) {
+        goto error;
+    }
+    tmp2 = _PyLong_Format(tmp1, 2);
+    Py_DECREF(tmp1);
+    if (tmp2 == NULL) {
+        goto error;
+    }
+    if (_hamt_dump_format(writer, "bitmap=%S id=%p):\n", tmp2, node)) {
+        Py_DECREF(tmp2);
+        goto error;
+    }
+    Py_DECREF(tmp2);
+
+    for (i = 0; i < Py_SIZE(node); i += 2) {
+        PyObject *key_or_null = node->b_array[i];
+        PyObject *val_or_node = node->b_array[i + 1];
+
+        if (_hamt_dump_ident(writer, level + 2)) {
+            goto error;
+        }
+
+        if (key_or_null == NULL) {
+            if (_hamt_dump_format(writer, "NULL:\n")) {
+                goto error;
+            }
+
+            if (hamt_node_dump((PyHamtNode *)val_or_node,
+                               writer, level + 2))
+            {
+                goto error;
+            }
+        }
+        else {
+            if (_hamt_dump_format(writer, "%R: %R", key_or_null,
+                                  val_or_node))
+            {
+                goto error;
+            }
+        }
+
+        if (_hamt_dump_format(writer, "\n")) {
+            goto error;
+        }
+    }
+
+    return 0;
+error:
+    return -1;
+}
+#endif  /* Py_DEBUG */
+
+
+/////////////////////////////////// Collision Node
+
+
+static PyHamtNode *
+hamt_node_collision_new(int32_t hash, Py_ssize_t size)
+{
+    /* Create a new Collision node. */
+
+    PyHamtNode_Collision *node;
+    Py_ssize_t i;
+
+    assert(size >= 4);
+    assert(size % 2 == 0);
+
+    node = PyObject_GC_NewVar(
+        PyHamtNode_Collision, &_PyHamt_CollisionNode_Type, size);
+    if (node == NULL) {
+        return NULL;
+    }
+
+    for (i = 0; i < size; i++) {
+        node->c_array[i] = NULL;
+    }
+
+    Py_SIZE(node) = size;
+    node->c_hash = hash;
+
+    _PyObject_GC_TRACK(node);
+
+    return (PyHamtNode *)node;
+}
+
+static hamt_find_t
+hamt_node_collision_find_index(PyHamtNode_Collision *self, PyObject *key,
+                               Py_ssize_t *idx)
+{
+    /* Lookup `key` in the Collision node `self`.  Set the index of the
+       found key to 'idx'. */
+
+    Py_ssize_t i;
+    PyObject *el;
+
+    for (i = 0; i < Py_SIZE(self); i += 2) {
+        el = self->c_array[i];
+
+        assert(el != NULL);
+        int cmp = PyObject_RichCompareBool(key, el, Py_EQ);
+        if (cmp < 0) {
+            return F_ERROR;
+        }
+        if (cmp == 1) {
+            *idx = i;
+            return F_FOUND;
+        }
+    }
+
+    return F_NOT_FOUND;
+}
+
+static PyHamtNode *
+hamt_node_collision_assoc(PyHamtNode_Collision *self,
+                          uint32_t shift, int32_t hash,
+                          PyObject *key, PyObject *val, int* added_leaf)
+{
+    /* Set a new key to this level (currently a Collision node)
+       of the tree. */
+
+    if (hash == self->c_hash) {
+        /* The hash of the 'key' we are adding matches the hash of
+           other keys in this Collision node. */
+
+        Py_ssize_t key_idx = -1;
+        hamt_find_t found;
+        PyHamtNode_Collision *new_node;
+        Py_ssize_t i;
+
+        /* Let's try to lookup the new 'key', maybe we already have it. */
+        found = hamt_node_collision_find_index(self, key, &key_idx);
+        switch (found) {
+            case F_ERROR:
+                /* Exception. */
+                return NULL;
+
+            case F_NOT_FOUND:
+                /* This is a totally new key.  Clone the current node,
+                   add a new key/value to the cloned node. */
+
+                new_node = (PyHamtNode_Collision *)hamt_node_collision_new(
+                    self->c_hash, Py_SIZE(self) + 2);
+                if (new_node == NULL) {
+                    return NULL;
+                }
+
+                for (i = 0; i < Py_SIZE(self); i++) {
+                    Py_INCREF(self->c_array[i]);
+                    new_node->c_array[i] = self->c_array[i];
+                }
+
+                Py_INCREF(key);
+                new_node->c_array[i] = key;
+                Py_INCREF(val);
+                new_node->c_array[i + 1] = val;
+
+                *added_leaf = 1;
+                return (PyHamtNode *)new_node;
+
+            case F_FOUND:
+                /* There's a key which is equal to the key we are adding. */
+
+                assert(key_idx >= 0);
+                assert(key_idx < Py_SIZE(self));
+                Py_ssize_t val_idx = key_idx + 1;
+
+                if (self->c_array[val_idx] == val) {
+                    /* We're setting a key/value pair that's already set. */
+                    Py_INCREF(self);
+                    return (PyHamtNode *)self;
+                }
+
+                /* We need to replace old value for the key
+                   with a new value.  Create a new Collision node.*/
+                new_node = (PyHamtNode_Collision *)hamt_node_collision_new(
+                    self->c_hash, Py_SIZE(self));
+                if (new_node == NULL) {
+                    return NULL;
+                }
+
+                /* Copy all elements of the old node to the new one. */
+                for (i = 0; i < Py_SIZE(self); i++) {
+                    Py_INCREF(self->c_array[i]);
+                    new_node->c_array[i] = self->c_array[i];
+                }
+
+                /* Replace the old value with the new value for the our key. */
+                Py_DECREF(new_node->c_array[val_idx]);
+                Py_INCREF(val);
+                new_node->c_array[val_idx] = val;
+
+                return (PyHamtNode *)new_node;
+
+            default:
+                Py_UNREACHABLE();
+        }
+    }
+    else {
+        /* The hash of the new key is different from the hash that
+           all keys of this Collision node have.
+
+           Create a Bitmap node inplace with two children:
+           key/value pair that we're adding, and the Collision node
+           we're replacing on this tree level.
+        */
+
+        PyHamtNode_Bitmap *new_node;
+        PyHamtNode *assoc_res;
+
+        new_node = (PyHamtNode_Bitmap *)hamt_node_bitmap_new(2);
+        if (new_node == NULL) {
+            return NULL;
+        }
+        new_node->b_bitmap = hamt_bitpos(self->c_hash, shift);
+        Py_INCREF(self);
+        new_node->b_array[1] = (PyObject*) self;
+
+        assoc_res = hamt_node_bitmap_assoc(
+            new_node, shift, hash, key, val, added_leaf);
+        Py_DECREF(new_node);
+        return assoc_res;
+    }
+}
+
+static inline Py_ssize_t
+hamt_node_collision_count(PyHamtNode_Collision *node)
+{
+    return Py_SIZE(node) / 2;
+}
+
+static hamt_without_t
+hamt_node_collision_without(PyHamtNode_Collision *self,
+                            uint32_t shift, int32_t hash,
+                            PyObject *key,
+                            PyHamtNode **new_node)
+{
+    if (hash != self->c_hash) {
+        return W_NOT_FOUND;
+    }
+
+    Py_ssize_t key_idx = -1;
+    hamt_find_t found = hamt_node_collision_find_index(self, key, &key_idx);
+
+    switch (found) {
+        case F_ERROR:
+            return W_ERROR;
+
+        case F_NOT_FOUND:
+            return W_NOT_FOUND;
+
+        case F_FOUND:
+            assert(key_idx >= 0);
+            assert(key_idx < Py_SIZE(self));
+
+            Py_ssize_t new_count = hamt_node_collision_count(self) - 1;
+
+            if (new_count == 0) {
+                /* The node has only one key/value pair and it's for the
+                   key we're trying to delete.  So a new node will be empty
+                   after the removal.
+                */
+                return W_EMPTY;
+            }
+
+            if (new_count == 1) {
+                /* The node has two keys, and after deletion the
+                   new Collision node would have one.  Collision nodes
+                   with one key shouldn't exist, co convert it to a
+                   Bitmap node.
+                */
+                PyHamtNode_Bitmap *node = (PyHamtNode_Bitmap *)
+                    hamt_node_bitmap_new(2);
+                if (node == NULL) {
+                    return W_ERROR;
+                }
+
+                if (key_idx == 0) {
+                    Py_INCREF(self->c_array[2]);
+                    node->b_array[0] = self->c_array[2];
+                    Py_INCREF(self->c_array[3]);
+                    node->b_array[1] = self->c_array[3];
+                }
+                else {
+                    assert(key_idx == 2);
+                    Py_INCREF(self->c_array[0]);
+                    node->b_array[0] = self->c_array[0];
+                    Py_INCREF(self->c_array[1]);
+                    node->b_array[1] = self->c_array[1];
+                }
+
+                node->b_bitmap = hamt_bitpos(hash, shift);
+
+                *new_node = (PyHamtNode *)node;
+                return W_NEWNODE;
+            }
+
+            /* Allocate a new Collision node with capacity for one
+               less key/value pair */
+            PyHamtNode_Collision *new = (PyHamtNode_Collision *)
+                hamt_node_collision_new(
+                    self->c_hash, Py_SIZE(self) - 2);
+
+            /* Copy all other keys from `self` to `new` */
+            Py_ssize_t i;
+            for (i = 0; i < key_idx; i++) {
+                Py_INCREF(self->c_array[i]);
+                new->c_array[i] = self->c_array[i];
+            }
+            for (i = key_idx + 2; i < Py_SIZE(self); i++) {
+                Py_INCREF(self->c_array[i]);
+                new->c_array[i - 2] = self->c_array[i];
+            }
+
+            *new_node = (PyHamtNode*)new;
+            return W_NEWNODE;
+
+        default:
+            Py_UNREACHABLE();
+    }
+}
+
+static hamt_find_t
+hamt_node_collision_find(PyHamtNode_Collision *self,
+                         uint32_t shift, int32_t hash,
+                         PyObject *key, PyObject **val)
+{
+    /* Lookup `key` in the Collision node `self`.  Set the value
+       for the found key to 'val'. */
+
+    Py_ssize_t idx = -1;
+    hamt_find_t res;
+
+    res = hamt_node_collision_find_index(self, key, &idx);
+    if (res == F_ERROR || res == F_NOT_FOUND) {
+        return res;
+    }
+
+    assert(idx >= 0);
+    assert(idx + 1 < Py_SIZE(self));
+
+    *val = self->c_array[idx + 1];
+    assert(*val != NULL);
+
+    return F_FOUND;
+}
+
+
+static int
+hamt_node_collision_traverse(PyHamtNode_Collision *self,
+                             visitproc visit, void *arg)
+{
+    /* Collision's tp_traverse */
+
+    Py_ssize_t i;
+
+    for (i = Py_SIZE(self); --i >= 0; ) {
+        Py_VISIT(self->c_array[i]);
+    }
+
+    return 0;
+}
+
+static void
+hamt_node_collision_dealloc(PyHamtNode_Collision *self)
+{
+    /* Collision's tp_dealloc */
+
+    Py_ssize_t len = Py_SIZE(self);
+
+    PyObject_GC_UnTrack(self);
+    Py_TRASHCAN_SAFE_BEGIN(self)
+
+    if (len > 0) {
+
+        while (--len >= 0) {
+            Py_XDECREF(self->c_array[len]);
+        }
+    }
+
+    Py_TYPE(self)->tp_free((PyObject *)self);
+    Py_TRASHCAN_SAFE_END(self)
+}
+
+#ifdef Py_DEBUG
+static int
+hamt_node_collision_dump(PyHamtNode_Collision *node,
+                         _PyUnicodeWriter *writer, int level)
+{
+    /* Debug build: __dump__() method implementation for Collision nodes. */
+
+    Py_ssize_t i;
+
+    if (_hamt_dump_ident(writer, level + 1)) {
+        goto error;
+    }
+
+    if (_hamt_dump_format(writer, "CollisionNode(size=%zd id=%p):\n",
+                          Py_SIZE(node), node))
+    {
+        goto error;
+    }
+
+    for (i = 0; i < Py_SIZE(node); i += 2) {
+        PyObject *key = node->c_array[i];
+        PyObject *val = node->c_array[i + 1];
+
+        if (_hamt_dump_ident(writer, level + 2)) {
+            goto error;
+        }
+
+        if (_hamt_dump_format(writer, "%R: %R\n", key, val)) {
+            goto error;
+        }
+    }
+
+    return 0;
+error:
+    return -1;
+}
+#endif  /* Py_DEBUG */
+
+
+/////////////////////////////////// Array Node
+
+
+static PyHamtNode *
+hamt_node_array_new(Py_ssize_t count)
+{
+    Py_ssize_t i;
+
+    PyHamtNode_Array *node = PyObject_GC_New(
+        PyHamtNode_Array, &_PyHamt_ArrayNode_Type);
+    if (node == NULL) {
+        return NULL;
+    }
+
+    for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        node->a_array[i] = NULL;
+    }
+
+    node->a_count = count;
+
+    _PyObject_GC_TRACK(node);
+    return (PyHamtNode *)node;
+}
+
+static PyHamtNode_Array *
+hamt_node_array_clone(PyHamtNode_Array *node)
+{
+    PyHamtNode_Array *clone;
+    Py_ssize_t i;
+
+    VALIDATE_ARRAY_NODE(node)
+
+    /* Create a new Array node. */
+    clone = (PyHamtNode_Array *)hamt_node_array_new(node->a_count);
+    if (clone == NULL) {
+        return NULL;
+    }
+
+    /* Copy all elements from the current Array node to the new one. */
+    for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        Py_XINCREF(node->a_array[i]);
+        clone->a_array[i] = node->a_array[i];
+    }
+
+    VALIDATE_ARRAY_NODE(clone)
+    return clone;
+}
+
+static PyHamtNode *
+hamt_node_array_assoc(PyHamtNode_Array *self,
+                      uint32_t shift, int32_t hash,
+                      PyObject *key, PyObject *val, int* added_leaf)
+{
+    /* Set a new key to this level (currently a Collision node)
+       of the tree.
+
+       Array nodes don't store values, they can only point to
+       other nodes.  They are simple arrays of 32 BaseNode pointers/
+     */
+
+    uint32_t idx = hamt_mask(hash, shift);
+    PyHamtNode *node = self->a_array[idx];
+    PyHamtNode *child_node;
+    PyHamtNode_Array *new_node;
+    Py_ssize_t i;
+
+    if (node == NULL) {
+        /* There's no child node for the given hash.  Create a new
+           Bitmap node for this key. */
+
+        PyHamtNode_Bitmap *empty = NULL;
+
+        /* Get an empty Bitmap node to work with. */
+        empty = (PyHamtNode_Bitmap *)hamt_node_bitmap_new(0);
+        if (empty == NULL) {
+            return NULL;
+        }
+
+        /* Set key/val to the newly created empty Bitmap, thus
+           creating a new Bitmap node with our key/value pair. */
+        child_node = hamt_node_bitmap_assoc(
+            empty,
+            shift + 5, hash, key, val, added_leaf);
+        Py_DECREF(empty);
+        if (child_node == NULL) {
+            return NULL;
+        }
+
+        /* Create a new Array node. */
+        new_node = (PyHamtNode_Array *)hamt_node_array_new(self->a_count + 1);
+        if (new_node == NULL) {
+            Py_DECREF(child_node);
+            return NULL;
+        }
+
+        /* Copy all elements from the current Array node to the
+           new one. */
+        for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+            Py_XINCREF(self->a_array[i]);
+            new_node->a_array[i] = self->a_array[i];
+        }
+
+        assert(new_node->a_array[idx] == NULL);
+        new_node->a_array[idx] = child_node;  /* borrow */
+        VALIDATE_ARRAY_NODE(new_node)
+    }
+    else {
+        /* There's a child node for the given hash.
+           Set the key to it./ */
+        child_node = hamt_node_assoc(
+            node, shift + 5, hash, key, val, added_leaf);
+        if (child_node == (PyHamtNode *)self) {
+            Py_DECREF(child_node);
+            return (PyHamtNode *)self;
+        }
+
+        new_node = hamt_node_array_clone(self);
+        if (new_node == NULL) {
+            Py_DECREF(child_node);
+            return NULL;
+        }
+
+        Py_SETREF(new_node->a_array[idx], child_node);  /* borrow */
+        VALIDATE_ARRAY_NODE(new_node)
+    }
+
+    return (PyHamtNode *)new_node;
+}
+
+static hamt_without_t
+hamt_node_array_without(PyHamtNode_Array *self,
+                        uint32_t shift, int32_t hash,
+                        PyObject *key,
+                        PyHamtNode **new_node)
+{
+    uint32_t idx = hamt_mask(hash, shift);
+    PyHamtNode *node = self->a_array[idx];
+
+    if (node == NULL) {
+        return W_NOT_FOUND;
+    }
+
+    PyHamtNode *sub_node = NULL;
+    hamt_without_t res = hamt_node_without(
+        (PyHamtNode *)node,
+        shift + 5, hash, key, &sub_node);
+
+    switch (res) {
+        case W_NOT_FOUND:
+        case W_ERROR:
+            assert(sub_node == NULL);
+            return res;
+
+        case W_NEWNODE: {
+            /* We need to replace a node at the `idx` index.
+               Clone this node and replace.
+            */
+            assert(sub_node != NULL);
+
+            PyHamtNode_Array *clone = hamt_node_array_clone(self);
+            if (clone == NULL) {
+                Py_DECREF(sub_node);
+                return W_ERROR;
+            }
+
+            Py_SETREF(clone->a_array[idx], sub_node);  /* borrow */
+            *new_node = (PyHamtNode*)clone;  /* borrow */
+            return W_NEWNODE;
+        }
+
+        case W_EMPTY: {
+            assert(sub_node == NULL);
+            /* We need to remove a node at the `idx` index.
+               Calculate the size of the replacement Array node.
+            */
+            Py_ssize_t new_count = self->a_count - 1;
+
+            if (new_count == 0) {
+                return W_EMPTY;
+            }
+
+            if (new_count >= 16) {
+                /* We convert Bitmap nodes to Array nodes, when a
+                   Bitmap node needs to store more than 15 key/value
+                   pairs.  So we will create a new Array node if we
+                   the number of key/values after deletion is still
+                   greater than 15.
+                */
+
+                PyHamtNode_Array *new = hamt_node_array_clone(self);
+                if (new == NULL) {
+                    return W_ERROR;
+                }
+                new->a_count = new_count;
+                Py_CLEAR(new->a_array[idx]);
+
+                *new_node = (PyHamtNode*)new;  /* borrow */
+                return W_NEWNODE;
+            }
+
+            /* New Array node would have less than 16 key/value
+               pairs.  We need to create a replacement Bitmap node. */
+
+            Py_ssize_t bitmap_size = new_count * 2;
+            uint32_t bitmap = 0;
+
+            PyHamtNode_Bitmap *new = (PyHamtNode_Bitmap *)
+                hamt_node_bitmap_new(bitmap_size);
+            if (new == NULL) {
+                return W_ERROR;
+            }
+
+            Py_ssize_t new_i = 0;
+            for (uint32_t i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+                if (i == idx) {
+                    /* Skip the node we are deleting. */
+                    continue;
+                }
+
+                PyHamtNode *node = self->a_array[i];
+                if (node == NULL) {
+                    /* Skip any missing nodes. */
+                    continue;
+                }
+
+                bitmap |= 1 << i;
+
+                if (IS_BITMAP_NODE(node)) {
+                    PyHamtNode_Bitmap *child = (PyHamtNode_Bitmap *)node;
+
+                    if (hamt_node_bitmap_count(child) == 1 &&
+                            child->b_array[0] != NULL)
+                    {
+                        /* node is a Bitmap with one key/value pair, just
+                           merge it into the new Bitmap node we're building.
+
+                           Note that we don't inline Bitmap nodes that
+                           have a NULL key -- those nodes point to another
+                           tree level, and we cannot simply move tree levels
+                           up or down.
+                        */
+                        PyObject *key = child->b_array[0];
+                        PyObject *val = child->b_array[1];
+
+                        Py_INCREF(key);
+                        new->b_array[new_i] = key;
+                        Py_INCREF(val);
+                        new->b_array[new_i + 1] = val;
+                    }
+                    else {
+                        new->b_array[new_i] = NULL;
+                        Py_INCREF(node);
+                        new->b_array[new_i + 1] = (PyObject*)node;
+                    }
+                }
+                else {
+
+#ifdef Py_DEBUG
+                    if (IS_COLLISION_NODE(node)) {
+                        Py_ssize_t child_count = hamt_node_collision_count(
+                            (PyHamtNode_Collision*)node);
+                        assert(child_count > 1);
+                    }
+                    else if (IS_ARRAY_NODE(node)) {
+                        assert(((PyHamtNode_Array*)node)->a_count >= 16);
+                    }
+#endif
+
+                    /* Just copy the node into our new Bitmap */
+                    new->b_array[new_i] = NULL;
+                    Py_INCREF(node);
+                    new->b_array[new_i + 1] = (PyObject*)node;
+                }
+
+                new_i += 2;
+            }
+
+            new->b_bitmap = bitmap;
+            *new_node = (PyHamtNode*)new;  /* borrow */
+            return W_NEWNODE;
+        }
+
+        default:
+            Py_UNREACHABLE();
+    }
+}
+
+static hamt_find_t
+hamt_node_array_find(PyHamtNode_Array *self,
+                     uint32_t shift, int32_t hash,
+                     PyObject *key, PyObject **val)
+{
+    /* Lookup `key` in the Array node `self`.  Set the value
+       for the found key to 'val'. */
+
+    uint32_t idx = hamt_mask(hash, shift);
+    PyHamtNode *node;
+
+    node = self->a_array[idx];
+    if (node == NULL) {
+        return F_NOT_FOUND;
+    }
+
+    /* Dispatch to the generic hamt_node_find */
+    return hamt_node_find(node, shift + 5, hash, key, val);
+}
+
+static int
+hamt_node_array_traverse(PyHamtNode_Array *self,
+                         visitproc visit, void *arg)
+{
+    /* Array's tp_traverse */
+
+    Py_ssize_t i;
+
+    for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        Py_VISIT(self->a_array[i]);
+    }
+
+    return 0;
+}
+
+static void
+hamt_node_array_dealloc(PyHamtNode_Array *self)
+{
+    /* Array's tp_dealloc */
+
+    Py_ssize_t i;
+
+    PyObject_GC_UnTrack(self);
+    Py_TRASHCAN_SAFE_BEGIN(self)
+
+    for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        Py_XDECREF(self->a_array[i]);
+    }
+
+    Py_TYPE(self)->tp_free((PyObject *)self);
+    Py_TRASHCAN_SAFE_END(self)
+}
+
+#ifdef Py_DEBUG
+static int
+hamt_node_array_dump(PyHamtNode_Array *node,
+                     _PyUnicodeWriter *writer, int level)
+{
+    /* Debug build: __dump__() method implementation for Array nodes. */
+
+    Py_ssize_t i;
+
+    if (_hamt_dump_ident(writer, level + 1)) {
+        goto error;
+    }
+
+    if (_hamt_dump_format(writer, "ArrayNode(id=%p):\n", node)) {
+        goto error;
+    }
+
+    for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        if (node->a_array[i] == NULL) {
+            continue;
+        }
+
+        if (_hamt_dump_ident(writer, level + 2)) {
+            goto error;
+        }
+
+        if (_hamt_dump_format(writer, "%d::\n", i)) {
+            goto error;
+        }
+
+        if (hamt_node_dump(node->a_array[i], writer, level + 1)) {
+            goto error;
+        }
+
+        if (_hamt_dump_format(writer, "\n")) {
+            goto error;
+        }
+    }
+
+    return 0;
+error:
+    return -1;
+}
+#endif  /* Py_DEBUG */
+
+
+/////////////////////////////////// Node Dispatch
+
+
+static PyHamtNode *
+hamt_node_assoc(PyHamtNode *node,
+                uint32_t shift, int32_t hash,
+                PyObject *key, PyObject *val, int* added_leaf)
+{
+    /* Set key/value to the 'node' starting with the given shift/hash.
+       Return a new node, or the same node if key/value already
+       set.
+
+       added_leaf will be set to 1 if key/value wasn't in the
+       tree before.
+
+       This method automatically dispatches to the suitable
+       hamt_node_{nodetype}_assoc method.
+    */
+
+    if (IS_BITMAP_NODE(node)) {
+        return hamt_node_bitmap_assoc(
+            (PyHamtNode_Bitmap *)node,
+            shift, hash, key, val, added_leaf);
+    }
+    else if (IS_ARRAY_NODE(node)) {
+        return hamt_node_array_assoc(
+            (PyHamtNode_Array *)node,
+            shift, hash, key, val, added_leaf);
+    }
+    else {
+        assert(IS_COLLISION_NODE(node));
+        return hamt_node_collision_assoc(
+            (PyHamtNode_Collision *)node,
+            shift, hash, key, val, added_leaf);
+    }
+}
+
+static hamt_without_t
+hamt_node_without(PyHamtNode *node,
+                  uint32_t shift, int32_t hash,
+                  PyObject *key,
+                  PyHamtNode **new_node)
+{
+    if (IS_BITMAP_NODE(node)) {
+        return hamt_node_bitmap_without(
+            (PyHamtNode_Bitmap *)node,
+            shift, hash, key,
+            new_node);
+    }
+    else if (IS_ARRAY_NODE(node)) {
+        return hamt_node_array_without(
+            (PyHamtNode_Array *)node,
+            shift, hash, key,
+            new_node);
+    }
+    else {
+        assert(IS_COLLISION_NODE(node));
+        return hamt_node_collision_without(
+            (PyHamtNode_Collision *)node,
+            shift, hash, key,
+            new_node);
+    }
+}
+
+static hamt_find_t
+hamt_node_find(PyHamtNode *node,
+               uint32_t shift, int32_t hash,
+               PyObject *key, PyObject **val)
+{
+    /* Find the key in the node starting with the given shift/hash.
+
+       If a value is found, the result will be set to F_FOUND, and
+       *val will point to the found value object.
+
+       If a value wasn't found, the result will be set to F_NOT_FOUND.
+
+       If an exception occurs during the call, the result will be F_ERROR.
+
+       This method automatically dispatches to the suitable
+       hamt_node_{nodetype}_find method.
+    */
+
+    if (IS_BITMAP_NODE(node)) {
+        return hamt_node_bitmap_find(
+            (PyHamtNode_Bitmap *)node,
+            shift, hash, key, val);
+
+    }
+    else if (IS_ARRAY_NODE(node)) {
+        return hamt_node_array_find(
+            (PyHamtNode_Array *)node,
+            shift, hash, key, val);
+    }
+    else {
+        assert(IS_COLLISION_NODE(node));
+        return hamt_node_collision_find(
+            (PyHamtNode_Collision *)node,
+            shift, hash, key, val);
+    }
+}
+
+#ifdef Py_DEBUG
+static int
+hamt_node_dump(PyHamtNode *node,
+               _PyUnicodeWriter *writer, int level)
+{
+    /* Debug build: __dump__() method implementation for a node.
+
+       This method automatically dispatches to the suitable
+       hamt_node_{nodetype})_dump method.
+    */
+
+    if (IS_BITMAP_NODE(node)) {
+        return hamt_node_bitmap_dump(
+            (PyHamtNode_Bitmap *)node, writer, level);
+    }
+    else if (IS_ARRAY_NODE(node)) {
+        return hamt_node_array_dump(
+            (PyHamtNode_Array *)node, writer, level);
+    }
+    else {
+        assert(IS_COLLISION_NODE(node));
+        return hamt_node_collision_dump(
+            (PyHamtNode_Collision *)node, writer, level);
+    }
+}
+#endif  /* Py_DEBUG */
+
+
+/////////////////////////////////// Iterators: Machinery
+
+
+static hamt_iter_t
+hamt_iterator_next(PyHamtIteratorState *iter, PyObject **key, PyObject **val);
+
+
+static void
+hamt_iterator_init(PyHamtIteratorState *iter, PyHamtNode *root)
+{
+    for (uint32_t i = 0; i < _Py_HAMT_MAX_TREE_DEPTH; i++) {
+        iter->i_nodes[i] = NULL;
+        iter->i_pos[i] = 0;
+    }
+
+    iter->i_level = 0;
+
+    /* Note: we don't incref/decref nodes in i_nodes. */
+    iter->i_nodes[0] = root;
+}
+
+static hamt_iter_t
+hamt_iterator_bitmap_next(PyHamtIteratorState *iter,
+                          PyObject **key, PyObject **val)
+{
+    int8_t level = iter->i_level;
+
+    PyHamtNode_Bitmap *node = (PyHamtNode_Bitmap *)(iter->i_nodes[level]);
+    Py_ssize_t pos = iter->i_pos[level];
+
+    if (pos + 1 >= Py_SIZE(node)) {
+#ifdef Py_DEBUG
+        assert(iter->i_level >= 0);
+        iter->i_nodes[iter->i_level] = NULL;
+#endif
+        iter->i_level--;
+        return hamt_iterator_next(iter, key, val);
+    }
+
+    if (node->b_array[pos] == NULL) {
+        iter->i_pos[level] = pos + 2;
+
+        int8_t next_level = level + 1;
+        assert(next_level < _Py_HAMT_MAX_TREE_DEPTH);
+        iter->i_level = next_level;
+        iter->i_pos[next_level] = 0;
+        iter->i_nodes[next_level] = (PyHamtNode *)
+            node->b_array[pos + 1];
+
+        return hamt_iterator_next(iter, key, val);
+    }
+
+    *key = node->b_array[pos];
+    *val = node->b_array[pos + 1];
+    iter->i_pos[level] = pos + 2;
+    return I_ITEM;
+}
+
+static hamt_iter_t
+hamt_iterator_collision_next(PyHamtIteratorState *iter,
+                             PyObject **key, PyObject **val)
+{
+    int8_t level = iter->i_level;
+
+    PyHamtNode_Collision *node = (PyHamtNode_Collision *)(iter->i_nodes[level]);
+    Py_ssize_t pos = iter->i_pos[level];
+
+    if (pos + 1 >= Py_SIZE(node)) {
+#ifdef Py_DEBUG
+        assert(iter->i_level >= 0);
+        iter->i_nodes[iter->i_level] = NULL;
+#endif
+        iter->i_level--;
+        return hamt_iterator_next(iter, key, val);
+    }
+
+    *key = node->c_array[pos];
+    *val = node->c_array[pos + 1];
+    iter->i_pos[level] = pos + 2;
+    return I_ITEM;
+}
+
+static hamt_iter_t
+hamt_iterator_array_next(PyHamtIteratorState *iter,
+                         PyObject **key, PyObject **val)
+{
+    int8_t level = iter->i_level;
+
+    PyHamtNode_Array *node = (PyHamtNode_Array *)(iter->i_nodes[level]);
+    Py_ssize_t pos = iter->i_pos[level];
+
+    if (pos >= HAMT_ARRAY_NODE_SIZE) {
+#ifdef Py_DEBUG
+        assert(iter->i_level >= 0);
+        iter->i_nodes[iter->i_level] = NULL;
+#endif
+        iter->i_level--;
+        return hamt_iterator_next(iter, key, val);
+    }
+
+    for (Py_ssize_t i = pos; i < HAMT_ARRAY_NODE_SIZE; i++) {
+        if (node->a_array[i] != NULL) {
+            iter->i_pos[level] = i + 1;
+
+            int8_t next_level = level + 1;
+            assert(next_level < _Py_HAMT_MAX_TREE_DEPTH);
+            iter->i_pos[next_level] = 0;
+            iter->i_nodes[next_level] = node->a_array[i];
+            iter->i_level = next_level;
+
+            return hamt_iterator_next(iter, key, val);
+        }
+    }
+
+#ifdef Py_DEBUG
+        assert(iter->i_level >= 0);
+        iter->i_nodes[iter->i_level] = NULL;
+#endif
+
+    iter->i_level--;
+    return hamt_iterator_next(iter, key, val);
+}
+
+static hamt_iter_t
+hamt_iterator_next(PyHamtIteratorState *iter, PyObject **key, PyObject **val)
+{
+    if (iter->i_level < 0) {
+        return I_END;
+    }
+
+    assert(iter->i_level < _Py_HAMT_MAX_TREE_DEPTH);
+
+    PyHamtNode *current = iter->i_nodes[iter->i_level];
+
+    if (IS_BITMAP_NODE(current)) {
+        return hamt_iterator_bitmap_next(iter, key, val);
+    }
+    else if (IS_ARRAY_NODE(current)) {
+        return hamt_iterator_array_next(iter, key, val);
+    }
+    else {
+        assert(IS_COLLISION_NODE(current));
+        return hamt_iterator_collision_next(iter, key, val);
+    }
+}
+
+
+/////////////////////////////////// HAMT high-level functions
+
+
+PyHamtObject *
+_PyHamt_Assoc(PyHamtObject *o, PyObject *key, PyObject *val)
+{
+    int32_t key_hash;
+    int added_leaf = 0;
+    PyHamtNode *new_root;
+    PyHamtObject *new_o;
+
+    key_hash = hamt_hash(key);
+    if (key_hash == -1) {
+        return NULL;
+    }
+
+    new_root = hamt_node_assoc(
+        (PyHamtNode *)(o->h_root),
+        0, key_hash, key, val, &added_leaf);
+    if (new_root == NULL) {
+        return NULL;
+    }
+
+    if (new_root == o->h_root) {
+        Py_DECREF(new_root);
+        Py_INCREF(o);
+        return o;
+    }
+
+    new_o = hamt_alloc();
+    if (new_o == NULL) {
+        Py_DECREF(new_root);
+        return NULL;
+    }
+
+    new_o->h_root = new_root;  /* borrow */
+    new_o->h_count = added_leaf ? o->h_count + 1 : o->h_count;
+
+    return new_o;
+}
+
+PyHamtObject *
+_PyHamt_Without(PyHamtObject *o, PyObject *key)
+{
+    int32_t key_hash = hamt_hash(key);
+    if (key_hash == -1) {
+        return NULL;
+    }
+
+    PyHamtNode *new_root;
+
+    hamt_without_t res = hamt_node_without(
+        (PyHamtNode *)(o->h_root),
+        0, key_hash, key,
+        &new_root);
+
+    switch (res) {
+        case W_ERROR:
+            return NULL;
+        case W_EMPTY:
+            return _PyHamt_New();
+        case W_NOT_FOUND:
+            Py_INCREF(o);
+            return o;
+        case W_NEWNODE: {
+            PyHamtObject *new_o = hamt_alloc();
+            if (new_o == NULL) {
+                Py_DECREF(new_root);
+                return NULL;
+            }
+
+            new_o->h_root = new_root;  /* borrow */
+            new_o->h_count = o->h_count - 1;
+            assert(new_o->h_count >= 0);
+            return new_o;
+        }
+        default:
+            Py_UNREACHABLE();
+    }
+}
+
+static hamt_find_t
+hamt_find(PyHamtObject *o, PyObject *key, PyObject **val)
+{
+    if (o->h_count == 0) {
+        return F_NOT_FOUND;
+    }
+
+    int32_t key_hash = hamt_hash(key);
+    if (key_hash == -1) {
+        return F_ERROR;
+    }
+
+    return hamt_node_find(o->h_root, 0, key_hash, key, val);
+}
+
+
+int
+_PyHamt_Find(PyHamtObject *o, PyObject *key, PyObject **val)
+{
+    hamt_find_t res = hamt_find(o, key, val);
+    switch (res) {
+        case F_ERROR:
+            return -1;
+        case F_NOT_FOUND:
+            return 0;
+        case F_FOUND:
+            return 1;
+        default:
+            Py_UNREACHABLE();
+    }
+}
+
+
+int
+_PyHamt_Eq(PyHamtObject *v, PyHamtObject *w)
+{
+    if (v == w) {
+        return 1;
+    }
+
+    if (v->h_count != w->h_count) {
+        return 0;
+    }
+
+    PyHamtIteratorState iter;
+    hamt_iter_t iter_res;
+    hamt_find_t find_res;
+    PyObject *v_key;
+    PyObject *v_val;
+    PyObject *w_val;
+
+    hamt_iterator_init(&iter, v->h_root);
+
+    do {
+        iter_res = hamt_iterator_next(&iter, &v_key, &v_val);
+        if (iter_res == I_ITEM) {
+            find_res = hamt_find(w, v_key, &w_val);
+            switch (find_res) {
+                case F_ERROR:
+                    return -1;
+
+                case F_NOT_FOUND:
+                    return 0;
+
+                case F_FOUND: {
+                    int cmp = PyObject_RichCompareBool(v_val, w_val, Py_EQ);
+                    if (cmp < 0) {
+                        return -1;
+                    }
+                    if (cmp == 0) {
+                        return 0;
+                    }
+                }
+            }
+        }
+    } while (iter_res != I_END);
+
+    return 1;
+}
+
+Py_ssize_t
+_PyHamt_Len(PyHamtObject *o)
+{
+    return o->h_count;
+}
+
+static PyHamtObject *
+hamt_alloc(void)
+{
+    PyHamtObject *o;
+    o = PyObject_GC_New(PyHamtObject, &_PyHamt_Type);
+    if (o == NULL) {
+        return NULL;
+    }
+    o->h_weakreflist = NULL;
+    PyObject_GC_Track(o);
+    return o;
+}
+
+PyHamtObject *
+_PyHamt_New(void)
+{
+    if (_empty_hamt != NULL) {
+        /* HAMT is an immutable object so we can easily cache an
+           empty instance. */
+        Py_INCREF(_empty_hamt);
+        return _empty_hamt;
+    }
+
+    PyHamtObject *o = hamt_alloc();
+    if (o == NULL) {
+        return NULL;
+    }
+
+    o->h_root = hamt_node_bitmap_new(0);
+    if (o->h_root == NULL) {
+        Py_DECREF(o);
+        return NULL;
+    }
+
+    o->h_count = 0;
+
+    if (_empty_hamt == NULL) {
+        Py_INCREF(o);
+        _empty_hamt = o;
+    }
+
+    return o;
+}
+
+#ifdef Py_DEBUG
+static PyObject *
+hamt_dump(PyHamtObject *self)
+{
+    _PyUnicodeWriter writer;
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_hamt_dump_format(&writer, "HAMT(len=%zd):\n", self->h_count)) {
+        goto error;
+    }
+
+    if (hamt_node_dump(self->h_root, &writer, 0)) {
+        goto error;
+    }
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+#endif  /* Py_DEBUG */
+
+
+/////////////////////////////////// Iterators: Shared Iterator Implementation
+
+
+static int
+hamt_baseiter_tp_clear(PyHamtIterator *it)
+{
+    Py_CLEAR(it->hi_obj);
+    return 0;
+}
+
+static void
+hamt_baseiter_tp_dealloc(PyHamtIterator *it)
+{
+    PyObject_GC_UnTrack(it);
+    (void)hamt_baseiter_tp_clear(it);
+    PyObject_GC_Del(it);
+}
+
+static int
+hamt_baseiter_tp_traverse(PyHamtIterator *it, visitproc visit, void *arg)
+{
+    Py_VISIT(it->hi_obj);
+    return 0;
+}
+
+static PyObject *
+hamt_baseiter_tp_iternext(PyHamtIterator *it)
+{
+    PyObject *key;
+    PyObject *val;
+    hamt_iter_t res = hamt_iterator_next(&it->hi_iter, &key, &val);
+
+    switch (res) {
+        case I_END:
+            PyErr_SetNone(PyExc_StopIteration);
+            return NULL;
+
+        case I_ITEM: {
+            return (*(it->hi_yield))(key, val);
+        }
+
+        default: {
+            Py_UNREACHABLE();
+        }
+    }
+}
+
+static Py_ssize_t
+hamt_baseiter_tp_len(PyHamtIterator *it)
+{
+    return it->hi_obj->h_count;
+}
+
+static PyMappingMethods PyHamtIterator_as_mapping = {
+    (lenfunc)hamt_baseiter_tp_len,
+};
+
+static PyObject *
+hamt_baseiter_new(PyTypeObject *type, binaryfunc yield, PyHamtObject *o)
+{
+    PyHamtIterator *it = PyObject_GC_New(PyHamtIterator, type);
+    if (it == NULL) {
+        return NULL;
+    }
+
+    Py_INCREF(o);
+    it->hi_obj = o;
+    it->hi_yield = yield;
+
+    hamt_iterator_init(&it->hi_iter, o->h_root);
+
+    return (PyObject*)it;
+}
+
+#define ITERATOR_TYPE_SHARED_SLOTS                              \
+    .tp_basicsize = sizeof(PyHamtIterator),                     \
+    .tp_itemsize = 0,                                           \
+    .tp_as_mapping = &PyHamtIterator_as_mapping,                \
+    .tp_dealloc = (destructor)hamt_baseiter_tp_dealloc,         \
+    .tp_getattro = PyObject_GenericGetAttr,                     \
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,        \
+    .tp_traverse = (traverseproc)hamt_baseiter_tp_traverse,     \
+    .tp_clear = (inquiry)hamt_baseiter_tp_clear,                \
+    .tp_iter = PyObject_SelfIter,                               \
+    .tp_iternext = (iternextfunc)hamt_baseiter_tp_iternext,
+
+
+/////////////////////////////////// _PyHamtItems_Type
+
+
+PyTypeObject _PyHamtItems_Type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "items",
+    ITERATOR_TYPE_SHARED_SLOTS
+};
+
+static PyObject *
+hamt_iter_yield_items(PyObject *key, PyObject *val)
+{
+    return PyTuple_Pack(2, key, val);
+}
+
+PyObject *
+_PyHamt_NewIterItems(PyHamtObject *o)
+{
+    return hamt_baseiter_new(
+        &_PyHamtItems_Type, hamt_iter_yield_items, o);
+}
+
+
+/////////////////////////////////// _PyHamtKeys_Type
+
+
+PyTypeObject _PyHamtKeys_Type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "keys",
+    ITERATOR_TYPE_SHARED_SLOTS
+};
+
+static PyObject *
+hamt_iter_yield_keys(PyObject *key, PyObject *val)
+{
+    Py_INCREF(key);
+    return key;
+}
+
+PyObject *
+_PyHamt_NewIterKeys(PyHamtObject *o)
+{
+    return hamt_baseiter_new(
+        &_PyHamtKeys_Type, hamt_iter_yield_keys, o);
+}
+
+
+/////////////////////////////////// _PyHamtValues_Type
+
+
+PyTypeObject _PyHamtValues_Type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "values",
+    ITERATOR_TYPE_SHARED_SLOTS
+};
+
+static PyObject *
+hamt_iter_yield_values(PyObject *key, PyObject *val)
+{
+    Py_INCREF(val);
+    return val;
+}
+
+PyObject *
+_PyHamt_NewIterValues(PyHamtObject *o)
+{
+    return hamt_baseiter_new(
+        &_PyHamtValues_Type, hamt_iter_yield_values, o);
+}
+
+
+/////////////////////////////////// _PyHamt_Type
+
+
+#ifdef Py_DEBUG
+static PyObject *
+hamt_dump(PyHamtObject *self);
+#endif
+
+
+static PyObject *
+hamt_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
+{
+    return (PyObject*)_PyHamt_New();
+}
+
+static int
+hamt_tp_clear(PyHamtObject *self)
+{
+    Py_CLEAR(self->h_root);
+    return 0;
+}
+
+
+static int
+hamt_tp_traverse(PyHamtObject *self, visitproc visit, void *arg)
+{
+    Py_VISIT(self->h_root);
+    return 0;
+}
+
+static void
+hamt_tp_dealloc(PyHamtObject *self)
+{
+    PyObject_GC_UnTrack(self);
+    if (self->h_weakreflist != NULL) {
+        PyObject_ClearWeakRefs((PyObject*)self);
+    }
+    (void)hamt_tp_clear(self);
+    Py_TYPE(self)->tp_free(self);
+}
+
+
+static PyObject *
+hamt_tp_richcompare(PyObject *v, PyObject *w, int op)
+{
+    if (!PyHamt_Check(v) || !PyHamt_Check(w) || (op != Py_EQ && op != Py_NE)) {
+        Py_RETURN_NOTIMPLEMENTED;
+    }
+
+    int res = _PyHamt_Eq((PyHamtObject *)v, (PyHamtObject *)w);
+    if (res < 0) {
+        return NULL;
+    }
+
+    if (op == Py_NE) {
+        res = !res;
+    }
+
+    if (res) {
+        Py_RETURN_TRUE;
+    }
+    else {
+        Py_RETURN_FALSE;
+    }
+}
+
+static int
+hamt_tp_contains(PyHamtObject *self, PyObject *key)
+{
+    PyObject *val;
+    return _PyHamt_Find(self, key, &val);
+}
+
+static PyObject *
+hamt_tp_subscript(PyHamtObject *self, PyObject *key)
+{
+    PyObject *val;
+    hamt_find_t res = hamt_find(self, key, &val);
+    switch (res) {
+        case F_ERROR:
+            return NULL;
+        case F_FOUND:
+            Py_INCREF(val);
+            return val;
+        case F_NOT_FOUND:
+            PyErr_SetObject(PyExc_KeyError, key);
+            return NULL;
+        default:
+            Py_UNREACHABLE();
+    }
+}
+
+static Py_ssize_t
+hamt_tp_len(PyHamtObject *self)
+{
+    return _PyHamt_Len(self);
+}
+
+static PyObject *
+hamt_tp_iter(PyHamtObject *self)
+{
+    return _PyHamt_NewIterKeys(self);
+}
+
+static PyObject *
+hamt_py_set(PyHamtObject *self, PyObject *args)
+{
+    PyObject *key;
+    PyObject *val;
+
+    if (!PyArg_UnpackTuple(args, "set", 2, 2, &key, &val)) {
+        return NULL;
+    }
+
+    return (PyObject *)_PyHamt_Assoc(self, key, val);
+}
+
+static PyObject *
+hamt_py_get(PyHamtObject *self, PyObject *args)
+{
+    PyObject *key;
+    PyObject *def = NULL;
+
+    if (!PyArg_UnpackTuple(args, "get", 1, 2, &key, &def)) {
+        return NULL;
+    }
+
+    PyObject *val = NULL;
+    hamt_find_t res = hamt_find(self, key, &val);
+    switch (res) {
+        case F_ERROR:
+            return NULL;
+        case F_FOUND:
+            Py_INCREF(val);
+            return val;
+        case F_NOT_FOUND:
+            if (def == NULL) {
+                Py_RETURN_NONE;
+            }
+            Py_INCREF(def);
+            return def;
+        default:
+            Py_UNREACHABLE();
+    }
+}
+
+static PyObject *
+hamt_py_delete(PyHamtObject *self, PyObject *key)
+{
+    return (PyObject *)_PyHamt_Without(self, key);
+}
+
+static PyObject *
+hamt_py_items(PyHamtObject *self, PyObject *args)
+{
+    return _PyHamt_NewIterItems(self);
+}
+
+static PyObject *
+hamt_py_values(PyHamtObject *self, PyObject *args)
+{
+    return _PyHamt_NewIterValues(self);
+}
+
+static PyObject *
+hamt_py_keys(PyHamtObject *self, PyObject *args)
+{
+    return _PyHamt_NewIterKeys(self);
+}
+
+#ifdef Py_DEBUG
+static PyObject *
+hamt_py_dump(PyHamtObject *self, PyObject *args)
+{
+    return hamt_dump(self);
+}
+#endif
+
+
+static PyMethodDef PyHamt_methods[] = {
+    {"set", (PyCFunction)hamt_py_set, METH_VARARGS, NULL},
+    {"get", (PyCFunction)hamt_py_get, METH_VARARGS, NULL},
+    {"delete", (PyCFunction)hamt_py_delete, METH_O, NULL},
+    {"items", (PyCFunction)hamt_py_items, METH_NOARGS, NULL},
+    {"keys", (PyCFunction)hamt_py_keys, METH_NOARGS, NULL},
+    {"values", (PyCFunction)hamt_py_values, METH_NOARGS, NULL},
+#ifdef Py_DEBUG
+    {"__dump__", (PyCFunction)hamt_py_dump, METH_NOARGS, NULL},
+#endif
+    {NULL, NULL}
+};
+
+static PySequenceMethods PyHamt_as_sequence = {
+    0,                                /* sq_length */
+    0,                                /* sq_concat */
+    0,                                /* sq_repeat */
+    0,                                /* sq_item */
+    0,                                /* sq_slice */
+    0,                                /* sq_ass_item */
+    0,                                /* sq_ass_slice */
+    (objobjproc)hamt_tp_contains,     /* sq_contains */
+    0,                                /* sq_inplace_concat */
+    0,                                /* sq_inplace_repeat */
+};
+
+static PyMappingMethods PyHamt_as_mapping = {
+    (lenfunc)hamt_tp_len,             /* mp_length */
+    (binaryfunc)hamt_tp_subscript,    /* mp_subscript */
+};
+
+PyTypeObject _PyHamt_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "hamt",
+    sizeof(PyHamtObject),
+    .tp_methods = PyHamt_methods,
+    .tp_as_mapping = &PyHamt_as_mapping,
+    .tp_as_sequence = &PyHamt_as_sequence,
+    .tp_iter = (getiterfunc)hamt_tp_iter,
+    .tp_dealloc = (destructor)hamt_tp_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_richcompare = hamt_tp_richcompare,
+    .tp_traverse = (traverseproc)hamt_tp_traverse,
+    .tp_clear = (inquiry)hamt_tp_clear,
+    .tp_new = hamt_tp_new,
+    .tp_weaklistoffset = offsetof(PyHamtObject, h_weakreflist),
+    .tp_hash = PyObject_HashNotImplemented,
+};
+
+
+/////////////////////////////////// Tree Node Types
+
+
+PyTypeObject _PyHamt_ArrayNode_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "hamt_array_node",
+    sizeof(PyHamtNode_Array),
+    0,
+    .tp_dealloc = (destructor)hamt_node_array_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_traverse = (traverseproc)hamt_node_array_traverse,
+    .tp_free = PyObject_GC_Del,
+    .tp_hash = PyObject_HashNotImplemented,
+};
+
+PyTypeObject _PyHamt_BitmapNode_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "hamt_bitmap_node",
+    sizeof(PyHamtNode_Bitmap) - sizeof(PyObject *),
+    sizeof(PyObject *),
+    .tp_dealloc = (destructor)hamt_node_bitmap_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_traverse = (traverseproc)hamt_node_bitmap_traverse,
+    .tp_free = PyObject_GC_Del,
+    .tp_hash = PyObject_HashNotImplemented,
+};
+
+PyTypeObject _PyHamt_CollisionNode_Type = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "hamt_collision_node",
+    sizeof(PyHamtNode_Collision) - sizeof(PyObject *),
+    sizeof(PyObject *),
+    .tp_dealloc = (destructor)hamt_node_collision_dealloc,
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
+    .tp_traverse = (traverseproc)hamt_node_collision_traverse,
+    .tp_free = PyObject_GC_Del,
+    .tp_hash = PyObject_HashNotImplemented,
+};
+
+
+int
+_PyHamt_Init(void)
+{
+    if ((PyType_Ready(&_PyHamt_Type) < 0) ||
+        (PyType_Ready(&_PyHamt_ArrayNode_Type) < 0) ||
+        (PyType_Ready(&_PyHamt_BitmapNode_Type) < 0) ||
+        (PyType_Ready(&_PyHamt_CollisionNode_Type) < 0) ||
+        (PyType_Ready(&_PyHamtKeys_Type) < 0) ||
+        (PyType_Ready(&_PyHamtValues_Type) < 0) ||
+        (PyType_Ready(&_PyHamtItems_Type) < 0))
+    {
+        return 0;
+    }
+
+    return 1;
+}
+
+void
+_PyHamt_Fini(void)
+{
+    Py_CLEAR(_empty_hamt);
+    Py_CLEAR(_empty_bitmap_node);
+}
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 2f61db0d826..d46784a2f6b 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -4,6 +4,8 @@
 
 #include "Python-ast.h"
 #undef Yield /* undefine macro conflicting with winbase.h */
+#include "internal/context.h"
+#include "internal/hamt.h"
 #include "internal/pystate.h"
 #include "grammar.h"
 #include "node.h"
@@ -758,6 +760,9 @@ _Py_InitializeCore(const _PyCoreConfig *core_config)
         return _Py_INIT_ERR("can't initialize warnings");
     }
 
+    if (!_PyContext_Init())
+        return _Py_INIT_ERR("can't init context");
+
     /* This call sets up builtin and frozen import support */
     if (!interp->core_config._disable_importlib) {
         err = initimport(interp, sysmod);
@@ -1176,6 +1181,7 @@ Py_FinalizeEx(void)
     _Py_HashRandomization_Fini();
     _PyArg_Fini();
     PyAsyncGen_Fini();
+    _PyContext_Fini();
 
     /* Cleanup Unicode implementation */
     _PyUnicode_Fini();
diff --git a/Python/pystate.c b/Python/pystate.c
index 9c25a26460e..909d831465d 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -173,6 +173,8 @@ PyInterpreterState_New(void)
     }
     HEAD_UNLOCK();
 
+    interp->tstate_next_unique_id = 0;
+
     return interp;
 }
 
@@ -313,6 +315,11 @@ new_threadstate(PyInterpreterState *interp, int init)
         tstate->async_gen_firstiter = NULL;
         tstate->async_gen_finalizer = NULL;
 
+        tstate->context = NULL;
+        tstate->context_ver = 1;
+
+        tstate->id = ++interp->tstate_next_unique_id;
+
         if (init)
             _PyThreadState_Init(tstate);
 
@@ -499,6 +506,8 @@ PyThreadState_Clear(PyThreadState *tstate)
     Py_CLEAR(tstate->coroutine_wrapper);
     Py_CLEAR(tstate->async_gen_firstiter);
     Py_CLEAR(tstate->async_gen_finalizer);
+
+    Py_CLEAR(tstate->context);
 }
 
 
diff --git a/Tools/msi/lib/lib_files.wxs b/Tools/msi/lib/lib_files.wxs
index 5a72612c6a5..46ddcb41e9a 100644
--- a/Tools/msi/lib/lib_files.wxs
+++ b/Tools/msi/lib/lib_files.wxs
@@ -1,6 +1,6 @@
 ?<?xml version="1.0" encoding="UTF-8"?>
 <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
-    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3;_asyncio;_queue;_distutils_findvs ?>
+    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3;_asyncio;_queue;_distutils_findvs;_contextvars ?>
     <Fragment>
         <ComponentGroup Id="lib_extensions">
             <?foreach ext in $(var.exts)?>
diff --git a/setup.py b/setup.py
index 1da40a426a2..258094e3ada 100644
--- a/setup.py
+++ b/setup.py
@@ -644,6 +644,9 @@ def detect_modules(self):
         # array objects
         exts.append( Extension('array', ['arraymodule.c']) )
 
+        # Context Variables
+        exts.append( Extension('_contextvars', ['_contextvarsmodule.c']) )
+
         shared_math = 'Modules/_math.o'
         # complex math library functions
         exts.append( Extension('cmath', ['cmathmodule.c'],


From lp_benchmark_robot at intel.com  Mon Jan 22 22:06:17 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Mon, 22 Jan 2018 19:06:17 -0800
Subject: [Python-checkins] [1 down,
 64 flat] Results for Python (master branch) 2018-01-22
Message-ID: <4a92bed1-3c47-4fb3-9897-12dfdf565c75@orsmsx155.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-22 03:04:23-08:00.
  - commit:		579e0b8
  - previous commit:	2507e29
  - revision date:	2018-01-22 16:45:31+09:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.922% |  +0.743%   |  +6.768%   |  +7.667%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.607% |  +0.870%   |  +22.210%  |  +11.484%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.944% |  +3.214%   |  +26.107%  |  +8.164%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.460% |  +1.280%   |  +21.981%  |  +12.742%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 4.180% |  +1.255%   |  +8.787%   |  +13.401%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.566% |  +1.803%   |  +12.241%  |  +11.391%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.009% |  +1.431%   |  +6.355%   |  +10.409%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.456% |  +0.591%   |  -0.444%   |  +8.476%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 2.914% |  -0.218%   |  +10.285%  |  +16.494%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 2.115% |  +4.436%   |  +21.383%  |  +14.141%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.316% |  +0.951%   |  +5.603%   |  +7.569%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.444% |  +0.775%   |  +5.965%   |  +4.987%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 2.030% |  -0.940%   |  +2.187%   |  +6.627%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.209% |  +0.607%   |  +14.519%  |  +9.201%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.475% |  -0.327%   |  +9.743%   |  +11.763%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.844% |  +0.832%   |  +7.078%   |  +10.370%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.456% |  +0.468%   |  +10.024%  |  +11.968%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 3.174% |  +0.909%   |  +8.994%   |  +10.199%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 2.120% |  +0.467%   |  +2.409%   |  +10.795%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 6.389% |  -5.476%   |  -6.366%   |  +18.510%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.488% |  -0.316%   |  +7.837%   |  +12.362%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.566% |  +0.238%   |  +47.073%  |  +12.336%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.584% |  -1.672%   |  +8.312%   |  +13.612%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.443% |  +0.519%   |  +17.699%  |  +12.923%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.859% |  -1.766%   |  +6.007%   |  +12.590%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 0.850% |  -1.457%   |  +2.236%   |  +6.668%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 1.310% |  -0.988%   |  -3.512%   |  +1.990%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 0.905% |  -0.171%   |  +4.901%   |  +8.608%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.276% |  +0.933%   |  +4.170%   |  +10.399%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.198% |  -0.371%   |  -2.094%   |  +24.216%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.394% |  +1.257%   |  +1.585%   |  +22.030%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.847% |  -0.426%   |  +1.653%   |  +21.290%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 3.034% |  -0.450%   |  +12.046%  |  +9.650%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.141% |  -0.273%   |  -0.064%   |  +10.204%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.120% |  +0.239%   |  +11.503%  |  +5.397%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.089% |  +0.211%   |  +2.442%   |  +5.285%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.113% |  -1.175%   |  +9.242%   |  +12.970%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.352% |  -0.189%   |  +3.739%   |  +10.572%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.274% |  -0.536%   |  -2.489%   |  +12.734%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.771% |  -0.657%   |  -2.965%   |  +4.076%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 1.175% |  +0.196%   |  +3.914%   |  +8.312%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 0.917% |  -0.072%   |  +8.449%   |  +15.987%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.380% |  +0.999%   |  -2.383%   |  +2.989%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.916% |  -0.381%   |  +26.021%  |  +10.070%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.179% |  +1.902%   |  +4.461%   |  +5.188%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 0.293% |  +1.515%   |  +14.878%  |  +8.569%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 1.909% |  +0.451%   |  -2.475%   |  -0.836%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 0.490% |  +0.520%   |  +5.981%   |  +0.826%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.431% |  +1.328%   |  +7.924%   |  +5.941%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.089% |  +1.415%   |  +7.567%   |  +3.192%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.027% |  +2.952%   |  +18.857%  |  +9.697%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 3.053% |  +0.802%   |  +15.117%  |  +6.694%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 2.661% |  +0.011%   |  +9.311%   |  +7.487%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.284% |  +1.464%   |  +16.261%  |  +6.879%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.126% |  +2.977%   |  +10.515%  |  +9.885%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 3.140% |  +2.527%   |  +18.792%  |  +12.456%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.977% |  +1.302%   |  +7.359%   |  +6.961%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.860% |  -0.813%   |  +1.339%   |  +0.150%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 7.275% |  +6.603%   |  +6.321%   |  +22.373%  |
+-----+------------------------+--------+------------+------------+------------+
| :-( |           unpickle_list| 0.701% |  -3.858%   |  -5.589%   |  +19.581%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.805% |  -0.318%   |  +8.072%   |  +6.912%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.466% |  -0.615%   |  +2.065%   |  +11.986%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 1.754% |  +2.018%   |  +4.059%   |  +6.110%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 2.909% |  +1.119%   |  -6.238%   |  +10.817%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.364% |  -0.590%   |  +3.735%   |  +12.195%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-down-64-flat-results-for-python-master-branch-2018-01-22

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Tue Jan 23 01:59:53 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Tue, 23 Jan 2018 06:59:53 -0000
Subject: [Python-checkins] Add myself as hamt/context (PEP 567) code owner
 (#5276)
Message-ID: <mailman.9.1516690794.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d83671e996f39484cff42e2f845e03ec2a64d71f
commit: d83671e996f39484cff42e2f845e03ec2a64d71f
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-23T01:59:50-05:00
summary:

Add myself as hamt/context (PEP 567) code owner (#5276)

files:
M .github/CODEOWNERS

diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index b42e1c93d8f..17d7ef4d7a2 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -8,7 +8,9 @@
 **/*asyncio*                  @1st1 @asvetlov
 
 # Core
+**/*context*                  @1st1
 **/*genobject*                @1st1
+**/*hamt*                     @1st1
 
 # Hashing
 **/*hashlib*                  @python/crypto-team


From webhook-mailer at python.org  Tue Jan 23 02:00:06 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Tue, 23 Jan 2018 07:00:06 -0000
Subject: [Python-checkins] bpo-32436: Remove a redundant assert (#5275)
Message-ID: <mailman.10.1516690808.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/83c8675edb4fe278c5d930f7865977a5d3c7168a
commit: 83c8675edb4fe278c5d930f7865977a5d3c7168a
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-23T02:00:03-05:00
summary:

bpo-32436: Remove a redundant assert (#5275)

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index 8ba5082026f..eb69fdd136b 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -1116,7 +1116,6 @@ hamt_node_bitmap_find(PyHamtNode_Bitmap *self,
     }
 
     idx = hamt_bitindex(self->b_bitmap, bit);
-    assert(idx >= 0);
     key_idx = idx * 2;
     val_idx = key_idx + 1;
 


From webhook-mailer at python.org  Tue Jan 23 04:09:38 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Tue, 23 Jan 2018 09:09:38 -0000
Subject: [Python-checkins] bpo-32633: Fix some warnings in
 test_asyncio.test_tasks (#5280)
Message-ID: <mailman.11.1516698580.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6934831e43d66222a626403dd775429d1c8963f3
commit: 6934831e43d66222a626403dd775429d1c8963f3
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-23T11:09:31+02:00
summary:

bpo-32633: Fix some warnings in test_asyncio.test_tasks (#5280)

files:
M Lib/test/test_asyncio/test_tasks.py

diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index 96d2658cb4c..1c361c8ec17 100644
--- a/Lib/test/test_asyncio/test_tasks.py
+++ b/Lib/test/test_asyncio/test_tasks.py
@@ -2342,7 +2342,8 @@ def test_set_result_causes_invalid_state(self):
             await asyncio.sleep(0.1, loop=self.loop)
             return 10
 
-        task = self.new_task(self.loop, foo())
+        coro = foo()
+        task = self.new_task(self.loop, coro)
         Future.set_result(task, 'spam')
 
         self.assertEqual(
@@ -2355,6 +2356,8 @@ def test_set_result_causes_invalid_state(self):
                                     r'step\(\): already done'):
             raise exc
 
+        coro.close()
+
     def test_set_exception_causes_invalid_state(self):
         class MyExc(Exception):
             pass
@@ -2366,7 +2369,8 @@ class MyExc(Exception):
             await asyncio.sleep(0.1, loop=self.loop)
             return 10
 
-        task = self.new_task(self.loop, foo())
+        coro = foo()
+        task = self.new_task(self.loop, coro)
         Future.set_exception(task, MyExc())
 
         with self.assertRaises(MyExc):
@@ -2378,6 +2382,8 @@ class MyExc(Exception):
                                     r'step\(\): already done'):
             raise exc
 
+        coro.close()
+
 
 @unittest.skipUnless(hasattr(futures, '_CFuture') and
                      hasattr(tasks, '_CTask'),


From webhook-mailer at python.org  Tue Jan 23 04:10:46 2018
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 23 Jan 2018 09:10:46 -0000
Subject: [Python-checkins] [3.5] bpo-32620: Remove failing pyenv call from
 CI config (#5274)
Message-ID: <mailman.12.1516698648.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/4a4c2743133e195cc3725b78a895d85d69e50089
commit: 4a4c2743133e195cc3725b78a895d85d69e50089
branch: 3.5
author: Nick Coghlan <ncoghlan at gmail.com>
committer: larryhastings <larry at hastings.org>
date: 2018-01-23T01:10:43-08:00
summary:

[3.5] bpo-32620: Remove failing pyenv call from CI config (#5274)

* [3.5] Remove failing pyenv call from CI config

* Backport XML RPC test skip to 3.5

The buildbot service upgrade removed the XML-RPC
interface, so this test no longer works (through no
fault of the standard library).

files:
M .travis.yml
M Lib/test/test_xmlrpc_net.py

diff --git a/.travis.yml b/.travis.yml
index fed0052b54a..eb98a657b27 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,9 +23,6 @@ matrix:
       # compiler here and the other to run the coverage build. Clang is preferred
       # in this instance for its better error messages.
       env: TESTING=cpython
-      before_install:
-        # work around https://github.com/travis-ci/travis-ci/issues/8363
-        - pyenv global system 3.5
     - os: osx
       language: c
       compiler: clang
diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py
index b60b82f3e23..c9119b04eb9 100644
--- a/Lib/test/test_xmlrpc_net.py
+++ b/Lib/test/test_xmlrpc_net.py
@@ -7,6 +7,7 @@
 
 import xmlrpc.client as xmlrpclib
 
+ at unittest.skip('XXX: buildbot.python.org/all/xmlrpc/ is gone')
 class PythonBuildersTest(unittest.TestCase):
 
     def test_python_builders(self):


From solipsis at pitrou.net  Tue Jan 23 04:20:27 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 23 Jan 2018 09:20:27 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=11
Message-ID: <20180123092027.1.8E6EFA83B8C78F35@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, 7] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [-2, 2, 0] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloga_lzAj', '--timeout', '7200']

From webhook-mailer at python.org  Tue Jan 23 05:24:46 2018
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 23 Jan 2018 10:24:46 -0000
Subject: [Python-checkins] [3.5] bpo-32563: Get expat to compile under C89
 (#5201)
Message-ID: <mailman.13.1516703089.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/57fa0ab8911a70d91e5b60b8dc91f1085442a9e7
commit: 57fa0ab8911a70d91e5b60b8dc91f1085442a9e7
branch: 3.5
author: Nick Coghlan <ncoghlan at gmail.com>
committer: larryhastings <larry at hastings.org>
date: 2018-01-23T02:24:39-08:00
summary:

[3.5] bpo-32563: Get expat to compile under C89 (#5201)

* [3.5] bpo-32563: Get expat to compile under C89

expat: Add artificial scopes in xmltok.c utf8_toUtf8() to fix c89 compilation.

Cherry-picked from libexpat commit e0b290eb3d8f4c4b45137a7d7f4f8db812145bd2

* Backport XML RPC test skip to 3.5

The buildbot service upgrade removed the XML-RPC
interface, so this test no longer works (through no
fault of the standard library).

* Don't configure a nonexistent Python

files:
M Modules/expat/xmltok.c

diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c
index 007aed0640a..70663ce9d4b 100644
--- a/Modules/expat/xmltok.c
+++ b/Modules/expat/xmltok.c
@@ -405,16 +405,20 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
   }
 
   /* Avoid copying partial characters (from incomplete input). */
-  const char * const fromLimBefore = fromLim;
-  align_limit_to_full_utf8_characters(*fromP, &fromLim);
-  if (fromLim < fromLimBefore) {
-    input_incomplete = true;
+  {
+    const char * const fromLimBefore = fromLim;
+    align_limit_to_full_utf8_characters(*fromP, &fromLim);
+    if (fromLim < fromLimBefore) {
+      input_incomplete = true;
+    }
   }
 
-  const ptrdiff_t bytesToCopy = fromLim - *fromP;
-  memcpy((void *)*toP, (const void *)*fromP, (size_t)bytesToCopy);
-  *fromP += bytesToCopy;
-  *toP += bytesToCopy;
+  {
+    const ptrdiff_t bytesToCopy = fromLim - *fromP;
+    memcpy((void *)*toP, (const void *)*fromP, (size_t)bytesToCopy);
+    *fromP += bytesToCopy;
+    *toP += bytesToCopy;
+  }
 
   if (output_exhausted)  // needs to go first
     return XML_CONVERT_OUTPUT_EXHAUSTED;


From webhook-mailer at python.org  Tue Jan 23 05:48:14 2018
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 23 Jan 2018 10:48:14 -0000
Subject: [Python-checkins] [3.5] bpo-32551: Consistently configure
 sys.path[0] (#5197)
Message-ID: <mailman.14.1516704498.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/891c91d8d38848377a9f475242507510873eb9c3
commit: 891c91d8d38848377a9f475242507510873eb9c3
branch: 3.5
author: Nick Coghlan <ncoghlan at gmail.com>
committer: larryhastings <larry at hastings.org>
date: 2018-01-23T02:48:11-08:00
summary:

[3.5] bpo-32551: Consistently configure sys.path[0] (#5197)

Directory and zipfile execution previously added
the parent directory of the directory or zipfile
as sys.path[0] and then subsequently overwrote
it with the directory or zipfile itself.

This caused problems in isolated mode, as it
overwrote the "stdlib as a zip archive" entry
in sys.path, as the parent directory was
never added.

The attempted fix to that issue in bpo-29319
created the opposite problem in *non*-isolated
mode, by potentially leaving the parent
directory on sys.path instead of overwriting it.

This change fixes the root cause of the problem
by removing the whole "add-and-overwrite" dance
for sys.path[0], and instead simply never adds
the parent directory to sys.path in the first
place.

(cherry picked from commit d2977a3ae2cc6802921b1e3b6e9d13fcfbda872d)

files:
A Misc/NEWS.d/next/Security/2018-01-16-16-05-37.bpo-32551.U0z4W-.rst
M Lib/test/test_cmd_line_script.py
M Modules/main.c

diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
index 919f3616c5b..75ca53bf352 100644
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -573,6 +573,73 @@ def test_syntaxerror_indented_caret_position(self):
             self.assertNotIn("\f", text)
             self.assertIn("\n    1 + 1 = 2\n    ^", text)
 
+    def test_consistent_sys_path_for_direct_execution(self):
+        # This test case ensures that the following all give the same
+        # sys.path configuration:
+        #
+        #    ./python -s script_dir/__main__.py
+        #    ./python -s script_dir
+        #    ./python -I script_dir
+        script = textwrap.dedent("""\
+            import sys
+            for entry in sys.path:
+                print(entry)
+            """)
+        # Always show full path diffs on errors
+        self.maxDiff = None
+        with support.temp_dir() as work_dir, support.temp_dir() as script_dir:
+            script_name = _make_test_script(script_dir, '__main__', script)
+            # Reference output comes from directly executing __main__.py
+            # We omit PYTHONPATH and user site to align with isolated mode
+            p = spawn_python("-Es", script_name, cwd=work_dir)
+            out_by_name = kill_python(p).decode().splitlines()
+            self.assertEqual(out_by_name[0], script_dir)
+            self.assertNotIn(work_dir, out_by_name)
+            # Directory execution should give the same output
+            p = spawn_python("-Es", script_dir, cwd=work_dir)
+            out_by_dir = kill_python(p).decode().splitlines()
+            self.assertEqual(out_by_dir, out_by_name)
+            # As should directory execution in isolated mode
+            p = spawn_python("-I", script_dir, cwd=work_dir)
+            out_by_dir_isolated = kill_python(p).decode().splitlines()
+            self.assertEqual(out_by_dir_isolated, out_by_dir, out_by_name)
+
+    def test_consistent_sys_path_for_module_execution(self):
+        # This test case ensures that the following both give the same
+        # sys.path configuration:
+        #    ./python -sm script_pkg.__main__
+        #    ./python -sm script_pkg
+        #
+        # And that this fails as unable to find the package:
+        #    ./python -Im script_pkg
+        script = textwrap.dedent("""\
+            import sys
+            for entry in sys.path:
+                print(entry)
+            """)
+        # Always show full path diffs on errors
+        self.maxDiff = None
+        with support.temp_dir() as work_dir:
+            script_dir = os.path.join(work_dir, "script_pkg")
+            os.mkdir(script_dir)
+            script_name = _make_test_script(script_dir, '__main__', script)
+            # Reference output comes from `-m script_pkg.__main__`
+            # We omit PYTHONPATH and user site to better align with the
+            # direct execution test cases
+            p = spawn_python("-sm", "script_pkg.__main__", cwd=work_dir)
+            out_by_module = kill_python(p).decode().splitlines()
+            self.assertEqual(out_by_module[0], '')
+            self.assertNotIn(script_dir, out_by_module)
+            # Package execution should give the same output
+            p = spawn_python("-sm", "script_pkg", cwd=work_dir)
+            out_by_package = kill_python(p).decode().splitlines()
+            self.assertEqual(out_by_package, out_by_module)
+            # Isolated mode should fail with an import error
+            exitcode, stdout, stderr = assert_python_failure(
+                "-Im", "script_pkg", cwd=work_dir
+            )
+            traceback_lines = stderr.decode().splitlines()
+            self.assertIn("No module named script_pkg", traceback_lines[-1])
 
 def test_main():
     support.run_unittest(CmdLineTest)
diff --git a/Misc/NEWS.d/next/Security/2018-01-16-16-05-37.bpo-32551.U0z4W-.rst b/Misc/NEWS.d/next/Security/2018-01-16-16-05-37.bpo-32551.U0z4W-.rst
new file mode 100644
index 00000000000..3ebcfbb5b75
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2018-01-16-16-05-37.bpo-32551.U0z4W-.rst
@@ -0,0 +1,13 @@
+The ``sys.path[0]`` initialization change for bpo-29139 caused a regression
+by revealing an inconsistency in how sys.path is initialized when executing
+``__main__`` from a zipfile, directory, or other import location. This is
+considered a potential security issue, as it may lead to privileged
+processes unexpectedly loading code from user controlled directories in
+situations where that was not previously the case.
+
+The interpreter now consistently avoids ever adding the import location's
+parent directory to ``sys.path``, and ensures no other ``sys.path`` entries
+are inadvertently modified when inserting the import location named on the
+command line. (Originally reported as bpo-29723 against Python 3.6rc1, but
+it was missed at the time that the then upcoming Python 3.5.4 release would
+also be affected)
diff --git a/Modules/main.c b/Modules/main.c
index d36db6d8b51..e1f55b4414c 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -220,55 +220,60 @@ static int RunModule(wchar_t *modname, int set_argv0)
     return 0;
 }
 
-static int
-RunMainFromImporter(wchar_t *filename)
+static PyObject *
+AsImportPathEntry(wchar_t *filename)
 {
-    PyObject *argv0 = NULL, *importer, *sys_path, *sys_path0;
-    int sts;
+    PyObject *sys_path0 = NULL, *importer;
 
-    argv0 = PyUnicode_FromWideChar(filename, wcslen(filename));
-    if (argv0 == NULL)
+    sys_path0 = PyUnicode_FromWideChar(filename, wcslen(filename));
+    if (sys_path0 == NULL)
         goto error;
 
-    importer = PyImport_GetImporter(argv0);
+    importer = PyImport_GetImporter(sys_path0);
     if (importer == NULL)
         goto error;
 
     if (importer == Py_None) {
-        Py_DECREF(argv0);
+        Py_DECREF(sys_path0);
         Py_DECREF(importer);
-        return -1;
+        return NULL;
     }
     Py_DECREF(importer);
+    return sys_path0;
+
+error:
+    Py_XDECREF(sys_path0);
+    PySys_WriteStderr("Failed checking if argv[0] is an import path entry\n");
+    PyErr_Print();
+    PyErr_Clear();
+    return NULL;
+}
+
+
+static int
+RunMainFromImporter(PyObject *sys_path0)
+{
+    PyObject *sys_path;
+    int sts;
 
-    /* argv0 is usable as an import source, so put it in sys.path[0]
-       and import __main__ */
+    /* Assume sys_path0 has already been checked by AsImportPathEntry,
+     * so put it in sys.path[0] and import __main__ */
     sys_path = PySys_GetObject("path");
     if (sys_path == NULL) {
         PyErr_SetString(PyExc_RuntimeError, "unable to get sys.path");
         goto error;
     }
-    sys_path0 = PyList_GetItem(sys_path, 0);
-    sts = 0;
-    if (!sys_path0) {
-        PyErr_Clear();
-        sts = PyList_Append(sys_path, argv0);
-    } else if (PyObject_IsTrue(sys_path0)) {
-        sts = PyList_Insert(sys_path, 0, argv0);
-    } else {
-        sts = PyList_SetItem(sys_path, 0, argv0);
-    }
+    sts = PyList_Insert(sys_path, 0, sys_path0);
     if (sts) {
-        argv0 = NULL;
+        sys_path0 = NULL;
         goto error;
     }
-    Py_INCREF(argv0);
 
     sts = RunModule(L"__main__", 0);
     return sts != 0;
 
 error:
-    Py_XDECREF(argv0);
+    Py_XDECREF(sys_path0);
     PyErr_Print();
     return 1;
 }
@@ -352,6 +357,7 @@ Py_Main(int argc, wchar_t **argv)
     int version = 0;
     int saw_unbuffered_flag = 0;
     PyCompilerFlags cf;
+    PyObject *main_importer_path = NULL;
     PyObject *warning_option = NULL;
     PyObject *warning_options = NULL;
 
@@ -700,7 +706,17 @@ Py_Main(int argc, wchar_t **argv)
         argv[_PyOS_optind] = L"-m";
     }
 
-    PySys_SetArgv(argc-_PyOS_optind, argv+_PyOS_optind);
+    if (filename != NULL) {
+        main_importer_path = AsImportPathEntry(filename);
+    }
+
+    if (main_importer_path != NULL) {
+        /* Let RunMainFromImporter adjust sys.path[0] later */
+        PySys_SetArgvEx(argc-_PyOS_optind, argv+_PyOS_optind, 0);
+    } else {
+        /* Use config settings to decide whether or not to update sys.path[0] */
+        PySys_SetArgv(argc-_PyOS_optind, argv+_PyOS_optind);
+    }
 
     if ((Py_InspectFlag || (command == NULL && filename == NULL && module == NULL)) &&
         isatty(fileno(stdin))) {
@@ -729,11 +745,11 @@ Py_Main(int argc, wchar_t **argv)
 
         sts = -1;               /* keep track of whether we've already run __main__ */
 
-        if (filename != NULL) {
-            sts = RunMainFromImporter(filename);
+        if (main_importer_path != NULL) {
+            sts = RunMainFromImporter(main_importer_path);
         }
 
-        if (sts==-1 && filename!=NULL) {
+        if (sts==-1 && filename != NULL) {
             fp = _Py_wfopen(filename, L"r");
             if (fp == NULL) {
                 char *cfilename_buffer;


From webhook-mailer at python.org  Tue Jan 23 06:21:23 2018
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 23 Jan 2018 11:21:23 -0000
Subject: [Python-checkins] [3.5] bpo-32072: Fix issues with binary plists.
 (GH-4455) (#4656)
Message-ID: <mailman.15.1516706486.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/43f014d3f12468edf61046f0612edc7660042fd5
commit: 43f014d3f12468edf61046f0612edc7660042fd5
branch: 3.5
author: Serhiy Storchaka <storchaka at gmail.com>
committer: larryhastings <larry at hastings.org>
date: 2018-01-23T03:21:21-08:00
summary:

[3.5] bpo-32072: Fix issues with binary plists. (GH-4455) (#4656)

* [3.5] bpo-32072: Fix issues with binary plists. (GH-4455)

* Fixed saving bytearrays.
* Identical objects will be saved only once.
* Equal references will be load as identical objects.
* Added support for saving and loading recursive data structures.
(cherry picked from commit a897aee)

* Fix implementation dependent assertion in test_plistlib. (#4813)

It is failed with an advanced optimizer.

files:
A Misc/NEWS.d/next/Library/2017-11-18-21-13-52.bpo-32072.nwDV8L.rst
M Lib/plistlib.py
M Lib/test/test_plistlib.py

diff --git a/Lib/plistlib.py b/Lib/plistlib.py
index 2502b39c7c1..1d27667a340 100644
--- a/Lib/plistlib.py
+++ b/Lib/plistlib.py
@@ -590,6 +590,8 @@ def __init__(self, message="Invalid file"):
 
 _BINARY_FORMAT = {1: 'B', 2: 'H', 4: 'L', 8: 'Q'}
 
+_undefined = object()
+
 class _BinaryPlistParser:
     """
     Read or write a binary plist file, following the description of the binary
@@ -620,7 +622,8 @@ def parse(self, fp):
             ) = struct.unpack('>6xBBQQQ', trailer)
             self._fp.seek(offset_table_offset)
             self._object_offsets = self._read_ints(num_objects, offset_size)
-            return self._read_object(self._object_offsets[top_object])
+            self._objects = [_undefined] * num_objects
+            return self._read_object(top_object)
 
         except (OSError, IndexError, struct.error):
             raise InvalidFileException()
@@ -646,62 +649,68 @@ def _read_ints(self, n, size):
     def _read_refs(self, n):
         return self._read_ints(n, self._ref_size)
 
-    def _read_object(self, offset):
+    def _read_object(self, ref):
         """
-        read the object at offset.
+        read the object by reference.
 
         May recursively read sub-objects (content of an array/dict/set)
         """
+        result = self._objects[ref]
+        if result is not _undefined:
+            return result
+
+        offset = self._object_offsets[ref]
         self._fp.seek(offset)
         token = self._fp.read(1)[0]
         tokenH, tokenL = token & 0xF0, token & 0x0F
 
         if token == 0x00:
-            return None
+            result = None
 
         elif token == 0x08:
-            return False
+            result = False
 
         elif token == 0x09:
-            return True
+            result = True
 
         # The referenced source code also mentions URL (0x0c, 0x0d) and
         # UUID (0x0e), but neither can be generated using the Cocoa libraries.
 
         elif token == 0x0f:
-            return b''
+            result = b''
 
         elif tokenH == 0x10:  # int
-            return int.from_bytes(self._fp.read(1 << tokenL),
-                                  'big', signed=tokenL >= 3)
+            result = int.from_bytes(self._fp.read(1 << tokenL),
+                                    'big', signed=tokenL >= 3)
 
         elif token == 0x22: # real
-            return struct.unpack('>f', self._fp.read(4))[0]
+            result = struct.unpack('>f', self._fp.read(4))[0]
 
         elif token == 0x23: # real
-            return struct.unpack('>d', self._fp.read(8))[0]
+            result = struct.unpack('>d', self._fp.read(8))[0]
 
         elif token == 0x33:  # date
             f = struct.unpack('>d', self._fp.read(8))[0]
             # timestamp 0 of binary plists corresponds to 1/1/2001
             # (year of Mac OS X 10.0), instead of 1/1/1970.
-            return datetime.datetime(2001, 1, 1) + datetime.timedelta(seconds=f)
+            result = (datetime.datetime(2001, 1, 1) +
+                      datetime.timedelta(seconds=f))
 
         elif tokenH == 0x40:  # data
             s = self._get_size(tokenL)
             if self._use_builtin_types:
-                return self._fp.read(s)
+                result = self._fp.read(s)
             else:
-                return Data(self._fp.read(s))
+                result = Data(self._fp.read(s))
 
         elif tokenH == 0x50:  # ascii string
             s = self._get_size(tokenL)
             result =  self._fp.read(s).decode('ascii')
-            return result
+            result = result
 
         elif tokenH == 0x60:  # unicode string
             s = self._get_size(tokenL)
-            return self._fp.read(s * 2).decode('utf-16be')
+            result = self._fp.read(s * 2).decode('utf-16be')
 
         # tokenH == 0x80 is documented as 'UID' and appears to be used for
         # keyed-archiving, not in plists.
@@ -709,8 +718,9 @@ def _read_object(self, offset):
         elif tokenH == 0xA0:  # array
             s = self._get_size(tokenL)
             obj_refs = self._read_refs(s)
-            return [self._read_object(self._object_offsets[x])
-                for x in obj_refs]
+            result = []
+            self._objects[ref] = result
+            result.extend(self._read_object(x) for x in obj_refs)
 
         # tokenH == 0xB0 is documented as 'ordset', but is not actually
         # implemented in the Apple reference code.
@@ -723,12 +733,15 @@ def _read_object(self, offset):
             key_refs = self._read_refs(s)
             obj_refs = self._read_refs(s)
             result = self._dict_type()
+            self._objects[ref] = result
             for k, o in zip(key_refs, obj_refs):
-                result[self._read_object(self._object_offsets[k])
-                    ] = self._read_object(self._object_offsets[o])
-            return result
+                result[self._read_object(k)] = self._read_object(o)
 
-        raise InvalidFileException()
+        else:
+            raise InvalidFileException()
+
+        self._objects[ref] = result
+        return result
 
 def _count_to_size(count):
     if count < 1 << 8:
@@ -743,6 +756,8 @@ def _count_to_size(count):
     else:
         return 8
 
+_scalars = (str, int, float, datetime.datetime, bytes)
+
 class _BinaryPlistWriter (object):
     def __init__(self, fp, sort_keys, skipkeys):
         self._fp = fp
@@ -798,8 +813,7 @@ def _flatten(self, value):
         # First check if the object is in the object table, not used for
         # containers to ensure that two subcontainers with the same contents
         # will be serialized as distinct values.
-        if isinstance(value, (
-                str, int, float, datetime.datetime, bytes, bytearray)):
+        if isinstance(value, _scalars):
             if (type(value), value) in self._objtable:
                 return
 
@@ -807,15 +821,17 @@ def _flatten(self, value):
             if (type(value.data), value.data) in self._objtable:
                 return
 
+        elif id(value) in self._objidtable:
+            return
+
         # Add to objectreference map
         refnum = len(self._objlist)
         self._objlist.append(value)
-        try:
-            if isinstance(value, Data):
-                self._objtable[(type(value.data), value.data)] = refnum
-            else:
-                self._objtable[(type(value), value)] = refnum
-        except TypeError:
+        if isinstance(value, _scalars):
+            self._objtable[(type(value), value)] = refnum
+        elif isinstance(value, Data):
+            self._objtable[(type(value.data), value.data)] = refnum
+        else:
             self._objidtable[id(value)] = refnum
 
         # And finally recurse into containers
@@ -842,12 +858,11 @@ def _flatten(self, value):
                 self._flatten(o)
 
     def _getrefnum(self, value):
-        try:
-            if isinstance(value, Data):
-                return self._objtable[(type(value.data), value.data)]
-            else:
-                return self._objtable[(type(value), value)]
-        except TypeError:
+        if isinstance(value, _scalars):
+            return self._objtable[(type(value), value)]
+        elif isinstance(value, Data):
+            return self._objtable[(type(value.data), value.data)]
+        else:
             return self._objidtable[id(value)]
 
     def _write_size(self, token, size):
diff --git a/Lib/test/test_plistlib.py b/Lib/test/test_plistlib.py
index 692cac4bdf4..c9719e37fd0 100644
--- a/Lib/test/test_plistlib.py
+++ b/Lib/test/test_plistlib.py
@@ -170,6 +170,17 @@ def test_int(self):
                     self.assertRaises(OverflowError, plistlib.dumps,
                                       pl, fmt=fmt)
 
+    def test_bytearray(self):
+        for pl in (b'<binary gunk>', b"<lots of binary gunk>\0\1\2\3" * 10):
+            for fmt in ALL_FORMATS:
+                with self.subTest(pl=pl, fmt=fmt):
+                    data = plistlib.dumps(bytearray(pl), fmt=fmt)
+                    pl2 = plistlib.loads(data)
+                    self.assertIsInstance(pl2, bytes)
+                    self.assertEqual(pl2, pl)
+                    data2 = plistlib.dumps(pl2, fmt=fmt)
+                    self.assertEqual(data, data2)
+
     def test_bytes(self):
         pl = self._create()
         data = plistlib.dumps(pl)
@@ -311,7 +322,8 @@ def test_tuple_members(self):
                     'second': [1, 2],
                     'third': [3, 4],
                 })
-                self.assertIsNot(pl2['first'], pl2['second'])
+                if fmt != plistlib.FMT_BINARY:
+                    self.assertIsNot(pl2['first'], pl2['second'])
 
     def test_list_members(self):
         pl = {
@@ -423,6 +435,9 @@ def test_xml_encodings(self):
                 pl2 = plistlib.loads(data)
                 self.assertEqual(dict(pl), dict(pl2))
 
+
+class TestBinaryPlistlib(unittest.TestCase):
+
     def test_nonstandard_refs_size(self):
         # Issue #21538: Refs and offsets are 24-bit integers
         data = (b'bplist00'
@@ -435,6 +450,47 @@ def test_nonstandard_refs_size(self):
                 b'\x00\x00\x00\x00\x00\x00\x00\x13')
         self.assertEqual(plistlib.loads(data), {'a': 'b'})
 
+    def test_dump_duplicates(self):
+        # Test effectiveness of saving duplicated objects
+        for x in (None, False, True, 12345, 123.45, 'abcde', b'abcde',
+                  datetime.datetime(2004, 10, 26, 10, 33, 33),
+                  plistlib.Data(b'abcde'), bytearray(b'abcde'),
+                  [12, 345], (12, 345), {'12': 345}):
+            with self.subTest(x=x):
+                data = plistlib.dumps([x]*1000, fmt=plistlib.FMT_BINARY)
+                self.assertLess(len(data), 1100, repr(data))
+
+    def test_identity(self):
+        for x in (None, False, True, 12345, 123.45, 'abcde', b'abcde',
+                  datetime.datetime(2004, 10, 26, 10, 33, 33),
+                  plistlib.Data(b'abcde'), bytearray(b'abcde'),
+                  [12, 345], (12, 345), {'12': 345}):
+            with self.subTest(x=x):
+                data = plistlib.dumps([x]*2, fmt=plistlib.FMT_BINARY)
+                a, b = plistlib.loads(data)
+                if isinstance(x, tuple):
+                    x = list(x)
+                self.assertEqual(a, x)
+                self.assertEqual(b, x)
+                self.assertIs(a, b)
+
+    def test_cycles(self):
+        # recursive list
+        a = []
+        a.append(a)
+        b = plistlib.loads(plistlib.dumps(a, fmt=plistlib.FMT_BINARY))
+        self.assertIs(b[0], b)
+        # recursive tuple
+        a = ([],)
+        a[0].append(a)
+        b = plistlib.loads(plistlib.dumps(a, fmt=plistlib.FMT_BINARY))
+        self.assertIs(b[0][0], b)
+        # recursive dict
+        a = {}
+        a['x'] = a
+        b = plistlib.loads(plistlib.dumps(a, fmt=plistlib.FMT_BINARY))
+        self.assertIs(b['x'], b)
+
     def test_large_timestamp(self):
         # Issue #26709: 32-bit timestamp out of range
         for ts in -2**31-1, 2**31:
diff --git a/Misc/NEWS.d/next/Library/2017-11-18-21-13-52.bpo-32072.nwDV8L.rst b/Misc/NEWS.d/next/Library/2017-11-18-21-13-52.bpo-32072.nwDV8L.rst
new file mode 100644
index 00000000000..6da5bb427d4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-11-18-21-13-52.bpo-32072.nwDV8L.rst
@@ -0,0 +1,6 @@
+Fixed issues with binary plists:
+
+* Fixed saving bytearrays.
+* Identical objects will be saved only once.
+* Equal references will be load as identical objects.
+* Added support for saving and loading recursive data structures.


From webhook-mailer at python.org  Tue Jan 23 07:14:14 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 23 Jan 2018 12:14:14 -0000
Subject: [Python-checkins] bpo-32593: Run autoconf (#5282)
Message-ID: <mailman.16.1516709655.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/872f841b122160987845db8fdddfaee1aaa203f1
commit: 872f841b122160987845db8fdddfaee1aaa203f1
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-23T13:14:08+01:00
summary:

bpo-32593: Run autoconf (#5282)

Update configure since configure.ac was modified to drop support for
FreeBSD 4.

files:
M configure

diff --git a/configure b/configure
index 420c07639b5..a9b0ab9646b 100755
--- a/configure
+++ b/configure
@@ -3355,10 +3355,6 @@ $as_echo "#define _BSD_SOURCE 1" >>confdefs.h
   # but used in struct sockaddr.sa_family. Reported by Tim Rice.
   SCO_SV/3.2)
     define_xopen_source=no;;
-  # On FreeBSD 4, the math functions C89 does not cover are never defined
-  # with _XOPEN_SOURCE and __BSD_VISIBLE does not re-enable them.
-  FreeBSD/4.*)
-    define_xopen_source=no;;
   # On MacOS X 10.2, a bug in ncurses.h means that it craps out if
   # _XOPEN_EXTENDED_SOURCE is defined. Apparently, this is fixed in 10.3, which
   # identifies itself as Darwin/7.*


From webhook-mailer at python.org  Tue Jan 23 07:30:56 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 23 Jan 2018 12:30:56 -0000
Subject: [Python-checkins] bpo-20361: Remove workaround for a now fixed bug
 (#5283)
Message-ID: <mailman.17.1516710657.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/dc6b9462c00873c8404a7966b7ca210717718af5
commit: dc6b9462c00873c8404a7966b7ca210717718af5
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-23T13:30:53+01:00
summary:

bpo-20361: Remove workaround for a now fixed bug (#5283)

"python3 -bb -Wd" now works as expected:
"python3 -bb -Wd -Werror::BytesWarning" is no more needed.

files:
M Tools/scripts/run_tests.py

diff --git a/Tools/scripts/run_tests.py b/Tools/scripts/run_tests.py
index 9fc37997683..3c1c3bd060b 100644
--- a/Tools/scripts/run_tests.py
+++ b/Tools/scripts/run_tests.py
@@ -30,9 +30,6 @@ def main(regrtest_args):
     # Allow user-specified interpreter options to override our defaults.
     args.extend(test.support.args_from_interpreter_flags())
 
-    # Workaround for issue #20361
-    args.extend(['-W', 'error::BytesWarning'])
-
     args.extend(['-m', 'test',    # Run the test suite
                  '-r',            # Randomize test order
                  '-w',            # Re-run failed tests in verbose mode


From webhook-mailer at python.org  Tue Jan 23 09:50:53 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Tue, 23 Jan 2018 14:50:53 -0000
Subject: [Python-checkins] bpo-32618: Fix test_mutatingdecodehandler not
 testing test.mutating (#5269)
Message-ID: <mailman.18.1516719055.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/370d04d1dcca50a52d59f40aff4d11434f71df6b
commit: 370d04d1dcca50a52d59f40aff4d11434f71df6b
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-23T22:50:50+08:00
summary:

bpo-32618: Fix test_mutatingdecodehandler not testing test.mutating (#5269)

* bpo-32618: Fix test_mutatingdecodehandler not testing test.mutating

It should test both test.replacing and test.mutating instead of test test.replacing twice.

files:
M Lib/test/test_codeccallbacks.py

diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
index 6a3e9932656..0c066e633fa 100644
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1032,7 +1032,7 @@ def replacing(exc):
 
         def mutating(exc):
             if isinstance(exc, UnicodeDecodeError):
-                exc.object[:] = b""
+                exc.object = b""
                 return ("\u4242", 0)
             else:
                 raise TypeError("don't know how to handle %r" % exc)
@@ -1042,8 +1042,7 @@ def mutating(exc):
         with test.support.check_warnings():
             # unicode-internal has been deprecated
             for (encoding, data) in baddata:
-                with self.assertRaises(TypeError):
-                    data.decode(encoding, "test.replacing")
+                self.assertEqual(data.decode(encoding, "test.mutating"), "\u4242")
 
     def test_fake_error_class(self):
         handlers = [


From webhook-mailer at python.org  Tue Jan 23 10:11:27 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 23 Jan 2018 15:11:27 -0000
Subject: [Python-checkins] bpo-32635: Fix a segfault when importing the
 crypt module with libxcrypt. (#5284)
Message-ID: <mailman.19.1516720288.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/e768c86ef442ef89004089a8a34ce5909ffb90f2
commit: e768c86ef442ef89004089a8a34ce5909ffb90f2
branch: master
author: stratakis <cstratak at redhat.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-23T16:11:24+01:00
summary:

 bpo-32635: Fix a segfault when importing the crypt module with libxcrypt. (#5284)

glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.

files:
A Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
M Include/Python.h
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Include/Python.h b/Include/Python.h
index dd595ea5e4c..1feb1531cc9 100644
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -35,6 +35,9 @@
 #ifdef HAVE_UNISTD_H
 #include <unistd.h>
 #endif
+#ifdef HAVE_CRYPT_H
+#include <crypt.h>
+#endif
 
 /* For size_t? */
 #ifdef HAVE_STDDEF_H
diff --git a/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst b/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
new file mode 100644
index 00000000000..d411890f5d8
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
@@ -0,0 +1,2 @@
+Fix segfault of the crypt module when libxcrypt is provided instead of
+libcrypt at the system.
diff --git a/configure b/configure
index a9b0ab9646b..5904b54c3ed 100755
--- a/configure
+++ b/configure
@@ -7682,7 +7682,7 @@ $as_echo "#define STDC_HEADERS 1" >>confdefs.h
 
 fi
 
-for ac_header in asm/types.h conio.h direct.h dlfcn.h errno.h \
+for ac_header in asm/types.h crypt.h conio.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h process.h pthread.h \
 sched.h shadow.h signal.h stropts.h termios.h \
diff --git a/configure.ac b/configure.ac
index 03b0f501aff..15ef872a53a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2041,7 +2041,7 @@ dnl AC_MSG_RESULT($cpp_type)
 
 # checks for header files
 AC_HEADER_STDC
-AC_CHECK_HEADERS(asm/types.h conio.h direct.h dlfcn.h errno.h \
+AC_CHECK_HEADERS(asm/types.h crypt.h conio.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h process.h pthread.h \
 sched.h shadow.h signal.h stropts.h termios.h \
diff --git a/pyconfig.h.in b/pyconfig.h.in
index dd7c62bad1b..fd5a0c38d89 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -140,6 +140,9 @@
 /* Define to 1 if you have the `copysign' function. */
 #undef HAVE_COPYSIGN
 
+/* Define to 1 if you have the <crypt.h> header file. */
+#undef HAVE_CRYPT_H
+
 /* Define to 1 if you have the `ctermid' function. */
 #undef HAVE_CTERMID
 


From webhook-mailer at python.org  Tue Jan 23 10:47:21 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Tue, 23 Jan 2018 15:47:21 -0000
Subject: [Python-checkins] bpo-32618: Fix test_mutatingdecodehandler not
 testing test.mutating (GH-5269) (#5285)
Message-ID: <mailman.20.1516722442.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6abbf14a876ee1e04d1493bb27025f2f0aa56430
commit: 6abbf14a876ee1e04d1493bb27025f2f0aa56430
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2018-01-23T23:47:15+08:00
summary:

bpo-32618: Fix test_mutatingdecodehandler not testing test.mutating (GH-5269) (#5285)

It should test both test.replacing and test.mutating instead of test test.replacing twice.
(cherry picked from commit 370d04d1dcca50a52d59f40aff4d11434f71df6b)

files:
M Lib/test/test_codeccallbacks.py

diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
index 6a3e9932656..0c066e633fa 100644
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1032,7 +1032,7 @@ def replacing(exc):
 
         def mutating(exc):
             if isinstance(exc, UnicodeDecodeError):
-                exc.object[:] = b""
+                exc.object = b""
                 return ("\u4242", 0)
             else:
                 raise TypeError("don't know how to handle %r" % exc)
@@ -1042,8 +1042,7 @@ def mutating(exc):
         with test.support.check_warnings():
             # unicode-internal has been deprecated
             for (encoding, data) in baddata:
-                with self.assertRaises(TypeError):
-                    data.decode(encoding, "test.replacing")
+                self.assertEqual(data.decode(encoding, "test.mutating"), "\u4242")
 
     def test_fake_error_class(self):
         handlers = [


From webhook-mailer at python.org  Tue Jan 23 15:10:06 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Tue, 23 Jan 2018 20:10:06 -0000
Subject: [Python-checkins] bpo-32296: Make get_running_loop() another 4-5x
 faster (#5277)
Message-ID: <mailman.21.1516738208.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/9d411c119fdd8e42209ec16be27686a843507f18
commit: 9d411c119fdd8e42209ec16be27686a843507f18
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-23T15:10:03-05:00
summary:

bpo-32296: Make get_running_loop() another 4-5x faster (#5277)

files:
M Modules/_asynciomodule.c

diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c
index f77ec999b09..a08096942b8 100644
--- a/Modules/_asynciomodule.c
+++ b/Modules/_asynciomodule.c
@@ -26,7 +26,6 @@ _Py_IDENTIFIER(_wakeup);
 /* State of the _asyncio module */
 static PyObject *asyncio_mod;
 static PyObject *inspect_isgenerator;
-static PyObject *os_getpid;
 static PyObject *traceback_extract_stack;
 static PyObject *asyncio_get_event_loop_policy;
 static PyObject *asyncio_future_repr_info_func;
@@ -38,6 +37,9 @@ static PyObject *asyncio_InvalidStateError;
 static PyObject *asyncio_CancelledError;
 static PyObject *context_kwname;
 
+static PyObject *cached_running_holder;
+static volatile uint64_t cached_running_holder_tsid;
+
 
 /* WeakSet containing all alive tasks. */
 static PyObject *all_tasks;
@@ -95,9 +97,18 @@ typedef struct {
     TaskObj *ww_task;
 } TaskWakeupMethWrapper;
 
+typedef struct {
+    PyObject_HEAD
+    PyObject *rl_loop;
+#if defined(HAVE_GETPID) && !defined(MS_WINDOWS)
+    pid_t rl_pid;
+#endif
+} PyRunningLoopHolder;
+
 
 static PyTypeObject FutureType;
 static PyTypeObject TaskType;
+static PyTypeObject PyRunningLoopHolder_Type;
 
 
 #define Future_CheckExact(obj) (Py_TYPE(obj) == &FutureType)
@@ -116,9 +127,11 @@ class _asyncio.Future "FutureObj *" "&Future_Type"
 
 
 /* Get FutureIter from Future */
-static PyObject* future_new_iter(PyObject *);
+static PyObject * future_new_iter(PyObject *);
 static inline int future_call_schedule_callbacks(FutureObj *);
 
+static PyRunningLoopHolder * new_running_loop_holder(PyObject *);
+
 
 static int
 _is_coroutine(PyObject *coro)
@@ -214,60 +227,57 @@ get_future_loop(PyObject *fut)
 static int
 get_running_loop(PyObject **loop)
 {
-    PyObject *ts_dict;
-    PyObject *running_tuple;
-    PyObject *running_loop;
-    PyObject *running_loop_pid;
-    PyObject *current_pid;
-    int same_pid;
+    PyObject *rl;
 
-    ts_dict = PyThreadState_GetDict();  // borrowed
-    if (ts_dict == NULL) {
-        PyErr_SetString(
-            PyExc_RuntimeError, "thread-local storage is not available");
-        goto error;
+    PyThreadState *ts = PyThreadState_Get();
+    if (ts->id == cached_running_holder_tsid && cached_running_holder != NULL) {
+        // Fast path, check the cache.
+        rl = cached_running_holder;  // borrowed
     }
+    else {
+        if (ts->dict == NULL) {
+            goto not_found;
+        }
 
-    running_tuple = _PyDict_GetItemId(
-        ts_dict, &PyId___asyncio_running_event_loop__);  // borrowed
-    if (running_tuple == NULL) {
-        /* _PyDict_GetItemId doesn't set an error if key is not found */
-        goto not_found;
+        rl = _PyDict_GetItemIdWithError(
+            ts->dict, &PyId___asyncio_running_event_loop__);  // borrowed
+        if (rl == NULL) {
+            if (PyErr_Occurred()) {
+                goto error;
+            }
+            else {
+                goto not_found;
+            }
+        }
+
+        cached_running_holder = rl;  // borrowed
+        cached_running_holder_tsid = ts->id;
     }
 
-    assert(PyTuple_CheckExact(running_tuple));
-    assert(PyTuple_Size(running_tuple) == 2);
-    running_loop = PyTuple_GET_ITEM(running_tuple, 0);  // borrowed
-    running_loop_pid = PyTuple_GET_ITEM(running_tuple, 1);  // borrowed
+    assert(Py_TYPE(rl) == &PyRunningLoopHolder_Type);
+    PyObject *running_loop = ((PyRunningLoopHolder *)rl)->rl_loop;
 
     if (running_loop == Py_None) {
         goto not_found;
     }
 
-    current_pid = _PyObject_CallNoArg(os_getpid);
-    if (current_pid == NULL) {
-        goto error;
-    }
-    same_pid = PyObject_RichCompareBool(current_pid, running_loop_pid, Py_EQ);
-    Py_DECREF(current_pid);
-    if (same_pid == -1) {
-        goto error;
+#if defined(HAVE_GETPID) && !defined(MS_WINDOWS)
+    /* On Windows there is no getpid, but there is also no os.fork(),
+       so there is no need for this check.
+    */
+    if (getpid() != ((PyRunningLoopHolder *)rl)->rl_pid) {
+        goto not_found;
     }
+#endif
 
-    if (same_pid) {
-        // current_pid == running_loop_pid
-        goto found;
-    }
+    Py_INCREF(running_loop);
+    *loop = running_loop;
+    return 0;
 
 not_found:
     *loop = NULL;
     return 0;
 
-found:
-    Py_INCREF(running_loop);
-    *loop = running_loop;
-    return 0;
-
 error:
     *loop = NULL;
     return -1;
@@ -277,38 +287,28 @@ get_running_loop(PyObject **loop)
 static int
 set_running_loop(PyObject *loop)
 {
-    PyObject *ts_dict;
-    PyObject *running_tuple;
-    PyObject *current_pid;
+    cached_running_holder = NULL;
+    cached_running_holder_tsid = 0;
 
-    ts_dict = PyThreadState_GetDict();  // borrowed
+    PyObject *ts_dict = PyThreadState_GetDict();  // borrowed
     if (ts_dict == NULL) {
         PyErr_SetString(
             PyExc_RuntimeError, "thread-local storage is not available");
         return -1;
     }
 
-    current_pid = _PyObject_CallNoArg(os_getpid);
-    if (current_pid == NULL) {
+    PyRunningLoopHolder *rl = new_running_loop_holder(loop);
+    if (rl == NULL) {
         return -1;
     }
 
-    running_tuple = PyTuple_New(2);
-    if (running_tuple == NULL) {
-        Py_DECREF(current_pid);
-        return -1;
-    }
-
-    Py_INCREF(loop);
-    PyTuple_SET_ITEM(running_tuple, 0, loop);
-    PyTuple_SET_ITEM(running_tuple, 1, current_pid);  // borrowed
-
     if (_PyDict_SetItemId(
-            ts_dict, &PyId___asyncio_running_event_loop__, running_tuple)) {
-        Py_DECREF(running_tuple);  // will cleanup loop & current_pid
+            ts_dict, &PyId___asyncio_running_event_loop__, (PyObject *)rl) < 0)
+    {
+        Py_DECREF(rl);  // will cleanup loop & current_pid
         return -1;
     }
-    Py_DECREF(running_tuple);
+    Py_DECREF(rl);
 
     return 0;
 }
@@ -3184,6 +3184,47 @@ _asyncio__leave_task_impl(PyObject *module, PyObject *loop, PyObject *task)
 }
 
 
+/*********************** PyRunningLoopHolder ********************/
+
+
+static PyRunningLoopHolder *
+new_running_loop_holder(PyObject *loop)
+{
+    PyRunningLoopHolder *rl = PyObject_New(
+        PyRunningLoopHolder, &PyRunningLoopHolder_Type);
+    if (rl == NULL) {
+        return NULL;
+    }
+
+#if defined(HAVE_GETPID) && !defined(MS_WINDOWS)
+    rl->rl_pid = getpid();
+#endif
+
+    Py_INCREF(loop);
+    rl->rl_loop = loop;
+
+    return rl;
+}
+
+
+static void
+PyRunningLoopHolder_tp_dealloc(PyRunningLoopHolder *rl)
+{
+    Py_CLEAR(rl->rl_loop);
+    PyObject_Free(rl);
+}
+
+
+static PyTypeObject PyRunningLoopHolder_Type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "_RunningLoopHolder",
+    sizeof(PyRunningLoopHolder),
+    .tp_getattro = PyObject_GenericGetAttr,
+    .tp_flags = Py_TPFLAGS_DEFAULT,
+    .tp_dealloc = (destructor)PyRunningLoopHolder_tp_dealloc,
+};
+
+
 /*********************** Module **************************/
 
 
@@ -3212,7 +3253,6 @@ module_free(void *m)
 {
     Py_CLEAR(asyncio_mod);
     Py_CLEAR(inspect_isgenerator);
-    Py_CLEAR(os_getpid);
     Py_CLEAR(traceback_extract_stack);
     Py_CLEAR(asyncio_future_repr_info_func);
     Py_CLEAR(asyncio_get_event_loop_policy);
@@ -3295,9 +3335,6 @@ module_init(void)
     WITH_MOD("inspect")
     GET_MOD_ATTR(inspect_isgenerator, "isgenerator")
 
-    WITH_MOD("os")
-    GET_MOD_ATTR(os_getpid, "getpid")
-
     WITH_MOD("traceback")
     GET_MOD_ATTR(traceback_extract_stack, "extract_stack")
 
@@ -3370,6 +3407,9 @@ PyInit__asyncio(void)
     if (PyType_Ready(&TaskType) < 0) {
         return NULL;
     }
+    if (PyType_Ready(&PyRunningLoopHolder_Type) < 0) {
+        return NULL;
+    }
 
     PyObject *m = PyModule_Create(&_asynciomodule);
     if (m == NULL) {


From webhook-mailer at python.org  Tue Jan 23 16:26:09 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Tue, 23 Jan 2018 21:26:09 -0000
Subject: [Python-checkins] bpo-32436: Fix potential NULL dereference (#5286)
Message-ID: <mailman.22.1516742772.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/0bad4d63c654d93e1f32ff35026405a3987db5ca
commit: 0bad4d63c654d93e1f32ff35026405a3987db5ca
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-23T16:26:07-05:00
summary:

bpo-32436: Fix potential NULL dereference (#5286)

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index eb69fdd136b..af3dfce43c8 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -1055,6 +1055,10 @@ hamt_node_bitmap_without(PyHamtNode_Bitmap *self,
 #endif
 
                 PyHamtNode_Bitmap *clone = hamt_node_bitmap_clone(self);
+                if (clone == NULL) {
+                    return W_ERROR;
+                }
+
                 Py_SETREF(clone->b_array[val_idx],
                           (PyObject *)sub_node);  /* borrow */
 


From webhook-mailer at python.org  Tue Jan 23 20:11:51 2018
From: webhook-mailer at python.org (Barry Warsaw)
Date: Wed, 24 Jan 2018 01:11:51 -0000
Subject: [Python-checkins] bpo-32502: Discard 64-bit (and other invalid)
 hardware addresses (#5254)
Message-ID: <mailman.23.1516756312.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6b273f7f4056f8276f61a97c789d6bb4425e653c
commit: 6b273f7f4056f8276f61a97c789d6bb4425e653c
branch: master
author: Bo Bayles <bbayles at gmail.com>
committer: Barry Warsaw <barry at python.org>
date: 2018-01-23T20:11:44-05:00
summary:

bpo-32502: Discard 64-bit (and other invalid) hardware addresses (#5254)

files:
A Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst
M Lib/test/test_uuid.py
M Lib/uuid.py

diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py
index f21bd6dfa15..7af1d7aec79 100644
--- a/Lib/test/test_uuid.py
+++ b/Lib/test/test_uuid.py
@@ -311,6 +311,32 @@ def test_getnode(self):
         node2 = self.uuid.getnode()
         self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2))
 
+    # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers
+    # need not necessarily be 48 bits (e.g., EUI-64).
+    def test_uuid1_eui64(self):
+        # Confirm that uuid.getnode ignores hardware addresses larger than 48
+        # bits. Mock out each platform's *_getnode helper functions to return
+        # something just larger than 48 bits to test. This will cause
+        # uuid.getnode to fall back on uuid._random_getnode, which will
+        # generate a valid value.
+        too_large_getter = lambda: 1 << 48
+        with unittest.mock.patch.multiple(
+            self.uuid,
+            _node=None,  # Ignore any cached node value.
+            _NODE_GETTERS_WIN32=[too_large_getter],
+            _NODE_GETTERS_UNIX=[too_large_getter],
+        ):
+            node = self.uuid.getnode()
+        self.assertTrue(0 < node < (1 << 48), '%012x' % node)
+
+        # Confirm that uuid1 can use the generated node, i.e., the that
+        # uuid.getnode fell back on uuid._random_getnode() rather than using
+        # the value from too_large_getter above.
+        try:
+            self.uuid.uuid1(node=node)
+        except ValueError as e:
+            self.fail('uuid1 was given an invalid node ID')
+
     def test_uuid1(self):
         equal = self.assertEqual
 
diff --git a/Lib/uuid.py b/Lib/uuid.py
index b7433cb7192..ef7b3b59241 100644
--- a/Lib/uuid.py
+++ b/Lib/uuid.py
@@ -656,7 +656,12 @@ def _random_getnode():
 
 _node = None
 
-def getnode():
+_NODE_GETTERS_WIN32 = [_windll_getnode, _netbios_getnode, _ipconfig_getnode]
+
+_NODE_GETTERS_UNIX = [_unix_getnode, _ifconfig_getnode, _ip_getnode,
+                      _arp_getnode, _lanscan_getnode, _netstat_getnode]
+
+def getnode(*, getters=None):
     """Get the hardware address as a 48-bit positive integer.
 
     The first time this runs, it may launch a separate program, which could
@@ -669,19 +674,18 @@ def getnode():
         return _node
 
     if sys.platform == 'win32':
-        getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode]
+        getters = _NODE_GETTERS_WIN32
     else:
-        getters = [_unix_getnode, _ifconfig_getnode, _ip_getnode,
-                   _arp_getnode, _lanscan_getnode, _netstat_getnode]
+        getters = _NODE_GETTERS_UNIX
 
     for getter in getters + [_random_getnode]:
         try:
             _node = getter()
         except:
             continue
-        if _node is not None:
+        if (_node is not None) and (0 <= _node < (1 << 48)):
             return _node
-    assert False, '_random_getnode() returned None'
+    assert False, '_random_getnode() returned invalid value: {}'.format(_node)
 
 
 _last_timestamp = None
diff --git a/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst b/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst
new file mode 100644
index 00000000000..8338632aa2c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst
@@ -0,0 +1,2 @@
+uuid.uuid1 no longer raises an exception if a 64-bit hardware address is
+encountered.


From lp_benchmark_robot at intel.com  Tue Jan 23 21:53:40 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Tue, 23 Jan 2018 18:53:40 -0800
Subject: [Python-checkins] [4 down, 1 up,
 60 flat] Results for Python (master branch) 2018-01-23
Message-ID: <a104c055-8ddc-4252-9d49-dfbbd21b5552@orsmsx106.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-23 03:04:25-08:00.
  - commit:		6934831
  - previous commit:	579e0b8
  - revision date:	2018-01-23 11:09:31+02:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.884% |  +0.291%   |  +7.039%   |  +7.818%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 1.204% |  +0.755%   |  +22.797%  |  +5.162%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.292% |  -3.012%   |  +23.881%  |  +4.311%   |
+-----+------------------------+--------+------------+------------+------------+
| :-( |     call_method_unknown| 1.692% |  -3.310%   |  +19.399%  |  +6.670%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 3.797% |  -2.134%   |  +6.840%   |  +12.675%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.522% |  -2.763%   |  +9.817%   |  +12.928%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 0.543% |  +0.342%   |  +6.675%   |  +10.262%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.670% |  -0.491%   |  -0.936%   |  +8.792%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 5.556% |  -3.669%   |  +6.993%   |  +20.480%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 1.472% |  -0.048%   |  +21.345%  |  +11.568%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.237% |  +0.263%   |  +5.851%   |  +7.295%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 1.111% |  -1.373%   |  +4.674%   |  +6.358%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.914% |  +0.503%   |  +2.680%   |  +6.350%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.175% |  -2.478%   |  +12.401%  |  +12.088%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.722% |  -0.572%   |  +9.228%   |  +11.316%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.974% |  -0.594%   |  +6.526%   |  +11.983%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.691% |  +0.082%   |  +10.098%  |  +12.498%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 2.647% |  +0.117%   |  +9.100%   |  +9.409%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 1.104% |  -0.151%   |  +2.261%   |  +10.518%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 1.883% |  +7.383%   |  +1.488%   |  +12.026%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.622% |  +0.281%   |  +8.096%   |  +9.837%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.755% |  -0.613%   |  +46.748%  |  +12.724%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.569% |  +1.343%   |  +9.543%   |  +10.916%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.345% |  -0.676%   |  +17.143%  |  +13.792%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.158% |  +1.489%   |  +7.407%   |  +10.706%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 3.260% |  +2.656%   |  +4.832%   |  +3.793%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.664% |  -0.367%   |  -3.892%   |  +3.412%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.557% |  -0.725%   |  +4.211%   |  +7.483%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 0.848% |  +0.835%   |  +4.970%   |  +9.952%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 0.792% |  +1.778%   |  -0.279%   |  +18.931%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.372% |  -1.049%   |  +0.553%   |  +17.566%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.727% |  +0.327%   |  +1.974%   |  +17.694%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 3.796% |  -0.866%   |  +11.284%  |  +11.049%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.153% |  +0.123%   |  +0.059%   |  +10.512%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.116% |  -0.308%   |  +11.230%  |  +4.960%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.089% |  +0.050%   |  +2.491%   |  +5.031%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 0.972% |  +0.275%   |  +9.492%   |  +12.878%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.833% |  -0.602%   |  +3.160%   |  +11.870%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.219% |  +0.413%   |  -2.066%   |  +11.902%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.751% |  -2.608%   |  -5.650%   |  +7.580%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 1.633% |  -0.564%   |  +3.372%   |  +7.116%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.590% |  +0.341%   |  +8.761%   |  +15.471%  |
+-----+------------------------+--------+------------+------------+------------+
| :-( |             scimark_fft| 0.518% |  -2.701%   |  -5.148%   |  +3.978%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 1.730% |  -0.512%   |  +25.642%  |  +10.883%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.116% |  +0.262%   |  +4.711%   |  +4.519%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 1.080% |  -1.599%   |  +13.517%  |  +8.364%   |
+-----+------------------------+--------+------------+------------+------------+
| :-( | scimark_sparse_mat_mult| 1.063% |  -9.820%   |  -12.537%  |  +2.923%   |
+-----+------------------------+--------+------------+------------+------------+
| :-( |           spectral_norm| 1.771% |  -3.618%   |  +2.580%   |  +5.264%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.041% |  -1.707%   |  +6.352%   |  +7.639%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 2.985% |  -0.358%   |  +7.236%   |  +4.703%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.144% |  -2.296%   |  +16.994%  |  +8.910%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.557% |  -0.217%   |  +14.934%  |  +6.548%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 1.938% |  +0.207%   |  +9.498%   |  +7.341%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.610% |  -0.697%   |  +15.678%  |  +6.978%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 6.147% |  -0.321%   |  +10.228%  |  +10.258%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 3.053% |  +3.248%   |  +21.430%  |  +8.450%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 0.873% |  -0.585%   |  +6.817%   |  +7.411%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 5.070% |  +0.465%   |  +1.798%   |  +0.384%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 6.988% |  +1.035%   |  +7.291%   |  +22.730%  |
+-----+------------------------+--------+------------+------------+------------+
| :-) |           unpickle_list| 1.033% |  +3.234%   |  -2.174%   |  +15.073%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.931% |  -0.322%   |  +7.776%   |  +6.964%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 0.892% |  +3.131%   |  +5.132%   |  +9.680%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.401% |  -1.144%   |  +2.962%   |  +8.053%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 3.540% |  -0.511%   |  -6.781%   |  +12.616%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.113% |  +2.310%   |  +5.959%   |  +10.098%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/4-down-1-up-60-flat-results-for-python-master-branch-2018-01-23

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Tue Jan 23 22:17:07 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Wed, 24 Jan 2018 03:17:07 -0000
Subject: [Python-checkins] bpo-32436: Don't use native popcount() (also
 fixes bpo-32641) (#5292)
Message-ID: <mailman.24.1516763830.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/b7a80d543e1e94475ab9c8214f7a9eab4e63c9ab
commit: b7a80d543e1e94475ab9c8214f7a9eab4e63c9ab
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-23T22:17:04-05:00
summary:

bpo-32436: Don't use native popcount() (also fixes bpo-32641) (#5292)

files:
M Python/context.c
M Python/hamt.c

diff --git a/Python/context.c b/Python/context.c
index 2f1d0f5c342..5439531cd1e 100644
--- a/Python/context.c
+++ b/Python/context.c
@@ -1171,7 +1171,7 @@ get_token_missing(void)
 int
 PyContext_ClearFreeList(void)
 {
-    int size = ctx_freelist_len;
+    Py_ssize_t size = ctx_freelist_len;
     while (ctx_freelist_len) {
         PyContext *ctx = ctx_freelist;
         ctx_freelist = (PyContext *)ctx->ctx_weakreflist;
diff --git a/Python/hamt.c b/Python/hamt.c
index af3dfce43c8..81629e9bf1c 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -4,11 +4,6 @@
 #include "internal/pystate.h"
 #include "internal/hamt.h"
 
-/* popcnt support in Visual Studio */
-#ifdef _MSC_VER
-#include <intrin.h>
-#endif
-
 /*
 This file provides an implemention of an immutable mapping using the
 Hash Array Mapped Trie (or HAMT) datastructure.
@@ -440,18 +435,21 @@ hamt_bitpos(int32_t hash, uint32_t shift)
 static inline uint32_t
 hamt_bitcount(uint32_t i)
 {
-#if defined(__GNUC__) && (__GNUC__ > 4)
-    return (uint32_t)__builtin_popcountl(i);
-#elif defined(__clang__) && (__clang_major__ > 3)
-    return (uint32_t)__builtin_popcountl(i);
-#elif defined(_MSC_VER)
-    return (uint32_t)__popcnt(i);
-#else
-    /* https://graphics.stanford.edu/~seander/bithacks.html */
+    /* We could use native popcount instruction but that would
+       require to either add configure flags to enable SSE4.2
+       support or to detect it dynamically.  Otherwise, we have
+       a risk of CPython not working properly on older hardware.
+
+       In practice, there's no observable difference in
+       performance between using a popcount instruction or the
+       following fallback code.
+
+       The algorithm is copied from:
+       https://graphics.stanford.edu/~seander/bithacks.html
+    */
     i = i - ((i >> 1) & 0x55555555);
     i = (i & 0x33333333) + ((i >> 2) & 0x33333333);
     return ((i + (i >> 4) & 0xF0F0F0F) * 0x1010101) >> 24;
-#endif
 }
 
 static inline uint32_t


From webhook-mailer at python.org  Wed Jan 24 02:50:03 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Wed, 24 Jan 2018 07:50:03 -0000
Subject: [Python-checkins] bpo-32360: unittest.util: Use Counter instead of
 custom count function (GH-4994)
Message-ID: <mailman.25.1516780203.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/018e1b7aad8d1a33ee14aae5c466d581d31e2369
commit: 018e1b7aad8d1a33ee14aae5c466d581d31e2369
branch: master
author: Srinivas Reddy Thatiparthy (?????????? ?????? ?????????) <srinivasreddy at users.noreply.github.com>
committer: INADA Naoki <methane at users.noreply.github.com>
date: 2018-01-24T16:49:58+09:00
summary:

bpo-32360: unittest.util: Use Counter instead of custom count function (GH-4994)

files:
M Lib/unittest/util.py

diff --git a/Lib/unittest/util.py b/Lib/unittest/util.py
index 45485dcb0df..050eaed0b3f 100644
--- a/Lib/unittest/util.py
+++ b/Lib/unittest/util.py
@@ -1,6 +1,6 @@
 """Various utility functions."""
 
-from collections import namedtuple, OrderedDict
+from collections import namedtuple, Counter
 from os.path import commonprefix
 
 __unittest = True
@@ -153,17 +153,10 @@ def _count_diff_all_purpose(actual, expected):
         result.append(diff)
     return result
 
-def _ordered_count(iterable):
-    'Return dict of element counts, in the order they were first seen'
-    c = OrderedDict()
-    for elem in iterable:
-        c[elem] = c.get(elem, 0) + 1
-    return c
-
 def _count_diff_hashable(actual, expected):
     'Returns list of (cnt_act, cnt_exp, elem) triples where the counts differ'
     # elements must be hashable
-    s, t = _ordered_count(actual), _ordered_count(expected)
+    s, t = Counter(actual), Counter(expected)
     result = []
     for elem, cnt_s in s.items():
         cnt_t = t.get(elem, 0)


From solipsis at pitrou.net  Wed Jan 24 04:28:24 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 24 Jan 2018 09:28:24 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-2
Message-ID: <20180124092824.1.1EE0C96BF3399FFC@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, -7, 1] memory blocks, sum=-6
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogXcYMBp', '--timeout', '7200']

From webhook-mailer at python.org  Wed Jan 24 04:51:42 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 24 Jan 2018 09:51:42 -0000
Subject: [Python-checkins] bpo-32635: Fix a segfault when importing the
 crypt module with libxcrypt. (GH-5284) (#5295)
Message-ID: <mailman.26.1516787503.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6ccdad7b1f6fb316e4e85aa1eba8fbdc3169741e
commit: 6ccdad7b1f6fb316e4e85aa1eba8fbdc3169741e
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-24T10:51:39+01:00
summary:

bpo-32635: Fix a segfault when importing the crypt module with libxcrypt. (GH-5284) (#5295)

glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.
(cherry picked from commit e768c86ef442ef89004089a8a34ce5909ffb90f2)

files:
A Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
M Include/Python.h
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Include/Python.h b/Include/Python.h
index 775412b8c4e..a9327b02c81 100644
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -43,6 +43,9 @@
 #ifdef HAVE_UNISTD_H
 #include <unistd.h>
 #endif
+#ifdef HAVE_CRYPT_H
+#include <crypt.h>
+#endif
 
 /* For size_t? */
 #ifdef HAVE_STDDEF_H
diff --git a/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst b/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
new file mode 100644
index 00000000000..d411890f5d8
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
@@ -0,0 +1,2 @@
+Fix segfault of the crypt module when libxcrypt is provided instead of
+libcrypt at the system.
diff --git a/configure b/configure
index 609b5c04fd2..34773076f56 100755
--- a/configure
+++ b/configure
@@ -7035,7 +7035,7 @@ $as_echo "#define STDC_HEADERS 1" >>confdefs.h
 
 fi
 
-for ac_header in asm/types.h conio.h direct.h dlfcn.h errno.h \
+for ac_header in asm/types.h crypt.h conio.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h poll.h process.h pthread.h \
 shadow.h signal.h stdint.h stropts.h termios.h thread.h \
diff --git a/configure.ac b/configure.ac
index 7243de205be..c6453402162 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1690,7 +1690,7 @@ dnl AC_MSG_RESULT($cpp_type)
 
 # checks for header files
 AC_HEADER_STDC
-AC_CHECK_HEADERS(asm/types.h conio.h direct.h dlfcn.h errno.h \
+AC_CHECK_HEADERS(asm/types.h crypt.h conio.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h poll.h process.h pthread.h \
 shadow.h signal.h stdint.h stropts.h termios.h thread.h \
diff --git a/pyconfig.h.in b/pyconfig.h.in
index 22dc2dba6d6..11c4a66873c 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -130,6 +130,9 @@
 /* Define to 1 if you have the `copysign' function. */
 #undef HAVE_COPYSIGN
 
+/* Define to 1 if you have the <crypt.h> header file. */
+#undef HAVE_CRYPT_H
+
 /* Define to 1 if you have the `ctermid' function. */
 #undef HAVE_CTERMID
 


From webhook-mailer at python.org  Wed Jan 24 05:11:20 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 24 Jan 2018 10:11:20 -0000
Subject: [Python-checkins] bpo-32635: Fix a segfault when importing the
 crypt module with libxcrypt. (GH-5284) (#5296)
Message-ID: <mailman.27.1516788682.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/8e230e1eb3d38ad557b5dc5c31166afa35c01ff3
commit: 8e230e1eb3d38ad557b5dc5c31166afa35c01ff3
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-24T11:11:16+01:00
summary:

bpo-32635: Fix a segfault when importing the crypt module with libxcrypt. (GH-5284) (#5296)

glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.
(cherry picked from commit e768c86ef442ef89004089a8a34ce5909ffb90f2)

files:
A Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
M Include/Python.h
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Include/Python.h b/Include/Python.h
index 4c7c9a48c81..6177bad869c 100644
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -35,6 +35,9 @@
 #ifdef HAVE_UNISTD_H
 #include <unistd.h>
 #endif
+#ifdef HAVE_CRYPT_H
+#include <crypt.h>
+#endif
 
 /* For size_t? */
 #ifdef HAVE_STDDEF_H
diff --git a/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst b/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
new file mode 100644
index 00000000000..d411890f5d8
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2018-01-23-15-33-40.bpo-32635.qHwIZy.rst
@@ -0,0 +1,2 @@
+Fix segfault of the crypt module when libxcrypt is provided instead of
+libcrypt at the system.
diff --git a/configure b/configure
index ea1baef55d3..1e66117fffb 100755
--- a/configure
+++ b/configure
@@ -7809,7 +7809,7 @@ $as_echo "#define STDC_HEADERS 1" >>confdefs.h
 
 fi
 
-for ac_header in asm/types.h conio.h direct.h dlfcn.h errno.h \
+for ac_header in asm/types.h crypt.h conio.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h process.h pthread.h \
 sched.h shadow.h signal.h stropts.h termios.h \
diff --git a/configure.ac b/configure.ac
index fc1dba6b029..2eb511bf7f1 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2055,7 +2055,7 @@ dnl AC_MSG_RESULT($cpp_type)
 
 # checks for header files
 AC_HEADER_STDC
-AC_CHECK_HEADERS(asm/types.h conio.h direct.h dlfcn.h errno.h \
+AC_CHECK_HEADERS(asm/types.h crypt.h conio.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
 ieeefp.h io.h langinfo.h libintl.h process.h pthread.h \
 sched.h shadow.h signal.h stropts.h termios.h \
diff --git a/pyconfig.h.in b/pyconfig.h.in
index bc2693b0b21..6cef7b37b7b 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -140,6 +140,9 @@
 /* Define to 1 if you have the `copysign' function. */
 #undef HAVE_COPYSIGN
 
+/* Define to 1 if you have the <crypt.h> header file. */
+#undef HAVE_CRYPT_H
+
 /* Define to 1 if you have the `ctermid' function. */
 #undef HAVE_CTERMID
 


From webhook-mailer at python.org  Wed Jan 24 07:57:53 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Wed, 24 Jan 2018 12:57:53 -0000
Subject: [Python-checkins] bpo-17799: Explain real behaviour of sys.settrace
 and sys.setprofile (#4056)
Message-ID: <mailman.28.1516798676.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/131fd7f96c619bc7eaea956e45c6337175f4b27f
commit: 131fd7f96c619bc7eaea956e45c6337175f4b27f
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2018-01-24T20:57:49+08:00
summary:

bpo-17799: Explain real behaviour of sys.settrace and sys.setprofile (#4056)

files:
A Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
M Doc/c-api/init.rst
M Doc/library/sys.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 4ee61e8a1cf..86faf47f53b 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1300,7 +1300,6 @@ Python-level trace functions in previous versions.
    | :const:`PyTrace_C_RETURN`    | Function object being called.        |
    +------------------------------+--------------------------------------+
 
-
 .. c:var:: int PyTrace_CALL
 
    The value of the *what* parameter to a :c:type:`Py_tracefunc` function when a new
@@ -1357,16 +1356,18 @@ Python-level trace functions in previous versions.
    function as its first parameter, and may be any Python object, or *NULL*.  If
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
-   profile function is called for all monitored events except the line-number
-   events.
+   profile function is called for all monitored events except :const:`PyTrace_LINE`
+   and :const:`PyTrace_EXCEPTION`.
 
 
 .. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj)
 
    Set the tracing function to *func*.  This is similar to
    :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number
-   events.
-
+   events and does not receive any event related to C function objects being called. Any
+   trace function registered using :c:func:`PyEval_SetTrace` will not receive
+   :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or :const:`PyTrace_C_RETURN`
+   as a value for the *what* parameter.
 
 .. _advanced-debugging:
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 54281a3f9a0..ab08f61f6c0 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1084,13 +1084,38 @@ always available.
    Set the system's profile function, which allows you to implement a Python source
    code profiler in Python.  See chapter :ref:`profile` for more information on the
    Python profiler.  The system's profile function is called similarly to the
-   system's trace function (see :func:`settrace`), but it isn't called for each
-   executed line of code (only on call and return, but the return event is reported
-   even when an exception has been set).  The function is thread-specific, but
-   there is no way for the profiler to know about context switches between threads,
-   so it does not make sense to use this in the presence of multiple threads. Also,
+   system's trace function (see :func:`settrace`), but it is called with different events,
+   for example it isn't called for each executed line of code (only on call and return,
+   but the return event is reported even when an exception has been set). The function is
+   thread-specific, but there is no way for the profiler to know about context switches between
+   threads, so it does not make sense to use this in the presence of multiple threads. Also,
    its return value is not used, so it can simply return ``None``.
 
+   Profile functions should have three arguments: *frame*, *event*, and
+   *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
+   ``'return'``, ``'c_call'``, ``'c_return'``, or ``'c_exception'``. *arg* depends
+   on the event type.
+
+   The events have the following meaning:
+
+   ``'call'``
+      A function is called (or some other code block entered).  The
+      profile function is called; *arg* is ``None``.
+
+   ``'return'``
+      A function (or other code block) is about to return.  The profile
+      function is called; *arg* is the value that will be returned, or ``None``
+      if the event is caused by an exception being raised.
+
+   ``'c_call'``
+      A C function is about to be called.  This may be an extension function or
+      a built-in.  *arg* is the C function object.
+
+   ``'c_return'``
+      A C function has returned. *arg* is the C function object.
+
+   ``'c_exception'``
+      A C function has raised an exception.  *arg* is the C function object.
 
 .. function:: setrecursionlimit(limit)
 
@@ -1137,8 +1162,8 @@ always available.
 
    Trace functions should have three arguments: *frame*, *event*, and
    *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
-   ``'line'``, ``'return'``, ``'exception'``, ``'c_call'``, ``'c_return'``, or
-   ``'c_exception'``, ``'opcode'``. *arg* depends on the event type.
+   ``'line'``, ``'return'``, ``'exception'`` or ``'opcode'``.  *arg* depends on
+   the event type.
 
    The trace function is invoked (with *event* set to ``'call'``) whenever a new
    local scope is entered; it should return a reference to a local trace
@@ -1175,16 +1200,6 @@ always available.
       tuple ``(exception, value, traceback)``; the return value specifies the
       new local trace function.
 
-   ``'c_call'``
-      A C function is about to be called.  This may be an extension function or
-      a built-in.  *arg* is the C function object.
-
-   ``'c_return'``
-      A C function has returned. *arg* is the C function object.
-
-   ``'c_exception'``
-      A C function has raised an exception.  *arg* is the C function object.
-
    ``'opcode'``
       The interpreter is about to execute a new opcode (see :mod:`dis` for
       opcode details).  The local trace function is called; *arg* is
diff --git a/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst b/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
new file mode 100644
index 00000000000..ccc52f60eee
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
@@ -0,0 +1,2 @@
+Explain real behaviour of sys.settrace and sys.setprofile and their C-API counterparts
+regarding which type of events are received in each function. Patch by Pablo Galindo Salgado.


From webhook-mailer at python.org  Wed Jan 24 08:27:20 2018
From: webhook-mailer at python.org (Barry Warsaw)
Date: Wed, 24 Jan 2018 13:27:20 -0000
Subject: [Python-checkins] [3.6] bpo-32502: Discard 64-bit (and other
 invalid) hardware addresses (GH-5254) (#5290)
Message-ID: <mailman.29.1516800444.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d69794f4df81de731cc66dc82136e28bee691e1e
commit: d69794f4df81de731cc66dc82136e28bee691e1e
branch: 3.6
author: Bo Bayles <bbayles at gmail.com>
committer: Barry Warsaw <barry at python.org>
date: 2018-01-24T08:27:14-05:00
summary:

[3.6] bpo-32502: Discard 64-bit (and other invalid) hardware addresses (GH-5254) (#5290)

* [3.6] bpo-32502: Discard 64-bit (and other invalid) hardware addresses (GH-5254).
(cherry picked from commit 6b273f7f4056f8276f61a97c789d6bb4425e653c)

files:
A Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst
M Lib/test/test_uuid.py
M Lib/uuid.py

diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py
index 4373fba12ac..aa3de74cef0 100644
--- a/Lib/test/test_uuid.py
+++ b/Lib/test/test_uuid.py
@@ -304,6 +304,32 @@ def test_getnode(self):
         node2 = uuid.getnode()
         self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2))
 
+    # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers
+    # need not necessarily be 48 bits (e.g., EUI-64).
+    def test_uuid1_eui64(self):
+        # Confirm that uuid.getnode ignores hardware addresses larger than 48
+        # bits. Mock out each platform's *_getnode helper functions to return
+        # something just larger than 48 bits to test. This will cause
+        # uuid.getnode to fall back on uuid._random_getnode, which will
+        # generate a valid value.
+        too_large_getter = lambda: 1 << 48
+        with unittest.mock.patch.multiple(
+            uuid,
+            _node=None,  # Ignore any cached node value.
+            _NODE_GETTERS_WIN32=[too_large_getter],
+            _NODE_GETTERS_UNIX=[too_large_getter],
+        ):
+            node = uuid.getnode()
+        self.assertTrue(0 < node < (1 << 48), '%012x' % node)
+
+        # Confirm that uuid1 can use the generated node, i.e., the that
+        # uuid.getnode fell back on uuid._random_getnode() rather than using
+        # the value from too_large_getter above.
+        try:
+            uuid.uuid1(node=node)
+        except ValueError as e:
+            self.fail('uuid1 was given an invalid node ID')
+
     @unittest.skipUnless(importable('ctypes'), 'requires ctypes')
     def test_uuid1(self):
         equal = self.assertEqual
diff --git a/Lib/uuid.py b/Lib/uuid.py
index 36c956e2b0f..3cd67fcfd09 100644
--- a/Lib/uuid.py
+++ b/Lib/uuid.py
@@ -540,6 +540,11 @@ def _random_getnode():
 
 _node = None
 
+_NODE_GETTERS_WIN32 = [_windll_getnode, _netbios_getnode, _ipconfig_getnode]
+
+_NODE_GETTERS_UNIX = [_unixdll_getnode, _ifconfig_getnode, _ip_getnode,
+                      _arp_getnode, _lanscan_getnode, _netstat_getnode]
+
 def getnode():
     """Get the hardware address as a 48-bit positive integer.
 
@@ -555,18 +560,18 @@ def getnode():
 
     import sys
     if sys.platform == 'win32':
-        getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode]
+        getters = _NODE_GETTERS_WIN32
     else:
-        getters = [_unixdll_getnode, _ifconfig_getnode, _ip_getnode,
-                   _arp_getnode, _lanscan_getnode, _netstat_getnode]
+        getters = _NODE_GETTERS_UNIX
 
     for getter in getters + [_random_getnode]:
         try:
             _node = getter()
         except:
             continue
-        if _node is not None:
+        if (_node is not None) and (0 <= _node < (1 << 48)):
             return _node
+    assert False, '_random_getnode() returned invalid value: {}'.format(_node)
 
 _last_timestamp = None
 
diff --git a/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst b/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst
new file mode 100644
index 00000000000..8338632aa2c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst
@@ -0,0 +1,2 @@
+uuid.uuid1 no longer raises an exception if a 64-bit hardware address is
+encountered.


From webhook-mailer at python.org  Wed Jan 24 09:44:55 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Wed, 24 Jan 2018 14:44:55 -0000
Subject: [Python-checkins] bpo-17799: Explain real behaviour of sys.settrace
 and sys.setprofile (GH-4056) (#5298)
Message-ID: <mailman.30.1516805099.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/fd844efa9c31e1f00e04b07940875b9dacff3d77
commit: fd844efa9c31e1f00e04b07940875b9dacff3d77
branch: 3.6
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-24T22:44:53+08:00
summary:

bpo-17799: Explain real behaviour of sys.settrace and sys.setprofile (GH-4056) (#5298)

(cherry picked from commit 131fd7f96c619bc7eaea956e45c6337175f4b27f)

files:
A Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
M Doc/c-api/init.rst
M Doc/library/sys.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 2965bc931a6..a2c55fea9f7 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1137,15 +1137,19 @@ Python-level trace functions in previous versions.
    function as its first parameter, and may be any Python object, or *NULL*.  If
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
-   profile function is called for all monitored events except the line-number
-   events.
+   profile function is called for all monitored events except :const:`PyTrace_LINE`
+   and :const:`PyTrace_EXCEPTION`.
 
 
 .. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj)
 
    Set the tracing function to *func*.  This is similar to
    :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number
-   events.
+   events and does not receive any event related to C function objects being called. Any
+   trace function registered using :c:func:`PyEval_SetTrace` will not receive
+   :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or :const:`PyTrace_C_RETURN`
+   as a value for the *what* parameter.
+
 
 .. c:function:: PyObject* PyEval_GetCallStats(PyObject *self)
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index dd51ffd56cc..68521df37eb 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1005,13 +1005,38 @@ always available.
    Set the system's profile function, which allows you to implement a Python source
    code profiler in Python.  See chapter :ref:`profile` for more information on the
    Python profiler.  The system's profile function is called similarly to the
-   system's trace function (see :func:`settrace`), but it isn't called for each
-   executed line of code (only on call and return, but the return event is reported
-   even when an exception has been set).  The function is thread-specific, but
-   there is no way for the profiler to know about context switches between threads,
-   so it does not make sense to use this in the presence of multiple threads. Also,
+   system's trace function (see :func:`settrace`), but it is called with different events,
+   for example it isn't called for each executed line of code (only on call and return,
+   but the return event is reported even when an exception has been set). The function is
+   thread-specific, but there is no way for the profiler to know about context switches between
+   threads, so it does not make sense to use this in the presence of multiple threads. Also,
    its return value is not used, so it can simply return ``None``.
 
+   Profile functions should have three arguments: *frame*, *event*, and
+   *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
+   ``'return'``, ``'c_call'``, ``'c_return'``, or ``'c_exception'``. *arg* depends
+   on the event type.
+
+   The events have the following meaning:
+
+   ``'call'``
+      A function is called (or some other code block entered).  The
+      profile function is called; *arg* is ``None``.
+
+   ``'return'``
+      A function (or other code block) is about to return.  The profile
+      function is called; *arg* is the value that will be returned, or ``None``
+      if the event is caused by an exception being raised.
+
+   ``'c_call'``
+      A C function is about to be called.  This may be an extension function or
+      a built-in.  *arg* is the C function object.
+
+   ``'c_return'``
+      A C function has returned. *arg* is the C function object.
+
+   ``'c_exception'``
+      A C function has raised an exception.  *arg* is the C function object.
 
 .. function:: setrecursionlimit(limit)
 
@@ -1058,8 +1083,8 @@ always available.
 
    Trace functions should have three arguments: *frame*, *event*, and
    *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
-   ``'line'``, ``'return'``, ``'exception'``, ``'c_call'``, ``'c_return'``, or
-   ``'c_exception'``. *arg* depends on the event type.
+   ``'line'``, ``'return'`` or ``'exception'``.  *arg* depends on
+   the event type.
 
    The trace function is invoked (with *event* set to ``'call'``) whenever a new
    local scope is entered; it should return a reference to a local trace
@@ -1094,16 +1119,6 @@ always available.
       tuple ``(exception, value, traceback)``; the return value specifies the
       new local trace function.
 
-   ``'c_call'``
-      A C function is about to be called.  This may be an extension function or
-      a built-in.  *arg* is the C function object.
-
-   ``'c_return'``
-      A C function has returned. *arg* is the C function object.
-
-   ``'c_exception'``
-      A C function has raised an exception.  *arg* is the C function object.
-
    Note that as an exception is propagated down the chain of callers, an
    ``'exception'`` event is generated at each level.
 
diff --git a/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst b/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
new file mode 100644
index 00000000000..ccc52f60eee
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
@@ -0,0 +1,2 @@
+Explain real behaviour of sys.settrace and sys.setprofile and their C-API counterparts
+regarding which type of events are received in each function. Patch by Pablo Galindo Salgado.


From webhook-mailer at python.org  Wed Jan 24 09:53:46 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Wed, 24 Jan 2018 14:53:46 -0000
Subject: [Python-checkins] bpo-17799: Explain real behaviour of sys.settrace
 and sys.setprofile (GH-4056). (#5299)
Message-ID: <mailman.31.1516805628.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/e64a47b37d0c592fd162b2f51e79ecfd046b45ec
commit: e64a47b37d0c592fd162b2f51e79ecfd046b45ec
branch: 2.7
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-24T22:53:42+08:00
summary:

bpo-17799: Explain real behaviour of sys.settrace and sys.setprofile (GH-4056). (#5299)

(cherry picked from commit 131fd7f96c619bc7eaea956e45c6337175f4b27f)

files:
A Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
M Doc/c-api/init.rst
M Doc/library/sys.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 7643a3786c4..eade4121a25 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1057,15 +1057,19 @@ in previous versions.
    function as its first parameter, and may be any Python object, or *NULL*.  If
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
-   profile function is called for all monitored events except the line-number
-   events.
+   profile function is called for all monitored events except :const:`PyTrace_LINE`
+   and :const:`PyTrace_EXCEPTION`.
 
 
 .. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj)
 
    Set the tracing function to *func*.  This is similar to
    :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number
-   events.
+   events and does not receive any event related to C function objects being called. Any
+   trace function registered using :c:func:`PyEval_SetTrace` will not receive
+   :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or :const:`PyTrace_C_RETURN`
+   as a value for the *what* parameter.
+
 
 .. c:function:: PyObject* PyEval_GetCallStats(PyObject *self)
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 31e644e3947..5a7647b7e99 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -857,13 +857,38 @@ always available.
    Set the system's profile function, which allows you to implement a Python source
    code profiler in Python.  See chapter :ref:`profile` for more information on the
    Python profiler.  The system's profile function is called similarly to the
-   system's trace function (see :func:`settrace`), but it isn't called for each
-   executed line of code (only on call and return, but the return event is reported
-   even when an exception has been set).  The function is thread-specific, but
-   there is no way for the profiler to know about context switches between threads,
-   so it does not make sense to use this in the presence of multiple threads. Also,
+   system's trace function (see :func:`settrace`), but it is called with different events,
+   for example it isn't called for each executed line of code (only on call and return,
+   but the return event is reported even when an exception has been set). The function is
+   thread-specific, but there is no way for the profiler to know about context switches between
+   threads, so it does not make sense to use this in the presence of multiple threads. Also,
    its return value is not used, so it can simply return ``None``.
 
+   Profile functions should have three arguments: *frame*, *event*, and
+   *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
+   ``'return'``, ``'c_call'``, ``'c_return'``, or ``'c_exception'``. *arg* depends
+   on the event type.
+
+   The events have the following meaning:
+
+   ``'call'``
+      A function is called (or some other code block entered).  The
+      profile function is called; *arg* is ``None``.
+
+   ``'return'``
+      A function (or other code block) is about to return.  The profile
+      function is called; *arg* is the value that will be returned, or ``None``
+      if the event is caused by an exception being raised.
+
+   ``'c_call'``
+      A C function is about to be called.  This may be an extension function or
+      a built-in.  *arg* is the C function object.
+
+   ``'c_return'``
+      A C function has returned. *arg* is the C function object.
+
+   ``'c_exception'``
+      A C function has raised an exception.  *arg* is the C function object.
 
 .. function:: setrecursionlimit(limit)
 
@@ -890,8 +915,8 @@ always available.
 
    Trace functions should have three arguments: *frame*, *event*, and
    *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
-   ``'line'``, ``'return'``, ``'exception'``, ``'c_call'``, ``'c_return'``, or
-   ``'c_exception'``. *arg* depends on the event type.
+   ``'line'``, ``'return'`` or ``'exception'``.  *arg* depends on
+   the event type.
 
    The trace function is invoked (with *event* set to ``'call'``) whenever a new
    local scope is entered; it should return a reference to a local trace
@@ -926,16 +951,6 @@ always available.
       tuple ``(exception, value, traceback)``; the return value specifies the
       new local trace function.
 
-   ``'c_call'``
-      A C function is about to be called.  This may be an extension function or
-      a built-in.  *arg* is the C function object.
-
-   ``'c_return'``
-      A C function has returned. *arg* is the C function object.
-
-   ``'c_exception'``
-      A C function has raised an exception.  *arg* is the C function object.
-
    Note that as an exception is propagated down the chain of callers, an
    ``'exception'`` event is generated at each level.
 
@@ -1078,4 +1093,3 @@ always available.
 .. rubric:: Citations
 
 .. [C99] ISO/IEC 9899:1999.  "Programming languages -- C."  A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ .
-
diff --git a/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst b/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
new file mode 100644
index 00000000000..ccc52f60eee
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2018-01-22-21-13-46.bpo-17799.rdZ-Vk.rst
@@ -0,0 +1,2 @@
+Explain real behaviour of sys.settrace and sys.setprofile and their C-API counterparts
+regarding which type of events are received in each function. Patch by Pablo Galindo Salgado.


From webhook-mailer at python.org  Wed Jan 24 11:02:51 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 24 Jan 2018 16:02:51 -0000
Subject: [Python-checkins] bpo-28046: Remove MACHDEPPATH from
 Modules/Setup.dist (#5289)
Message-ID: <mailman.32.1516809772.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/5de15f1d5ff09085620f63f0597d0920d75be719
commit: 5de15f1d5ff09085620f63f0597d0920d75be719
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-24T17:02:41+01:00
summary:

bpo-28046: Remove MACHDEPPATH from Modules/Setup.dist (#5289)

Makefile $MACHDEPPATH uses $PLATDIR, but this variable was removed.
So remove also $MACHDEPPATH.

Remove also $EXTRAMACHDEPPATH.

files:
M Modules/Setup.dist

diff --git a/Modules/Setup.dist b/Modules/Setup.dist
index 239550e26b5..4b463b28d67 100644
--- a/Modules/Setup.dist
+++ b/Modules/Setup.dist
@@ -93,11 +93,7 @@ SITEPATH=
 # Standard path components for test modules
 TESTPATH=
 
-# Path components for machine- or system-dependent modules and shared libraries
-MACHDEPPATH=:$(PLATDIR)
-EXTRAMACHDEPPATH=
-
-COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH)$(MACHDEPPATH)$(EXTRAMACHDEPPATH)
+COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH)
 PYTHONPATH=$(COREPYTHONPATH)
 
 


From webhook-mailer at python.org  Wed Jan 24 11:03:42 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 24 Jan 2018 16:03:42 -0000
Subject: [Python-checkins] bpo-32030: Add _PyCoreConfig.module_search_paths
 (#4954)
Message-ID: <mailman.33.1516809823.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/8ded5b803705328749622256701b3f08a9d6c5ab
commit: 8ded5b803705328749622256701b3f08a9d6c5ab
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-24T17:03:28+01:00
summary:

bpo-32030: Add _PyCoreConfig.module_search_paths (#4954)

_PyCoreConfig_Read() is now responsible to compute sys.path.
So sys.path is now computed before calling _Py_InitializeCore().

Changes:

* Add module_search_path, module_search_paths, executable, prefix,
  base_prefix, exec_prefix and base_exec_prefix to _PyCoreConfig.
* _PyMainInterpreterConfig_Read() now only converts wchar_t** lists
  into a Python list, it doesn't compute sys.path anymore.

files:
M Include/pystate.h
M Modules/main.c
M PC/getpathp.c

diff --git a/Include/pystate.h b/Include/pystate.h
index d004be5e906..a19c1ed3b81 100644
--- a/Include/pystate.h
+++ b/Include/pystate.h
@@ -26,29 +26,24 @@ typedef PyObject* (*_PyFrameEvalFunction)(struct _frame *, int);
 
 typedef struct {
     int install_signal_handlers;  /* Install signal handlers? -1 means unset */
-    int ignore_environment; /* -E */
+
+    int ignore_environment; /* -E, Py_IgnoreEnvironmentFlag */
     int use_hash_seed;      /* PYTHONHASHSEED=x */
     unsigned long hash_seed;
-    int _disable_importlib; /* Needed by freeze_importlib */
     const char *allocator;  /* Memory allocator: _PyMem_SetupAllocators() */
-    int dev_mode;           /* -X dev */
-    int faulthandler;       /* -X faulthandler */
-    int tracemalloc;        /* -X tracemalloc=N */
-    int import_time;        /* -X importtime */
+    int dev_mode;           /* PYTHONDEVMODE, -X dev */
+    int faulthandler;       /* PYTHONFAULTHANDLER, -X faulthandler */
+    int tracemalloc;        /* PYTHONTRACEMALLOC, -X tracemalloc=N */
+    int import_time;        /* PYTHONPROFILEIMPORTTIME, -X importtime */
     int show_ref_count;     /* -X showrefcount */
     int show_alloc_count;   /* -X showalloccount */
     int dump_refs;          /* PYTHONDUMPREFS */
     int malloc_stats;       /* PYTHONMALLOCSTATS */
     int coerce_c_locale;    /* PYTHONCOERCECLOCALE, -1 means unknown */
     int coerce_c_locale_warn; /* PYTHONCOERCECLOCALE=warn */
-    int utf8_mode;          /* -X utf8 or PYTHONUTF8 environment variable,
-                               -1 means unknown */
+    int utf8_mode;          /* PYTHONUTF8, -X utf8; -1 means unknown */
 
-    wchar_t *module_search_path_env; /* PYTHONPATH environment variable */
-    wchar_t *home;          /* PYTHONHOME environment variable,
-                               see also Py_SetPythonHome(). */
     wchar_t *program_name;  /* Program name, see also Py_GetProgramName() */
-
     int argc;               /* Number of command line arguments,
                                -1 means unset */
     wchar_t **argv;         /* Command line arguments */
@@ -59,6 +54,24 @@ typedef struct {
 
     int nwarnoption;        /* Number of warnings options */
     wchar_t **warnoptions;  /* Warnings options */
+
+    /* Path configuration inputs */
+    wchar_t *module_search_path_env; /* PYTHONPATH environment variable */
+    wchar_t *home;          /* PYTHONHOME environment variable,
+                               see also Py_SetPythonHome(). */
+
+    /* Path configuration outputs */
+    int nmodule_search_path;        /* Number of sys.path paths,
+                                       -1 means unset */
+    wchar_t **module_search_paths;  /* sys.path paths */
+    wchar_t *executable;    /* sys.executable */
+    wchar_t *prefix;        /* sys.prefix */
+    wchar_t *base_prefix;   /* sys.base_prefix */
+    wchar_t *exec_prefix;   /* sys.exec_prefix */
+    wchar_t *base_exec_prefix;  /* sys.base_exec_prefix */
+
+    /* Private fields */
+    int _disable_importlib; /* Needed by freeze_importlib */
 } _PyCoreConfig;
 
 #define _PyCoreConfig_INIT \
@@ -67,7 +80,8 @@ typedef struct {
         .use_hash_seed = -1, \
         .coerce_c_locale = -1, \
         .utf8_mode = -1, \
-        .argc = -1}
+        .argc = -1, \
+        .nmodule_search_path = -1}
 /* Note: _PyCoreConfig_INIT sets other fields to 0/NULL */
 
 /* Placeholders while working on the new configuration API
diff --git a/Modules/main.c b/Modules/main.c
index dff7894bf35..2e632e292c2 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -1048,36 +1048,6 @@ config_init_warnoptions(_PyCoreConfig *config, _Py_CommandLineDetails *cmdline)
 }
 
 
-static int
-main_config_init_warnoptions(_PyMainInterpreterConfig *main_config,
-                             const _PyCoreConfig *config)
-{
-    PyObject *warnoptions = PyList_New(0);
-    if (warnoptions == NULL) {
-        return -1;
-    }
-
-    for (int i = 0; i < config->nwarnoption; i++) {
-        PyObject *option = PyUnicode_FromWideChar(config->warnoptions[i], -1);
-        if (option == NULL) {
-            goto error;
-        }
-        if (PyList_Append(warnoptions, option)) {
-            Py_DECREF(option);
-            goto error;
-        }
-        Py_DECREF(option);
-    }
-
-    main_config->warnoptions = warnoptions;
-    return 0;
-
-error:
-    Py_DECREF(warnoptions);
-    return -1;
-}
-
-
 /* Get warning options from PYTHONWARNINGS environment variable.
    Return 0 on success.
    Set pymain->err and return -1 on error. */
@@ -1324,34 +1294,25 @@ pymain_init_core_argv(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
 }
 
 
-static int
-main_config_init_argv(_PyMainInterpreterConfig *main_config,
-                      const _PyCoreConfig *config)
+static PyObject*
+wstrlist_as_pylist(int len, wchar_t **list)
 {
-    if (config->argc < 0) {
-        return 0;
-    }
-
-    int argc = config->argc;
-    wchar_t** argv = config->argv;
-    assert(argc >= 1 && argv != NULL);
+    assert(list != NULL || len < 1);
 
-    PyObject *list = PyList_New(argc);
-    if (list == NULL) {
-        return -1;
+    PyObject *pylist = PyList_New(len);
+    if (pylist == NULL) {
+        return NULL;
     }
 
-    for (int i = 0; i < argc; i++) {
-        PyObject *v = PyUnicode_FromWideChar(argv[i], -1);
+    for (int i = 0; i < len; i++) {
+        PyObject *v = PyUnicode_FromWideChar(list[i], -1);
         if (v == NULL) {
-            Py_DECREF(list);
-            return -1;
+            Py_DECREF(pylist);
+            return NULL;
         }
-        PyList_SET_ITEM(list, i, v);
+        PyList_SET_ITEM(pylist, i, v);
     }
-
-    main_config->argv = list;
-    return 0;
+    return pylist;
 }
 
 
@@ -1948,11 +1909,22 @@ pymain_read_conf_impl(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
         return -1;
     }
 
+    /* Global configuration variables should be set to read the core
+       configuration, and then get again to get updated values.
+
+       _PyPathConfig_Init() tests !Py_FrozenFlag to avoid some warnings.
+       Moreover, on Windows, it modifies Py_IsolatedFlag and Py_NoSiteFlag
+       variables if a "._pth" file is found. */
+    pymain_set_global_config(pymain, cmdline);
+
     err = _PyCoreConfig_Read(config);
     if (_Py_INIT_FAILED(err)) {
         pymain->err = err;
         return -1;
     }
+
+    Py_UTF8Mode = pymain->config.utf8_mode;
+    pymain_get_global_config(pymain, cmdline);
     return 0;
 }
 
@@ -1976,6 +1948,8 @@ pymain_read_conf(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
     int locale_coerced = 0;
     int loops = 0;
     int init_ignore_env = pymain->config.ignore_environment;
+    int init_isolated = cmdline->isolated;
+    int init_no_site = cmdline->no_site_import;
 
     while (1) {
         int utf8_mode = pymain->config.utf8_mode;
@@ -2033,9 +2007,12 @@ pymain_read_conf(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
 
         /* Reset the configuration, except UTF-8 Mode. Set Py_UTF8Mode for
            Py_DecodeLocale(). Reset Py_IgnoreEnvironmentFlag, modified by
-           pymain_read_conf_impl(). */
+           pymain_read_conf_impl(). Reset Py_IsolatedFlag and Py_NoSiteFlag
+           modified by _PyCoreConfig_Read(). */
         Py_UTF8Mode = pymain->config.utf8_mode;
         Py_IgnoreEnvironmentFlag = init_ignore_env;
+        Py_IsolatedFlag = init_isolated;
+        Py_NoSiteFlag = init_no_site;
         _PyCoreConfig_Clear(&pymain->config);
         pymain_clear_cmdline(pymain, cmdline);
         pymain_get_global_config(pymain, cmdline);
@@ -2083,6 +2060,101 @@ config_init_locale(_PyCoreConfig *config)
 }
 
 
+static _PyInitError
+config_init_module_search_paths(_PyCoreConfig *config)
+{
+    assert(config->module_search_paths == NULL);
+    assert(config->nmodule_search_path < 0);
+
+    config->nmodule_search_path = 0;
+
+    const wchar_t *sys_path = Py_GetPath();
+    const wchar_t delim = DELIM;
+    const wchar_t *p = sys_path;
+    while (1) {
+        p = wcschr(sys_path, delim);
+        if (p == NULL) {
+            p = sys_path + wcslen(sys_path); /* End of string */
+        }
+
+        size_t path_len = (p - sys_path);
+        wchar_t *path = PyMem_RawMalloc((path_len + 1) * sizeof(wchar_t));
+        if (path == NULL) {
+            return _Py_INIT_NO_MEMORY();
+        }
+        memcpy(path, sys_path, path_len * sizeof(wchar_t));
+        path[path_len] = L'\0';
+
+        _PyInitError err = wstrlist_append(&config->nmodule_search_path,
+                                           &config->module_search_paths,
+                                           path);
+        PyMem_RawFree(path);
+        if (_Py_INIT_FAILED(err)) {
+            return err;
+        }
+
+        if (*p == '\0') {
+            break;
+        }
+        sys_path = p + 1;
+    }
+    return _Py_INIT_OK();
+}
+
+
+static _PyInitError
+config_init_path_config(_PyCoreConfig *config)
+{
+    _PyInitError err = _PyPathConfig_Init(config);
+    if (_Py_INIT_FAILED(err)) {
+        return err;
+    }
+
+    if (config->nmodule_search_path < 0) {
+        err = config_init_module_search_paths(config);
+        if (_Py_INIT_FAILED(err)) {
+            return err;
+        }
+    }
+
+    if (config->executable == NULL) {
+        config->executable = _PyMem_RawWcsdup(Py_GetProgramFullPath());
+        if (config->executable == NULL) {
+            return _Py_INIT_NO_MEMORY();
+        }
+    }
+
+    if (config->prefix == NULL) {
+        config->prefix = _PyMem_RawWcsdup(Py_GetPrefix());
+        if (config->prefix == NULL) {
+            return _Py_INIT_NO_MEMORY();
+        }
+    }
+
+    if (config->exec_prefix == NULL) {
+        config->exec_prefix = _PyMem_RawWcsdup(Py_GetExecPrefix());
+        if (config->exec_prefix == NULL) {
+            return _Py_INIT_NO_MEMORY();
+        }
+    }
+
+    if (config->base_prefix == NULL) {
+        config->base_prefix = _PyMem_RawWcsdup(config->prefix);
+        if (config->base_prefix == NULL) {
+            return _Py_INIT_NO_MEMORY();
+        }
+    }
+
+    if (config->base_exec_prefix == NULL) {
+        config->base_exec_prefix = _PyMem_RawWcsdup(config->exec_prefix);
+        if (config->base_exec_prefix == NULL) {
+            return _Py_INIT_NO_MEMORY();
+        }
+    }
+
+    return _Py_INIT_OK();
+}
+
 /* Read configuration settings from standard locations
  *
  * This function doesn't make any changes to the interpreter state - it
@@ -2140,6 +2212,12 @@ _PyCoreConfig_Read(_PyCoreConfig *config)
         config->install_signal_handlers = 1;
     }
 
+    if (!config->_disable_importlib) {
+        err = config_init_path_config(config);
+        if (_Py_INIT_FAILED(err)) {
+            return err;
+        }
+    }
     return _Py_INIT_OK();
 }
 
@@ -2152,26 +2230,33 @@ _PyCoreConfig_Clear(_PyCoreConfig *config)
         PyMem_RawFree(ATTR); \
         ATTR = NULL; \
     } while (0)
+#define CLEAR_WSTRLIST(LEN, LIST) \
+    do { \
+        clear_wstrlist(LEN, LIST); \
+        LEN = 0; \
+        LIST = NULL; \
+    } while (0)
 
     CLEAR(config->module_search_path_env);
     CLEAR(config->home);
     CLEAR(config->program_name);
     CLEAR(config->program);
 
-    if (config->argc >= 0) {
-        clear_wstrlist(config->argc, config->argv);
-        config->argc = -1;
-        config->argv = NULL;
-    }
+    CLEAR_WSTRLIST(config->argc, config->argv);
+    config->argc = -1;
 
-    clear_wstrlist(config->nwarnoption, config->warnoptions);
-    config->nwarnoption = 0;
-    config->warnoptions = NULL;
+    CLEAR_WSTRLIST(config->nwarnoption, config->warnoptions);
+    CLEAR_WSTRLIST(config->nxoption, config->xoptions);
+    CLEAR_WSTRLIST(config->nmodule_search_path, config->module_search_paths);
+    config->nmodule_search_path = -1;
 
-    clear_wstrlist(config->nxoption, config->xoptions);
-    config->nxoption = 0;
-    config->xoptions = NULL;
+    CLEAR(config->executable);
+    CLEAR(config->prefix);
+    CLEAR(config->base_prefix);
+    CLEAR(config->exec_prefix);
+    CLEAR(config->base_exec_prefix);
 #undef CLEAR
+#undef CLEAR_WSTRLIST
 }
 
 
@@ -2190,6 +2275,16 @@ _PyCoreConfig_Copy(_PyCoreConfig *config, const _PyCoreConfig *config2)
             } \
         } \
     } while (0)
+#define COPY_WSTRLIST(LEN, LIST) \
+    do { \
+        if (config2->LIST != NULL) { \
+            config->LIST = copy_wstrlist(config2->LEN, config2->LIST); \
+            if (config->LIST == NULL) { \
+                return -1; \
+            } \
+        } \
+        config->LEN = config2->LEN; \
+    } while (0)
 
     COPY_ATTR(ignore_environment);
     COPY_ATTR(use_hash_seed);
@@ -2211,33 +2306,20 @@ _PyCoreConfig_Copy(_PyCoreConfig *config, const _PyCoreConfig *config2)
     COPY_STR_ATTR(program_name);
     COPY_STR_ATTR(program);
 
-    if (config2->argc >= 0) {
-        wchar_t **argv = copy_wstrlist(config2->argc, config2->argv);
-        if (argv == NULL) {
-            return -1;
-        }
-        config->argv = argv;
-    }
-    COPY_ATTR(argc);
-
-    if (config2->nwarnoption > 0) {
-        config->warnoptions = copy_wstrlist(config2->nwarnoption, config2->warnoptions);
-        if (config->warnoptions == NULL) {
-            return -1;
-        }
-    }
-    COPY_ATTR(nwarnoption);
+    COPY_WSTRLIST(argc, argv);
+    COPY_WSTRLIST(nwarnoption, warnoptions);
+    COPY_WSTRLIST(nxoption, xoptions);
+    COPY_WSTRLIST(nmodule_search_path, module_search_paths);
 
-    if (config2->nxoption > 0) {
-        config->xoptions = copy_wstrlist(config2->nxoption, config2->xoptions);
-        if (config->xoptions == NULL) {
-            return -1;
-        }
-    }
-    COPY_ATTR(nxoption);
+    COPY_STR_ATTR(executable);
+    COPY_STR_ATTR(prefix);
+    COPY_STR_ATTR(base_prefix);
+    COPY_STR_ATTR(exec_prefix);
+    COPY_STR_ATTR(base_exec_prefix);
 
 #undef COPY_ATTR
 #undef COPY_STR_ATTR
+#undef COPY_WSTRLIST
     return 0;
 }
 
@@ -2313,52 +2395,10 @@ _PyMainInterpreterConfig_Copy(_PyMainInterpreterConfig *config,
 
 
 
-static PyObject *
-create_path_list(const wchar_t *path, wchar_t delim)
-{
-    int i, n;
-    const wchar_t *p;
-    PyObject *v;
-
-    n = 1;
-    p = path;
-    while ((p = wcschr(p, delim)) != NULL) {
-        n++;
-        p++;
-    }
-    v = PyList_New(n);
-    if (v == NULL) {
-        return NULL;
-    }
-    for (i = 0; ; i++) {
-        p = wcschr(path, delim);
-        if (p == NULL) {
-            p = path + wcslen(path); /* End of string */
-        }
-        PyObject *w = PyUnicode_FromWideChar(path, (Py_ssize_t)(p - path));
-        if (w == NULL) {
-            Py_DECREF(v);
-            return NULL;
-        }
-        PyList_SET_ITEM(v, i, w);
-        if (*p == '\0') {
-            break;
-        }
-        path = p+1;
-    }
-    return v;
-}
-
-
 _PyInitError
 _PyMainInterpreterConfig_Read(_PyMainInterpreterConfig *main_config,
                               const _PyCoreConfig *config)
 {
-    _PyInitError err = _PyPathConfig_Init(config);
-    if (_Py_INIT_FAILED(err)) {
-        return err;
-    }
-
     if (main_config->install_signal_handlers < 0) {
         main_config->install_signal_handlers = config->install_signal_handlers;
     }
@@ -2370,59 +2410,46 @@ _PyMainInterpreterConfig_Read(_PyMainInterpreterConfig *main_config,
         }
     }
 
-    if (main_config->argv == NULL) {
-        if (main_config_init_argv(main_config, config) < 0) {
-            return _Py_INIT_ERR("failed to create sys.argv");
-        }
-    }
-
-    if (main_config->warnoptions == NULL) {
-        if (main_config_init_warnoptions(main_config, config) < 0) {
-            return _Py_INIT_NO_MEMORY();
-        }
-    }
-
-    if (main_config->module_search_path == NULL &&
-        !config->_disable_importlib)
-    {
-        wchar_t *sys_path = Py_GetPath();
-        main_config->module_search_path = create_path_list(sys_path, DELIM);
-        if (main_config->module_search_path == NULL) {
-            return _Py_INIT_NO_MEMORY();
-        }
-    }
-
-    if (main_config->executable == NULL) {
-        main_config->executable = PyUnicode_FromWideChar(Py_GetProgramFullPath(), -1);
-        if (main_config->executable == NULL) {
-            return _Py_INIT_NO_MEMORY();
-        }
-    }
+#define COPY_WSTR(ATTR) \
+    do { \
+        if (main_config->ATTR == NULL) { \
+            main_config->ATTR = PyUnicode_FromWideChar(config->ATTR, -1); \
+            if (main_config->ATTR == NULL) { \
+                return _Py_INIT_NO_MEMORY(); \
+            } \
+        } \
+    } while (0)
+#define COPY_WSTRLIST(ATTR, LEN, LIST) \
+    do { \
+        if (ATTR == NULL) { \
+            ATTR = wstrlist_as_pylist(LEN, LIST); \
+            if (ATTR == NULL) { \
+                return _Py_INIT_NO_MEMORY(); \
+            } \
+        } \
+    } while (0)
 
-    if (main_config->prefix == NULL) {
-        main_config->prefix = PyUnicode_FromWideChar(Py_GetPrefix(), -1);
-        if (main_config->prefix == NULL) {
-            return _Py_INIT_NO_MEMORY();
-        }
+    COPY_WSTRLIST(main_config->warnoptions,
+                  config->nwarnoption, config->warnoptions);
+    if (config->argc >= 0) {
+        COPY_WSTRLIST(main_config->argv,
+                      config->argc, config->argv);
     }
 
-    if (main_config->exec_prefix == NULL) {
-        main_config->exec_prefix = PyUnicode_FromWideChar(Py_GetExecPrefix(), -1);
-        if (main_config->exec_prefix == NULL) {
-            return _Py_INIT_NO_MEMORY();
-        }
-    }
+    if (!config->_disable_importlib) {
+        COPY_WSTR(executable);
+        COPY_WSTR(prefix);
+        COPY_WSTR(base_prefix);
+        COPY_WSTR(exec_prefix);
+        COPY_WSTR(base_exec_prefix);
 
-    if (main_config->base_prefix == NULL) {
-        Py_INCREF(main_config->prefix);
-        main_config->base_prefix = main_config->prefix;
+        COPY_WSTRLIST(main_config->module_search_path,
+                      config->nmodule_search_path, config->module_search_paths);
     }
 
-    if (main_config->base_exec_prefix == NULL) {
-        Py_INCREF(main_config->exec_prefix);
-        main_config->base_exec_prefix = main_config->exec_prefix;
-    }
     return _Py_INIT_OK();
+#undef COPY_WSTR
+#undef COPY_WSTRLIST
 }
 
 
diff --git a/PC/getpathp.c b/PC/getpathp.c
index c4977e79979..e90a643ab82 100644
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -692,6 +692,9 @@ calculate_pth_file(_PyPathConfig *config, wchar_t *prefix)
         return 0;
     }
 
+    /* FIXME, bpo-32030: Global configuration variables should not be modified
+       here, _PyPathConfig_Init() is called early in Python initialization:
+       see pymain_cmdline(). */
     return read_pth_file(config, prefix, spbuffer,
                          &Py_IsolatedFlag, &Py_NoSiteFlag);
 }


From webhook-mailer at python.org  Wed Jan 24 12:51:32 2018
From: webhook-mailer at python.org (Barry Warsaw)
Date: Wed, 24 Jan 2018 17:51:32 -0000
Subject: [Python-checkins] bpo-32651 Recommend getpass.getuser() (#5301)
Message-ID: <mailman.34.1516816294.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d499031f5f33c3ef71be7c81ef52eee6b47c42e1
commit: d499031f5f33c3ef71be7c81ef52eee6b47c42e1
branch: master
author: Barry Warsaw <barry at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-24T12:51:29-05:00
summary:

bpo-32651 Recommend getpass.getuser() (#5301)

* bpo-32651 - In the documentation for os.getlogin(), recommend getpass.getuser()

files:
M Doc/library/getpass.rst
M Doc/library/os.rst

diff --git a/Doc/library/getpass.rst b/Doc/library/getpass.rst
index 5eb9f04a8da..82b11919a3d 100644
--- a/Doc/library/getpass.rst
+++ b/Doc/library/getpass.rst
@@ -42,8 +42,10 @@ The :mod:`getpass` module provides two functions:
    Return the "login name" of the user.
 
    This function checks the environment variables :envvar:`LOGNAME`,
-   :envvar:`USER`, :envvar:`LNAME` and :envvar:`USERNAME`, in order, and returns
-   the value of the first one which is set to a non-empty string.  If none are set,
-   the login name from the password database is returned on systems which support
-   the :mod:`pwd` module, otherwise, an exception is raised.
+   :envvar:`USER`, :envvar:`LNAME` and :envvar:`USERNAME`, in order, and
+   returns the value of the first one which is set to a non-empty string.  If
+   none are set, the login name from the password database is returned on
+   systems which support the :mod:`pwd` module, otherwise, an exception is
+   raised.
 
+   In general, this function should be preferred over :func:`os.getlogin()`.
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index ee08853087f..9cb34deb042 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -325,10 +325,11 @@ process and user.
 .. function:: getlogin()
 
    Return the name of the user logged in on the controlling terminal of the
-   process.  For most purposes, it is more useful to use the environment
-   variables :envvar:`LOGNAME` or :envvar:`USERNAME` to find out who the user
-   is, or ``pwd.getpwuid(os.getuid())[0]`` to get the login name of the current
-   real user id.
+   process.  For most purposes, it is more useful to use
+   :func:`getpass.getuser` since the latter checks the environment variables
+   :envvar:`LOGNAME` or :envvar:`USERNAME` to find out who the user is, and
+   falls back to ``pwd.getpwuid(os.getuid())[0]`` to get the login name of the
+   current real user id.
 
    Availability: Unix, Windows.
 


From webhook-mailer at python.org  Wed Jan 24 14:10:00 2018
From: webhook-mailer at python.org (Barry Warsaw)
Date: Wed, 24 Jan 2018 19:10:00 -0000
Subject: [Python-checkins] bpo-32651 Recommend getpass.getuser() (GH-5301)
 (#5304)
Message-ID: <mailman.35.1516821002.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/196b8cbab2b5044b92077f0b3c07c798fff68bc6
commit: 196b8cbab2b5044b92077f0b3c07c798fff68bc6
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Barry Warsaw <barry at python.org>
date: 2018-01-24T14:09:56-05:00
summary:

bpo-32651 Recommend getpass.getuser() (GH-5301) (#5304)

* bpo-32651 - In the documentation for os.getlogin(), recommend getpass.getuser()
(cherry picked from commit d499031f5f33c3ef71be7c81ef52eee6b47c42e1)

files:
M Doc/library/getpass.rst
M Doc/library/os.rst

diff --git a/Doc/library/getpass.rst b/Doc/library/getpass.rst
index 5eb9f04a8da..82b11919a3d 100644
--- a/Doc/library/getpass.rst
+++ b/Doc/library/getpass.rst
@@ -42,8 +42,10 @@ The :mod:`getpass` module provides two functions:
    Return the "login name" of the user.
 
    This function checks the environment variables :envvar:`LOGNAME`,
-   :envvar:`USER`, :envvar:`LNAME` and :envvar:`USERNAME`, in order, and returns
-   the value of the first one which is set to a non-empty string.  If none are set,
-   the login name from the password database is returned on systems which support
-   the :mod:`pwd` module, otherwise, an exception is raised.
+   :envvar:`USER`, :envvar:`LNAME` and :envvar:`USERNAME`, in order, and
+   returns the value of the first one which is set to a non-empty string.  If
+   none are set, the login name from the password database is returned on
+   systems which support the :mod:`pwd` module, otherwise, an exception is
+   raised.
 
+   In general, this function should be preferred over :func:`os.getlogin()`.
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index 974ab2d481e..b2722168bc2 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -325,10 +325,11 @@ process and user.
 .. function:: getlogin()
 
    Return the name of the user logged in on the controlling terminal of the
-   process.  For most purposes, it is more useful to use the environment
-   variables :envvar:`LOGNAME` or :envvar:`USERNAME` to find out who the user
-   is, or ``pwd.getpwuid(os.getuid())[0]`` to get the login name of the current
-   real user id.
+   process.  For most purposes, it is more useful to use
+   :func:`getpass.getuser` since the latter checks the environment variables
+   :envvar:`LOGNAME` or :envvar:`USERNAME` to find out who the user is, and
+   falls back to ``pwd.getpwuid(os.getuid())[0]`` to get the login name of the
+   current real user id.
 
    Availability: Unix, Windows.
 


From webhook-mailer at python.org  Wed Jan 24 14:40:38 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Wed, 24 Jan 2018 19:40:38 -0000
Subject: [Python-checkins] bpo-32410: Improve sock_sendfile tests (#5294)
Message-ID: <mailman.36.1516822841.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/0a5e71b4c70aab87125a54d7a59765e18d7583a4
commit: 0a5e71b4c70aab87125a54d7a59765e18d7583a4
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-24T21:40:35+02:00
summary:

bpo-32410: Improve sock_sendfile tests (#5294)

* Rename sock_sendfile test methods
* Make sock_sendfile tests more stable

files:
M Lib/test/test_asyncio/test_unix_events.py

diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py
index 8646bf765e9..5bd76d30d2d 100644
--- a/Lib/test/test_asyncio/test_unix_events.py
+++ b/Lib/test/test_asyncio/test_unix_events.py
@@ -484,8 +484,11 @@ def prepare(self):
         self.run_loop(self.loop.sock_connect(sock, (support.HOST, port)))
 
         def cleanup():
-            proto.transport.close()
-            self.run_loop(proto.wait_closed())
+            if proto.transport is not None:
+                # can be None if the task was cancelled before
+                # connection_made callback
+                proto.transport.close()
+                self.run_loop(proto.wait_closed())
 
             server.close()
             self.run_loop(server.wait_closed())
@@ -494,7 +497,7 @@ def cleanup():
 
         return sock, proto
 
-    def test_success(self):
+    def test_sock_sendfile_success(self):
         sock, proto = self.prepare()
         ret = self.run_loop(self.loop.sock_sendfile(sock, self.file))
         sock.close()
@@ -504,7 +507,7 @@ def test_success(self):
         self.assertEqual(proto.data, self.DATA)
         self.assertEqual(self.file.tell(), len(self.DATA))
 
-    def test_with_offset_and_count(self):
+    def test_sock_sendfile_with_offset_and_count(self):
         sock, proto = self.prepare()
         ret = self.run_loop(self.loop.sock_sendfile(sock, self.file,
                                                     1000, 2000))
@@ -515,7 +518,7 @@ def test_with_offset_and_count(self):
         self.assertEqual(self.file.tell(), 3000)
         self.assertEqual(ret, 2000)
 
-    def test_sendfile_not_available(self):
+    def test_sock_sendfile_not_available(self):
         sock, proto = self.prepare()
         with mock.patch('asyncio.unix_events.os', spec=[]):
             with self.assertRaisesRegex(events.SendfileNotAvailableError,
@@ -524,7 +527,7 @@ def test_sendfile_not_available(self):
                                                               0, None))
         self.assertEqual(self.file.tell(), 0)
 
-    def test_sendfile_not_a_file(self):
+    def test_sock_sendfile_not_a_file(self):
         sock, proto = self.prepare()
         f = object()
         with self.assertRaisesRegex(events.SendfileNotAvailableError,
@@ -533,7 +536,7 @@ def test_sendfile_not_a_file(self):
                                                           0, None))
         self.assertEqual(self.file.tell(), 0)
 
-    def test_sendfile_iobuffer(self):
+    def test_sock_sendfile_iobuffer(self):
         sock, proto = self.prepare()
         f = io.BytesIO()
         with self.assertRaisesRegex(events.SendfileNotAvailableError,
@@ -542,7 +545,7 @@ def test_sendfile_iobuffer(self):
                                                           0, None))
         self.assertEqual(self.file.tell(), 0)
 
-    def test_sendfile_not_regular_file(self):
+    def test_sock_sendfile_not_regular_file(self):
         sock, proto = self.prepare()
         f = mock.Mock()
         f.fileno.return_value = -1
@@ -552,7 +555,7 @@ def test_sendfile_not_regular_file(self):
                                                           0, None))
         self.assertEqual(self.file.tell(), 0)
 
-    def test_sendfile_zero_size(self):
+    def test_sock_sendfile_zero_size(self):
         sock, proto = self.prepare()
         fname = support.TESTFN + '.suffix'
         with open(fname, 'wb') as f:
@@ -568,7 +571,7 @@ def test_sendfile_zero_size(self):
         self.assertEqual(ret, 0)
         self.assertEqual(self.file.tell(), 0)
 
-    def test_mix_sendfile_and_regular_send(self):
+    def test_sock_sendfile_mix_with_regular_send(self):
         buf = b'1234567890' * 1024 * 1024  # 10 MB
         sock, proto = self.prepare()
         self.run_loop(self.loop.sock_sendall(sock, buf))
@@ -582,7 +585,7 @@ def test_mix_sendfile_and_regular_send(self):
         self.assertEqual(proto.data, expected)
         self.assertEqual(self.file.tell(), len(self.DATA))
 
-    def test_cancel1(self):
+    def test_sock_sendfile_cancel1(self):
         sock, proto = self.prepare()
 
         fut = self.loop.create_future()
@@ -595,7 +598,7 @@ def test_cancel1(self):
         with self.assertRaises(KeyError):
             self.loop._selector.get_key(sock)
 
-    def test_cancel2(self):
+    def test_sock_sendfile_cancel2(self):
         sock, proto = self.prepare()
 
         fut = self.loop.create_future()
@@ -608,7 +611,7 @@ def test_cancel2(self):
         with self.assertRaises(KeyError):
             self.loop._selector.get_key(sock)
 
-    def test_blocking_error(self):
+    def test_sock_sendfile_blocking_error(self):
         sock, proto = self.prepare()
 
         fileno = self.file.fileno()
@@ -621,7 +624,7 @@ def test_blocking_error(self):
         self.assertIsNotNone(key)
         fut.add_done_callback.assert_called_once_with(mock.ANY)
 
-    def test_os_error_first_call(self):
+    def test_sock_sendfile_os_error_first_call(self):
         sock, proto = self.prepare()
 
         fileno = self.file.fileno()
@@ -635,7 +638,7 @@ def test_os_error_first_call(self):
         self.assertIsInstance(exc, events.SendfileNotAvailableError)
         self.assertEqual(0, self.file.tell())
 
-    def test_os_error_next_call(self):
+    def test_sock_sendfile_os_error_next_call(self):
         sock, proto = self.prepare()
 
         fileno = self.file.fileno()
@@ -652,7 +655,7 @@ def test_os_error_next_call(self):
         self.assertIs(exc, err)
         self.assertEqual(1000, self.file.tell())
 
-    def test_exception(self):
+    def test_sock_sendfile_exception(self):
         sock, proto = self.prepare()
 
         fileno = self.file.fileno()


From webhook-mailer at python.org  Wed Jan 24 15:14:35 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Wed, 24 Jan 2018 20:14:35 -0000
Subject: [Python-checkins] bpo-32636: Fix @asyncio.coroutine debug mode bug
 exposed by gh-5250 (#5291)
Message-ID: <mailman.37.1516824877.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/fb5a7ad421ac20c49218ee4b86fb0d85ca4cd664
commit: fb5a7ad421ac20c49218ee4b86fb0d85ca4cd664
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-24T15:14:33-05:00
summary:

bpo-32636: Fix @asyncio.coroutine debug mode bug exposed by gh-5250 (#5291)

files:
M Lib/asyncio/coroutines.py
M Lib/test/test_asyncio/test_tasks.py

diff --git a/Lib/asyncio/coroutines.py b/Lib/asyncio/coroutines.py
index 5a29100321f..c7fcd442558 100644
--- a/Lib/asyncio/coroutines.py
+++ b/Lib/asyncio/coroutines.py
@@ -132,8 +132,9 @@ def coro(*args, **kw):
                         res = yield from await_meth()
             return res
 
+    coro = types.coroutine(coro)
     if not _DEBUG:
-        wrapper = types.coroutine(coro)
+        wrapper = coro
     else:
         @functools.wraps(func)
         def wrapper(*args, **kwds):
diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index daa1ff927e6..5e83a54ff28 100644
--- a/Lib/test/test_asyncio/test_tasks.py
+++ b/Lib/test/test_asyncio/test_tasks.py
@@ -9,6 +9,7 @@
 import random
 import re
 import sys
+import textwrap
 import types
 import unittest
 import weakref
@@ -3090,6 +3091,22 @@ def coro2():
         result = self.loop.run_until_complete(inner())
         self.assertEqual(['ok1', 'ok2'], result)
 
+    def test_debug_mode_interop(self):
+        # https://bugs.python.org/issue32636
+        code = textwrap.dedent("""
+            import asyncio
+
+            async def native_coro():
+                pass
+
+            @asyncio.coroutine
+            def old_style_coro():
+                yield from native_coro()
+
+            asyncio.run(old_style_coro())
+        """)
+        assert_python_ok("-c", code, PYTHONASYNCIODEBUG="1")
+
 
 if __name__ == '__main__':
     unittest.main()


From webhook-mailer at python.org  Wed Jan 24 15:15:16 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Wed, 24 Jan 2018 20:15:16 -0000
Subject: [Python-checkins] bpo-32636: Fix two bugs in test_asyncio (#5302)
Message-ID: <mailman.38.1516824918.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/789e359f51d2b27bea01b8c6c3bf090aaedf8839
commit: 789e359f51d2b27bea01b8c6c3bf090aaedf8839
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-24T15:15:14-05:00
summary:

bpo-32636: Fix two bugs in test_asyncio (#5302)

files:
M Lib/test/test_asyncio/test_base_events.py
M Lib/test/test_asyncio/test_tasks.py

diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index 8d72df6a72e..6489f50f272 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -1917,6 +1917,7 @@ def test_nonbinary_file(self):
 
     def test_nonstream_socket(self):
         sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        sock.setblocking(False)
         self.addCleanup(sock.close)
         with self.assertRaisesRegex(ValueError, "only SOCK_STREAM type"):
             self.run_loop(self.loop.sock_sendfile(sock, self.file))
diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index 5e83a54ff28..822338f932e 100644
--- a/Lib/test/test_asyncio/test_tasks.py
+++ b/Lib/test/test_asyncio/test_tasks.py
@@ -2448,7 +2448,7 @@ def get_loop(self):
         self.loop = asyncio.new_event_loop()
         try:
             fut = Fut(loop=self.loop)
-            self.loop.call_later(0.1, fut.set_result(1))
+            self.loop.call_later(0.1, fut.set_result, 1)
             task = asyncio.Task(coro(), loop=self.loop)
             res = self.loop.run_until_complete(task)
         finally:


From webhook-mailer at python.org  Wed Jan 24 15:36:29 2018
From: webhook-mailer at python.org (Barry Warsaw)
Date: Wed, 24 Jan 2018 20:36:29 -0000
Subject: [Python-checkins] bpo-32248 - Implement `ResourceReader` and
 `get_resource_reader()` for zipimport (#5248)
Message-ID: <mailman.39.1516826190.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6f6eb35f9bee18f54945f09664344f2d118ed89f
commit: 6f6eb35f9bee18f54945f09664344f2d118ed89f
branch: master
author: Barry Warsaw <barry at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-24T15:36:21-05:00
summary:

 bpo-32248 - Implement `ResourceReader` and `get_resource_reader()` for zipimport (#5248)

files:
M Doc/whatsnew/3.7.rst
M Lib/importlib/resources.py
M Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
M Modules/clinic/zipimport.c.h
M Modules/zipimport.c

diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 0418aa6d7ad..20be158ab0a 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -328,8 +328,12 @@ importlib.resources
 This module provides several new APIs and one new ABC for access to, opening,
 and reading *resources* inside packages.  Resources are roughly akin to files
 inside of packages, but they needn't be actual files on the physical file
-system.  Module loaders can provide :class:`importlib.abc.ResourceReader`
-implementations to support this new module's API.
+system.  Module loaders can provide a :meth:`get_resource_reader()` function
+which returns a :class:`importlib.abc.ResourceReader` instance to support this
+new API.  Built-in file path loaders and zip file loaders both support this.
+(see the PyPI package
+`importlib_resources <http://importlib-resources.readthedocs.io/en/latest/>`_
+as a compatible back port for older Python versions).
 
 
 Improved Modules
diff --git a/Lib/importlib/resources.py b/Lib/importlib/resources.py
index 20888dfa8ea..bf6d7035850 100644
--- a/Lib/importlib/resources.py
+++ b/Lib/importlib/resources.py
@@ -12,7 +12,7 @@
 from typing import Iterator, Optional, Set, Union   # noqa: F401
 from typing import cast
 from typing.io import BinaryIO, TextIO
-from zipfile import ZipFile
+from zipimport import ZipImportError
 
 
 Package = Union[str, ModuleType]
@@ -216,38 +216,7 @@ def is_resource(package: Package, name: str) -> bool:
     # contents doesn't necessarily mean it's a resource.  Directories are not
     # resources, so let's try to find out if it's a directory or not.
     path = Path(package.__spec__.origin).parent / name
-    if path.is_file():
-        return True
-    if path.is_dir():
-        return False
-    # If it's not a file and it's not a directory, what is it?  Well, this
-    # means the file doesn't exist on the file system, so it probably lives
-    # inside a zip file.  We have to crack open the zip, look at its table of
-    # contents, and make sure that this entry doesn't have sub-entries.
-    archive_path = package.__spec__.loader.archive   # type: ignore
-    package_directory = Path(package.__spec__.origin).parent
-    with ZipFile(archive_path) as zf:
-        toc = zf.namelist()
-    relpath = package_directory.relative_to(archive_path)
-    candidate_path = relpath / name
-    for entry in toc:
-        try:
-            relative_to_candidate = Path(entry).relative_to(candidate_path)
-        except ValueError:
-            # The two paths aren't relative to each other so we can ignore it.
-            continue
-        # Since directories aren't explicitly listed in the zip file, we must
-        # infer their 'directory-ness' by looking at the number of path
-        # components in the path relative to the package resource we're
-        # looking up.  If there are zero additional parts, it's a file, i.e. a
-        # resource.  If there are more than zero it's a directory, i.e. not a
-        # resource.  It has to be one of these two cases.
-        return len(relative_to_candidate.parts) == 0
-    # I think it's impossible to get here.  It would mean that we are looking
-    # for a resource in a zip file, there's an entry matching it in the return
-    # value of contents(), but we never actually found it in the zip's table of
-    # contents.
-    raise AssertionError('Impossible situation')
+    return path.is_file()
 
 
 def contents(package: Package) -> Iterator[str]:
@@ -268,38 +237,85 @@ def contents(package: Package) -> Iterator[str]:
             not package.__spec__.has_location):
         return []
     package_directory = Path(package.__spec__.origin).parent
-    try:
-        yield from os.listdir(str(package_directory))
-    except (NotADirectoryError, FileNotFoundError):
-        # The package is probably in a zip file.
-        archive_path = getattr(package.__spec__.loader, 'archive', None)
-        if archive_path is None:
-            raise
-        relpath = package_directory.relative_to(archive_path)
-        with ZipFile(archive_path) as zf:
-            toc = zf.namelist()
-        subdirs_seen = set()                        # type: Set
-        for filename in toc:
-            path = Path(filename)
-            # Strip off any path component parts that are in common with the
-            # package directory, relative to the zip archive's file system
-            # path.  This gives us all the parts that live under the named
-            # package inside the zip file.  If the length of these subparts is
-            # exactly 1, then it is situated inside the package.  The resulting
-            # length will be 0 if it's above the package, and it will be
-            # greater than 1 if it lives in a subdirectory of the package
-            # directory.
-            #
-            # However, since directories themselves don't appear in the zip
-            # archive as a separate entry, we need to return the first path
-            # component for any case that has > 1 subparts -- but only once!
-            if path.parts[:len(relpath.parts)] != relpath.parts:
+    yield from os.listdir(str(package_directory))
+
+
+# Private implementation of ResourceReader and get_resource_reader() for
+# zipimport.  Don't use these directly!  We're implementing these in Python
+# because 1) it's easier, 2) zipimport will likely get rewritten in Python
+# itself at some point, so doing this all in C would just be a waste of
+# effort.
+
+class _ZipImportResourceReader(resources_abc.ResourceReader):
+    """Private class used to support ZipImport.get_resource_reader().
+
+    This class is allowed to reference all the innards and private parts of
+    the zipimporter.
+    """
+
+    def __init__(self, zipimporter, fullname):
+        self.zipimporter = zipimporter
+        self.fullname = fullname
+
+    def open_resource(self, resource):
+        path = f'{self.fullname}/{resource}'
+        try:
+            return BytesIO(self.zipimporter.get_data(path))
+        except OSError:
+            raise FileNotFoundError
+
+    def resource_path(self, resource):
+        # All resources are in the zip file, so there is no path to the file.
+        # Raising FileNotFoundError tells the higher level API to extract the
+        # binary data and create a temporary file.
+        raise FileNotFoundError
+
+    def is_resource(self, name):
+        # Maybe we could do better, but if we can get the data, it's a
+        # resource.  Otherwise it isn't.
+        path = f'{self.fullname}/{name}'
+        try:
+            self.zipimporter.get_data(path)
+        except OSError:
+            return False
+        return True
+
+    def contents(self):
+        # This is a bit convoluted, because fullname will be a module path,
+        # but _files is a list of file names relative to the top of the
+        # archive's namespace.  We want to compare file paths to find all the
+        # names of things inside the module represented by fullname.  So we
+        # turn the module path of fullname into a file path relative to the
+        # top of the archive, and then we iterate through _files looking for
+        # names inside that "directory".
+        fullname_path = Path(self.zipimporter.get_filename(self.fullname))
+        relative_path = fullname_path.relative_to(self.zipimporter.archive)
+        # Don't forget that fullname names a package, so its path will include
+        # __init__.py, which we want to ignore.
+        assert relative_path.name == '__init__.py'
+        package_path = relative_path.parent
+        subdirs_seen = set()
+        for filename in self.zipimporter._files:
+            try:
+                relative = Path(filename).relative_to(package_path)
+            except ValueError:
                 continue
-            subparts = path.parts[len(relpath.parts):]
-            if len(subparts) == 1:
-                yield subparts[0]
-            elif len(subparts) > 1:
-                subdir = subparts[0]
-                if subdir not in subdirs_seen:
-                    subdirs_seen.add(subdir)
-                    yield subdir
+            # If the path of the file (which is relative to the top of the zip
+            # namespace), relative to the package given when the resource
+            # reader was created, has a parent, then it's a name in a
+            # subdirectory and thus we skip it.
+            parent_name = relative.parent.name
+            if len(parent_name) == 0:
+                yield relative.name
+            elif parent_name not in subdirs_seen:
+                subdirs_seen.add(parent_name)
+                yield parent_name
+
+
+def _zipimport_get_resource_reader(zipimporter, fullname):
+    try:
+        if not zipimporter.is_package(fullname):
+            return None
+    except ZipImportError:
+        return None
+    return _ZipImportResourceReader(zipimporter, fullname)
diff --git a/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst b/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
index a41fde94b3d..d821cf91bdd 100644
--- a/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
+++ b/Misc/NEWS.d/next/Library/2017-12-15-15-34-12.bpo-32248.zmO8G2.rst
@@ -1,13 +1,6 @@
-Add :class:`importlib.abc.ResourceReader` as an ABC to provide a
-unified API for reading resources contained within packages. Loaders
-wishing to support resource reading are expected to implement the
-``get_resource_reader(fullname)`` method.
-
-Also add :mod:`importlib.resources` as the stdlib port of the
-``importlib_resources`` PyPI package. The modules provides a high-level
-API for end-users to read resources in a nicer fashion than having to
-directly interact with low-level details such as loaders.
-
-Thanks to this work, :class:`importlib.abc.ResourceLoader` has now
-been documented as deprecated due to its under-specified nature and
-lack of features as provided by :class:`importlib.abc.ResourceReader`.
+Add :mod:`importlib.resources` and :class:`importlib.abc.ResourceReader` as
+the unified API for reading resources contained within packages.  Loaders
+wishing to support resource reading must implement the
+:meth:`get_resource_reader()` method.  File-based and zipimport-based loaders
+both implement these APIs.  :class:`importlib.abc.ResourceLoader` is
+deprecated in favor of these new APIs.
diff --git a/Modules/clinic/zipimport.c.h b/Modules/clinic/zipimport.c.h
index 548aade6efa..565b0653036 100644
--- a/Modules/clinic/zipimport.c.h
+++ b/Modules/clinic/zipimport.c.h
@@ -291,4 +291,35 @@ zipimport_zipimporter_get_source(ZipImporter *self, PyObject *arg)
 exit:
     return return_value;
 }
-/*[clinic end generated code: output=93cb62a3a9752b9f input=a9049054013a1b77]*/
+
+PyDoc_STRVAR(zipimport_zipimporter_get_resource_reader__doc__,
+"get_resource_reader($self, fullname, /)\n"
+"--\n"
+"\n"
+"Return the ResourceReader for a package in a zip file.\n"
+"\n"
+"If \'fullname\' is a package within the zip file, return the \'ResourceReader\'\n"
+"object for the package.  Otherwise return None.");
+
+#define ZIPIMPORT_ZIPIMPORTER_GET_RESOURCE_READER_METHODDEF    \
+    {"get_resource_reader", (PyCFunction)zipimport_zipimporter_get_resource_reader, METH_O, zipimport_zipimporter_get_resource_reader__doc__},
+
+static PyObject *
+zipimport_zipimporter_get_resource_reader_impl(ZipImporter *self,
+                                               PyObject *fullname);
+
+static PyObject *
+zipimport_zipimporter_get_resource_reader(ZipImporter *self, PyObject *arg)
+{
+    PyObject *return_value = NULL;
+    PyObject *fullname;
+
+    if (!PyArg_Parse(arg, "U:get_resource_reader", &fullname)) {
+        goto exit;
+    }
+    return_value = zipimport_zipimporter_get_resource_reader_impl(self, fullname);
+
+exit:
+    return return_value;
+}
+/*[clinic end generated code: output=0b57adfe21373512 input=a9049054013a1b77]*/
diff --git a/Modules/zipimport.c b/Modules/zipimport.c
index 1d0e0ba9137..85013665d1f 100644
--- a/Modules/zipimport.c
+++ b/Modules/zipimport.c
@@ -784,6 +784,35 @@ zipimport_zipimporter_get_source_impl(ZipImporter *self, PyObject *fullname)
     Py_RETURN_NONE;
 }
 
+/*[clinic input]
+zipimport.zipimporter.get_resource_reader
+
+    fullname: unicode
+    /
+
+Return the ResourceReader for a package in a zip file.
+
+If 'fullname' is a package within the zip file, return the 'ResourceReader'
+object for the package.  Otherwise return None.
+
+[clinic start generated code]*/
+
+static PyObject *
+zipimport_zipimporter_get_resource_reader_impl(ZipImporter *self,
+                                               PyObject *fullname)
+/*[clinic end generated code: output=5e367d431f830726 input=bfab94d736e99151]*/
+{
+    PyObject *module = PyImport_ImportModule("importlib.resources");
+    if (module == NULL) {
+        return NULL;
+    }
+    PyObject *retval = PyObject_CallMethod(
+        module, "_zipimport_get_resource_reader",
+        "OO", (PyObject *)self, fullname);
+    Py_DECREF(module);
+    return retval;
+}
+
 
 static PyMethodDef zipimporter_methods[] = {
     ZIPIMPORT_ZIPIMPORTER_FIND_MODULE_METHODDEF
@@ -794,6 +823,7 @@ static PyMethodDef zipimporter_methods[] = {
     ZIPIMPORT_ZIPIMPORTER_GET_DATA_METHODDEF
     ZIPIMPORT_ZIPIMPORTER_GET_CODE_METHODDEF
     ZIPIMPORT_ZIPIMPORTER_GET_SOURCE_METHODDEF
+    ZIPIMPORT_ZIPIMPORTER_GET_RESOURCE_READER_METHODDEF
     {NULL,              NULL}   /* sentinel */
 };
 


From webhook-mailer at python.org  Wed Jan 24 16:26:22 2018
From: webhook-mailer at python.org (Brett Cannon)
Date: Wed, 24 Jan 2018 21:26:22 -0000
Subject: [Python-checkins] bpo-29708: Setting SOURCE_DATE_EPOCH forces
 hash-based .pyc files (GH-5200)
Message-ID: <mailman.40.1516829185.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/ccbe5818af20f8c12043f5c30c277a74714405e0
commit: ccbe5818af20f8c12043f5c30c277a74714405e0
branch: master
author: Bernhard M. Wiedemann <githubbmw at lsmod.de>
committer: Brett Cannon <brettcannon at users.noreply.github.com>
date: 2018-01-24T13:26:18-08:00
summary:

bpo-29708: Setting SOURCE_DATE_EPOCH forces hash-based .pyc files (GH-5200)

To support reproducible builds, the setting of of SOURCE_DATE_EPOCH triggers the py_compile module -- and by extension, compileall -- to forcibly compile with hash-based .pyc files. This eliminates the possibility of timestamp-based .pyc files which vary between builds.

files:
A Misc/NEWS.d/next/Build/2018-01-16-08-32-49.bpo-29708.YCaHEx.rst
M Doc/library/py_compile.rst
M Lib/py_compile.py
M Lib/test/test_py_compile.py

diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst
index a4f06de597b..d720e010505 100644
--- a/Doc/library/py_compile.rst
+++ b/Doc/library/py_compile.rst
@@ -55,7 +55,9 @@ byte-code cache files in the directory containing the source code.
 
    *invalidation_mode* should be a member of the :class:`PycInvalidationMode`
    enum and controls how the generated ``.pyc`` files are invalidated at
-   runtime.
+   runtime. If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
+   *invalidation_mode* will be forced to
+   :attr:`PycInvalidationMode.CHECKED_HASH`.
 
    .. versionchanged:: 3.2
       Changed default value of *cfile* to be :PEP:`3147`-compliant.  Previous
@@ -71,6 +73,9 @@ byte-code cache files in the directory containing the source code.
 
    .. versionchanged:: 3.7
       The *invalidation_mode* parameter was added as specified in :pep:`552`.
+      If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
+      *invalidation_mode* will be forced to
+      :attr:`PycInvalidationMode.CHECKED_HASH`.
 
 
 .. class:: PycInvalidationMode
diff --git a/Lib/py_compile.py b/Lib/py_compile.py
index a0f4defdce6..16dc0a011ff 100644
--- a/Lib/py_compile.py
+++ b/Lib/py_compile.py
@@ -112,6 +112,8 @@ def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1,
     the resulting file would be regular and thus not the same type of file as
     it was previously.
     """
+    if os.environ.get('SOURCE_DATE_EPOCH'):
+        invalidation_mode = PycInvalidationMode.CHECKED_HASH
     if cfile is None:
         if optimize >= 0:
             optimization = optimize if optimize >= 1 else ''
diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
index bcb686c86b5..8fc0b3308c9 100644
--- a/Lib/test/test_py_compile.py
+++ b/Lib/test/test_py_compile.py
@@ -98,6 +98,18 @@ def test_bad_coding(self):
         self.assertFalse(os.path.exists(
             importlib.util.cache_from_source(bad_coding)))
 
+    def test_source_date_epoch(self):
+        testtime = 123456789
+        with support.EnvironmentVarGuard() as env:
+            env["SOURCE_DATE_EPOCH"] = str(testtime)
+            py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+        with open(self.pyc_path, 'rb') as fp:
+            flags = importlib._bootstrap_external._classify_pyc(
+                fp.read(), 'test', {})
+        self.assertEqual(flags, 0b11)
+
     @unittest.skipIf(sys.flags.optimize > 0, 'test does not work with -O')
     def test_double_dot_no_clobber(self):
         # http://bugs.python.org/issue22966
diff --git a/Misc/NEWS.d/next/Build/2018-01-16-08-32-49.bpo-29708.YCaHEx.rst b/Misc/NEWS.d/next/Build/2018-01-16-08-32-49.bpo-29708.YCaHEx.rst
new file mode 100644
index 00000000000..e52c0018e5e
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2018-01-16-08-32-49.bpo-29708.YCaHEx.rst
@@ -0,0 +1,2 @@
+If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
+:mod:`py_compile` will always create hash-based ``.pyc`` files.


From webhook-mailer at python.org  Wed Jan 24 17:29:37 2018
From: webhook-mailer at python.org (Alexander Belopolsky)
Date: Wed, 24 Jan 2018 22:29:37 -0000
Subject: [Python-checkins] bpo-10381: Add timezone to datetime C API (#5032)
Message-ID: <mailman.41.1516832980.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/04af5b1ba9eb546a29735fac6cb5298159069b53
commit: 04af5b1ba9eb546a29735fac6cb5298159069b53
branch: master
author: Paul Ganssle <pganssle at users.noreply.github.com>
committer: Alexander Belopolsky <abalkin at users.noreply.github.com>
date: 2018-01-24T17:29:30-05:00
summary:

bpo-10381: Add timezone to datetime C API (#5032)

* Add timezone to datetime C API

* Add documentation for timezone C API macros

* Add dedicated tests for datetime type check macros

* Remove superfluous C API test

* Drop support for TimeZoneType in datetime C API

* Expose UTC singleton to the datetime C API

* Update datetime C-API documentation to include links

* Add reference count information for timezone constructors

files:
A Misc/NEWS.d/next/C API/2017-12-28-15-22-05.bpo-10381.a1E6aF.rst
M Doc/c-api/datetime.rst
M Doc/data/refcounts.dat
M Include/datetime.h
M Lib/test/datetimetester.py
M Modules/_datetimemodule.c
M Modules/_testcapimodule.c

diff --git a/Doc/c-api/datetime.rst b/Doc/c-api/datetime.rst
index 305e990368c..78724619ea3 100644
--- a/Doc/c-api/datetime.rst
+++ b/Doc/c-api/datetime.rst
@@ -13,6 +13,16 @@ the module initialisation function.  The macro puts a pointer to a C structure
 into a static variable, :c:data:`PyDateTimeAPI`, that is used by the following
 macros.
 
+Macro for access to the UTC singleton:
+
+.. c:var:: PyObject* PyDateTime_TimeZone_UTC
+
+   Returns the time zone singleton representing UTC, the same object as
+   :attr:`datetime.timezone.utc`.
+
+   .. versionadded:: 3.7
+
+
 Type-check macros:
 
 .. c:function:: int PyDate_Check(PyObject *ob)
@@ -79,27 +89,41 @@ Macros to create objects:
 
 .. c:function:: PyObject* PyDate_FromDate(int year, int month, int day)
 
-   Return a ``datetime.date`` object with the specified year, month and day.
+   Return a :class:`datetime.date` object with the specified year, month and day.
 
 
 .. c:function:: PyObject* PyDateTime_FromDateAndTime(int year, int month, int day, int hour, int minute, int second, int usecond)
 
-   Return a ``datetime.datetime`` object with the specified year, month, day, hour,
+   Return a :class:`datetime.datetime` object with the specified year, month, day, hour,
    minute, second and microsecond.
 
 
 .. c:function:: PyObject* PyTime_FromTime(int hour, int minute, int second, int usecond)
 
-   Return a ``datetime.time`` object with the specified hour, minute, second and
+   Return a :class:`datetime.time` object with the specified hour, minute, second and
    microsecond.
 
 
 .. c:function:: PyObject* PyDelta_FromDSU(int days, int seconds, int useconds)
 
-   Return a ``datetime.timedelta`` object representing the given number of days,
-   seconds and microseconds.  Normalization is performed so that the resulting
-   number of microseconds and seconds lie in the ranges documented for
-   ``datetime.timedelta`` objects.
+   Return a :class:`datetime.timedelta` object representing the given number
+   of days, seconds and microseconds.  Normalization is performed so that the
+   resulting number of microseconds and seconds lie in the ranges documented for
+   :class:`datetime.timedelta` objects.
+
+.. c:function:: PyObject* PyTimeZone_FromOffset(PyDateTime_DeltaType* offset)
+
+   Return a :class:`datetime.timezone` object with an unnamed fixed offset
+   represented by the *offset* argument.
+
+   .. versionadded:: 3.7
+
+.. c:function:: PyObject* PyTimeZone_FromOffsetAndName(PyDateTime_DeltaType* offset, PyUnicode* name)
+
+   Return a :class:`datetime.timezone` object with a fixed offset represented
+   by the *offset* argument and with tzname *name*.
+
+   .. versionadded:: 3.7
 
 
 Macros to extract fields from date objects.  The argument must be an instance of
@@ -199,11 +223,11 @@ Macros for the convenience of modules implementing the DB API:
 
 .. c:function:: PyObject* PyDateTime_FromTimestamp(PyObject *args)
 
-   Create and return a new ``datetime.datetime`` object given an argument tuple
-   suitable for passing to ``datetime.datetime.fromtimestamp()``.
+   Create and return a new :class:`datetime.datetime` object given an argument
+   tuple suitable for passing to :meth:`datetime.datetime.fromtimestamp()`.
 
 
 .. c:function:: PyObject* PyDate_FromTimestamp(PyObject *args)
 
-   Create and return a new ``datetime.date`` object given an argument tuple
-   suitable for passing to ``datetime.date.fromtimestamp()``.
+   Create and return a new :class:`datetime.date` object given an argument
+   tuple suitable for passing to :meth:`datetime.date.fromtimestamp()`.
diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat
index b1cad48c3e5..6dc86fc5e54 100644
--- a/Doc/data/refcounts.dat
+++ b/Doc/data/refcounts.dat
@@ -177,6 +177,14 @@ PyDelta_FromDSU:int:days::
 PyDelta_FromDSU:int:seconds::
 PyDelta_FromDSU:int:useconds::
 
+PyTimeZone_FromOffset:PyObject*::+1:
+PyTimeZone_FromOffset:PyDateTime_DeltaType*:offset:+1:Reference count not increased if offset is +00:00
+
+PyTimeZone_FromOffsetAndName:PyObject*::+1:
+PyTimeZone_FromOffsetAndName:PyDateTime_DeltaType*:offset:+1:Reference count not increased if offset is +00:00 and name == NULL
+PyTimeZone_FromOffsetAndName:PyUnicode*:name:+1:
+
+
 PyDescr_NewClassMethod:PyObject*::+1:
 PyDescr_NewClassMethod:PyTypeObject*:type::
 PyDescr_NewClassMethod:PyMethodDef*:method::
diff --git a/Include/datetime.h b/Include/datetime.h
index 3bf35cbb7f2..059d5ecf7a2 100644
--- a/Include/datetime.h
+++ b/Include/datetime.h
@@ -155,12 +155,16 @@ typedef struct {
     PyTypeObject *DeltaType;
     PyTypeObject *TZInfoType;
 
+    /* singletons */
+    PyObject *TimeZone_UTC;
+
     /* constructors */
     PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*);
     PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, int, int,
         PyObject*, PyTypeObject*);
     PyObject *(*Time_FromTime)(int, int, int, int, PyObject*, PyTypeObject*);
     PyObject *(*Delta_FromDelta)(int, int, int, int, PyTypeObject*);
+    PyObject *(*TimeZone_FromTimeZone)(PyObject *offset, PyObject *name);
 
     /* constructors for the DB API */
     PyObject *(*DateTime_FromTimestamp)(PyObject*, PyObject*, PyObject*);
@@ -202,6 +206,9 @@ static PyDateTime_CAPI *PyDateTimeAPI = NULL;
 #define PyDateTime_IMPORT \
     PyDateTimeAPI = (PyDateTime_CAPI *)PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)
 
+/* Macro for access to the UTC singleton */
+#define PyDateTime_TimeZone_UTC PyDateTimeAPI->TimeZone_UTC
+
 /* Macros for type checking when not building the Python core. */
 #define PyDate_Check(op) PyObject_TypeCheck(op, PyDateTimeAPI->DateType)
 #define PyDate_CheckExact(op) (Py_TYPE(op) == PyDateTimeAPI->DateType)
@@ -242,6 +249,12 @@ static PyDateTime_CAPI *PyDateTimeAPI = NULL;
     PyDateTimeAPI->Delta_FromDelta(days, seconds, useconds, 1, \
         PyDateTimeAPI->DeltaType)
 
+#define PyTimeZone_FromOffset(offset) \
+    PyDateTimeAPI->TimeZone_FromTimeZone(offset, NULL)
+
+#define PyTimeZone_FromOffsetAndName(offset, name) \
+    PyDateTimeAPI->TimeZone_FromTimeZone(offset, name)
+
 /* Macros supporting the DB API. */
 #define PyDateTime_FromTimestamp(args) \
     PyDateTimeAPI->DateTime_FromTimestamp( \
diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py
index e8ed79e8b32..a0883b9f421 100644
--- a/Lib/test/datetimetester.py
+++ b/Lib/test/datetimetester.py
@@ -31,6 +31,8 @@
 from datetime import date, datetime
 import time as _time
 
+import _testcapi
+
 # Needed by test_datetime
 import _strptime
 #
@@ -5443,6 +5445,185 @@ def __init__(self):
 class IranTest(ZoneInfoTest):
     zonename = 'Asia/Tehran'
 
+
+class CapiTest(unittest.TestCase):
+    def setUp(self):
+        # Since the C API is not present in the _Pure tests, skip all tests
+        if self.__class__.__name__.endswith('Pure'):
+            self.skipTest('Not relevant in pure Python')
+
+        # This *must* be called, and it must be called first, so until either
+        # restriction is loosened, we'll call it as part of test setup
+        _testcapi.test_datetime_capi()
+
+    def test_utc_capi(self):
+        for use_macro in (True, False):
+            capi_utc = _testcapi.get_timezone_utc_capi(use_macro)
+
+            with self.subTest(use_macro=use_macro):
+                self.assertIs(capi_utc, timezone.utc)
+
+    def test_timezones_capi(self):
+        est_capi, est_macro, est_macro_nn = _testcapi.make_timezones_capi()
+
+        exp_named = timezone(timedelta(hours=-5), "EST")
+        exp_unnamed = timezone(timedelta(hours=-5))
+
+        cases = [
+            ('est_capi', est_capi, exp_named),
+            ('est_macro', est_macro, exp_named),
+            ('est_macro_nn', est_macro_nn, exp_unnamed)
+        ]
+
+        for name, tz_act, tz_exp in cases:
+            with self.subTest(name=name):
+                self.assertEqual(tz_act, tz_exp)
+
+                dt1 = datetime(2000, 2, 4, tzinfo=tz_act)
+                dt2 = datetime(2000, 2, 4, tzinfo=tz_exp)
+
+                self.assertEqual(dt1, dt2)
+                self.assertEqual(dt1.tzname(), dt2.tzname())
+
+                dt_utc = datetime(2000, 2, 4, 5, tzinfo=timezone.utc)
+
+                self.assertEqual(dt1.astimezone(timezone.utc), dt_utc)
+
+    def test_check_date(self):
+        class DateSubclass(date):
+            pass
+
+        d = date(2011, 1, 1)
+        ds = DateSubclass(2011, 1, 1)
+        dt = datetime(2011, 1, 1)
+
+        is_date = _testcapi.datetime_check_date
+
+        # Check the ones that should be valid
+        self.assertTrue(is_date(d))
+        self.assertTrue(is_date(dt))
+        self.assertTrue(is_date(ds))
+        self.assertTrue(is_date(d, True))
+
+        # Check that the subclasses do not match exactly
+        self.assertFalse(is_date(dt, True))
+        self.assertFalse(is_date(ds, True))
+
+        # Check that various other things are not dates at all
+        args = [tuple(), list(), 1, '2011-01-01',
+                timedelta(1), timezone.utc, time(12, 00)]
+        for arg in args:
+            for exact in (True, False):
+                with self.subTest(arg=arg, exact=exact):
+                    self.assertFalse(is_date(arg, exact))
+
+    def test_check_time(self):
+        class TimeSubclass(time):
+            pass
+
+        t = time(12, 30)
+        ts = TimeSubclass(12, 30)
+
+        is_time = _testcapi.datetime_check_time
+
+        # Check the ones that should be valid
+        self.assertTrue(is_time(t))
+        self.assertTrue(is_time(ts))
+        self.assertTrue(is_time(t, True))
+
+        # Check that the subclass does not match exactly
+        self.assertFalse(is_time(ts, True))
+
+        # Check that various other things are not times
+        args = [tuple(), list(), 1, '2011-01-01',
+                timedelta(1), timezone.utc, date(2011, 1, 1)]
+
+        for arg in args:
+            for exact in (True, False):
+                with self.subTest(arg=arg, exact=exact):
+                    self.assertFalse(is_time(arg, exact))
+
+    def test_check_datetime(self):
+        class DateTimeSubclass(datetime):
+            pass
+
+        dt = datetime(2011, 1, 1, 12, 30)
+        dts = DateTimeSubclass(2011, 1, 1, 12, 30)
+
+        is_datetime = _testcapi.datetime_check_datetime
+
+        # Check the ones that should be valid
+        self.assertTrue(is_datetime(dt))
+        self.assertTrue(is_datetime(dts))
+        self.assertTrue(is_datetime(dt, True))
+
+        # Check that the subclass does not match exactly
+        self.assertFalse(is_datetime(dts, True))
+
+        # Check that various other things are not datetimes
+        args = [tuple(), list(), 1, '2011-01-01',
+                timedelta(1), timezone.utc, date(2011, 1, 1)]
+
+        for arg in args:
+            for exact in (True, False):
+                with self.subTest(arg=arg, exact=exact):
+                    self.assertFalse(is_datetime(arg, exact))
+
+    def test_check_delta(self):
+        class TimeDeltaSubclass(timedelta):
+            pass
+
+        td = timedelta(1)
+        tds = TimeDeltaSubclass(1)
+
+        is_timedelta = _testcapi.datetime_check_delta
+
+        # Check the ones that should be valid
+        self.assertTrue(is_timedelta(td))
+        self.assertTrue(is_timedelta(tds))
+        self.assertTrue(is_timedelta(td, True))
+
+        # Check that the subclass does not match exactly
+        self.assertFalse(is_timedelta(tds, True))
+
+        # Check that various other things are not timedeltas
+        args = [tuple(), list(), 1, '2011-01-01',
+                timezone.utc, date(2011, 1, 1), datetime(2011, 1, 1)]
+
+        for arg in args:
+            for exact in (True, False):
+                with self.subTest(arg=arg, exact=exact):
+                    self.assertFalse(is_timedelta(arg, exact))
+
+    def test_check_tzinfo(self):
+        class TZInfoSubclass(tzinfo):
+            pass
+
+        tzi = tzinfo()
+        tzis = TZInfoSubclass()
+        tz = timezone(timedelta(hours=-5))
+
+        is_tzinfo = _testcapi.datetime_check_tzinfo
+
+        # Check the ones that should be valid
+        self.assertTrue(is_tzinfo(tzi))
+        self.assertTrue(is_tzinfo(tz))
+        self.assertTrue(is_tzinfo(tzis))
+        self.assertTrue(is_tzinfo(tzi, True))
+
+        # Check that the subclasses do not match exactly
+        self.assertFalse(is_tzinfo(tz, True))
+        self.assertFalse(is_tzinfo(tzis, True))
+
+        # Check that various other things are not tzinfos
+        args = [tuple(), list(), 1, '2011-01-01',
+                date(2011, 1, 1), datetime(2011, 1, 1)]
+
+        for arg in args:
+            for exact in (True, False):
+                with self.subTest(arg=arg, exact=exact):
+                    self.assertFalse(is_tzinfo(arg, exact))
+
 def load_tests(loader, standard_tests, pattern):
     standard_tests.addTest(ZoneInfoCompleteTest())
     return standard_tests
diff --git a/Misc/NEWS.d/next/C API/2017-12-28-15-22-05.bpo-10381.a1E6aF.rst b/Misc/NEWS.d/next/C API/2017-12-28-15-22-05.bpo-10381.a1E6aF.rst
new file mode 100644
index 00000000000..26717692d03
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2017-12-28-15-22-05.bpo-10381.a1E6aF.rst	
@@ -0,0 +1,2 @@
+Add C API access to the ``datetime.timezone`` constructor and
+``datetime.timzone.UTC`` singleton.
diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
index d1f48e5bd04..4a33f2d8964 100644
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -6036,10 +6036,12 @@ static PyDateTime_CAPI CAPI = {
     &PyDateTime_TimeType,
     &PyDateTime_DeltaType,
     &PyDateTime_TZInfoType,
+    NULL,                       // PyDatetime_TimeZone_UTC not initialized yet
     new_date_ex,
     new_datetime_ex,
     new_time_ex,
     new_delta_ex,
+    new_timezone,
     datetime_fromtimestamp,
     date_fromtimestamp,
     new_datetime_ex2,
@@ -6168,6 +6170,7 @@ PyInit__datetime(void)
     if (x == NULL || PyDict_SetItemString(d, "utc", x) < 0)
         return NULL;
     PyDateTime_TimeZone_UTC = x;
+    CAPI.TimeZone_UTC = PyDateTime_TimeZone_UTC;
 
     delta = new_delta(-1, 60, 0, 1); /* -23:59 */
     if (delta == NULL)
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index e3be7d3d829..3f41134a345 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2220,12 +2220,92 @@ test_datetime_capi(PyObject *self, PyObject *args) {
     }
     test_run_counter++;
     PyDateTime_IMPORT;
+
     if (PyDateTimeAPI)
         Py_RETURN_NONE;
     else
         return NULL;
 }
 
+/* Functions exposing the C API type checking for testing */
+#define MAKE_DATETIME_CHECK_FUNC(check_method, exact_method)    \
+    PyObject *obj;                                              \
+    int exact = 0;                                              \
+    if (!PyArg_ParseTuple(args, "O|p", &obj, &exact)) {         \
+        return NULL;                                            \
+    }                                                           \
+    int rv = exact?exact_method(obj):check_method(obj);         \
+    if (rv) {                                                   \
+        Py_RETURN_TRUE;                                         \
+    } else {                                                    \
+        Py_RETURN_FALSE;                                        \
+    }
+
+static PyObject *
+datetime_check_date(PyObject *self, PyObject *args) {
+    MAKE_DATETIME_CHECK_FUNC(PyDate_Check, PyDate_CheckExact)
+}
+
+static PyObject *
+datetime_check_time(PyObject *self, PyObject *args) {
+    MAKE_DATETIME_CHECK_FUNC(PyTime_Check, PyTime_CheckExact)
+}
+
+static PyObject *
+datetime_check_datetime(PyObject *self, PyObject *args) {
+    MAKE_DATETIME_CHECK_FUNC(PyDateTime_Check, PyDateTime_CheckExact)
+}
+
+static PyObject *
+datetime_check_delta(PyObject *self, PyObject *args) {
+    MAKE_DATETIME_CHECK_FUNC(PyDelta_Check, PyDelta_CheckExact)
+}
+
+static PyObject *
+datetime_check_tzinfo(PyObject *self, PyObject *args) {
+    MAKE_DATETIME_CHECK_FUNC(PyTZInfo_Check, PyTZInfo_CheckExact)
+}
+
+
+/* Makes three variations on timezone representing UTC-5:
+   1. timezone with offset and name from PyDateTimeAPI
+   2. timezone with offset and name from PyTimeZone_FromOffsetAndName
+   3. timezone with offset (no name) from PyTimeZone_FromOffset
+*/
+static PyObject *
+make_timezones_capi(PyObject *self, PyObject *args) {
+    PyObject *offset = PyDelta_FromDSU(0, -18000, 0);
+    PyObject *name = PyUnicode_FromString("EST");
+
+    PyObject *est_zone_capi = PyDateTimeAPI->TimeZone_FromTimeZone(offset, name);
+    PyObject *est_zone_macro = PyTimeZone_FromOffsetAndName(offset, name);
+    PyObject *est_zone_macro_noname = PyTimeZone_FromOffset(offset);
+
+    Py_DecRef(offset);
+    Py_DecRef(name);
+
+    PyObject *rv = PyTuple_New(3);
+
+    PyTuple_SET_ITEM(rv, 0, est_zone_capi);
+    PyTuple_SET_ITEM(rv, 1, est_zone_macro);
+    PyTuple_SET_ITEM(rv, 2, est_zone_macro_noname);
+
+    return rv;
+}
+
+static PyObject *
+get_timezone_utc_capi(PyObject* self, PyObject *args) {
+    int macro = 0;
+    if (!PyArg_ParseTuple(args, "|p", &macro)) {
+        return NULL;
+    }
+    if (macro) {
+        return PyDateTime_TimeZone_UTC;
+    } else {
+        return PyDateTimeAPI->TimeZone_UTC;
+    }
+}
+
 
 /* test_thread_state spawns a thread of its own, and that thread releases
  * `thread_done` when it's finished.  The driver code has to know when the
@@ -4452,6 +4532,13 @@ static PyMethodDef TestMethods[] = {
     {"test_config",             (PyCFunction)test_config,        METH_NOARGS},
     {"test_sizeof_c_types",     (PyCFunction)test_sizeof_c_types, METH_NOARGS},
     {"test_datetime_capi",  test_datetime_capi,              METH_NOARGS},
+    {"datetime_check_date",     datetime_check_date,             METH_VARARGS},
+    {"datetime_check_time",     datetime_check_time,             METH_VARARGS},
+    {"datetime_check_datetime",     datetime_check_datetime,     METH_VARARGS},
+    {"datetime_check_delta",     datetime_check_delta,           METH_VARARGS},
+    {"datetime_check_tzinfo",     datetime_check_tzinfo,         METH_VARARGS},
+    {"make_timezones_capi",     make_timezones_capi,             METH_NOARGS},
+    {"get_timezone_utc_capi",    get_timezone_utc_capi,            METH_VARARGS},
     {"test_list_api",           (PyCFunction)test_list_api,      METH_NOARGS},
     {"test_dict_iteration",     (PyCFunction)test_dict_iteration,METH_NOARGS},
     {"dict_getitem_knownhash",  dict_getitem_knownhash,          METH_VARARGS},


From webhook-mailer at python.org  Wed Jan 24 17:30:33 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Wed, 24 Jan 2018 22:30:33 -0000
Subject: [Python-checkins] bpo-32391: Implement StreamWriter.wait_closed()
 (#5281)
Message-ID: <mailman.42.1516833036.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/fe133aad52222949db7309c26c58b066c22e714e
commit: fe133aad52222949db7309c26c58b066c22e714e
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-25T00:30:30+02:00
summary:

bpo-32391: Implement StreamWriter.wait_closed() (#5281)

files:
A Misc/NEWS.d/next/Library/2018-01-24-15-20-12.bpo-32391.0f8MY9.rst
M Doc/library/asyncio-stream.rst
M Lib/asyncio/streams.py
M Lib/test/test_asyncio/test_streams.py

diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst
index 6d5cbbc5bd5..099b59ee582 100644
--- a/Doc/library/asyncio-stream.rst
+++ b/Doc/library/asyncio-stream.rst
@@ -201,6 +201,21 @@ StreamWriter
 
       Close the transport: see :meth:`BaseTransport.close`.
 
+   .. method:: is_closing()
+
+      Return ``True`` if the writer is closing or is closed.
+
+      .. versionadded:: 3.7
+
+   .. coroutinemethod:: wait_closed()
+
+      Wait until the writer is closed.
+
+      Should be called after :meth:`close`  to wait until the underlying
+      connection (and the associated transport/protocol pair) is closed.
+
+      .. versionadded:: 3.7
+
    .. coroutinemethod:: drain()
 
       Let the write buffer of the underlying transport a chance to be flushed.
diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
index eef2b895f1e..9a53ee48247 100644
--- a/Lib/asyncio/streams.py
+++ b/Lib/asyncio/streams.py
@@ -224,6 +224,7 @@ def __init__(self, stream_reader, client_connected_cb=None, loop=None):
         self._stream_writer = None
         self._client_connected_cb = client_connected_cb
         self._over_ssl = False
+        self._closed = self._loop.create_future()
 
     def connection_made(self, transport):
         self._stream_reader.set_transport(transport)
@@ -243,6 +244,11 @@ def connection_lost(self, exc):
                 self._stream_reader.feed_eof()
             else:
                 self._stream_reader.set_exception(exc)
+        if not self._closed.done():
+            if exc is None:
+                self._closed.set_result(None)
+            else:
+                self._closed.set_exception(exc)
         super().connection_lost(exc)
         self._stream_reader = None
         self._stream_writer = None
@@ -259,6 +265,13 @@ def eof_received(self):
             return False
         return True
 
+    def __del__(self):
+        # Prevent reports about unhandled exceptions.
+        # Better than self._closed._log_traceback = False hack
+        closed = self._closed
+        if closed.done() and not closed.cancelled():
+            closed.exception()
+
 
 class StreamWriter:
     """Wraps a Transport.
@@ -303,6 +316,12 @@ def can_write_eof(self):
     def close(self):
         return self._transport.close()
 
+    def is_closing(self):
+        return self._transport.is_closing()
+
+    async def wait_closed(self):
+        await self._protocol._closed
+
     def get_extra_info(self, name, default=None):
         return self._transport.get_extra_info(name, default)
 
@@ -318,15 +337,14 @@ def get_extra_info(self, name, default=None):
             exc = self._reader.exception()
             if exc is not None:
                 raise exc
-        if self._transport is not None:
-            if self._transport.is_closing():
-                # Yield to the event loop so connection_lost() may be
-                # called.  Without this, _drain_helper() would return
-                # immediately, and code that calls
-                #     write(...); await drain()
-                # in a loop would never call connection_lost(), so it
-                # would not see an error when the socket is closed.
-                await sleep(0, loop=self._loop)
+        if self._transport.is_closing():
+            # Yield to the event loop so connection_lost() may be
+            # called.  Without this, _drain_helper() would return
+            # immediately, and code that calls
+            #     write(...); await drain()
+            # in a loop would never call connection_lost(), so it
+            # would not see an error when the socket is closed.
+            await sleep(0, loop=self._loop)
         await self._protocol._drain_helper()
 
 
diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
index 7a0762cfa3b..63fa13f79e2 100644
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -19,7 +19,7 @@
 from test.test_asyncio import utils as test_utils
 
 
-class StreamReaderTests(test_utils.TestCase):
+class StreamTests(test_utils.TestCase):
 
     DATA = b'line1\nline2\nline3\n'
 
@@ -860,6 +860,35 @@ def test_LimitOverrunError_pickleable(self):
                 self.assertEqual(str(e), str(e2))
                 self.assertEqual(e.consumed, e2.consumed)
 
+    def test_wait_closed_on_close(self):
+        with test_utils.run_test_server() as httpd:
+            rd, wr = self.loop.run_until_complete(
+                asyncio.open_connection(*httpd.address, loop=self.loop))
+
+            wr.write(b'GET / HTTP/1.0\r\n\r\n')
+            f = rd.readline()
+            data = self.loop.run_until_complete(f)
+            self.assertEqual(data, b'HTTP/1.0 200 OK\r\n')
+            f = rd.read()
+            data = self.loop.run_until_complete(f)
+            self.assertTrue(data.endswith(b'\r\n\r\nTest message'))
+            self.assertFalse(wr.is_closing())
+            wr.close()
+            self.assertTrue(wr.is_closing())
+            self.loop.run_until_complete(wr.wait_closed())
+
+    def test_wait_closed_on_close_with_unread_data(self):
+        with test_utils.run_test_server() as httpd:
+            rd, wr = self.loop.run_until_complete(
+                asyncio.open_connection(*httpd.address, loop=self.loop))
+
+            wr.write(b'GET / HTTP/1.0\r\n\r\n')
+            f = rd.readline()
+            data = self.loop.run_until_complete(f)
+            self.assertEqual(data, b'HTTP/1.0 200 OK\r\n')
+            wr.close()
+            self.loop.run_until_complete(wr.wait_closed())
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/Misc/NEWS.d/next/Library/2018-01-24-15-20-12.bpo-32391.0f8MY9.rst b/Misc/NEWS.d/next/Library/2018-01-24-15-20-12.bpo-32391.0f8MY9.rst
new file mode 100644
index 00000000000..6e0922774d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-24-15-20-12.bpo-32391.0f8MY9.rst
@@ -0,0 +1 @@
+Implement :meth:`asyncio.StreamWriter.wait_closed` and :meth:`asyncio.StreamWriter.is_closing` methods


From webhook-mailer at python.org  Wed Jan 24 20:13:26 2018
From: webhook-mailer at python.org (Brett Cannon)
Date: Thu, 25 Jan 2018 01:13:26 -0000
Subject: [Python-checkins] bpo-29708: Add What's New entries for
 SOURCE_DATE_EPOCH and py_compile (GH-5306)
Message-ID: <mailman.43.1516842808.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/cab0b2b053970982b760048acc3046363615a8dd
commit: cab0b2b053970982b760048acc3046363615a8dd
branch: master
author: Brett Cannon <brettcannon at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-24T17:13:24-08:00
summary:

bpo-29708: Add What's New entries for SOURCE_DATE_EPOCH and py_compile (GH-5306)

files:
M Doc/whatsnew/3.7.rst

diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 20be158ab0a..e52e32c4736 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -479,6 +479,17 @@ begins.  (Contributed by Barry Warsaw in :issue:`31389`.)
 pdb command line now accepts `-m module_name` as an alternative to
 script file. (Contributed by Mario Corchero in :issue:`32206`.)
 
+py_compile
+----------
+
+:func:`py_compile.compile` -- and by extension, :mod:`compileall` -- now
+respects the :envvar:`SOURCE_DATE_EPOCH` environment variable by
+unconditionally creating ``.pyc`` files for hash-based validation.
+This allows for guaranteeing
+`reproducible builds <https://reproducible-builds.org/>`_ of ``.pyc``
+files when they are created eagerly. (Contributed by Bernhard M. Wiedemann
+in :issue:`29708`.)
+
 
 re
 --
@@ -544,6 +555,12 @@ Added functions :func:`time.thread_time` and :func:`time.thread_time_ns`
 to get per-thread CPU time measurements.
 (Contributed by Antoine Pitrou in :issue:`32025`.)
 
+unicodedata
+-----------
+
+The internal :mod:`unicodedata` database has been upgraded to use `Unicode 10
+<http://www.unicode.org/versions/Unicode10.0.0/>`_. (Contributed by Benjamin
+Peterson.)
 
 unittest
 --------
@@ -565,20 +582,6 @@ children by preventing to get or set any new attribute on the sealed mock.
 The sealing process is performed recursively. (Contributed by Mario Corchero
 in :issue:`30541`.)
 
-xmlrpc.server
--------------
-
-:meth:`register_function` of :class:`xmlrpc.server.SimpleXMLRPCDispatcher` and
-its subclasses can be used as a decorator. (Contributed by Xiang Zhang in
-:issue:`7769`.)
-
-unicodedata
------------
-
-The internal :mod:`unicodedata` database has been upgraded to use `Unicode 10
-<http://www.unicode.org/versions/Unicode10.0.0/>`_. (Contributed by Benjamin
-Peterson.)
-
 urllib.parse
 ------------
 
@@ -624,6 +627,13 @@ methods can now compare text of the current node with ``[. = "text"]``,
 not only text in children.  Predicates also allow adding spaces for
 better readability.  (Contributed by Stefan Behnel in :issue:`31648`.)
 
+xmlrpc.server
+-------------
+
+:meth:`register_function` of :class:`xmlrpc.server.SimpleXMLRPCDispatcher` and
+its subclasses can be used as a decorator. (Contributed by Xiang Zhang in
+:issue:`7769`.)
+
 zipapp
 ------
 
@@ -680,10 +690,16 @@ Optimizations
 Build and C API Changes
 =======================
 
+* :mod:`py_compile` and :mod:`compileall` now support the
+  :envvar:`SOURCE_DATE_EPOCH` environment variable by unconditionally
+  building ``.pyc`` files for hash verification instead of potentially
+  timestamp-based ``.pyc`` files. See the notes for the `py_compile`_
+  improvement notes for more details.
+
 * A full copy of libffi is no longer bundled for use when building the
   :mod:`_ctypes <ctypes>` module on non-OSX UNIX platforms.  An installed copy
   of libffi is now required when building ``_ctypes`` on such platforms.
-  Contributed by Zachary Ware in :issue:`27979`.
+  (Contributed by Zachary Ware in :issue:`27979`.)
 
 * The fields :c:member:`name` and :c:member:`doc` of structures
   :c:type:`PyMemberDef`, :c:type:`PyGetSetDef`,


From lp_benchmark_robot at intel.com  Wed Jan 24 20:45:05 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Wed, 24 Jan 2018 17:45:05 -0800
Subject: [Python-checkins] [1 up,
 64 flat] Results for Python (master branch) 2018-01-24
Message-ID: <8bae4f1f-a035-4f7d-80d1-cb4beeb65e04@orsmsx104.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-24 03:04:26-08:00.
  - commit:		018e1b7
  - previous commit:	6934831
  - revision date:	2018-01-24 16:49:58+09:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.947% |  -0.613%   |  +6.469%   |  +8.160%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_method| 0.945% |  +0.074%   |  +22.854%  |  +12.191%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.425% |  +0.046%   |  +23.917%  |  +11.147%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 2.641% |  -1.312%   |  +18.341%  |  +16.197%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 2.358% |  +1.278%   |  +8.031%   |  +13.829%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.392% |  +1.097%   |  +10.806%  |  +12.209%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 0.655% |  -1.755%   |  +5.036%   |  +11.586%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.549% |  -0.198%   |  -1.136%   |  +9.110%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 4.836% |  +1.863%   |  +8.726%   |  +19.424%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 4.390% |  -3.032%   |  +18.960%  |  +15.542%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.149% |  -0.625%   |  +5.262%   |  +7.614%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 0.221% |  -0.570%   |  +4.131%   |  +7.004%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.946% |  -0.153%   |  +2.531%   |  +6.959%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 1.417% |  +0.567%   |  +12.898%  |  +11.592%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.209% |  +0.756%   |  +9.914%   |  +10.971%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 1.056% |  -0.038%   |  +6.490%   |  +12.522%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 0.418% |  -0.588%   |  +9.569%   |  +12.389%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 2.628% |  -0.963%   |  +8.225%   |  +10.006%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 2.011% |  +0.297%   |  +2.552%   |  +10.304%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 3.847% |  -0.567%   |  +0.929%   |  +12.124%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.384% |  -1.557%   |  +6.666%   |  +13.655%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.699% |  -0.134%   |  +46.677%  |  +13.342%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_simple| 1.642% |  -3.006%   |  +6.824%   |  +14.947%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.314% |  -0.140%   |  +17.027%  |  +14.912%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 2.080% |  -0.070%   |  +7.342%   |  +11.663%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 3.247% |  +0.143%   |  +4.969%   |  +4.383%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.934% |  -0.710%   |  -4.630%   |  +4.031%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.082% |  +0.357%   |  +4.553%   |  +7.778%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.164% |  -2.053%   |  +3.019%   |  +10.568%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 4.816% |  -0.928%   |  -1.210%   |  +21.511%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.257% |  +0.027%   |  +0.579%   |  +22.249%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.747% |  -0.047%   |  +1.928%   |  +20.732%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 4.740% |  -0.737%   |  +10.630%  |  +11.362%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.135% |  +0.002%   |  +0.061%   |  +10.512%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          python_startup| 0.120% |  -0.207%   |  +11.047%  |  +5.491%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.093% |  -0.212%   |  +2.284%   |  +5.632%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.162% |  -0.153%   |  +9.354%   |  +13.801%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.116% |  -0.612%   |  +2.568%   |  +12.999%  |
+-----+------------------------+--------+------------+------------+------------+
| :-) |               regex_dna| 0.418% |  +1.881%   |  -0.146%   |  +8.974%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.813% |  -4.373%   |  -10.270%  |  +10.015%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 2.097% |  +4.037%   |  +7.273%   |  +6.771%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 1.249% |  -0.096%   |  +8.674%   |  +15.288%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_fft| 0.408% |  +0.450%   |  -4.675%   |  +5.336%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 2.290% |  +0.924%   |  +26.329%  |  +10.519%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.263% |  -0.441%   |  +4.291%   |  +2.673%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 1.613% |  -0.439%   |  +13.137%  |  +8.591%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.658% |  +2.618%   |  -9.591%   |  +3.207%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 1.385% |  -0.230%   |  +2.356%   |  +4.921%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 0.922% |  +0.205%   |  +6.543%   |  +7.726%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 3.352% |  -0.953%   |  +6.352%   |  +5.000%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 4.790% |  +1.860%   |  +18.538%  |  +8.359%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.938% |  +0.260%   |  +15.155%  |  +6.775%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 1.720% |  +0.538%   |  +9.986%   |  +8.014%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.094% |  +0.271%   |  +15.906%  |  +8.529%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 5.348% |  -2.118%   |  +8.326%   |  +13.193%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 3.421% |  +0.512%   |  +21.832%  |  +9.170%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.060% |  -0.587%   |  +6.270%   |  +8.179%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.565% |  +0.186%   |  +1.981%   |  +3.430%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 6.144% |  +0.778%   |  +8.013%   |  +21.401%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 1.315% |  +0.260%   |  -1.908%   |  +15.972%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.983% |  -0.671%   |  +7.157%   |  +7.994%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 1.277% |  -0.365%   |  +4.786%   |  +9.943%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.619% |  +0.392%   |  +3.342%   |  +8.407%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 1.765% |  +0.774%   |  -5.954%   |  +12.497%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.137% |  -0.287%   |  +5.689%   |  +10.713%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-up-64-flat-results-for-python-master-branch-2018-01-24

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Thu Jan 25 03:18:40 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 25 Jan 2018 08:18:40 -0000
Subject: [Python-checkins] bpo-32652: Defer pymain_set_global_config() call
 (#5303)
Message-ID: <mailman.44.1516868321.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/2b822a0bb1de2612c85d8f75e3ce89eda2ac9f68
commit: 2b822a0bb1de2612c85d8f75e3ce89eda2ac9f68
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-25T09:18:36+01:00
summary:

bpo-32652: Defer pymain_set_global_config() call (#5303)

In Py_Main(), don't call pymain_set_global_config() early: only call
it when the whole configuration has been read.

Add an unit test to prevent future regression.

files:
M Lib/test/test_utf8_mode.py
M Modules/main.c

diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 73d1bd424ca..26e2e13ec53 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -223,6 +223,21 @@ def check(utf8_opt, expected, **kw):
             c_arg = arg_ascii
         check('utf8=0', [c_arg], LC_ALL='C')
 
+    def test_optim_level(self):
+        # CPython: check that Py_Main() doesn't increment Py_OptimizeFlag
+        # twice when -X utf8 requires to parse the configuration twice (when
+        # the encoding changes after reading the configuration, the
+        # configuration is read again with the new encoding).
+        code = 'import sys; print(sys.flags.optimize)'
+        out = self.get_output('-X', 'utf8', '-O', '-c', code)
+        self.assertEqual(out, '1')
+        out = self.get_output('-X', 'utf8', '-OO', '-c', code)
+        self.assertEqual(out, '2')
+
+        code = 'import sys; print(sys.flags.ignore_environment)'
+        out = self.get_output('-X', 'utf8', '-E', '-c', code)
+        self.assertEqual(out, '1')
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Modules/main.c b/Modules/main.c
index 2e632e292c2..1ce075f98e3 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -1909,22 +1909,24 @@ pymain_read_conf_impl(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
         return -1;
     }
 
-    /* Global configuration variables should be set to read the core
-       configuration, and then get again to get updated values.
-
-       _PyPathConfig_Init() tests !Py_FrozenFlag to avoid some warnings.
-       Moreover, on Windows, it modifies Py_IsolatedFlag and Py_NoSiteFlag
-       variables if a "._pth" file is found. */
-    pymain_set_global_config(pymain, cmdline);
+    /* On Windows, _PyPathConfig_Init() modifies Py_IsolatedFlag and
+       Py_NoSiteFlag variables if a "._pth" file is found. */
+    int init_isolated = Py_IsolatedFlag;
+    int init_no_site = Py_NoSiteFlag;
+    Py_IsolatedFlag = cmdline->isolated;
+    Py_NoSiteFlag = cmdline->no_site_import;
 
     err = _PyCoreConfig_Read(config);
+
+    cmdline->isolated = Py_IsolatedFlag;
+    cmdline->no_site_import = Py_NoSiteFlag;
+    Py_IsolatedFlag = init_isolated;
+    Py_NoSiteFlag = init_no_site;
+
     if (_Py_INIT_FAILED(err)) {
         pymain->err = err;
         return -1;
     }
-
-    Py_UTF8Mode = pymain->config.utf8_mode;
-    pymain_get_global_config(pymain, cmdline);
     return 0;
 }
 
@@ -1948,8 +1950,6 @@ pymain_read_conf(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
     int locale_coerced = 0;
     int loops = 0;
     int init_ignore_env = pymain->config.ignore_environment;
-    int init_isolated = cmdline->isolated;
-    int init_no_site = cmdline->no_site_import;
 
     while (1) {
         int utf8_mode = pymain->config.utf8_mode;
@@ -2011,8 +2011,6 @@ pymain_read_conf(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
            modified by _PyCoreConfig_Read(). */
         Py_UTF8Mode = pymain->config.utf8_mode;
         Py_IgnoreEnvironmentFlag = init_ignore_env;
-        Py_IsolatedFlag = init_isolated;
-        Py_NoSiteFlag = init_no_site;
         _PyCoreConfig_Clear(&pymain->config);
         pymain_clear_cmdline(pymain, cmdline);
         pymain_get_global_config(pymain, cmdline);


From webhook-mailer at python.org  Thu Jan 25 03:49:46 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Thu, 25 Jan 2018 08:49:46 -0000
Subject: [Python-checkins] bpo-32571: Avoid raising unneeded AttributeError
 and silencing it in C code (GH-5222)
Message-ID: <mailman.45.1516870188.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/f320be77ffb73e3b9e7fc98c37b8df3975d84b40
commit: f320be77ffb73e3b9e7fc98c37b8df3975d84b40
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: INADA Naoki <methane at users.noreply.github.com>
date: 2018-01-25T17:49:40+09:00
summary:

 bpo-32571: Avoid raising unneeded AttributeError and silencing it in C code (GH-5222)

Add two new private APIs: _PyObject_LookupAttr() and _PyObject_LookupAttrId()

files:
M Include/object.h
M Modules/_collectionsmodule.c
M Modules/_io/bufferedio.c
M Modules/_io/fileio.c
M Modules/_io/iobase.c
M Modules/_io/textio.c
M Modules/_pickle.c
M Modules/arraymodule.c
M Modules/itertoolsmodule.c
M Objects/abstract.c
M Objects/classobject.c
M Objects/dictobject.c
M Objects/genobject.c
M Objects/object.c
M Objects/odictobject.c
M Objects/typeobject.c
M Parser/asdl_c.py
M Python/Python-ast.c
M Python/_warnings.c
M Python/bltinmodule.c
M Python/ceval.c
M Python/codecs.c

diff --git a/Include/object.h b/Include/object.h
index 274b67656ee..c772deaf57d 100644
--- a/Include/object.h
+++ b/Include/object.h
@@ -536,11 +536,20 @@ PyAPI_FUNC(int) PyObject_SetAttr(PyObject *, PyObject *, PyObject *);
 PyAPI_FUNC(int) PyObject_HasAttr(PyObject *, PyObject *);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(int) _PyObject_IsAbstract(PyObject *);
-/* Same as PyObject_GetAttr(), but don't raise AttributeError. */
-PyAPI_FUNC(PyObject *) _PyObject_GetAttrWithoutError(PyObject *, PyObject *);
 PyAPI_FUNC(PyObject *) _PyObject_GetAttrId(PyObject *, struct _Py_Identifier *);
 PyAPI_FUNC(int) _PyObject_SetAttrId(PyObject *, struct _Py_Identifier *, PyObject *);
 PyAPI_FUNC(int) _PyObject_HasAttrId(PyObject *, struct _Py_Identifier *);
+/* Replacements of PyObject_GetAttr() and _PyObject_GetAttrId() which
+   don't raise AttributeError.
+
+   Return 1 and set *result != NULL if an attribute is found.
+   Return 0 and set *result == NULL if an attribute is not found;
+   an AttributeError is silenced.
+   Return -1 and set *result == NULL if an error other than AttributeError
+   is raised.
+*/
+PyAPI_FUNC(int) _PyObject_LookupAttr(PyObject *, PyObject *, PyObject **);
+PyAPI_FUNC(int) _PyObject_LookupAttrId(PyObject *, struct _Py_Identifier *, PyObject **);
 PyAPI_FUNC(PyObject **) _PyObject_GetDictPtr(PyObject *);
 #endif
 PyAPI_FUNC(PyObject *) PyObject_SelfIter(PyObject *);
diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c
index 222cace5af7..5753dd946a5 100644
--- a/Modules/_collectionsmodule.c
+++ b/Modules/_collectionsmodule.c
@@ -1339,12 +1339,10 @@ deque_reduce(dequeobject *deque)
     PyObject *dict, *it;
     _Py_IDENTIFIER(__dict__);
 
-    dict = _PyObject_GetAttrId((PyObject *)deque, &PyId___dict__);
+    if (_PyObject_LookupAttrId((PyObject *)deque, &PyId___dict__, &dict) < 0) {
+        return NULL;
+    }
     if (dict == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            return NULL;
-        }
-        PyErr_Clear();
         dict = Py_None;
         Py_INCREF(dict);
     }
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
index b81abde5c0a..2b7aaaf7c7d 100644
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -1541,7 +1541,9 @@ _bufferedreader_read_all(buffered *self)
     }
     _bufferedreader_reset_buf(self);
 
-    readall = _PyObject_GetAttrWithoutError(self->raw, _PyIO_str_readall);
+    if (_PyObject_LookupAttr(self->raw, _PyIO_str_readall, &readall) < 0) {
+        goto cleanup;
+    }
     if (readall) {
         tmp = _PyObject_CallNoArg(readall);
         Py_DECREF(readall);
@@ -1561,9 +1563,6 @@ _bufferedreader_read_all(buffered *self)
         }
         goto cleanup;
     }
-    else if (PyErr_Occurred()) {
-        goto cleanup;
-    }
 
     chunks = PyList_New(0);
     if (chunks == NULL)
diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c
index 269142cfee5..0d5bf3b2211 100644
--- a/Modules/_io/fileio.c
+++ b/Modules/_io/fileio.c
@@ -1073,12 +1073,10 @@ fileio_repr(fileio *self)
     if (self->fd < 0)
         return PyUnicode_FromFormat("<_io.FileIO [closed]>");
 
-    nameobj = _PyObject_GetAttrId((PyObject *) self, &PyId_name);
+    if (_PyObject_LookupAttrId((PyObject *) self, &PyId_name, &nameobj) < 0) {
+        return NULL;
+    }
     if (nameobj == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        else
-            return NULL;
         res = PyUnicode_FromFormat(
             "<_io.FileIO fd=%d mode='%s' closefd=%s>",
             self->fd, mode_string(self), self->closefd ? "True" : "False");
diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
index 348d449a272..0c329bff840 100644
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -133,18 +133,12 @@ static int
 iobase_is_closed(PyObject *self)
 {
     PyObject *res;
+    int ret;
     /* This gets the derived attribute, which is *not* __IOBase_closed
        in most cases! */
-    res = _PyObject_GetAttrId(self, &PyId___IOBase_closed);
-    if (res == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            return -1;
-        }
-        PyErr_Clear();
-        return 0;
-    }
-    Py_DECREF(res);
-    return 1;
+    ret = _PyObject_LookupAttrId(self, &PyId___IOBase_closed, &res);
+    Py_XDECREF(res);
+    return ret;
 }
 
 /* Flush and close methods */
@@ -190,20 +184,16 @@ iobase_check_closed(PyObject *self)
     int closed;
     /* This gets the derived attribute, which is *not* __IOBase_closed
        in most cases! */
-    res = _PyObject_GetAttrWithoutError(self, _PyIO_str_closed);
-    if (res == NULL) {
-        if (PyErr_Occurred()) {
+    closed = _PyObject_LookupAttr(self, _PyIO_str_closed, &res);
+    if (closed > 0) {
+        closed = PyObject_IsTrue(res);
+        Py_DECREF(res);
+        if (closed > 0) {
+            PyErr_SetString(PyExc_ValueError, "I/O operation on closed file.");
             return -1;
         }
-        return 0;
-    }
-    closed = PyObject_IsTrue(res);
-    Py_DECREF(res);
-    if (closed <= 0) {
-        return closed;
     }
-    PyErr_SetString(PyExc_ValueError, "I/O operation on closed file.");
-    return -1;
+    return closed;
 }
 
 PyObject *
@@ -273,8 +263,7 @@ iobase_finalize(PyObject *self)
 
     /* If `closed` doesn't exist or can't be evaluated as bool, then the
        object is probably in an unusable state, so ignore. */
-    res = _PyObject_GetAttrWithoutError(self, _PyIO_str_closed);
-    if (res == NULL) {
+    if (_PyObject_LookupAttr(self, _PyIO_str_closed, &res) <= 0) {
         PyErr_Clear();
         closed = -1;
     }
@@ -538,8 +527,7 @@ _io__IOBase_readline_impl(PyObject *self, Py_ssize_t limit)
     PyObject *peek, *buffer, *result;
     Py_ssize_t old_size = -1;
 
-    peek = _PyObject_GetAttrWithoutError(self, _PyIO_str_peek);
-    if (peek == NULL && PyErr_Occurred()) {
+    if (_PyObject_LookupAttr(self, _PyIO_str_peek, &peek) < 0) {
         return NULL;
     }
 
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
index 9f3fd2d2346..717b56ab319 100644
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -924,14 +924,10 @@ _textiowrapper_set_encoder(textio *self, PyObject *codec_info,
         return -1;
 
     /* Get the normalized named of the codec */
-    res = _PyObject_GetAttrId(codec_info, &PyId_name);
-    if (res == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        else
-            return -1;
+    if (_PyObject_LookupAttrId(codec_info, &PyId_name, &res) < 0) {
+        return -1;
     }
-    else if (PyUnicode_Check(res)) {
+    if (res != NULL && PyUnicode_Check(res)) {
         const encodefuncentry *e = encodefuncs;
         while (e->name != NULL) {
             if (_PyUnicode_EqualToASCIIString(res, e->name)) {
@@ -1177,19 +1173,17 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
 
     if (Py_TYPE(buffer) == &PyBufferedReader_Type ||
         Py_TYPE(buffer) == &PyBufferedWriter_Type ||
-        Py_TYPE(buffer) == &PyBufferedRandom_Type) {
-        raw = _PyObject_GetAttrId(buffer, &PyId_raw);
+        Py_TYPE(buffer) == &PyBufferedRandom_Type)
+    {
+        if (_PyObject_LookupAttrId(buffer, &PyId_raw, &raw) < 0)
+            goto error;
         /* Cache the raw FileIO object to speed up 'closed' checks */
-        if (raw == NULL) {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError))
-                PyErr_Clear();
+        if (raw != NULL) {
+            if (Py_TYPE(raw) == &PyFileIO_Type)
+                self->raw = raw;
             else
-                goto error;
+                Py_DECREF(raw);
         }
-        else if (Py_TYPE(raw) == &PyFileIO_Type)
-            self->raw = raw;
-        else
-            Py_DECREF(raw);
     }
 
     res = _PyObject_CallMethodId(buffer, &PyId_seekable, NULL);
@@ -1201,17 +1195,12 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
         goto error;
     self->seekable = self->telling = r;
 
-    res = _PyObject_GetAttrWithoutError(buffer, _PyIO_str_read1);
-    if (res != NULL) {
-        Py_DECREF(res);
-        self->has_read1 = 1;
-    }
-    else if (!PyErr_Occurred()) {
-        self->has_read1 = 0;
-    }
-    else {
+    r = _PyObject_LookupAttr(buffer, _PyIO_str_read1, &res);
+    if (r < 0) {
         goto error;
     }
+    Py_XDECREF(res);
+    self->has_read1 = r;
 
     self->encoding_start_of_stream = 0;
     if (_textiowrapper_fix_encoder_state(self) < 0) {
@@ -3020,10 +3009,9 @@ textiowrapper_newlines_get(textio *self, void *context)
 {
     PyObject *res;
     CHECK_ATTACHED(self);
-    if (self->decoder == NULL)
-        Py_RETURN_NONE;
-    res = _PyObject_GetAttrWithoutError(self->decoder, _PyIO_str_newlines);
-    if (res == NULL && !PyErr_Occurred()) {
+    if (self->decoder == NULL ||
+        _PyObject_LookupAttr(self->decoder, _PyIO_str_newlines, &res) == 0)
+    {
         Py_RETURN_NONE;
     }
     return res;
diff --git a/Modules/_pickle.c b/Modules/_pickle.c
index f06a87d04f2..05f9a023d92 100644
--- a/Modules/_pickle.c
+++ b/Modules/_pickle.c
@@ -367,18 +367,15 @@ init_method_ref(PyObject *self, _Py_Identifier *name,
                 PyObject **method_func, PyObject **method_self)
 {
     PyObject *func, *func2;
+    int ret;
 
     /* *method_func and *method_self should be consistent.  All refcount decrements
        should be occurred after setting *method_self and *method_func. */
-    func = _PyObject_GetAttrId(self, name);
+    ret = _PyObject_LookupAttrId(self, name, &func);
     if (func == NULL) {
         *method_self = NULL;
         Py_CLEAR(*method_func);
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            return -1;
-        }
-        PyErr_Clear();
-        return 0;
+        return ret;
     }
 
     if (PyMethod_Check(func) && PyMethod_GET_SELF(func) == self) {
@@ -1155,11 +1152,12 @@ _Pickler_SetOutputStream(PicklerObject *self, PyObject *file)
 {
     _Py_IDENTIFIER(write);
     assert(file != NULL);
-    self->write = _PyObject_GetAttrId(file, &PyId_write);
+    if (_PyObject_LookupAttrId(file, &PyId_write, &self->write) < 0) {
+        return -1;
+    }
     if (self->write == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_SetString(PyExc_TypeError,
-                            "file must have a 'write' attribute");
+        PyErr_SetString(PyExc_TypeError,
+                        "file must have a 'write' attribute");
         return -1;
     }
 
@@ -1504,19 +1502,16 @@ _Unpickler_SetInputStream(UnpicklerObject *self, PyObject *file)
     _Py_IDENTIFIER(read);
     _Py_IDENTIFIER(readline);
 
-    self->peek = _PyObject_GetAttrId(file, &PyId_peek);
-    if (self->peek == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        else
-            return -1;
+    if (_PyObject_LookupAttrId(file, &PyId_peek, &self->peek) < 0) {
+        return -1;
     }
-    self->read = _PyObject_GetAttrId(file, &PyId_read);
-    self->readline = _PyObject_GetAttrId(file, &PyId_readline);
+    (void)_PyObject_LookupAttrId(file, &PyId_read, &self->read);
+    (void)_PyObject_LookupAttrId(file, &PyId_readline, &self->readline);
     if (self->readline == NULL || self->read == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
+        if (!PyErr_Occurred()) {
             PyErr_SetString(PyExc_TypeError,
                             "file must have 'read' and 'readline' attributes");
+        }
         Py_CLEAR(self->read);
         Py_CLEAR(self->readline);
         Py_CLEAR(self->peek);
@@ -1691,7 +1686,7 @@ get_deep_attribute(PyObject *obj, PyObject *names, PyObject **pparent)
         PyObject *name = PyList_GET_ITEM(names, i);
         Py_XDECREF(parent);
         parent = obj;
-        obj = PyObject_GetAttr(parent, name);
+        (void)_PyObject_LookupAttr(parent, name, &obj);
         if (obj == NULL) {
             Py_DECREF(parent);
             return NULL;
@@ -1704,16 +1699,6 @@ get_deep_attribute(PyObject *obj, PyObject *names, PyObject **pparent)
     return obj;
 }
 
-static void
-reformat_attribute_error(PyObject *obj, PyObject *name)
-{
-    if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-        PyErr_Clear();
-        PyErr_Format(PyExc_AttributeError,
-                     "Can't get attribute %R on %R", name, obj);
-    }
-}
-
 
 static PyObject *
 getattribute(PyObject *obj, PyObject *name, int allow_qualname)
@@ -1727,10 +1712,13 @@ getattribute(PyObject *obj, PyObject *name, int allow_qualname)
         attr = get_deep_attribute(obj, dotted_path, NULL);
         Py_DECREF(dotted_path);
     }
-    else
-        attr = PyObject_GetAttr(obj, name);
-    if (attr == NULL)
-        reformat_attribute_error(obj, name);
+    else {
+        (void)_PyObject_LookupAttr(obj, name, &attr);
+    }
+    if (attr == NULL && !PyErr_Occurred()) {
+        PyErr_Format(PyExc_AttributeError,
+                     "Can't get attribute %R on %R", name, obj);
+    }
     return attr;
 }
 
@@ -1748,9 +1736,6 @@ _checkmodule(PyObject *module_name, PyObject *module,
 
     PyObject *candidate = get_deep_attribute(module, dotted_path, NULL);
     if (candidate == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-        }
         return -1;
     }
     if (candidate != global) {
@@ -1772,14 +1757,10 @@ whichmodule(PyObject *global, PyObject *dotted_path)
     _Py_IDENTIFIER(modules);
     _Py_IDENTIFIER(__main__);
 
-    module_name = _PyObject_GetAttrId(global, &PyId___module__);
-
-    if (module_name == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-            return NULL;
-        PyErr_Clear();
+    if (_PyObject_LookupAttrId(global, &PyId___module__, &module_name) < 0) {
+        return NULL;
     }
-    else {
+    if (module_name) {
         /* In some rare cases (e.g., bound methods of extension types),
            __module__ can be None. If it is so, then search sys.modules for
            the module of global. */
@@ -3328,12 +3309,8 @@ save_global(PicklerObject *self, PyObject *obj, PyObject *name)
         global_name = name;
     }
     else {
-        global_name = _PyObject_GetAttrId(obj, &PyId___qualname__);
-        if (global_name == NULL) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-                goto error;
-            PyErr_Clear();
-        }
+        if (_PyObject_LookupAttrId(obj, &PyId___qualname__, &global_name) < 0)
+            goto error;
         if (global_name == NULL) {
             global_name = _PyObject_GetAttrId(obj, &PyId___name__);
             if (global_name == NULL)
@@ -3656,13 +3633,9 @@ get_class(PyObject *obj)
     PyObject *cls;
     _Py_IDENTIFIER(__class__);
 
-    cls = _PyObject_GetAttrId(obj, &PyId___class__);
-    if (cls == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-            cls = (PyObject *) Py_TYPE(obj);
-            Py_INCREF(cls);
-        }
+    if (_PyObject_LookupAttrId(obj, &PyId___class__, &cls) == 0) {
+        cls = (PyObject *) Py_TYPE(obj);
+        Py_INCREF(cls);
     }
     return cls;
 }
@@ -3734,14 +3707,10 @@ save_reduce(PicklerObject *self, PyObject *args, PyObject *obj)
         PyObject *name;
         _Py_IDENTIFIER(__name__);
 
-        name = _PyObject_GetAttrId(callable, &PyId___name__);
-        if (name == NULL) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                return -1;
-            }
-            PyErr_Clear();
+        if (_PyObject_LookupAttrId(callable, &PyId___name__, &name) < 0) {
+            return -1;
         }
-        else if (PyUnicode_Check(name)) {
+        if (name != NULL && PyUnicode_Check(name)) {
             _Py_IDENTIFIER(__newobj_ex__);
             use_newobj_ex = _PyUnicode_EqualToASCIIId(
                     name, &PyId___newobj_ex__);
@@ -4108,7 +4077,9 @@ save(PicklerObject *self, PyObject *obj, int pers_save)
            don't actually have to check for a __reduce__ method. */
 
         /* Check for a __reduce_ex__ method. */
-        reduce_func = _PyObject_GetAttrId(obj, &PyId___reduce_ex__);
+        if (_PyObject_LookupAttrId(obj, &PyId___reduce_ex__, &reduce_func) < 0) {
+            goto error;
+        }
         if (reduce_func != NULL) {
             PyObject *proto;
             proto = PyLong_FromLong(self->proto);
@@ -4119,12 +4090,6 @@ save(PicklerObject *self, PyObject *obj, int pers_save)
         else {
             PickleState *st = _Pickle_GetGlobalState();
 
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_Clear();
-            }
-            else {
-                goto error;
-            }
             /* Check for a __reduce__ method. */
             reduce_func = _PyObject_GetAttrId(obj, &PyId___reduce__);
             if (reduce_func != NULL) {
@@ -4401,13 +4366,9 @@ _pickle_Pickler___init___impl(PicklerObject *self, PyObject *file,
         return -1;
     }
 
-    self->dispatch_table = _PyObject_GetAttrId((PyObject *)self,
-                                               &PyId_dispatch_table);
-    if (self->dispatch_table == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            return -1;
-        }
-        PyErr_Clear();
+    if (_PyObject_LookupAttrId((PyObject *)self,
+                                    &PyId_dispatch_table, &self->dispatch_table) < 0) {
+        return -1;
     }
 
     return 0;
@@ -5370,12 +5331,11 @@ instantiate(PyObject *cls, PyObject *args)
     if (!PyTuple_GET_SIZE(args) && PyType_Check(cls)) {
         _Py_IDENTIFIER(__getinitargs__);
         _Py_IDENTIFIER(__new__);
-        PyObject *func = _PyObject_GetAttrId(cls, &PyId___getinitargs__);
+        PyObject *func;
+        if (_PyObject_LookupAttrId(cls, &PyId___getinitargs__, &func) < 0) {
+            return NULL;
+        }
         if (func == NULL) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                return NULL;
-            }
-            PyErr_Clear();
             return _PyObject_CallMethodIdObjArgs(cls, &PyId___new__, cls, NULL);
         }
         Py_DECREF(func);
@@ -6225,16 +6185,11 @@ load_build(UnpicklerObject *self)
 
     inst = self->stack->data[Py_SIZE(self->stack) - 1];
 
-    setstate = _PyObject_GetAttrId(inst, &PyId___setstate__);
-    if (setstate == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        else {
-            Py_DECREF(state);
-            return -1;
-        }
+    if (_PyObject_LookupAttrId(inst, &PyId___setstate__, &setstate) < 0) {
+        Py_DECREF(state);
+        return -1;
     }
-    else {
+    if (setstate != NULL) {
         PyObject *result;
 
         /* The explicit __setstate__ is responsible for everything. */
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
index 8c3f0a1c6c6..900c374c585 100644
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2203,11 +2203,10 @@ array_array___reduce_ex__(arrayobject *self, PyObject *value)
     if (protocol == -1 && PyErr_Occurred())
         return NULL;
 
-    dict = _PyObject_GetAttrId((PyObject *)self, &PyId___dict__);
+    if (_PyObject_LookupAttrId((PyObject *)self, &PyId___dict__, &dict) < 0) {
+        return NULL;
+    }
     if (dict == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-            return NULL;
-        PyErr_Clear();
         dict = Py_None;
         Py_INCREF(dict);
     }
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
index 985915f09d5..1113fb6b76c 100644
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -830,17 +830,15 @@ tee(PyObject *self, PyObject *args)
         return NULL;
     }
 
-    copyfunc = _PyObject_GetAttrId(it, &PyId___copy__);
-    if (copyfunc != NULL) {
-        copyable = it;
-    }
-    else if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
+    if (_PyObject_LookupAttrId(it, &PyId___copy__, &copyfunc) < 0) {
         Py_DECREF(it);
         Py_DECREF(result);
         return NULL;
     }
+    if (copyfunc != NULL) {
+        copyable = it;
+    }
     else {
-        PyErr_Clear();
         copyable = tee_fromiterable(it);
         Py_DECREF(it);
         if (copyable == NULL) {
diff --git a/Objects/abstract.c b/Objects/abstract.c
index a68253bdb43..93eda62a274 100644
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -171,16 +171,14 @@ PyObject_GetItem(PyObject *o, PyObject *key)
     if (PyType_Check(o)) {
         PyObject *meth, *result, *stack[1] = {key};
         _Py_IDENTIFIER(__class_getitem__);
-        meth = _PyObject_GetAttrId(o, &PyId___class_getitem__);
+        if (_PyObject_LookupAttrId(o, &PyId___class_getitem__, &meth) < 0) {
+            return NULL;
+        }
         if (meth) {
             result = _PyObject_FastCall(meth, stack, 1);
             Py_DECREF(meth);
             return result;
         }
-        else if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            return NULL;
-        }
-        PyErr_Clear();
     }
 
     return type_error("'%.200s' object is not subscriptable", o);
@@ -2268,14 +2266,9 @@ abstract_get_bases(PyObject *cls)
     PyObject *bases;
 
     Py_ALLOW_RECURSION
-    bases = _PyObject_GetAttrId(cls, &PyId___bases__);
+    (void)_PyObject_LookupAttrId(cls, &PyId___bases__, &bases);
     Py_END_ALLOW_RECURSION
-    if (bases == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        return NULL;
-    }
-    if (!PyTuple_Check(bases)) {
+    if (bases != NULL && !PyTuple_Check(bases)) {
         Py_DECREF(bases);
         return NULL;
     }
@@ -2338,26 +2331,23 @@ static int
 recursive_isinstance(PyObject *inst, PyObject *cls)
 {
     PyObject *icls;
-    int retval = 0;
+    int retval;
     _Py_IDENTIFIER(__class__);
 
     if (PyType_Check(cls)) {
         retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls);
         if (retval == 0) {
-            PyObject *c = _PyObject_GetAttrId(inst, &PyId___class__);
-            if (c == NULL) {
-                if (PyErr_ExceptionMatches(PyExc_AttributeError))
-                    PyErr_Clear();
-                else
-                    retval = -1;
-            }
-            else {
-                if (c != (PyObject *)(inst->ob_type) &&
-                    PyType_Check(c))
+            retval = _PyObject_LookupAttrId(inst, &PyId___class__, &icls);
+            if (icls != NULL) {
+                if (icls != (PyObject *)(inst->ob_type) && PyType_Check(icls)) {
                     retval = PyType_IsSubtype(
-                        (PyTypeObject *)c,
+                        (PyTypeObject *)icls,
                         (PyTypeObject *)cls);
-                Py_DECREF(c);
+                }
+                else {
+                    retval = 0;
+                }
+                Py_DECREF(icls);
             }
         }
     }
@@ -2365,14 +2355,8 @@ recursive_isinstance(PyObject *inst, PyObject *cls)
         if (!check_class(cls,
             "isinstance() arg 2 must be a type or tuple of types"))
             return -1;
-        icls = _PyObject_GetAttrId(inst, &PyId___class__);
-        if (icls == NULL) {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError))
-                PyErr_Clear();
-            else
-                retval = -1;
-        }
-        else {
+        retval = _PyObject_LookupAttrId(inst, &PyId___class__, &icls);
+        if (icls != NULL) {
             retval = abstract_issubclass(icls, cls);
             Py_DECREF(icls);
         }
diff --git a/Objects/classobject.c b/Objects/classobject.c
index e88c95cbfbd..3dc23b796c2 100644
--- a/Objects/classobject.c
+++ b/Objects/classobject.c
@@ -246,21 +246,14 @@ method_repr(PyMethodObject *a)
 {
     PyObject *self = a->im_self;
     PyObject *func = a->im_func;
-    PyObject *funcname = NULL, *result = NULL;
+    PyObject *funcname, *result;
     const char *defname = "?";
 
-    funcname = _PyObject_GetAttrId(func, &PyId___qualname__);
-    if (funcname == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-            return NULL;
-        PyErr_Clear();
-
-        funcname = _PyObject_GetAttrId(func, &PyId___name__);
-        if (funcname == NULL) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-                return NULL;
-            PyErr_Clear();
-        }
+    if (_PyObject_LookupAttrId(func, &PyId___qualname__, &funcname) < 0 ||
+        (funcname == NULL &&
+         _PyObject_LookupAttrId(func, &PyId___name__, &funcname) < 0))
+    {
+        return NULL;
     }
 
     if (funcname != NULL && !PyUnicode_Check(funcname)) {
@@ -542,7 +535,7 @@ static PyObject *
 instancemethod_repr(PyObject *self)
 {
     PyObject *func = PyInstanceMethod_Function(self);
-    PyObject *funcname = NULL , *result = NULL;
+    PyObject *funcname, *result;
     const char *defname = "?";
 
     if (func == NULL) {
@@ -550,13 +543,10 @@ instancemethod_repr(PyObject *self)
         return NULL;
     }
 
-    funcname = _PyObject_GetAttrId(func, &PyId___name__);
-    if (funcname == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-            return NULL;
-        PyErr_Clear();
+    if (_PyObject_LookupAttrId(func, &PyId___name__, &funcname) < 0) {
+        return NULL;
     }
-    else if (!PyUnicode_Check(funcname)) {
+    if (funcname != NULL && !PyUnicode_Check(funcname)) {
         Py_DECREF(funcname);
         funcname = NULL;
     }
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 259fa255653..0d25739f72c 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -2234,17 +2234,16 @@ dict_update_common(PyObject *self, PyObject *args, PyObject *kwds,
     }
     else if (arg != NULL) {
         _Py_IDENTIFIER(keys);
-        PyObject *func = _PyObject_GetAttrId(arg, &PyId_keys);
-        if (func != NULL) {
+        PyObject *func;
+        if (_PyObject_LookupAttrId(arg, &PyId_keys, &func) < 0) {
+            result = -1;
+        }
+        else if (func != NULL) {
             Py_DECREF(func);
             result = PyDict_Merge(self, arg, 1);
         }
-        else if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-            result = PyDict_MergeFromSeq2(self, arg, 1);
-        }
         else {
-            result = -1;
+            result = PyDict_MergeFromSeq2(self, arg, 1);
         }
     }
 
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 0a34c1f6a97..7baffa7fc02 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -352,13 +352,11 @@ gen_close_iter(PyObject *yf)
             return -1;
     }
     else {
-        PyObject *meth = _PyObject_GetAttrId(yf, &PyId_close);
-        if (meth == NULL) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-                PyErr_WriteUnraisable(yf);
-            PyErr_Clear();
+        PyObject *meth;
+        if (_PyObject_LookupAttrId(yf, &PyId_close, &meth) < 0) {
+            PyErr_WriteUnraisable(yf);
         }
-        else {
+        if (meth) {
             retval = _PyObject_CallNoArg(meth);
             Py_DECREF(meth);
             if (retval == NULL)
@@ -471,13 +469,12 @@ _gen_throw(PyGenObject *gen, int close_on_genexit,
             gen->gi_running = 0;
         } else {
             /* `yf` is an iterator or a coroutine-like object. */
-            PyObject *meth = _PyObject_GetAttrId(yf, &PyId_throw);
+            PyObject *meth;
+            if (_PyObject_LookupAttrId(yf, &PyId_throw, &meth) < 0) {
+                Py_DECREF(yf);
+                return NULL;
+            }
             if (meth == NULL) {
-                if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                    Py_DECREF(yf);
-                    return NULL;
-                }
-                PyErr_Clear();
                 Py_DECREF(yf);
                 goto throw_here;
             }
diff --git a/Objects/object.c b/Objects/object.c
index 8cec6e2f122..7b2adbea1a1 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -817,16 +817,11 @@ _PyObject_IsAbstract(PyObject *obj)
     if (obj == NULL)
         return 0;
 
-    isabstract = _PyObject_GetAttrId(obj, &PyId___isabstractmethod__);
-    if (isabstract == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-            return 0;
-        }
-        return -1;
+    res = _PyObject_LookupAttrId(obj, &PyId___isabstractmethod__, &isabstract);
+    if (res > 0) {
+        res = PyObject_IsTrue(isabstract);
+        Py_DECREF(isabstract);
     }
-    res = PyObject_IsTrue(isabstract);
-    Py_DECREF(isabstract);
     return res;
 }
 
@@ -888,47 +883,74 @@ PyObject_GetAttr(PyObject *v, PyObject *name)
     return NULL;
 }
 
-PyObject *
-_PyObject_GetAttrWithoutError(PyObject *v, PyObject *name)
+int
+_PyObject_LookupAttr(PyObject *v, PyObject *name, PyObject **result)
 {
     PyTypeObject *tp = Py_TYPE(v);
-    PyObject *ret = NULL;
 
     if (!PyUnicode_Check(name)) {
         PyErr_Format(PyExc_TypeError,
                      "attribute name must be string, not '%.200s'",
                      name->ob_type->tp_name);
-        return NULL;
+        *result = NULL;
+        return -1;
     }
 
     if (tp->tp_getattro == PyObject_GenericGetAttr) {
-        return _PyObject_GenericGetAttrWithDict(v, name, NULL, 1);
+        *result = _PyObject_GenericGetAttrWithDict(v, name, NULL, 1);
+        if (*result != NULL) {
+            return 1;
+        }
+        if (PyErr_Occurred()) {
+            return -1;
+        }
+        return 0;
     }
     if (tp->tp_getattro != NULL) {
-        ret = (*tp->tp_getattro)(v, name);
+        *result = (*tp->tp_getattro)(v, name);
     }
     else if (tp->tp_getattr != NULL) {
         const char *name_str = PyUnicode_AsUTF8(name);
-        if (name_str == NULL)
-            return NULL;
-        ret = (*tp->tp_getattr)(v, (char *)name_str);
+        if (name_str == NULL) {
+            *result = NULL;
+            return -1;
+        }
+        *result = (*tp->tp_getattr)(v, (char *)name_str);
     }
-    if (ret == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
-        PyErr_Clear();
+    if (*result != NULL) {
+        return 1;
     }
-    return ret;
+    if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
+        return -1;
+    }
+    PyErr_Clear();
+    return 0;
+}
+
+int
+_PyObject_LookupAttrId(PyObject *v, _Py_Identifier *name, PyObject **result)
+{
+    PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
+    if (!oname) {
+        *result = NULL;
+        return -1;
+    }
+    return  _PyObject_LookupAttr(v, oname, result);
 }
 
 int
 PyObject_HasAttr(PyObject *v, PyObject *name)
 {
-    PyObject *res = _PyObject_GetAttrWithoutError(v, name);
-    if (res != NULL) {
-        Py_DECREF(res);
-        return 1;
+    PyObject *res;
+    if (_PyObject_LookupAttr(v, name, &res) < 0) {
+        PyErr_Clear();
+        return 0;
     }
-    PyErr_Clear();
-    return 0;
+    if (res == NULL) {
+        return 0;
+    }
+    Py_DECREF(res);
+    return 1;
 }
 
 int
diff --git a/Objects/odictobject.c b/Objects/odictobject.c
index 218aa2c5d9c..bf19fedb07f 100644
--- a/Objects/odictobject.c
+++ b/Objects/odictobject.c
@@ -2359,7 +2359,10 @@ mutablemapping_update(PyObject *self, PyObject *args, PyObject *kwargs)
             goto handle_kwargs;
         }
 
-        func = _PyObject_GetAttrId(other, &PyId_keys);
+        if (_PyObject_LookupAttrId(other, &PyId_keys, &func) < 0) {
+            Py_DECREF(other);
+            return NULL;
+        }
         if (func != NULL) {
             PyObject *keys, *iterator, *key;
             keys = _PyObject_CallNoArg(func);
@@ -2391,15 +2394,11 @@ mutablemapping_update(PyObject *self, PyObject *args, PyObject *kwargs)
                 return NULL;
             goto handle_kwargs;
         }
-        else if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
+
+        if (_PyObject_LookupAttrId(other, &PyId_items, &func) < 0) {
             Py_DECREF(other);
             return NULL;
         }
-        else {
-            PyErr_Clear();
-        }
-
-        func = _PyObject_GetAttrId(other, &PyId_items);
         if (func != NULL) {
             PyObject *items;
             Py_DECREF(other);
@@ -2413,13 +2412,6 @@ mutablemapping_update(PyObject *self, PyObject *args, PyObject *kwargs)
                 return NULL;
             goto handle_kwargs;
         }
-        else if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            Py_DECREF(other);
-            return NULL;
-        }
-        else {
-            PyErr_Clear();
-        }
 
         res = mutablemapping_add_pairs(self, other);
         Py_DECREF(other);
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index 0c8f0adfb78..a22173878e8 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -2403,7 +2403,9 @@ type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
             if (PyType_Check(tmp)) {
                 continue;
             }
-            tmp = _PyObject_GetAttrId(tmp, &PyId___mro_entries__);
+            if (_PyObject_LookupAttrId(tmp, &PyId___mro_entries__, &tmp) < 0) {
+                return NULL;
+            }
             if (tmp != NULL) {
                 PyErr_SetString(PyExc_TypeError,
                                 "type() doesn't support MRO entry resolution; "
@@ -2411,12 +2413,6 @@ type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
                 Py_DECREF(tmp);
                 return NULL;
             }
-            else if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_Clear();
-            }
-            else {
-                return NULL;
-            }
         }
         /* Search the bases for the proper metatype to deal with this: */
         winner = _PyType_CalculateMetaclass(metatype, bases);
@@ -4099,15 +4095,12 @@ _PyObject_GetState(PyObject *obj, int required)
     PyObject *getstate;
     _Py_IDENTIFIER(__getstate__);
 
-    getstate = _PyObject_GetAttrId(obj, &PyId___getstate__);
+    if (_PyObject_LookupAttrId(obj, &PyId___getstate__, &getstate) < 0) {
+        return NULL;
+    }
     if (getstate == NULL) {
         PyObject *slotnames;
 
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            return NULL;
-        }
-        PyErr_Clear();
-
         if (required && obj->ob_type->tp_itemsize) {
             PyErr_Format(PyExc_TypeError,
                          "can't pickle %.200s objects",
@@ -4174,14 +4167,12 @@ _PyObject_GetState(PyObject *obj, int required)
 
                 name = PyList_GET_ITEM(slotnames, i);
                 Py_INCREF(name);
-                value = PyObject_GetAttr(obj, name);
+                if (_PyObject_LookupAttr(obj, name, &value) < 0) {
+                    goto error;
+                }
                 if (value == NULL) {
                     Py_DECREF(name);
-                    if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                        goto error;
-                    }
                     /* It is not an error if the attribute is not present. */
-                    PyErr_Clear();
                 }
                 else {
                     int err = PyDict_SetItem(slots, name, value);
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
index 69583d914f6..399e79e0597 100644
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -497,11 +497,27 @@ def isSimpleType(self, field):
 
     def visitField(self, field, name, sum=None, prod=None, depth=0):
         ctype = get_c_type(field.type)
+        self.emit("if (_PyObject_LookupAttrId(obj, &PyId_%s, &tmp) < 0) {" % field.name, depth)
+        self.emit("return 1;", depth+1)
+        self.emit("}", depth)
         if not field.opt:
-            self.emit("tmp = _PyObject_GetAttrId(obj, &PyId_%s);" % field.name, depth)
+            self.emit("if (tmp == NULL) {", depth)
+            message = "required field \\\"%s\\\" missing from %s" % (field.name, name)
+            format = "PyErr_SetString(PyExc_TypeError, \"%s\");"
+            self.emit(format % message, depth+1, reflow=False)
+            self.emit("return 1;", depth+1)
         else:
-            self.emit("tmp = get_not_none(obj, &PyId_%s);" % field.name, depth)
-        self.emit("if (tmp != NULL) {", depth)
+            self.emit("if (tmp == NULL || tmp == Py_None) {", depth)
+            self.emit("Py_CLEAR(tmp);", depth+1)
+            if self.isNumeric(field):
+                self.emit("%s = 0;" % field.name, depth+1)
+            elif not self.isSimpleType(field):
+                self.emit("%s = NULL;" % field.name, depth+1)
+            else:
+                raise TypeError("could not determine the default value for %s" % field.name)
+        self.emit("}", depth)
+        self.emit("else {", depth)
+
         self.emit("int res;", depth+1)
         if field.seq:
             self.emit("Py_ssize_t len;", depth+1)
@@ -539,25 +555,6 @@ def visitField(self, field, name, sum=None, prod=None, depth=0):
             self.emit("if (res != 0) goto failed;", depth+1)
 
         self.emit("Py_CLEAR(tmp);", depth+1)
-        if not field.opt:
-            self.emit("} else {", depth)
-            self.emit("if (PyErr_ExceptionMatches(PyExc_AttributeError)) {", depth+1)
-            message = "required field \\\"%s\\\" missing from %s" % (field.name, name)
-            format = "PyErr_SetString(PyExc_TypeError, \"%s\");"
-            self.emit(format % message, depth+2, reflow=False)
-            self.emit("}", depth+1)
-            self.emit("return 1;", depth+1)
-        else:
-            self.emit("} else if (PyErr_Occurred()) {", depth)
-            self.emit("return 1;", depth+1)
-            self.emit("} else {", depth)
-
-            if self.isNumeric(field):
-                self.emit("%s = 0;" % field.name, depth+1)
-            elif not self.isSimpleType(field):
-                self.emit("%s = NULL;" % field.name, depth+1)
-            else:
-                raise TypeError("could not determine the default value for %s" % field.name)
         self.emit("}", depth)
 
 
@@ -662,18 +659,14 @@ def visitModule(self, mod):
     Py_ssize_t i, numfields = 0;
     int res = -1;
     PyObject *key, *value, *fields;
-    fields = _PyObject_GetAttrId((PyObject*)Py_TYPE(self), &PyId__fields);
+    if (_PyObject_LookupAttrId((PyObject*)Py_TYPE(self), &PyId__fields, &fields) < 0) {
+        goto cleanup;
+    }
     if (fields) {
         numfields = PySequence_Size(fields);
         if (numfields == -1)
             goto cleanup;
     }
-    else if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-        PyErr_Clear();
-    }
-    else {
-        goto cleanup;
-    }
 
     res = 0; /* if no error occurs, this stays 0 to the end */
     if (numfields < PyTuple_GET_SIZE(args)) {
@@ -713,19 +706,13 @@ def visitModule(self, mod):
 static PyObject *
 ast_type_reduce(PyObject *self, PyObject *unused)
 {
-    PyObject *res;
     _Py_IDENTIFIER(__dict__);
-    PyObject *dict = _PyObject_GetAttrId(self, &PyId___dict__);
-    if (dict == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        else
-            return NULL;
+    PyObject *dict;
+    if (_PyObject_LookupAttrId(self, &PyId___dict__, &dict) < 0) {
+        return NULL;
     }
     if (dict) {
-        res = Py_BuildValue("O()O", Py_TYPE(self), dict);
-        Py_DECREF(dict);
-        return res;
+        return Py_BuildValue("O()N", Py_TYPE(self), dict);
     }
     return Py_BuildValue("O()", Py_TYPE(self));
 }
@@ -965,22 +952,6 @@ def visitModule(self, mod):
     return 0;
 }
 
-static PyObject *get_not_none(PyObject *obj, _Py_Identifier *id)
-{
-    PyObject *attr = _PyObject_GetAttrId(obj, id);
-    if (!attr) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-        }
-        return NULL;
-    }
-    else if (attr == Py_None) {
-        Py_DECREF(attr);
-        return NULL;
-    }
-    return attr;
-}
-
 """, 0, reflow=False)
 
         self.emit("static int init_types(void)",0)
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
index 81226c8a060..f7a056845e9 100644
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -545,18 +545,14 @@ ast_type_init(PyObject *self, PyObject *args, PyObject *kw)
     Py_ssize_t i, numfields = 0;
     int res = -1;
     PyObject *key, *value, *fields;
-    fields = _PyObject_GetAttrId((PyObject*)Py_TYPE(self), &PyId__fields);
+    if (_PyObject_LookupAttrId((PyObject*)Py_TYPE(self), &PyId__fields, &fields) < 0) {
+        goto cleanup;
+    }
     if (fields) {
         numfields = PySequence_Size(fields);
         if (numfields == -1)
             goto cleanup;
     }
-    else if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-        PyErr_Clear();
-    }
-    else {
-        goto cleanup;
-    }
 
     res = 0; /* if no error occurs, this stays 0 to the end */
     if (numfields < PyTuple_GET_SIZE(args)) {
@@ -596,19 +592,13 @@ ast_type_init(PyObject *self, PyObject *args, PyObject *kw)
 static PyObject *
 ast_type_reduce(PyObject *self, PyObject *unused)
 {
-    PyObject *res;
     _Py_IDENTIFIER(__dict__);
-    PyObject *dict = _PyObject_GetAttrId(self, &PyId___dict__);
-    if (dict == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError))
-            PyErr_Clear();
-        else
-            return NULL;
+    PyObject *dict;
+    if (_PyObject_LookupAttrId(self, &PyId___dict__, &dict) < 0) {
+        return NULL;
     }
     if (dict) {
-        res = Py_BuildValue("O()O", Py_TYPE(self), dict);
-        Py_DECREF(dict);
-        return res;
+        return Py_BuildValue("O()N", Py_TYPE(self), dict);
     }
     return Py_BuildValue("O()", Py_TYPE(self));
 }
@@ -848,22 +838,6 @@ static int add_ast_fields(void)
     return 0;
 }
 
-static PyObject *get_not_none(PyObject *obj, _Py_Identifier *id)
-{
-    PyObject *attr = _PyObject_GetAttrId(obj, id);
-    if (!attr) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-        }
-        return NULL;
-    }
-    else if (attr == Py_None) {
-        Py_DECREF(attr);
-        return NULL;
-    }
-    return attr;
-}
-
 
 static int init_types(void)
 {
@@ -4012,8 +3986,14 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
         asdl_seq* body;
         string docstring;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Module");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4035,22 +4015,19 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Module");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_docstring, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_docstring);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            docstring = NULL;
+        }
+        else {
             int res;
             res = obj2ast_string(tmp, &docstring, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            docstring = NULL;
         }
         *out = Module(body, docstring, arena);
         if (*out == NULL) goto failed;
@@ -4063,8 +4040,14 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* body;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Interactive");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4086,11 +4069,6 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Interactive");
-            }
-            return 1;
         }
         *out = Interactive(body, arena);
         if (*out == NULL) goto failed;
@@ -4103,17 +4081,18 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty body;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Expression");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &body, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Expression");
-            }
-            return 1;
         }
         *out = Expression(body, arena);
         if (*out == NULL) goto failed;
@@ -4126,8 +4105,14 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* body;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Suite");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4149,11 +4134,6 @@ obj2ast_mod(PyObject* obj, mod_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Suite");
-            }
-            return 1;
         }
         *out = Suite(body, arena);
         if (*out == NULL) goto failed;
@@ -4179,29 +4159,31 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         *out = NULL;
         return 0;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_lineno, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from stmt");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_int(tmp, &lineno, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from stmt");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_col_offset, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from stmt");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_int(tmp, &col_offset, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from stmt");
-        }
-        return 1;
     }
     isinstance = PyObject_IsInstance(obj, (PyObject*)FunctionDef_type);
     if (isinstance == -1) {
@@ -4215,32 +4197,40 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         expr_ty returns;
         string docstring;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_name);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_name, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from FunctionDef");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_identifier(tmp, &name, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from FunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_args, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from FunctionDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_args);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_arguments(tmp, &args, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from FunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from FunctionDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4262,14 +4252,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from FunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_decorator_list, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from FunctionDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_decorator_list);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4291,33 +4282,32 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(decorator_list, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from FunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_returns, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_returns);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            returns = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &returns, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_docstring, &tmp) < 0) {
             return 1;
-        } else {
-            returns = NULL;
         }
-        tmp = get_not_none(obj, &PyId_docstring);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            docstring = NULL;
+        }
+        else {
             int res;
             res = obj2ast_string(tmp, &docstring, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            docstring = NULL;
         }
         *out = FunctionDef(name, args, body, decorator_list, returns,
                            docstring, lineno, col_offset, arena);
@@ -4336,32 +4326,40 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         expr_ty returns;
         string docstring;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_name);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_name, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from AsyncFunctionDef");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_identifier(tmp, &name, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from AsyncFunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_args, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from AsyncFunctionDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_args);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_arguments(tmp, &args, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from AsyncFunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFunctionDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4383,14 +4381,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_decorator_list, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from AsyncFunctionDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_decorator_list);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4412,33 +4411,32 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(decorator_list, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from AsyncFunctionDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_returns, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_returns);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            returns = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &returns, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_docstring, &tmp) < 0) {
             return 1;
-        } else {
-            returns = NULL;
         }
-        tmp = get_not_none(obj, &PyId_docstring);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            docstring = NULL;
+        }
+        else {
             int res;
             res = obj2ast_string(tmp, &docstring, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            docstring = NULL;
         }
         *out = AsyncFunctionDef(name, args, body, decorator_list, returns,
                                 docstring, lineno, col_offset, arena);
@@ -4457,20 +4455,27 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* decorator_list;
         string docstring;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_name);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_name, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from ClassDef");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_identifier(tmp, &name, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from ClassDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_bases, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"bases\" missing from ClassDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_bases);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4492,14 +4497,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(bases, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"bases\" missing from ClassDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_keywords, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from ClassDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_keywords);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4521,14 +4527,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(keywords, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from ClassDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ClassDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4550,14 +4557,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ClassDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_decorator_list, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from ClassDef");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_decorator_list);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4579,22 +4587,19 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(decorator_list, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from ClassDef");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_docstring, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_docstring);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            docstring = NULL;
+        }
+        else {
             int res;
             res = obj2ast_string(tmp, &docstring, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            docstring = NULL;
         }
         *out = ClassDef(name, bases, keywords, body, decorator_list, docstring,
                         lineno, col_offset, arena);
@@ -4608,16 +4613,18 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty value;
 
-        tmp = get_not_none(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            value = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            value = NULL;
         }
         *out = Return(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -4630,8 +4637,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* targets;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_targets);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_targets, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Delete");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4653,11 +4666,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(targets, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Delete");
-            }
-            return 1;
         }
         *out = Delete(targets, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -4671,8 +4679,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* targets;
         expr_ty value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_targets);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_targets, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Assign");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4694,23 +4708,19 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(targets, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Assign");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Assign");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Assign");
-            }
-            return 1;
         }
         *out = Assign(targets, value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -4725,41 +4735,44 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         operator_ty op;
         expr_ty value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_target);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_target, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AugAssign");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &target, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AugAssign");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_op, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_op);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from AugAssign");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_operator(tmp, &op, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from AugAssign");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from AugAssign");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from AugAssign");
-            }
-            return 1;
         }
         *out = AugAssign(target, op, value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -4775,52 +4788,57 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         expr_ty value;
         int simple;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_target);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_target, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AnnAssign");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &target, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AnnAssign");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_annotation, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"annotation\" missing from AnnAssign");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_annotation);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &annotation, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"annotation\" missing from AnnAssign");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            value = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_simple, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"simple\" missing from AnnAssign");
             return 1;
-        } else {
-            value = NULL;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_simple);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_int(tmp, &simple, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"simple\" missing from AnnAssign");
-            }
-            return 1;
         }
         *out = AnnAssign(target, annotation, value, simple, lineno, col_offset,
                          arena);
@@ -4837,32 +4855,40 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* body;
         asdl_seq* orelse;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_target);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_target, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from For");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &target, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from For");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_iter, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from For");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_iter);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &iter, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from For");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from For");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4884,14 +4910,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from For");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_orelse, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from For");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4913,11 +4940,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(orelse, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from For");
-            }
-            return 1;
         }
         *out = For(target, iter, body, orelse, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -4933,32 +4955,40 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* body;
         asdl_seq* orelse;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_target);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_target, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AsyncFor");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &target, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AsyncFor");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_iter, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from AsyncFor");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_iter);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &iter, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from AsyncFor");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFor");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -4980,14 +5010,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFor");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_orelse, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from AsyncFor");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5009,11 +5040,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(orelse, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from AsyncFor");
-            }
-            return 1;
         }
         *out = AsyncFor(target, iter, body, orelse, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5028,20 +5054,27 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* body;
         asdl_seq* orelse;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_test);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_test, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from While");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &test, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from While");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from While");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5063,14 +5096,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from While");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_orelse, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from While");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5092,11 +5126,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(orelse, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from While");
-            }
-            return 1;
         }
         *out = While(test, body, orelse, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5111,21 +5140,28 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* body;
         asdl_seq* orelse;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_test);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_test, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from If");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &test, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from If");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
-            int res;
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from If");
+            return 1;
+        }
+        else {
+            int res;
             Py_ssize_t len;
             Py_ssize_t i;
             if (!PyList_Check(tmp)) {
@@ -5146,14 +5182,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from If");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_orelse, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from If");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5175,11 +5212,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(orelse, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from If");
-            }
-            return 1;
         }
         *out = If(test, body, orelse, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5193,8 +5225,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* items;
         asdl_seq* body;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_items);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_items, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from With");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5216,14 +5254,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(items, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from With");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from With");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5245,11 +5284,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from With");
-            }
-            return 1;
         }
         *out = With(items, body, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5263,8 +5297,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* items;
         asdl_seq* body;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_items);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_items, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from AsyncWith");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5286,14 +5326,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(items, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from AsyncWith");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncWith");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5315,11 +5356,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncWith");
-            }
-            return 1;
         }
         *out = AsyncWith(items, body, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5333,27 +5369,31 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         expr_ty exc;
         expr_ty cause;
 
-        tmp = get_not_none(obj, &PyId_exc);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_exc, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            exc = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &exc, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_cause, &tmp) < 0) {
             return 1;
-        } else {
-            exc = NULL;
         }
-        tmp = get_not_none(obj, &PyId_cause);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            cause = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &cause, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            cause = NULL;
         }
         *out = Raise(exc, cause, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5369,8 +5409,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* orelse;
         asdl_seq* finalbody;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Try");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5392,14 +5438,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Try");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_handlers, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_handlers);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"handlers\" missing from Try");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5421,14 +5468,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(handlers, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"handlers\" missing from Try");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_orelse, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from Try");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5450,14 +5498,15 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(orelse, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from Try");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_finalbody, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"finalbody\" missing from Try");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_finalbody);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5479,11 +5528,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(finalbody, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"finalbody\" missing from Try");
-            }
-            return 1;
         }
         *out = Try(body, handlers, orelse, finalbody, lineno, col_offset,
                    arena);
@@ -5498,28 +5542,31 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         expr_ty test;
         expr_ty msg;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_test);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_test, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from Assert");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &test, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from Assert");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_msg, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_msg);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            msg = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &msg, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            msg = NULL;
         }
         *out = Assert(test, msg, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5532,8 +5579,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* names;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_names);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_names, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Import");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5555,11 +5608,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(names, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Import");
-            }
-            return 1;
         }
         *out = Import(names, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5574,19 +5622,27 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
         asdl_seq* names;
         int level;
 
-        tmp = get_not_none(obj, &PyId_module);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_module, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            module = NULL;
+        }
+        else {
             int res;
             res = obj2ast_identifier(tmp, &module, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_names, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from ImportFrom");
             return 1;
-        } else {
-            module = NULL;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_names);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5608,22 +5664,19 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(names, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from ImportFrom");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_level, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_level);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            level = 0;
+        }
+        else {
             int res;
             res = obj2ast_int(tmp, &level, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            level = 0;
         }
         *out = ImportFrom(module, names, level, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5636,8 +5689,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* names;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_names);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_names, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Global");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5659,11 +5718,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(names, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Global");
-            }
-            return 1;
         }
         *out = Global(names, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5676,8 +5730,14 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* names;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_names);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_names, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Nonlocal");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5699,11 +5759,6 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
                 asdl_seq_SET(names, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Nonlocal");
-            }
-            return 1;
         }
         *out = Nonlocal(names, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5716,17 +5771,18 @@ obj2ast_stmt(PyObject* obj, stmt_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Expr");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Expr");
-            }
-            return 1;
         }
         *out = Expr(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5782,29 +5838,31 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         *out = NULL;
         return 0;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_lineno, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from expr");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_int(tmp, &lineno, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from expr");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_col_offset, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from expr");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_int(tmp, &col_offset, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from expr");
-        }
-        return 1;
     }
     isinstance = PyObject_IsInstance(obj, (PyObject*)BoolOp_type);
     if (isinstance == -1) {
@@ -5814,20 +5872,27 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         boolop_ty op;
         asdl_seq* values;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_op);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_op, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BoolOp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_boolop(tmp, &op, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BoolOp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_values, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_values);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from BoolOp");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -5849,11 +5914,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(values, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from BoolOp");
-            }
-            return 1;
         }
         *out = BoolOp(op, values, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5868,41 +5928,44 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         operator_ty op;
         expr_ty right;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_left);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_left, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from BinOp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &left, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from BinOp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_op, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BinOp");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_op);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_operator(tmp, &op, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BinOp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_right, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"right\" missing from BinOp");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_right);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &right, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"right\" missing from BinOp");
-            }
-            return 1;
         }
         *out = BinOp(left, op, right, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5916,29 +5979,31 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         unaryop_ty op;
         expr_ty operand;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_op);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_op, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from UnaryOp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_unaryop(tmp, &op, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from UnaryOp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_operand, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"operand\" missing from UnaryOp");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_operand);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &operand, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"operand\" missing from UnaryOp");
-            }
-            return 1;
         }
         *out = UnaryOp(op, operand, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5952,29 +6017,31 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         arguments_ty args;
         expr_ty body;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_args);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_args, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Lambda");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_arguments(tmp, &args, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Lambda");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Lambda");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &body, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Lambda");
-            }
-            return 1;
         }
         *out = Lambda(args, body, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -5989,41 +6056,44 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         expr_ty body;
         expr_ty orelse;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_test);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_test, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from IfExp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &test, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from IfExp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from IfExp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &body, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from IfExp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_orelse, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from IfExp");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &orelse, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from IfExp");
-            }
-            return 1;
         }
         *out = IfExp(test, body, orelse, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6037,8 +6107,14 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         asdl_seq* keys;
         asdl_seq* values;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_keys);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_keys, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"keys\" missing from Dict");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6060,14 +6136,15 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(keys, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"keys\" missing from Dict");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_values, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from Dict");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_values);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6089,11 +6166,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(values, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from Dict");
-            }
-            return 1;
         }
         *out = Dict(keys, values, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6106,8 +6178,14 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* elts;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_elts);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_elts, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Set");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6129,11 +6207,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(elts, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Set");
-            }
-            return 1;
         }
         *out = Set(elts, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6147,20 +6220,27 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         expr_ty elt;
         asdl_seq* generators;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_elt);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_elt, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from ListComp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &elt, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from ListComp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_generators, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_generators);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from ListComp");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6182,11 +6262,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(generators, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from ListComp");
-            }
-            return 1;
         }
         *out = ListComp(elt, generators, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6200,20 +6275,27 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         expr_ty elt;
         asdl_seq* generators;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_elt);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_elt, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from SetComp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &elt, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from SetComp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_generators, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from SetComp");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_generators);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6235,11 +6317,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(generators, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from SetComp");
-            }
-            return 1;
         }
         *out = SetComp(elt, generators, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6254,32 +6331,40 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         expr_ty value;
         asdl_seq* generators;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_key);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_key, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"key\" missing from DictComp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &key, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"key\" missing from DictComp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from DictComp");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from DictComp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_generators, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_generators);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from DictComp");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6301,11 +6386,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(generators, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from DictComp");
-            }
-            return 1;
         }
         *out = DictComp(key, value, generators, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6319,20 +6399,27 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         expr_ty elt;
         asdl_seq* generators;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_elt);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_elt, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from GeneratorExp");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &elt, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from GeneratorExp");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_generators, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_generators);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from GeneratorExp");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6354,11 +6441,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(generators, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from GeneratorExp");
-            }
-            return 1;
         }
         *out = GeneratorExp(elt, generators, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6371,17 +6453,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Await");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Await");
-            }
-            return 1;
         }
         *out = Await(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6394,16 +6477,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty value;
 
-        tmp = get_not_none(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            value = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            value = NULL;
         }
         *out = Yield(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6416,17 +6501,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from YieldFrom");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from YieldFrom");
-            }
-            return 1;
         }
         *out = YieldFrom(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6441,20 +6527,27 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         asdl_int_seq* ops;
         asdl_seq* comparators;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_left);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_left, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from Compare");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &left, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from Compare");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ops, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ops\" missing from Compare");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ops);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6476,14 +6569,15 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(ops, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ops\" missing from Compare");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_comparators, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_comparators);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"comparators\" missing from Compare");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6505,11 +6599,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(comparators, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"comparators\" missing from Compare");
-            }
-            return 1;
         }
         *out = Compare(left, ops, comparators, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6524,20 +6613,27 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         asdl_seq* args;
         asdl_seq* keywords;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_func);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_func, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"func\" missing from Call");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &func, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"func\" missing from Call");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_args, &tmp) < 0) {
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_args);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Call");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6559,14 +6655,15 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(args, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Call");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_keywords, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from Call");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_keywords);
-        if (tmp != NULL) {
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6588,11 +6685,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(keywords, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from Call");
-            }
-            return 1;
         }
         *out = Call(func, args, keywords, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6605,17 +6697,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         object n;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_n);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_n, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"n\" missing from Num");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_object(tmp, &n, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"n\" missing from Num");
-            }
-            return 1;
         }
         *out = Num(n, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6628,17 +6721,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         string s;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_s);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_s, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Str");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_string(tmp, &s, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Str");
-            }
-            return 1;
         }
         *out = Str(s, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6653,39 +6747,44 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         int conversion;
         expr_ty format_spec;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from FormattedValue");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from FormattedValue");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_conversion, &tmp) < 0) {
             return 1;
         }
-        tmp = get_not_none(obj, &PyId_conversion);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            conversion = 0;
+        }
+        else {
             int res;
             res = obj2ast_int(tmp, &conversion, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_format_spec, &tmp) < 0) {
             return 1;
-        } else {
-            conversion = 0;
         }
-        tmp = get_not_none(obj, &PyId_format_spec);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            format_spec = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &format_spec, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            format_spec = NULL;
         }
         *out = FormattedValue(value, conversion, format_spec, lineno,
                               col_offset, arena);
@@ -6699,8 +6798,14 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* values;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_values);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_values, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from JoinedStr");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -6722,11 +6827,6 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(values, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from JoinedStr");
-            }
-            return 1;
         }
         *out = JoinedStr(values, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6739,17 +6839,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         bytes s;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_s);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_s, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Bytes");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_bytes(tmp, &s, arena);
             if (res != 0) goto failed;
-            Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Bytes");
-            }
-            return 1;
+            Py_CLEAR(tmp);
         }
         *out = Bytes(s, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6762,17 +6863,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         singleton value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from NameConstant");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_singleton(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from NameConstant");
-            }
-            return 1;
         }
         *out = NameConstant(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6795,17 +6897,18 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
     if (isinstance) {
         constant value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Constant");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_constant(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Constant");
-            }
-            return 1;
         }
         *out = Constant(value, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6820,41 +6923,44 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         identifier attr;
         expr_context_ty ctx;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Attribute");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Attribute");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_attr, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"attr\" missing from Attribute");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_attr);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_identifier(tmp, &attr, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"attr\" missing from Attribute");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ctx, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Attribute");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr_context(tmp, &ctx, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Attribute");
-            }
-            return 1;
         }
         *out = Attribute(value, attr, ctx, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6869,41 +6975,44 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         slice_ty slice;
         expr_context_ty ctx;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Subscript");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Subscript");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_slice, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"slice\" missing from Subscript");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_slice);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_slice(tmp, &slice, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"slice\" missing from Subscript");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ctx, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Subscript");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr_context(tmp, &ctx, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Subscript");
-            }
-            return 1;
         }
         *out = Subscript(value, slice, ctx, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6917,29 +7026,31 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         expr_ty value;
         expr_context_ty ctx;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Starred");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Starred");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ctx, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Starred");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr_context(tmp, &ctx, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Starred");
-            }
-            return 1;
         }
         *out = Starred(value, ctx, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6953,29 +7064,31 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         identifier id;
         expr_context_ty ctx;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_id);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_id, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"id\" missing from Name");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_identifier(tmp, &id, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"id\" missing from Name");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ctx, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Name");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr_context(tmp, &ctx, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Name");
-            }
-            return 1;
         }
         *out = Name(id, ctx, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -6989,8 +7102,14 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         asdl_seq* elts;
         expr_context_ty ctx;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_elts);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_elts, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from List");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -7012,23 +7131,19 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(elts, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from List");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ctx, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from List");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr_context(tmp, &ctx, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from List");
-            }
-            return 1;
         }
         *out = List(elts, ctx, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -7042,8 +7157,14 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
         asdl_seq* elts;
         expr_context_ty ctx;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_elts);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_elts, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Tuple");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -7065,23 +7186,19 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena)
                 asdl_seq_SET(elts, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Tuple");
-            }
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_ctx, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Tuple");
             return 1;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
-        if (tmp != NULL) {
+        else {
             int res;
             res = obj2ast_expr_context(tmp, &ctx, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Tuple");
-            }
-            return 1;
         }
         *out = Tuple(elts, ctx, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -7172,38 +7289,44 @@ obj2ast_slice(PyObject* obj, slice_ty* out, PyArena* arena)
         expr_ty upper;
         expr_ty step;
 
-        tmp = get_not_none(obj, &PyId_lower);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_lower, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            lower = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &lower, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_upper, &tmp) < 0) {
             return 1;
-        } else {
-            lower = NULL;
         }
-        tmp = get_not_none(obj, &PyId_upper);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            upper = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &upper, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_step, &tmp) < 0) {
             return 1;
-        } else {
-            upper = NULL;
         }
-        tmp = get_not_none(obj, &PyId_step);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            step = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &step, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
-            return 1;
-        } else {
-            step = NULL;
         }
         *out = Slice(lower, upper, step, arena);
         if (*out == NULL) goto failed;
@@ -7216,8 +7339,14 @@ obj2ast_slice(PyObject* obj, slice_ty* out, PyArena* arena)
     if (isinstance) {
         asdl_seq* dims;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_dims);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_dims, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"dims\" missing from ExtSlice");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -7239,11 +7368,6 @@ obj2ast_slice(PyObject* obj, slice_ty* out, PyArena* arena)
                 asdl_seq_SET(dims, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"dims\" missing from ExtSlice");
-            }
-            return 1;
         }
         *out = ExtSlice(dims, arena);
         if (*out == NULL) goto failed;
@@ -7256,17 +7380,18 @@ obj2ast_slice(PyObject* obj, slice_ty* out, PyArena* arena)
     if (isinstance) {
         expr_ty value;
 
-        tmp = _PyObject_GetAttrId(obj, &PyId_value);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Index");
+            return 1;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &value, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Index");
-            }
-            return 1;
         }
         *out = Index(value, arena);
         if (*out == NULL) goto failed;
@@ -7560,32 +7685,40 @@ obj2ast_comprehension(PyObject* obj, comprehension_ty* out, PyArena* arena)
     asdl_seq* ifs;
     int is_async;
 
-    tmp = _PyObject_GetAttrId(obj, &PyId_target);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_target, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from comprehension");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_expr(tmp, &target, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from comprehension");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_iter, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from comprehension");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_iter);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_expr(tmp, &iter, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from comprehension");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_ifs, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"ifs\" missing from comprehension");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_ifs);
-    if (tmp != NULL) {
+    else {
         int res;
         Py_ssize_t len;
         Py_ssize_t i;
@@ -7607,23 +7740,19 @@ obj2ast_comprehension(PyObject* obj, comprehension_ty* out, PyArena* arena)
             asdl_seq_SET(ifs, i, val);
         }
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"ifs\" missing from comprehension");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_is_async, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"is_async\" missing from comprehension");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_is_async);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_int(tmp, &is_async, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"is_async\" missing from comprehension");
-        }
-        return 1;
     }
     *out = comprehension(target, iter, ifs, is_async, arena);
     return 0;
@@ -7645,29 +7774,31 @@ obj2ast_excepthandler(PyObject* obj, excepthandler_ty* out, PyArena* arena)
         *out = NULL;
         return 0;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_lineno, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from excepthandler");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_int(tmp, &lineno, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from excepthandler");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_col_offset, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from excepthandler");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_int(tmp, &col_offset, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from excepthandler");
-        }
-        return 1;
     }
     isinstance = PyObject_IsInstance(obj, (PyObject*)ExceptHandler_type);
     if (isinstance == -1) {
@@ -7678,30 +7809,40 @@ obj2ast_excepthandler(PyObject* obj, excepthandler_ty* out, PyArena* arena)
         identifier name;
         asdl_seq* body;
 
-        tmp = get_not_none(obj, &PyId_type);
-        if (tmp != NULL) {
+        if (_PyObject_LookupAttrId(obj, &PyId_type, &tmp) < 0) {
+            return 1;
+        }
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            type = NULL;
+        }
+        else {
             int res;
             res = obj2ast_expr(tmp, &type, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_name, &tmp) < 0) {
             return 1;
-        } else {
-            type = NULL;
         }
-        tmp = get_not_none(obj, &PyId_name);
-        if (tmp != NULL) {
+        if (tmp == NULL || tmp == Py_None) {
+            Py_CLEAR(tmp);
+            name = NULL;
+        }
+        else {
             int res;
             res = obj2ast_identifier(tmp, &name, arena);
             if (res != 0) goto failed;
             Py_CLEAR(tmp);
-        } else if (PyErr_Occurred()) {
+        }
+        if (_PyObject_LookupAttrId(obj, &PyId_body, &tmp) < 0) {
             return 1;
-        } else {
-            name = NULL;
         }
-        tmp = _PyObject_GetAttrId(obj, &PyId_body);
-        if (tmp != NULL) {
+        if (tmp == NULL) {
+            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ExceptHandler");
+            return 1;
+        }
+        else {
             int res;
             Py_ssize_t len;
             Py_ssize_t i;
@@ -7723,11 +7864,6 @@ obj2ast_excepthandler(PyObject* obj, excepthandler_ty* out, PyArena* arena)
                 asdl_seq_SET(body, i, val);
             }
             Py_CLEAR(tmp);
-        } else {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ExceptHandler");
-            }
-            return 1;
         }
         *out = ExceptHandler(type, name, body, lineno, col_offset, arena);
         if (*out == NULL) goto failed;
@@ -7751,8 +7887,14 @@ obj2ast_arguments(PyObject* obj, arguments_ty* out, PyArena* arena)
     arg_ty kwarg;
     asdl_seq* defaults;
 
-    tmp = _PyObject_GetAttrId(obj, &PyId_args);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_args, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from arguments");
+        return 1;
+    }
+    else {
         int res;
         Py_ssize_t len;
         Py_ssize_t i;
@@ -7774,25 +7916,28 @@ obj2ast_arguments(PyObject* obj, arguments_ty* out, PyArena* arena)
             asdl_seq_SET(args, i, val);
         }
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from arguments");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_vararg, &tmp) < 0) {
         return 1;
     }
-    tmp = get_not_none(obj, &PyId_vararg);
-    if (tmp != NULL) {
+    if (tmp == NULL || tmp == Py_None) {
+        Py_CLEAR(tmp);
+        vararg = NULL;
+    }
+    else {
         int res;
         res = obj2ast_arg(tmp, &vararg, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else if (PyErr_Occurred()) {
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_kwonlyargs, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"kwonlyargs\" missing from arguments");
         return 1;
-    } else {
-        vararg = NULL;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_kwonlyargs);
-    if (tmp != NULL) {
+    else {
         int res;
         Py_ssize_t len;
         Py_ssize_t i;
@@ -7814,14 +7959,15 @@ obj2ast_arguments(PyObject* obj, arguments_ty* out, PyArena* arena)
             asdl_seq_SET(kwonlyargs, i, val);
         }
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"kwonlyargs\" missing from arguments");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_kw_defaults, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"kw_defaults\" missing from arguments");
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_kw_defaults);
-    if (tmp != NULL) {
+    else {
         int res;
         Py_ssize_t len;
         Py_ssize_t i;
@@ -7843,25 +7989,28 @@ obj2ast_arguments(PyObject* obj, arguments_ty* out, PyArena* arena)
             asdl_seq_SET(kw_defaults, i, val);
         }
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"kw_defaults\" missing from arguments");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_kwarg, &tmp) < 0) {
         return 1;
     }
-    tmp = get_not_none(obj, &PyId_kwarg);
-    if (tmp != NULL) {
+    if (tmp == NULL || tmp == Py_None) {
+        Py_CLEAR(tmp);
+        kwarg = NULL;
+    }
+    else {
         int res;
         res = obj2ast_arg(tmp, &kwarg, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else if (PyErr_Occurred()) {
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_defaults, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"defaults\" missing from arguments");
         return 1;
-    } else {
-        kwarg = NULL;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_defaults);
-    if (tmp != NULL) {
+    else {
         int res;
         Py_ssize_t len;
         Py_ssize_t i;
@@ -7883,11 +8032,6 @@ obj2ast_arguments(PyObject* obj, arguments_ty* out, PyArena* arena)
             asdl_seq_SET(defaults, i, val);
         }
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"defaults\" missing from arguments");
-        }
-        return 1;
     }
     *out = arguments(args, vararg, kwonlyargs, kw_defaults, kwarg, defaults,
                      arena);
@@ -7906,52 +8050,57 @@ obj2ast_arg(PyObject* obj, arg_ty* out, PyArena* arena)
     int lineno;
     int col_offset;
 
-    tmp = _PyObject_GetAttrId(obj, &PyId_arg);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_arg, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"arg\" missing from arg");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_identifier(tmp, &arg, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"arg\" missing from arg");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_annotation, &tmp) < 0) {
         return 1;
     }
-    tmp = get_not_none(obj, &PyId_annotation);
-    if (tmp != NULL) {
+    if (tmp == NULL || tmp == Py_None) {
+        Py_CLEAR(tmp);
+        annotation = NULL;
+    }
+    else {
         int res;
         res = obj2ast_expr(tmp, &annotation, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else if (PyErr_Occurred()) {
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_lineno, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from arg");
         return 1;
-    } else {
-        annotation = NULL;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_int(tmp, &lineno, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from arg");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_col_offset, &tmp) < 0) {
         return 1;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
-    if (tmp != NULL) {
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from arg");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_int(tmp, &col_offset, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from arg");
-        }
-        return 1;
     }
     *out = arg(arg, annotation, lineno, col_offset, arena);
     return 0;
@@ -7967,28 +8116,31 @@ obj2ast_keyword(PyObject* obj, keyword_ty* out, PyArena* arena)
     identifier arg;
     expr_ty value;
 
-    tmp = get_not_none(obj, &PyId_arg);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_arg, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL || tmp == Py_None) {
+        Py_CLEAR(tmp);
+        arg = NULL;
+    }
+    else {
         int res;
         res = obj2ast_identifier(tmp, &arg, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else if (PyErr_Occurred()) {
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_value, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from keyword");
         return 1;
-    } else {
-        arg = NULL;
     }
-    tmp = _PyObject_GetAttrId(obj, &PyId_value);
-    if (tmp != NULL) {
+    else {
         int res;
         res = obj2ast_expr(tmp, &value, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from keyword");
-        }
-        return 1;
     }
     *out = keyword(arg, value, arena);
     return 0;
@@ -8004,28 +8156,31 @@ obj2ast_alias(PyObject* obj, alias_ty* out, PyArena* arena)
     identifier name;
     identifier asname;
 
-    tmp = _PyObject_GetAttrId(obj, &PyId_name);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_name, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from alias");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_identifier(tmp, &name, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from alias");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_asname, &tmp) < 0) {
         return 1;
     }
-    tmp = get_not_none(obj, &PyId_asname);
-    if (tmp != NULL) {
+    if (tmp == NULL || tmp == Py_None) {
+        Py_CLEAR(tmp);
+        asname = NULL;
+    }
+    else {
         int res;
         res = obj2ast_identifier(tmp, &asname, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else if (PyErr_Occurred()) {
-        return 1;
-    } else {
-        asname = NULL;
     }
     *out = alias(name, asname, arena);
     return 0;
@@ -8041,28 +8196,31 @@ obj2ast_withitem(PyObject* obj, withitem_ty* out, PyArena* arena)
     expr_ty context_expr;
     expr_ty optional_vars;
 
-    tmp = _PyObject_GetAttrId(obj, &PyId_context_expr);
-    if (tmp != NULL) {
+    if (_PyObject_LookupAttrId(obj, &PyId_context_expr, &tmp) < 0) {
+        return 1;
+    }
+    if (tmp == NULL) {
+        PyErr_SetString(PyExc_TypeError, "required field \"context_expr\" missing from withitem");
+        return 1;
+    }
+    else {
         int res;
         res = obj2ast_expr(tmp, &context_expr, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_SetString(PyExc_TypeError, "required field \"context_expr\" missing from withitem");
-        }
+    }
+    if (_PyObject_LookupAttrId(obj, &PyId_optional_vars, &tmp) < 0) {
         return 1;
     }
-    tmp = get_not_none(obj, &PyId_optional_vars);
-    if (tmp != NULL) {
+    if (tmp == NULL || tmp == Py_None) {
+        Py_CLEAR(tmp);
+        optional_vars = NULL;
+    }
+    else {
         int res;
         res = obj2ast_expr(tmp, &optional_vars, arena);
         if (res != 0) goto failed;
         Py_CLEAR(tmp);
-    } else if (PyErr_Occurred()) {
-        return 1;
-    } else {
-        optional_vars = NULL;
     }
     *out = withitem(context_expr, optional_vars, arena);
     return 0;
diff --git a/Python/_warnings.c b/Python/_warnings.c
index 20ba7a16616..16ae932450c 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -80,11 +80,8 @@ get_warnings_attr(_Py_Identifier *attr_id, int try_import)
             return NULL;
     }
 
-    obj = _PyObject_GetAttrId(warnings_module, attr_id);
+    (void)_PyObject_LookupAttrId(warnings_module, attr_id, &obj);
     Py_DECREF(warnings_module);
-    if (obj == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
-        PyErr_Clear();
-    }
     return obj;
 }
 
@@ -893,13 +890,10 @@ get_source_line(PyObject *module_globals, int lineno)
     Py_INCREF(module_name);
 
     /* Make sure the loader implements the optional get_source() method. */
-    get_source = _PyObject_GetAttrId(loader, &PyId_get_source);
+    (void)_PyObject_LookupAttrId(loader, &PyId_get_source, &get_source);
     Py_DECREF(loader);
     if (!get_source) {
         Py_DECREF(module_name);
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-        }
         return NULL;
     }
     /* Call get_source() to get the source code. */
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
index 7fc2261ec64..d8971bef2ae 100644
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -71,12 +71,10 @@ update_bases(PyObject *bases, PyObject *const *args, Py_ssize_t nargs)
             }
             continue;
         }
-        meth = _PyObject_GetAttrId(base, &PyId___mro_entries__);
+        if (_PyObject_LookupAttrId(base, &PyId___mro_entries__, &meth) < 0) {
+            goto error;
+        }
         if (!meth) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                goto error;
-            }
-            PyErr_Clear();
             if (new_bases) {
                 if (PyList_Append(new_bases, base) < 0) {
                     goto error;
@@ -218,18 +216,11 @@ builtin___build_class__(PyObject *self, PyObject *const *args, Py_ssize_t nargs,
     }
     /* else: meta is not a class, so we cannot do the metaclass
        calculation, so we will use the explicitly given object as it is */
-    prep = _PyObject_GetAttrId(meta, &PyId___prepare__);
-    if (prep == NULL) {
-        if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-            PyErr_Clear();
-            ns = PyDict_New();
-        }
-        else {
-            Py_DECREF(meta);
-            Py_XDECREF(mkw);
-            Py_DECREF(bases);
-            return NULL;
-        }
+    if (_PyObject_LookupAttrId(meta, &PyId___prepare__, &prep) < 0) {
+        ns = NULL;
+    }
+    else if (prep == NULL) {
+        ns = PyDict_New();
     }
     else {
         PyObject *pargs[2] = {name, bases};
@@ -1127,8 +1118,7 @@ builtin_getattr(PyObject *self, PyObject *const *args, Py_ssize_t nargs)
         return NULL;
     }
     if (dflt != NULL) {
-        result = _PyObject_GetAttrWithoutError(v, name);
-        if (result == NULL && !PyErr_Occurred()) {
+        if (_PyObject_LookupAttr(v, name, &result) == 0) {
             Py_INCREF(dflt);
             return dflt;
         }
@@ -1191,13 +1181,12 @@ builtin_hasattr_impl(PyObject *module, PyObject *obj, PyObject *name)
                         "hasattr(): attribute name must be string");
         return NULL;
     }
-    v = _PyObject_GetAttrWithoutError(obj, name);
-    if (v == NULL) {
-        if (!PyErr_Occurred()) {
-            Py_RETURN_FALSE;
-        }
+    if (_PyObject_LookupAttr(obj, name, &v) < 0) {
         return NULL;
     }
+    if (v == NULL) {
+        Py_RETURN_FALSE;
+    }
     Py_DECREF(v);
     Py_RETURN_TRUE;
 }
diff --git a/Python/ceval.c b/Python/ceval.c
index 2b7c0c80242..3d7bed0111b 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -4813,13 +4813,12 @@ import_from(PyObject *v, PyObject *name)
     _Py_IDENTIFIER(__name__);
     PyObject *fullmodname, *pkgname, *pkgpath, *pkgname_or_unknown, *errmsg;
 
-    x = PyObject_GetAttr(v, name);
-    if (x != NULL || !PyErr_ExceptionMatches(PyExc_AttributeError))
+    if (_PyObject_LookupAttr(v, name, &x) != 0) {
         return x;
+    }
     /* Issue #17636: in case this failed because of a circular relative
        import, try to fallback on reading the module directly from
        sys.modules. */
-    PyErr_Clear();
     pkgname = _PyObject_GetAttrId(v, &PyId___name__);
     if (pkgname == NULL) {
         goto error;
@@ -4881,21 +4880,20 @@ import_all_from(PyObject *locals, PyObject *v)
 {
     _Py_IDENTIFIER(__all__);
     _Py_IDENTIFIER(__dict__);
-    PyObject *all = _PyObject_GetAttrId(v, &PyId___all__);
-    PyObject *dict, *name, *value;
+    PyObject *all, *dict, *name, *value;
     int skip_leading_underscores = 0;
     int pos, err;
 
+    if (_PyObject_LookupAttrId(v, &PyId___all__, &all) < 0) {
+        return -1; /* Unexpected error */
+    }
     if (all == NULL) {
-        if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-            return -1; /* Unexpected error */
-        PyErr_Clear();
-        dict = _PyObject_GetAttrId(v, &PyId___dict__);
+        if (_PyObject_LookupAttrId(v, &PyId___dict__, &dict) < 0) {
+            return -1;
+        }
         if (dict == NULL) {
-            if (!PyErr_ExceptionMatches(PyExc_AttributeError))
-                return -1;
             PyErr_SetString(PyExc_ImportError,
-            "from-import-* object has no __dict__ and no __all__");
+                    "from-import-* object has no __dict__ and no __all__");
             return -1;
         }
         all = PyMapping_Keys(dict);
diff --git a/Python/codecs.c b/Python/codecs.c
index 18edfbdab94..223ccca603f 100644
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -540,15 +540,11 @@ PyObject * _PyCodec_LookupTextEncoding(const char *encoding,
      * attribute.
      */
     if (!PyTuple_CheckExact(codec)) {
-        attr = _PyObject_GetAttrId(codec, &PyId__is_text_encoding);
-        if (attr == NULL) {
-            if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
-                PyErr_Clear();
-            } else {
-                Py_DECREF(codec);
-                return NULL;
-            }
-        } else {
+        if (_PyObject_LookupAttrId(codec, &PyId__is_text_encoding, &attr) < 0) {
+            Py_DECREF(codec);
+            return NULL;
+        }
+        if (attr != NULL) {
             is_text_codec = PyObject_IsTrue(attr);
             Py_DECREF(attr);
             if (is_text_codec <= 0) {


From solipsis at pitrou.net  Thu Jan 25 04:16:18 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 25 Jan 2018 09:16:18 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=4
Message-ID: <20180125091618.1.304EAB15ED13FF31@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogijTKf4', '--timeout', '7200']

From webhook-mailer at python.org  Thu Jan 25 05:53:02 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Thu, 25 Jan 2018 10:53:02 -0000
Subject: [Python-checkins] bpo-32360: Remove OrderedDict usage from
 json.tool (GH-5315)
Message-ID: <mailman.46.1516877584.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/2812d3d99287c50bab99625d7240bcf1c2e32369
commit: 2812d3d99287c50bab99625d7240bcf1c2e32369
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-25T19:52:58+09:00
summary:

bpo-32360: Remove OrderedDict usage from json.tool (GH-5315)

`object_pairs_hook=OrderedDict` idiom is deprecated.

files:
M Lib/json/tool.py

diff --git a/Lib/json/tool.py b/Lib/json/tool.py
index 4f3182c0c1e..5932f4ecded 100644
--- a/Lib/json/tool.py
+++ b/Lib/json/tool.py
@@ -11,7 +11,6 @@
 
 """
 import argparse
-import collections
 import json
 import sys
 
@@ -34,11 +33,7 @@ def main():
     sort_keys = options.sort_keys
     with infile:
         try:
-            if sort_keys:
-                obj = json.load(infile)
-            else:
-                obj = json.load(infile,
-                                object_pairs_hook=collections.OrderedDict)
+            obj = json.load(infile)
         except ValueError as e:
             raise SystemExit(e)
     with outfile:


From webhook-mailer at python.org  Thu Jan 25 08:58:10 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 25 Jan 2018 13:58:10 -0000
Subject: [Python-checkins] Cleanup dangling reference in
 get_timezone_utc_capi (#5317)
Message-ID: <mailman.47.1516888691.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/58dc03c737a71de93edef7723c9f6186116288ef
commit: 58dc03c737a71de93edef7723c9f6186116288ef
branch: master
author: Paul Ganssle <pganssle at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-25T14:58:07+01:00
summary:

Cleanup dangling reference in get_timezone_utc_capi (#5317)

files:
M Modules/_testcapimodule.c

diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 3f41134a345..2ad4322ecaa 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2300,8 +2300,10 @@ get_timezone_utc_capi(PyObject* self, PyObject *args) {
         return NULL;
     }
     if (macro) {
+        Py_INCREF(PyDateTime_TimeZone_UTC);
         return PyDateTime_TimeZone_UTC;
     } else {
+        Py_INCREF(PyDateTimeAPI->TimeZone_UTC);
         return PyDateTimeAPI->TimeZone_UTC;
     }
 }


From webhook-mailer at python.org  Thu Jan 25 12:54:45 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 25 Jan 2018 17:54:45 -0000
Subject: [Python-checkins] Fix 3rd level node's binary representation in
 HAMT algorithm description (#5319)
Message-ID: <mailman.48.1516902886.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/93a6119f081aa6e95a3f22466966cdc8820e37f9
commit: 93a6119f081aa6e95a3f22466966cdc8820e37f9
branch: master
author: Dmitry Alimov <dvalimov at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-25T12:54:41-05:00
summary:

Fix 3rd level node's binary representation in HAMT algorithm description (#5319)

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index 81629e9bf1c..df3b1092cbe 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -56,7 +56,7 @@ tree structure:
                      +---+ -- +----+----+----+ -- +----+
                                       |
                      +---+ -- +----+----+----+ -- +----+
-  a 3rd level node   | 0 | .. | 04 | 05 | 06 | .. | 31 |   0b01011 = 5  (3)
+  a 3rd level node   | 0 | .. | 04 | 05 | 06 | .. | 31 |   0b00101 = 5  (3)
                      +---+ -- +----+----+----+ -- +----+
                                       |
                      +---+ -- +----+----+----+----+


From webhook-mailer at python.org  Thu Jan 25 13:06:08 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 25 Jan 2018 18:06:08 -0000
Subject: [Python-checkins] bpo-32667: Fix tests when $PATH contains a file
 (#5322)
Message-ID: <mailman.49.1516903570.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/b31206a223955d614d7769f95fb979d60f77bf87
commit: b31206a223955d614d7769f95fb979d60f77bf87
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-25T19:06:05+01:00
summary:

bpo-32667: Fix tests when $PATH contains a file (#5322)

Some tests failed when the PATH environment variable contained a path
to an existing file. Fix tests to ignore also NotADirectoryError, not
only FileNotFoundError and PermissionError.

files:
M Lib/test/test_dtrace.py
M Lib/test/test_subprocess.py

diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py
index 47a501018a3..8612e276d76 100644
--- a/Lib/test/test_dtrace.py
+++ b/Lib/test/test_dtrace.py
@@ -79,7 +79,7 @@ def assert_usable(self):
         try:
             output = self.trace(abspath("assert_usable" + self.EXTENSION))
             output = output.strip()
-        except (FileNotFoundError, PermissionError) as fnfe:
+        except (FileNotFoundError, NotADirectoryError, PermissionError) as fnfe:
             output = str(fnfe)
         if output != "probe: success":
             raise unittest.SkipTest(
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 540ad34d3f5..2e2721e5e99 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -46,6 +46,8 @@
     SETBINARY = ''
 
 NONEXISTING_CMD = ('nonexisting_i_hope',)
+# Ignore errors that indicate the command was not found
+NONEXISTING_ERRORS = (FileNotFoundError, NotADirectoryError, PermissionError)
 
 
 class BaseTestCase(unittest.TestCase):
@@ -306,9 +308,9 @@ def test_executable_takes_precedence(self):
         # Verify first that the call succeeds without the executable arg.
         pre_args = [sys.executable, "-c"]
         self._assert_python(pre_args)
-        self.assertRaises((FileNotFoundError, PermissionError),
+        self.assertRaises(NONEXISTING_ERRORS,
                           self._assert_python, pre_args,
-                          executable="doesnotexist")
+                          executable=NONEXISTING_CMD[0])
 
     @unittest.skipIf(mswindows, "executable argument replaces shell")
     def test_executable_replaces_shell(self):
@@ -1146,13 +1148,10 @@ def test_leaking_fds_on_error(self):
         # value for that limit, but Windows has 2048, so we loop
         # 1024 times (each call leaked two fds).
         for i in range(1024):
-            with self.assertRaises(OSError) as c:
+            with self.assertRaises(NONEXISTING_ERRORS):
                 subprocess.Popen(NONEXISTING_CMD,
                                  stdout=subprocess.PIPE,
                                  stderr=subprocess.PIPE)
-            # ignore errors that indicate the command was not found
-            if c.exception.errno not in (errno.ENOENT, errno.EACCES):
-                raise c.exception
 
     def test_nonexisting_with_pipes(self):
         # bpo-30121: Popen with pipes must close properly pipes on error.
@@ -2533,7 +2532,7 @@ def test_leak_fast_process_del_killed(self):
         # let some time for the process to exit, and create a new Popen: this
         # should trigger the wait() of p
         time.sleep(0.2)
-        with self.assertRaises(OSError) as c:
+        with self.assertRaises(OSError):
             with subprocess.Popen(NONEXISTING_CMD,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE) as proc:
@@ -3044,7 +3043,7 @@ def test_communicate_stdin(self):
             self.assertEqual(proc.returncode, 1)
 
     def test_invalid_args(self):
-        with self.assertRaises((FileNotFoundError, PermissionError)) as c:
+        with self.assertRaises(NONEXISTING_ERRORS):
             with subprocess.Popen(NONEXISTING_CMD,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE) as proc:


From webhook-mailer at python.org  Thu Jan 25 14:18:58 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 25 Jan 2018 19:18:58 -0000
Subject: [Python-checkins] bpo-32436: Fix a refleak; var GC tracking;
 a GCC warning (#5326)
Message-ID: <mailman.50.1516907940.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6ab62920c87930dedc31fe633ecda3e51d3d7503
commit: 6ab62920c87930dedc31fe633ecda3e51d3d7503
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-25T14:18:55-05:00
summary:

bpo-32436: Fix a refleak; var GC tracking; a GCC warning (#5326)

The refleak in question wasn't really important, as context vars
are usually created at the toplevel and live as long as the interpreter
lives, so the context var name isn't ever GCed anyways.

files:
M Python/context.c
M Python/hamt.c

diff --git a/Python/context.c b/Python/context.c
index 5439531cd1e..d90ed25c4d4 100644
--- a/Python/context.c
+++ b/Python/context.c
@@ -134,7 +134,9 @@ PyContextVar_New(const char *name, PyObject *def)
     if (pyname == NULL) {
         return NULL;
     }
-    return contextvar_new(pyname, def);
+    PyContextVar *var = contextvar_new(pyname, def);
+    Py_DECREF(pyname);
+    return var;
 }
 
 
@@ -741,8 +743,8 @@ contextvar_new(PyObject *name, PyObject *def)
     var->var_cached_tsid = 0;
     var->var_cached_tsver = 0;
 
-    if (_PyObject_GC_IS_TRACKED(name) ||
-            (def != NULL && _PyObject_GC_IS_TRACKED(def)))
+    if (_PyObject_GC_MAY_BE_TRACKED(name) ||
+            (def != NULL && _PyObject_GC_MAY_BE_TRACKED(def)))
     {
         PyObject_GC_Track(var);
     }
diff --git a/Python/hamt.c b/Python/hamt.c
index df3b1092cbe..79c42c0f433 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -449,7 +449,7 @@ hamt_bitcount(uint32_t i)
     */
     i = i - ((i >> 1) & 0x55555555);
     i = (i & 0x33333333) + ((i >> 2) & 0x33333333);
-    return ((i + (i >> 4) & 0xF0F0F0F) * 0x1010101) >> 24;
+    return (((i + (i >> 4)) & 0xF0F0F0F) * 0x1010101) >> 24;
 }
 
 static inline uint32_t


From webhook-mailer at python.org  Thu Jan 25 15:51:27 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 25 Jan 2018 20:51:27 -0000
Subject: [Python-checkins] bpo-29302: Implement contextlib.AsyncExitStack.
 (#4790)
Message-ID: <mailman.52.1516913490.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/1aa094f74039cd20fdc7df56c68f6848c18ce4dd
commit: 1aa094f74039cd20fdc7df56c68f6848c18ce4dd
branch: master
author: Ilya Kulakov <kulakov.ilya at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-25T15:51:18-05:00
summary:

bpo-29302: Implement contextlib.AsyncExitStack. (#4790)

files:
A Misc/NEWS.d/next/Library/2017-12-11-15-14-55.bpo-29302.Nczj9l.rst
M Doc/library/contextlib.rst
M Doc/whatsnew/3.7.rst
M Lib/contextlib.py
M Lib/test/test_contextlib.py
M Lib/test/test_contextlib_async.py

diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst
index faa6c8ac23b..54d3a8e4296 100644
--- a/Doc/library/contextlib.rst
+++ b/Doc/library/contextlib.rst
@@ -435,6 +435,44 @@ Functions and classes provided:
       callbacks registered, the arguments passed in will indicate that no
       exception occurred.
 
+.. class:: AsyncExitStack()
+
+   An :ref:`asynchronous context manager <async-context-managers>`, similar
+   to :class:`ExitStack`, that supports combining both synchronous and
+   asynchronous context managers, as well as having coroutines for
+   cleanup logic.
+
+   The :meth:`close` method is not implemented, :meth:`aclose` must be used
+   instead.
+
+   .. method:: enter_async_context(cm)
+
+      Similar to :meth:`enter_context` but expects an asynchronous context
+      manager.
+
+   .. method:: push_async_exit(exit)
+
+      Similar to :meth:`push` but expects either an asynchronous context manager
+      or a coroutine.
+
+   .. method:: push_async_callback(callback, *args, **kwds)
+
+      Similar to :meth:`callback` but expects a coroutine.
+
+   .. method:: aclose()
+
+      Similar to :meth:`close` but properly handles awaitables.
+
+   Continuing the example for :func:`asynccontextmanager`::
+
+      async with AsyncExitStack() as stack:
+          connections = [await stack.enter_async_context(get_connection())
+              for i in range(5)]
+          # All opened connections will automatically be released at the end of
+          # the async with statement, even if attempts to open a connection
+          # later in the list raise an exception.
+
+   .. versionadded:: 3.7
 
 Examples and Recipes
 --------------------
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index e52e32c4736..e9f057029eb 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -379,6 +379,9 @@ contextlib
 :class:`~contextlib.AbstractAsyncContextManager` have been added. (Contributed
 by Jelle Zijlstra in :issue:`29679` and :issue:`30241`.)
 
+:class:`contextlib.AsyncExitStack` has been added. (Contributed by
+Alexander Mohr and Ilya Kulakov in :issue:`29302`.)
+
 cProfile
 --------
 
diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 96c8c22084a..ef8f8c9f55b 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -7,7 +7,7 @@
 
 __all__ = ["asynccontextmanager", "contextmanager", "closing", "nullcontext",
            "AbstractContextManager", "AbstractAsyncContextManager",
-           "ContextDecorator", "ExitStack",
+           "AsyncExitStack", "ContextDecorator", "ExitStack",
            "redirect_stdout", "redirect_stderr", "suppress"]
 
 
@@ -365,85 +365,102 @@ def __exit__(self, exctype, excinst, exctb):
         return exctype is not None and issubclass(exctype, self._exceptions)
 
 
-# Inspired by discussions on http://bugs.python.org/issue13585
-class ExitStack(AbstractContextManager):
-    """Context manager for dynamic management of a stack of exit callbacks
+class _BaseExitStack:
+    """A base class for ExitStack and AsyncExitStack."""
 
-    For example:
+    @staticmethod
+    def _create_exit_wrapper(cm, cm_exit):
+        def _exit_wrapper(exc_type, exc, tb):
+            return cm_exit(cm, exc_type, exc, tb)
+        return _exit_wrapper
 
-        with ExitStack() as stack:
-            files = [stack.enter_context(open(fname)) for fname in filenames]
-            # All opened files will automatically be closed at the end of
-            # the with statement, even if attempts to open files later
-            # in the list raise an exception
+    @staticmethod
+    def _create_cb_wrapper(callback, *args, **kwds):
+        def _exit_wrapper(exc_type, exc, tb):
+            callback(*args, **kwds)
+        return _exit_wrapper
 
-    """
     def __init__(self):
         self._exit_callbacks = deque()
 
     def pop_all(self):
-        """Preserve the context stack by transferring it to a new instance"""
+        """Preserve the context stack by transferring it to a new instance."""
         new_stack = type(self)()
         new_stack._exit_callbacks = self._exit_callbacks
         self._exit_callbacks = deque()
         return new_stack
 
-    def _push_cm_exit(self, cm, cm_exit):
-        """Helper to correctly register callbacks to __exit__ methods"""
-        def _exit_wrapper(*exc_details):
-            return cm_exit(cm, *exc_details)
-        _exit_wrapper.__self__ = cm
-        self.push(_exit_wrapper)
-
     def push(self, exit):
-        """Registers a callback with the standard __exit__ method signature
-
-        Can suppress exceptions the same way __exit__ methods can.
+        """Registers a callback with the standard __exit__ method signature.
 
+        Can suppress exceptions the same way __exit__ method can.
         Also accepts any object with an __exit__ method (registering a call
-        to the method instead of the object itself)
+        to the method instead of the object itself).
         """
         # We use an unbound method rather than a bound method to follow
-        # the standard lookup behaviour for special methods
+        # the standard lookup behaviour for special methods.
         _cb_type = type(exit)
+
         try:
             exit_method = _cb_type.__exit__
         except AttributeError:
-            # Not a context manager, so assume its a callable
-            self._exit_callbacks.append(exit)
+            # Not a context manager, so assume it's a callable.
+            self._push_exit_callback(exit)
         else:
             self._push_cm_exit(exit, exit_method)
-        return exit # Allow use as a decorator
-
-    def callback(self, callback, *args, **kwds):
-        """Registers an arbitrary callback and arguments.
-
-        Cannot suppress exceptions.
-        """
-        def _exit_wrapper(exc_type, exc, tb):
-            callback(*args, **kwds)
-        # We changed the signature, so using @wraps is not appropriate, but
-        # setting __wrapped__ may still help with introspection
-        _exit_wrapper.__wrapped__ = callback
-        self.push(_exit_wrapper)
-        return callback # Allow use as a decorator
+        return exit  # Allow use as a decorator.
 
     def enter_context(self, cm):
-        """Enters the supplied context manager
+        """Enters the supplied context manager.
 
         If successful, also pushes its __exit__ method as a callback and
         returns the result of the __enter__ method.
         """
-        # We look up the special methods on the type to match the with statement
+        # We look up the special methods on the type to match the with
+        # statement.
         _cm_type = type(cm)
         _exit = _cm_type.__exit__
         result = _cm_type.__enter__(cm)
         self._push_cm_exit(cm, _exit)
         return result
 
-    def close(self):
-        """Immediately unwind the context stack"""
-        self.__exit__(None, None, None)
+    def callback(self, callback, *args, **kwds):
+        """Registers an arbitrary callback and arguments.
+
+        Cannot suppress exceptions.
+        """
+        _exit_wrapper = self._create_cb_wrapper(callback, *args, **kwds)
+
+        # We changed the signature, so using @wraps is not appropriate, but
+        # setting __wrapped__ may still help with introspection.
+        _exit_wrapper.__wrapped__ = callback
+        self._push_exit_callback(_exit_wrapper)
+        return callback  # Allow use as a decorator
+
+    def _push_cm_exit(self, cm, cm_exit):
+        """Helper to correctly register callbacks to __exit__ methods."""
+        _exit_wrapper = self._create_exit_wrapper(cm, cm_exit)
+        _exit_wrapper.__self__ = cm
+        self._push_exit_callback(_exit_wrapper, True)
+
+    def _push_exit_callback(self, callback, is_sync=True):
+        self._exit_callbacks.append((is_sync, callback))
+
+
+# Inspired by discussions on http://bugs.python.org/issue13585
+class ExitStack(_BaseExitStack, AbstractContextManager):
+    """Context manager for dynamic management of a stack of exit callbacks.
+
+    For example:
+        with ExitStack() as stack:
+            files = [stack.enter_context(open(fname)) for fname in filenames]
+            # All opened files will automatically be closed at the end of
+            # the with statement, even if attempts to open files later
+            # in the list raise an exception.
+    """
+
+    def __enter__(self):
+        return self
 
     def __exit__(self, *exc_details):
         received_exc = exc_details[0] is not None
@@ -470,7 +487,8 @@ def _fix_exception_context(new_exc, old_exc):
         suppressed_exc = False
         pending_raise = False
         while self._exit_callbacks:
-            cb = self._exit_callbacks.pop()
+            is_sync, cb = self._exit_callbacks.pop()
+            assert is_sync
             try:
                 if cb(*exc_details):
                     suppressed_exc = True
@@ -493,6 +511,147 @@ def _fix_exception_context(new_exc, old_exc):
                 raise
         return received_exc and suppressed_exc
 
+    def close(self):
+        """Immediately unwind the context stack."""
+        self.__exit__(None, None, None)
+
+
+# Inspired by discussions on https://bugs.python.org/issue29302
+class AsyncExitStack(_BaseExitStack, AbstractAsyncContextManager):
+    """Async context manager for dynamic management of a stack of exit
+    callbacks.
+
+    For example:
+        async with AsyncExitStack() as stack:
+            connections = [await stack.enter_async_context(get_connection())
+                for i in range(5)]
+            # All opened connections will automatically be released at the
+            # end of the async with statement, even if attempts to open a
+            # connection later in the list raise an exception.
+    """
+
+    @staticmethod
+    def _create_async_exit_wrapper(cm, cm_exit):
+        async def _exit_wrapper(exc_type, exc, tb):
+            return await cm_exit(cm, exc_type, exc, tb)
+        return _exit_wrapper
+
+    @staticmethod
+    def _create_async_cb_wrapper(callback, *args, **kwds):
+        async def _exit_wrapper(exc_type, exc, tb):
+            await callback(*args, **kwds)
+        return _exit_wrapper
+
+    async def enter_async_context(self, cm):
+        """Enters the supplied async context manager.
+
+        If successful, also pushes its __aexit__ method as a callback and
+        returns the result of the __aenter__ method.
+        """
+        _cm_type = type(cm)
+        _exit = _cm_type.__aexit__
+        result = await _cm_type.__aenter__(cm)
+        self._push_async_cm_exit(cm, _exit)
+        return result
+
+    def push_async_exit(self, exit):
+        """Registers a coroutine function with the standard __aexit__ method
+        signature.
+
+        Can suppress exceptions the same way __aexit__ method can.
+        Also accepts any object with an __aexit__ method (registering a call
+        to the method instead of the object itself).
+        """
+        _cb_type = type(exit)
+        try:
+            exit_method = _cb_type.__aexit__
+        except AttributeError:
+            # Not an async context manager, so assume it's a coroutine function
+            self._push_exit_callback(exit, False)
+        else:
+            self._push_async_cm_exit(exit, exit_method)
+        return exit  # Allow use as a decorator
+
+    def push_async_callback(self, callback, *args, **kwds):
+        """Registers an arbitrary coroutine function and arguments.
+
+        Cannot suppress exceptions.
+        """
+        _exit_wrapper = self._create_async_cb_wrapper(callback, *args, **kwds)
+
+        # We changed the signature, so using @wraps is not appropriate, but
+        # setting __wrapped__ may still help with introspection.
+        _exit_wrapper.__wrapped__ = callback
+        self._push_exit_callback(_exit_wrapper, False)
+        return callback  # Allow use as a decorator
+
+    async def aclose(self):
+        """Immediately unwind the context stack."""
+        await self.__aexit__(None, None, None)
+
+    def _push_async_cm_exit(self, cm, cm_exit):
+        """Helper to correctly register coroutine function to __aexit__
+        method."""
+        _exit_wrapper = self._create_async_exit_wrapper(cm, cm_exit)
+        _exit_wrapper.__self__ = cm
+        self._push_exit_callback(_exit_wrapper, False)
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *exc_details):
+        received_exc = exc_details[0] is not None
+
+        # We manipulate the exception state so it behaves as though
+        # we were actually nesting multiple with statements
+        frame_exc = sys.exc_info()[1]
+        def _fix_exception_context(new_exc, old_exc):
+            # Context may not be correct, so find the end of the chain
+            while 1:
+                exc_context = new_exc.__context__
+                if exc_context is old_exc:
+                    # Context is already set correctly (see issue 20317)
+                    return
+                if exc_context is None or exc_context is frame_exc:
+                    break
+                new_exc = exc_context
+            # Change the end of the chain to point to the exception
+            # we expect it to reference
+            new_exc.__context__ = old_exc
+
+        # Callbacks are invoked in LIFO order to match the behaviour of
+        # nested context managers
+        suppressed_exc = False
+        pending_raise = False
+        while self._exit_callbacks:
+            is_sync, cb = self._exit_callbacks.pop()
+            try:
+                if is_sync:
+                    cb_suppress = cb(*exc_details)
+                else:
+                    cb_suppress = await cb(*exc_details)
+
+                if cb_suppress:
+                    suppressed_exc = True
+                    pending_raise = False
+                    exc_details = (None, None, None)
+            except:
+                new_exc_details = sys.exc_info()
+                # simulate the stack of exceptions by setting the context
+                _fix_exception_context(new_exc_details[1], exc_details[1])
+                pending_raise = True
+                exc_details = new_exc_details
+        if pending_raise:
+            try:
+                # bare "raise exc_details[1]" replaces our carefully
+                # set-up context
+                fixed_ctx = exc_details[1].__context__
+                raise exc_details[1]
+            except BaseException:
+                exc_details[1].__context__ = fixed_ctx
+                raise
+        return received_exc and suppressed_exc
+
 
 class nullcontext(AbstractContextManager):
     """Context manager that does no additional processing.
diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py
index 1a5e6edad9b..96ceaa7f060 100644
--- a/Lib/test/test_contextlib.py
+++ b/Lib/test/test_contextlib.py
@@ -1,5 +1,6 @@
 """Unit tests for contextlib.py, and other context managers."""
 
+import asyncio
 import io
 import sys
 import tempfile
@@ -505,17 +506,18 @@ def test(x):
         self.assertEqual(state, [1, 'something else', 999])
 
 
-class TestExitStack(unittest.TestCase):
+class TestBaseExitStack:
+    exit_stack = None
 
     @support.requires_docstrings
     def test_instance_docs(self):
         # Issue 19330: ensure context manager instances have good docstrings
-        cm_docstring = ExitStack.__doc__
-        obj = ExitStack()
+        cm_docstring = self.exit_stack.__doc__
+        obj = self.exit_stack()
         self.assertEqual(obj.__doc__, cm_docstring)
 
     def test_no_resources(self):
-        with ExitStack():
+        with self.exit_stack():
             pass
 
     def test_callback(self):
@@ -531,7 +533,7 @@ def test_callback(self):
         def _exit(*args, **kwds):
             """Test metadata propagation"""
             result.append((args, kwds))
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             for args, kwds in reversed(expected):
                 if args and kwds:
                     f = stack.callback(_exit, *args, **kwds)
@@ -543,9 +545,9 @@ def _exit(*args, **kwds):
                     f = stack.callback(_exit)
                 self.assertIs(f, _exit)
             for wrapper in stack._exit_callbacks:
-                self.assertIs(wrapper.__wrapped__, _exit)
-                self.assertNotEqual(wrapper.__name__, _exit.__name__)
-                self.assertIsNone(wrapper.__doc__, _exit.__doc__)
+                self.assertIs(wrapper[1].__wrapped__, _exit)
+                self.assertNotEqual(wrapper[1].__name__, _exit.__name__)
+                self.assertIsNone(wrapper[1].__doc__, _exit.__doc__)
         self.assertEqual(result, expected)
 
     def test_push(self):
@@ -565,21 +567,21 @@ def __enter__(self):
                 self.fail("Should not be called!")
             def __exit__(self, *exc_details):
                 self.check_exc(*exc_details)
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             stack.push(_expect_ok)
-            self.assertIs(stack._exit_callbacks[-1], _expect_ok)
+            self.assertIs(stack._exit_callbacks[-1][1], _expect_ok)
             cm = ExitCM(_expect_ok)
             stack.push(cm)
-            self.assertIs(stack._exit_callbacks[-1].__self__, cm)
+            self.assertIs(stack._exit_callbacks[-1][1].__self__, cm)
             stack.push(_suppress_exc)
-            self.assertIs(stack._exit_callbacks[-1], _suppress_exc)
+            self.assertIs(stack._exit_callbacks[-1][1], _suppress_exc)
             cm = ExitCM(_expect_exc)
             stack.push(cm)
-            self.assertIs(stack._exit_callbacks[-1].__self__, cm)
+            self.assertIs(stack._exit_callbacks[-1][1].__self__, cm)
             stack.push(_expect_exc)
-            self.assertIs(stack._exit_callbacks[-1], _expect_exc)
+            self.assertIs(stack._exit_callbacks[-1][1], _expect_exc)
             stack.push(_expect_exc)
-            self.assertIs(stack._exit_callbacks[-1], _expect_exc)
+            self.assertIs(stack._exit_callbacks[-1][1], _expect_exc)
             1/0
 
     def test_enter_context(self):
@@ -591,19 +593,19 @@ def __exit__(self, *exc_details):
 
         result = []
         cm = TestCM()
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             @stack.callback  # Registered first => cleaned up last
             def _exit():
                 result.append(4)
             self.assertIsNotNone(_exit)
             stack.enter_context(cm)
-            self.assertIs(stack._exit_callbacks[-1].__self__, cm)
+            self.assertIs(stack._exit_callbacks[-1][1].__self__, cm)
             result.append(2)
         self.assertEqual(result, [1, 2, 3, 4])
 
     def test_close(self):
         result = []
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             @stack.callback
             def _exit():
                 result.append(1)
@@ -614,7 +616,7 @@ def _exit():
 
     def test_pop_all(self):
         result = []
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             @stack.callback
             def _exit():
                 result.append(3)
@@ -627,12 +629,12 @@ def _exit():
 
     def test_exit_raise(self):
         with self.assertRaises(ZeroDivisionError):
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.push(lambda *exc: False)
                 1/0
 
     def test_exit_suppress(self):
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             stack.push(lambda *exc: True)
             1/0
 
@@ -696,7 +698,7 @@ def suppress_exc(*exc_details):
             return True
 
         try:
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.callback(raise_exc, IndexError)
                 stack.callback(raise_exc, KeyError)
                 stack.callback(raise_exc, AttributeError)
@@ -724,7 +726,7 @@ def suppress_exc(*exc_details):
             return True
 
         try:
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.callback(lambda: None)
                 stack.callback(raise_exc, IndexError)
         except Exception as exc:
@@ -733,7 +735,7 @@ def suppress_exc(*exc_details):
             self.fail("Expected IndexError, but no exception was raised")
 
         try:
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.callback(raise_exc, KeyError)
                 stack.push(suppress_exc)
                 stack.callback(raise_exc, IndexError)
@@ -760,7 +762,7 @@ def gets_the_context_right(exc):
         # fix, ExitStack would try to fix it *again* and get into an
         # infinite self-referential loop
         try:
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.enter_context(gets_the_context_right(exc4))
                 stack.enter_context(gets_the_context_right(exc3))
                 stack.enter_context(gets_the_context_right(exc2))
@@ -787,7 +789,7 @@ def raise_nested(inner_exc, outer_exc):
         exc4 = Exception(4)
         exc5 = Exception(5)
         try:
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.callback(raise_nested, exc4, exc5)
                 stack.callback(raise_nested, exc2, exc3)
                 raise exc1
@@ -801,27 +803,25 @@ def raise_nested(inner_exc, outer_exc):
             self.assertIsNone(
                 exc.__context__.__context__.__context__.__context__.__context__)
 
-
-
     def test_body_exception_suppress(self):
         def suppress_exc(*exc_details):
             return True
         try:
-            with ExitStack() as stack:
+            with self.exit_stack() as stack:
                 stack.push(suppress_exc)
                 1/0
         except IndexError as exc:
             self.fail("Expected no exception, got IndexError")
 
     def test_exit_exception_chaining_suppress(self):
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             stack.push(lambda *exc: True)
             stack.push(lambda *exc: 1/0)
             stack.push(lambda *exc: {}[1])
 
     def test_excessive_nesting(self):
         # The original implementation would die with RecursionError here
-        with ExitStack() as stack:
+        with self.exit_stack() as stack:
             for i in range(10000):
                 stack.callback(int)
 
@@ -829,10 +829,10 @@ def test_instance_bypass(self):
         class Example(object): pass
         cm = Example()
         cm.__exit__ = object()
-        stack = ExitStack()
+        stack = self.exit_stack()
         self.assertRaises(AttributeError, stack.enter_context, cm)
         stack.push(cm)
-        self.assertIs(stack._exit_callbacks[-1], cm)
+        self.assertIs(stack._exit_callbacks[-1][1], cm)
 
     def test_dont_reraise_RuntimeError(self):
         # https://bugs.python.org/issue27122
@@ -856,7 +856,7 @@ def first():
         # The UniqueRuntimeError should be caught by second()'s exception
         # handler which chain raised a new UniqueException.
         with self.assertRaises(UniqueException) as err_ctx:
-            with ExitStack() as es_ctx:
+            with self.exit_stack() as es_ctx:
                 es_ctx.enter_context(second())
                 es_ctx.enter_context(first())
                 raise UniqueRuntimeError("please no infinite loop.")
@@ -869,6 +869,10 @@ def first():
         self.assertIs(exc.__cause__, exc.__context__)
 
 
+class TestExitStack(TestBaseExitStack, unittest.TestCase):
+    exit_stack = ExitStack
+
+
 class TestRedirectStream:
 
     redirect_stream = None
diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py
index 447ca965122..879ddbe0e11 100644
--- a/Lib/test/test_contextlib_async.py
+++ b/Lib/test/test_contextlib_async.py
@@ -1,9 +1,11 @@
 import asyncio
-from contextlib import asynccontextmanager, AbstractAsyncContextManager
+from contextlib import asynccontextmanager, AbstractAsyncContextManager, AsyncExitStack
 import functools
 from test import support
 import unittest
 
+from .test_contextlib import TestBaseExitStack
+
 
 def _async_test(func):
     """Decorator to turn an async function into a test case."""
@@ -255,5 +257,168 @@ def test_contextmanager_doc_attrib(self):
             self.assertEqual(target, (11, 22, 33, 44))
 
 
+class TestAsyncExitStack(TestBaseExitStack, unittest.TestCase):
+    class SyncAsyncExitStack(AsyncExitStack):
+        @staticmethod
+        def run_coroutine(coro):
+            loop = asyncio.get_event_loop()
+
+            f = asyncio.ensure_future(coro)
+            f.add_done_callback(lambda f: loop.stop())
+            loop.run_forever()
+
+            exc = f.exception()
+
+            if not exc:
+                return f.result()
+            else:
+                context = exc.__context__
+
+                try:
+                    raise exc
+                except:
+                    exc.__context__ = context
+                    raise exc
+
+        def close(self):
+            return self.run_coroutine(self.aclose())
+
+        def __enter__(self):
+            return self.run_coroutine(self.__aenter__())
+
+        def __exit__(self, *exc_details):
+            return self.run_coroutine(self.__aexit__(*exc_details))
+
+    exit_stack = SyncAsyncExitStack
+
+    def setUp(self):
+        self.loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(self.loop)
+        self.addCleanup(self.loop.close)
+
+    @_async_test
+    async def test_async_callback(self):
+        expected = [
+            ((), {}),
+            ((1,), {}),
+            ((1,2), {}),
+            ((), dict(example=1)),
+            ((1,), dict(example=1)),
+            ((1,2), dict(example=1)),
+        ]
+        result = []
+        async def _exit(*args, **kwds):
+            """Test metadata propagation"""
+            result.append((args, kwds))
+
+        async with AsyncExitStack() as stack:
+            for args, kwds in reversed(expected):
+                if args and kwds:
+                    f = stack.push_async_callback(_exit, *args, **kwds)
+                elif args:
+                    f = stack.push_async_callback(_exit, *args)
+                elif kwds:
+                    f = stack.push_async_callback(_exit, **kwds)
+                else:
+                    f = stack.push_async_callback(_exit)
+                self.assertIs(f, _exit)
+            for wrapper in stack._exit_callbacks:
+                self.assertIs(wrapper[1].__wrapped__, _exit)
+                self.assertNotEqual(wrapper[1].__name__, _exit.__name__)
+                self.assertIsNone(wrapper[1].__doc__, _exit.__doc__)
+
+        self.assertEqual(result, expected)
+
+    @_async_test
+    async def test_async_push(self):
+        exc_raised = ZeroDivisionError
+        async def _expect_exc(exc_type, exc, exc_tb):
+            self.assertIs(exc_type, exc_raised)
+        async def _suppress_exc(*exc_details):
+            return True
+        async def _expect_ok(exc_type, exc, exc_tb):
+            self.assertIsNone(exc_type)
+            self.assertIsNone(exc)
+            self.assertIsNone(exc_tb)
+        class ExitCM(object):
+            def __init__(self, check_exc):
+                self.check_exc = check_exc
+            async def __aenter__(self):
+                self.fail("Should not be called!")
+            async def __aexit__(self, *exc_details):
+                await self.check_exc(*exc_details)
+
+        async with self.exit_stack() as stack:
+            stack.push_async_exit(_expect_ok)
+            self.assertIs(stack._exit_callbacks[-1][1], _expect_ok)
+            cm = ExitCM(_expect_ok)
+            stack.push_async_exit(cm)
+            self.assertIs(stack._exit_callbacks[-1][1].__self__, cm)
+            stack.push_async_exit(_suppress_exc)
+            self.assertIs(stack._exit_callbacks[-1][1], _suppress_exc)
+            cm = ExitCM(_expect_exc)
+            stack.push_async_exit(cm)
+            self.assertIs(stack._exit_callbacks[-1][1].__self__, cm)
+            stack.push_async_exit(_expect_exc)
+            self.assertIs(stack._exit_callbacks[-1][1], _expect_exc)
+            stack.push_async_exit(_expect_exc)
+            self.assertIs(stack._exit_callbacks[-1][1], _expect_exc)
+            1/0
+
+    @_async_test
+    async def test_async_enter_context(self):
+        class TestCM(object):
+            async def __aenter__(self):
+                result.append(1)
+            async def __aexit__(self, *exc_details):
+                result.append(3)
+
+        result = []
+        cm = TestCM()
+
+        async with AsyncExitStack() as stack:
+            @stack.push_async_callback  # Registered first => cleaned up last
+            async def _exit():
+                result.append(4)
+            self.assertIsNotNone(_exit)
+            await stack.enter_async_context(cm)
+            self.assertIs(stack._exit_callbacks[-1][1].__self__, cm)
+            result.append(2)
+
+        self.assertEqual(result, [1, 2, 3, 4])
+
+    @_async_test
+    async def test_async_exit_exception_chaining(self):
+        # Ensure exception chaining matches the reference behaviour
+        async def raise_exc(exc):
+            raise exc
+
+        saved_details = None
+        async def suppress_exc(*exc_details):
+            nonlocal saved_details
+            saved_details = exc_details
+            return True
+
+        try:
+            async with self.exit_stack() as stack:
+                stack.push_async_callback(raise_exc, IndexError)
+                stack.push_async_callback(raise_exc, KeyError)
+                stack.push_async_callback(raise_exc, AttributeError)
+                stack.push_async_exit(suppress_exc)
+                stack.push_async_callback(raise_exc, ValueError)
+                1 / 0
+        except IndexError as exc:
+            self.assertIsInstance(exc.__context__, KeyError)
+            self.assertIsInstance(exc.__context__.__context__, AttributeError)
+            # Inner exceptions were suppressed
+            self.assertIsNone(exc.__context__.__context__.__context__)
+        else:
+            self.fail("Expected IndexError, but no exception was raised")
+        # Check the inner exceptions
+        inner_exc = saved_details[1]
+        self.assertIsInstance(inner_exc, ValueError)
+        self.assertIsInstance(inner_exc.__context__, ZeroDivisionError)
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/Misc/NEWS.d/next/Library/2017-12-11-15-14-55.bpo-29302.Nczj9l.rst b/Misc/NEWS.d/next/Library/2017-12-11-15-14-55.bpo-29302.Nczj9l.rst
new file mode 100644
index 00000000000..0030e2ce367
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-11-15-14-55.bpo-29302.Nczj9l.rst
@@ -0,0 +1 @@
+Add contextlib.AsyncExitStack. Patch by Alexander Mohr and Ilya Kulakov.


From webhook-mailer at python.org  Thu Jan 25 16:39:44 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 25 Jan 2018 21:39:44 -0000
Subject: [Python-checkins] bpo-32667: Fix tests when $PATH contains a file
 (GH-5322) (#5323)
Message-ID: <mailman.53.1516916386.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/255dbd2102d5dec5ffbd0b94084377e98c3b56c4
commit: 255dbd2102d5dec5ffbd0b94084377e98c3b56c4
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-25T22:39:36+01:00
summary:

bpo-32667: Fix tests when $PATH contains a file (GH-5322) (#5323)

Some tests failed when the PATH environment variable contained a path
to an existing file. Fix tests to ignore also NotADirectoryError, not
only FileNotFoundError and PermissionError.
(cherry picked from commit b31206a223955d614d7769f95fb979d60f77bf87)

files:
M Lib/test/test_dtrace.py
M Lib/test/test_subprocess.py

diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py
index 47a501018a3..8612e276d76 100644
--- a/Lib/test/test_dtrace.py
+++ b/Lib/test/test_dtrace.py
@@ -79,7 +79,7 @@ def assert_usable(self):
         try:
             output = self.trace(abspath("assert_usable" + self.EXTENSION))
             output = output.strip()
-        except (FileNotFoundError, PermissionError) as fnfe:
+        except (FileNotFoundError, NotADirectoryError, PermissionError) as fnfe:
             output = str(fnfe)
         if output != "probe: success":
             raise unittest.SkipTest(
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 5016513decd..d7eb982dc57 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -50,6 +50,8 @@
     SETBINARY = ''
 
 NONEXISTING_CMD = ('nonexisting_i_hope',)
+# Ignore errors that indicate the command was not found
+NONEXISTING_ERRORS = (FileNotFoundError, NotADirectoryError, PermissionError)
 
 
 class BaseTestCase(unittest.TestCase):
@@ -310,9 +312,9 @@ def test_executable_takes_precedence(self):
         # Verify first that the call succeeds without the executable arg.
         pre_args = [sys.executable, "-c"]
         self._assert_python(pre_args)
-        self.assertRaises((FileNotFoundError, PermissionError),
+        self.assertRaises(NONEXISTING_ERRORS,
                           self._assert_python, pre_args,
-                          executable="doesnotexist")
+                          executable=NONEXISTING_CMD[0])
 
     @unittest.skipIf(mswindows, "executable argument replaces shell")
     def test_executable_replaces_shell(self):
@@ -1150,13 +1152,10 @@ def test_leaking_fds_on_error(self):
         # value for that limit, but Windows has 2048, so we loop
         # 1024 times (each call leaked two fds).
         for i in range(1024):
-            with self.assertRaises(OSError) as c:
+            with self.assertRaises(NONEXISTING_ERRORS):
                 subprocess.Popen(NONEXISTING_CMD,
                                  stdout=subprocess.PIPE,
                                  stderr=subprocess.PIPE)
-            # ignore errors that indicate the command was not found
-            if c.exception.errno not in (errno.ENOENT, errno.EACCES):
-                raise c.exception
 
     def test_nonexisting_with_pipes(self):
         # bpo-30121: Popen with pipes must close properly pipes on error.
@@ -2539,7 +2538,7 @@ def test_leak_fast_process_del_killed(self):
         # let some time for the process to exit, and create a new Popen: this
         # should trigger the wait() of p
         time.sleep(0.2)
-        with self.assertRaises(OSError) as c:
+        with self.assertRaises(OSError):
             with subprocess.Popen(NONEXISTING_CMD,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE) as proc:
@@ -2978,7 +2977,7 @@ def test_communicate_stdin(self):
             self.assertEqual(proc.returncode, 1)
 
     def test_invalid_args(self):
-        with self.assertRaises((FileNotFoundError, PermissionError)) as c:
+        with self.assertRaises(NONEXISTING_ERRORS):
             with subprocess.Popen(NONEXISTING_CMD,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE) as proc:


From webhook-mailer at python.org  Thu Jan 25 16:41:41 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 25 Jan 2018 21:41:41 -0000
Subject: [Python-checkins] bpo-32667: Fix tests when $PATH contains a file
 (#5324)
Message-ID: <mailman.54.1516916503.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/6996f284d4d90aa05c46d9fe6f38d1030454b224
commit: 6996f284d4d90aa05c46d9fe6f38d1030454b224
branch: 2.7
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-25T22:41:38+01:00
summary:

bpo-32667: Fix tests when $PATH contains a file (#5324)

test_subprocess.test_leaking_fds_on_error() failed when the PATH
environment variable contains a path to an existing file. Fix the
test: ignore also ENOTDIR, not only ENOENT and EACCES.

files:
M Lib/test/test_subprocess.py

diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index a5a727efd62..ee2383b8a93 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -697,7 +697,7 @@ def test_leaking_fds_on_error(self):
                                  stdout=subprocess.PIPE,
                                  stderr=subprocess.PIPE)
             # ignore errors that indicate the command was not found
-            if c.exception.errno not in (errno.ENOENT, errno.EACCES):
+            if c.exception.errno not in (errno.ENOENT, errno.ENOTDIR, errno.EACCES):
                 raise c.exception
 
     @unittest.skipIf(threading is None, "threading required")


From webhook-mailer at python.org  Thu Jan 25 18:08:12 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 25 Jan 2018 23:08:12 -0000
Subject: [Python-checkins] bpo-32662: Implement Server.start_serving() and
 Server.serve_forever() (#5312)
Message-ID: <mailman.55.1516921694.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/c9070d03f5169ad6e171e641b7fa8feab18bf229
commit: c9070d03f5169ad6e171e641b7fa8feab18bf229
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-25T18:08:09-05:00
summary:

bpo-32662: Implement Server.start_serving() and Server.serve_forever() (#5312)

* bpo-32662: Implement Server.start_serving() and Server.serve_forever()

New methods:

* Server.start_serving(),
* Server.serve_forever(), and
* Server.is_serving().

Add 'start_serving' keyword parameter to loop.create_server() and
loop.create_unix_server().

files:
A Lib/test/test_asyncio/test_server.py
A Misc/NEWS.d/next/Library/2018-01-25-01-45-30.bpo-32662.oabhd8.rst
M Doc/library/asyncio-eventloop.rst
M Lib/asyncio/base_events.py
M Lib/asyncio/events.py
M Lib/asyncio/unix_events.py

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 6cee171b65a..834a4e85c2f 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -424,7 +424,7 @@ Creating connections
 Creating listening connections
 ------------------------------
 
-.. coroutinemethod:: AbstractEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None)
+.. coroutinemethod:: AbstractEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, start_serving=True)
 
    Create a TCP server (socket type :data:`~socket.SOCK_STREAM`) bound to
    *host* and *port*.
@@ -472,9 +472,15 @@ Creating listening connections
      for the SSL handshake to complete before aborting the connection.
      ``10.0`` seconds if ``None`` (default).
 
+   * *start_serving* set to ``True`` (the default) causes the created server
+     to start accepting connections immediately.  When set to ``False``,
+     the user should await on :meth:`Server.start_serving` or
+     :meth:`Server.serve_forever` to make the server to start accepting
+     connections.
+
    .. versionadded:: 3.7
 
-      The *ssl_handshake_timeout* parameter.
+      *ssl_handshake_timeout* and *start_serving* parameters.
 
    .. versionchanged:: 3.5
 
@@ -490,7 +496,7 @@ Creating listening connections
       The *host* parameter can now be a sequence of strings.
 
 
-.. coroutinemethod:: AbstractEventLoop.create_unix_server(protocol_factory, path=None, \*, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None)
+.. coroutinemethod:: AbstractEventLoop.create_unix_server(protocol_factory, path=None, \*, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, start_serving=True)
 
    Similar to :meth:`AbstractEventLoop.create_server`, but specific to the
    socket family :py:data:`~socket.AF_UNIX`.
@@ -929,8 +935,26 @@ Server
 
    Server listening on sockets.
 
-   Object created by the :meth:`AbstractEventLoop.create_server` method and the
-   :func:`start_server` function. Don't instantiate the class directly.
+   Object created by :meth:`AbstractEventLoop.create_server`,
+   :meth:`AbstractEventLoop.create_unix_server`, :func:`start_server`,
+   and :func:`start_unix_server` functions.  Don't instantiate the class
+   directly.
+
+   *Server* objects are asynchronous context managers.  When used in an
+   ``async with`` statement, it's guaranteed that the Server object is
+   closed and not accepting new connections when the ``async with``
+   statement is completed::
+
+      srv = await loop.create_server(...)
+
+      async with srv:
+          # some code
+
+      # At this point, srv is closed and no longer accepts new connections.
+
+
+   .. versionchanged:: 3.7
+      Server object is an asynchronous context manager since Python 3.7.
 
    .. method:: close()
 
@@ -949,6 +973,54 @@ Server
 
       .. versionadded:: 3.7
 
+   .. coroutinemethod:: start_serving()
+
+      Start accepting connections.
+
+      This method is idempotent, so it can be called when
+      the server is already being serving.
+
+      The new *start_serving* keyword-only parameter to
+      :meth:`AbstractEventLoop.create_server` and
+      :meth:`asyncio.start_server` allows to create a Server object
+      that is not accepting connections right away.  In which case
+      this method, or :meth:`Server.serve_forever` can be used
+      to make the Server object to start accepting connections.
+
+      .. versionadded:: 3.7
+
+   .. coroutinemethod:: serve_forever()
+
+      Start accepting connections until the coroutine is cancelled.
+      Cancellation of ``serve_forever`` task causes the server
+      to be closed.
+
+      This method can be called if the server is already accepting
+      connections.  Only one ``serve_forever`` task can exist per
+      one *Server* object.
+
+      Example::
+
+          async def client_connected(reader, writer):
+              # Communicate with the client with
+              # reader/writer streams.  For example:
+              await reader.readline()
+
+          async def main(host, port):
+              srv = await asyncio.start_server(
+                  client_connected, host, port)
+              await loop.serve_forever()
+
+          asyncio.run(main('127.0.0.1', 0))
+
+      .. versionadded:: 3.7
+
+   .. method:: is_serving()
+
+      Return ``True`` if the server is accepting new connections.
+
+      .. versionadded:: 3.7
+
    .. coroutinemethod:: wait_closed()
 
       Wait until the :meth:`close` method completes.
@@ -958,6 +1030,11 @@ Server
       List of :class:`socket.socket` objects the server is listening to, or
       ``None`` if the server is closed.
 
+      .. versionchanged:: 3.7
+         Prior to Python 3.7 ``Server.sockets`` used to return the
+         internal list of server's sockets directly.  In 3.7 a copy
+         of that list is returned.
+
 
 Handle
 ------
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index e722cf26b51..94eb3089e93 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -157,47 +157,106 @@ def _run_until_complete_cb(fut):
 
 class Server(events.AbstractServer):
 
-    def __init__(self, loop, sockets):
+    def __init__(self, loop, sockets, protocol_factory, ssl_context, backlog,
+                 ssl_handshake_timeout):
         self._loop = loop
-        self.sockets = sockets
+        self._sockets = sockets
         self._active_count = 0
         self._waiters = []
+        self._protocol_factory = protocol_factory
+        self._backlog = backlog
+        self._ssl_context = ssl_context
+        self._ssl_handshake_timeout = ssl_handshake_timeout
+        self._serving = False
+        self._serving_forever_fut = None
 
     def __repr__(self):
         return f'<{self.__class__.__name__} sockets={self.sockets!r}>'
 
     def _attach(self):
-        assert self.sockets is not None
+        assert self._sockets is not None
         self._active_count += 1
 
     def _detach(self):
         assert self._active_count > 0
         self._active_count -= 1
-        if self._active_count == 0 and self.sockets is None:
+        if self._active_count == 0 and self._sockets is None:
             self._wakeup()
 
+    def _wakeup(self):
+        waiters = self._waiters
+        self._waiters = None
+        for waiter in waiters:
+            if not waiter.done():
+                waiter.set_result(waiter)
+
+    def _start_serving(self):
+        if self._serving:
+            return
+        self._serving = True
+        for sock in self._sockets:
+            sock.listen(self._backlog)
+            self._loop._start_serving(
+                self._protocol_factory, sock, self._ssl_context,
+                self, self._backlog, self._ssl_handshake_timeout)
+
+    def get_loop(self):
+        return self._loop
+
+    def is_serving(self):
+        return self._serving
+
+    @property
+    def sockets(self):
+        if self._sockets is None:
+            return []
+        return list(self._sockets)
+
     def close(self):
-        sockets = self.sockets
+        sockets = self._sockets
         if sockets is None:
             return
-        self.sockets = None
+        self._sockets = None
+
         for sock in sockets:
             self._loop._stop_serving(sock)
+
+        self._serving = False
+
+        if (self._serving_forever_fut is not None and
+                not self._serving_forever_fut.done()):
+            self._serving_forever_fut.cancel()
+            self._serving_forever_fut = None
+
         if self._active_count == 0:
             self._wakeup()
 
-    def get_loop(self):
-        return self._loop
+    async def start_serving(self):
+        self._start_serving()
 
-    def _wakeup(self):
-        waiters = self._waiters
-        self._waiters = None
-        for waiter in waiters:
-            if not waiter.done():
-                waiter.set_result(waiter)
+    async def serve_forever(self):
+        if self._serving_forever_fut is not None:
+            raise RuntimeError(
+                f'server {self!r} is already being awaited on serve_forever()')
+        if self._sockets is None:
+            raise RuntimeError(f'server {self!r} is closed')
+
+        self._start_serving()
+        self._serving_forever_fut = self._loop.create_future()
+
+        try:
+            await self._serving_forever_fut
+        except futures.CancelledError:
+            try:
+                self.close()
+                await self.wait_closed()
+            finally:
+                raise
+        finally:
+            self._serving_forever_fut = None
 
     async def wait_closed(self):
-        if self.sockets is None or self._waiters is None:
+        if self._sockets is None or self._waiters is None:
             return
         waiter = self._loop.create_future()
         self._waiters.append(waiter)
@@ -1059,7 +1118,8 @@ def _check_sendfile_params(self, sock, file, offset, count):
             ssl=None,
             reuse_address=None,
             reuse_port=None,
-            ssl_handshake_timeout=None):
+            ssl_handshake_timeout=None,
+            start_serving=True):
         """Create a TCP server.
 
         The host parameter can be a string, in that case the TCP server is
@@ -1149,12 +1209,14 @@ def _check_sendfile_params(self, sock, file, offset, count):
                 raise ValueError(f'A Stream Socket was expected, got {sock!r}')
             sockets = [sock]
 
-        server = Server(self, sockets)
         for sock in sockets:
-            sock.listen(backlog)
             sock.setblocking(False)
-            self._start_serving(protocol_factory, sock, ssl, server, backlog,
-                                ssl_handshake_timeout)
+
+        server = Server(self, sockets, protocol_factory,
+                        ssl, backlog, ssl_handshake_timeout)
+        if start_serving:
+            server._start_serving()
+
         if self._debug:
             logger.info("%r is serving", server)
         return server
diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index 5c68d4cb97d..7aa3de02c95 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -164,13 +164,39 @@ def close(self):
         """Stop serving.  This leaves existing connections open."""
         raise NotImplementedError
 
+    def get_loop(self):
+        """Get the event loop the Server object is attached to."""
+        raise NotImplementedError
+
+    def is_serving(self):
+        """Return True if the server is accepting connections."""
+        raise NotImplementedError
+
+    async def start_serving(self):
+        """Start accepting connections.
+
+        This method is idempotent, so it can be called when
+        the server is already being serving.
+        """
+        raise NotImplementedError
+
+    async def serve_forever(self):
+        """Start accepting connections until the coroutine is cancelled.
+
+        The server is closed when the coroutine is cancelled.
+        """
+        raise NotImplementedError
+
     async def wait_closed(self):
         """Coroutine to wait until service is closed."""
         raise NotImplementedError
 
-    def get_loop(self):
-        """ Get the event loop the Server object is attached to."""
-        raise NotImplementedError
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *exc):
+        self.close()
+        await self.wait_closed()
 
 
 class AbstractEventLoop:
@@ -279,7 +305,8 @@ def set_default_executor(self, executor):
             *, family=socket.AF_UNSPEC,
             flags=socket.AI_PASSIVE, sock=None, backlog=100,
             ssl=None, reuse_address=None, reuse_port=None,
-            ssl_handshake_timeout=None):
+            ssl_handshake_timeout=None,
+            start_serving=True):
         """A coroutine which creates a TCP server bound to host and port.
 
         The return value is a Server object which can be used to stop
@@ -319,6 +346,11 @@ def set_default_executor(self, executor):
         will wait for completion of the SSL handshake before aborting the
         connection. Default is 10s, longer timeouts may increase vulnerability
         to DoS attacks (see https://support.f5.com/csp/article/K13834)
+
+        start_serving set to True (default) causes the created server
+        to start accepting connections immediately.  When set to False,
+        the user should await Server.start_serving() or Server.serve_forever()
+        to make the server to start accepting connections.
         """
         raise NotImplementedError
 
@@ -343,7 +375,8 @@ def set_default_executor(self, executor):
     async def create_unix_server(
             self, protocol_factory, path=None, *,
             sock=None, backlog=100, ssl=None,
-            ssl_handshake_timeout=None):
+            ssl_handshake_timeout=None,
+            start_serving=True):
         """A coroutine which creates a UNIX Domain Socket server.
 
         The return value is a Server object, which can be used to stop
@@ -363,6 +396,11 @@ def set_default_executor(self, executor):
 
         ssl_handshake_timeout is the time in seconds that an SSL server
         will wait for the SSL handshake to complete (defaults to 10s).
+
+        start_serving set to True (default) causes the created server
+        to start accepting connections immediately.  When set to False,
+        the user should await Server.start_serving() or Server.serve_forever()
+        to make the server to start accepting connections.
         """
         raise NotImplementedError
 
diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
index 9b9d0043b50..a4d892acad0 100644
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -250,7 +250,8 @@ def _child_watcher_callback(self, pid, returncode, transp):
     async def create_unix_server(
             self, protocol_factory, path=None, *,
             sock=None, backlog=100, ssl=None,
-            ssl_handshake_timeout=None):
+            ssl_handshake_timeout=None,
+            start_serving=True):
         if isinstance(ssl, bool):
             raise TypeError('ssl argument must be an SSLContext or None')
 
@@ -302,11 +303,12 @@ def _child_watcher_callback(self, pid, returncode, transp):
                 raise ValueError(
                     f'A UNIX Domain Stream Socket was expected, got {sock!r}')
 
-        server = base_events.Server(self, [sock])
-        sock.listen(backlog)
         sock.setblocking(False)
-        self._start_serving(protocol_factory, sock, ssl, server,
-                            ssl_handshake_timeout=ssl_handshake_timeout)
+        server = base_events.Server(self, [sock], protocol_factory,
+                                    ssl, backlog, ssl_handshake_timeout)
+        if start_serving:
+            server._start_serving()
+
         return server
 
     async def _sock_sendfile_native(self, sock, file, offset, count):
diff --git a/Lib/test/test_asyncio/test_server.py b/Lib/test/test_asyncio/test_server.py
new file mode 100644
index 00000000000..44d135db105
--- /dev/null
+++ b/Lib/test/test_asyncio/test_server.py
@@ -0,0 +1,117 @@
+import asyncio
+import socket
+import threading
+import unittest
+
+from test.test_asyncio import utils as test_utils
+from test.test_asyncio import functional as func_tests
+
+
+class BaseStartServer(func_tests.FunctionalTestCaseMixin):
+
+    def new_loop(self):
+        raise NotImplementedError
+
+    def test_start_server_1(self):
+        HELLO_MSG = b'1' * 1024 * 5 + b'\n'
+
+        def client(sock, addr):
+            sock.connect(addr)
+            sock.send(HELLO_MSG)
+            sock.recv_all(1)
+            sock.close()
+
+        async def serve(reader, writer):
+            await reader.readline()
+            main_task.cancel()
+            writer.write(b'1')
+            writer.close()
+            await writer.wait_closed()
+
+        async def main(srv):
+            async with srv:
+                await srv.serve_forever()
+
+        srv = self.loop.run_until_complete(asyncio.start_server(
+            serve, '127.0.0.1', 0, loop=self.loop, start_serving=False))
+
+        self.assertFalse(srv.is_serving())
+
+        main_task = self.loop.create_task(main(srv))
+
+        addr = srv.sockets[0].getsockname()
+        with self.assertRaises(asyncio.CancelledError):
+            with self.tcp_client(lambda sock: client(sock, addr)):
+                self.loop.run_until_complete(main_task)
+
+        self.assertEqual(srv.sockets, [])
+
+        self.assertIsNone(srv._sockets)
+        self.assertIsNone(srv._waiters)
+        self.assertFalse(srv.is_serving())
+
+        with self.assertRaisesRegex(RuntimeError, r'is closed'):
+            self.loop.run_until_complete(srv.serve_forever())
+
+
+class SelectorStartServerTests(BaseStartServer, unittest.TestCase):
+
+    def new_loop(self):
+        return asyncio.SelectorEventLoop()
+
+    @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'no Unix sockets')
+    def test_start_unix_server_1(self):
+        HELLO_MSG = b'1' * 1024 * 5 + b'\n'
+        started = threading.Event()
+
+        def client(sock, addr):
+            started.wait(5)
+            sock.connect(addr)
+            sock.send(HELLO_MSG)
+            sock.recv_all(1)
+            sock.close()
+
+        async def serve(reader, writer):
+            await reader.readline()
+            main_task.cancel()
+            writer.write(b'1')
+            writer.close()
+            await writer.wait_closed()
+
+        async def main(srv):
+            async with srv:
+                self.assertFalse(srv.is_serving())
+                await srv.start_serving()
+                self.assertTrue(srv.is_serving())
+                started.set()
+                await srv.serve_forever()
+
+        with test_utils.unix_socket_path() as addr:
+            srv = self.loop.run_until_complete(asyncio.start_unix_server(
+                serve, addr, loop=self.loop, start_serving=False))
+
+            main_task = self.loop.create_task(main(srv))
+
+            with self.assertRaises(asyncio.CancelledError):
+                with self.unix_client(lambda sock: client(sock, addr)):
+                    self.loop.run_until_complete(main_task)
+
+            self.assertEqual(srv.sockets, [])
+
+            self.assertIsNone(srv._sockets)
+            self.assertIsNone(srv._waiters)
+            self.assertFalse(srv.is_serving())
+
+            with self.assertRaisesRegex(RuntimeError, r'is closed'):
+                self.loop.run_until_complete(srv.serve_forever())
+
+
+ at unittest.skipUnless(hasattr(asyncio, 'ProactorEventLoop'), 'Windows only')
+class ProactorStartServerTests(BaseStartServer, unittest.TestCase):
+
+    def new_loop(self):
+        return asyncio.ProactorEventLoop()
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/Misc/NEWS.d/next/Library/2018-01-25-01-45-30.bpo-32662.oabhd8.rst b/Misc/NEWS.d/next/Library/2018-01-25-01-45-30.bpo-32662.oabhd8.rst
new file mode 100644
index 00000000000..44c8b951030
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-25-01-45-30.bpo-32662.oabhd8.rst
@@ -0,0 +1,3 @@
+Implement Server.start_serving(), Server.serve_forever(), and
+Server.is_serving() methods.  Add 'start_serving' keyword parameter to
+loop.create_server() and loop.create_unix_server().


From webhook-mailer at python.org  Thu Jan 25 18:45:52 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 25 Jan 2018 23:45:52 -0000
Subject: [Python-checkins] bpo-32574: Fix leaks in asyncio.Queue.put() and
 .get() (#5208)
Message-ID: <mailman.56.1516923954.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/c47dacb69054f6fe1c2465df585985430f7fe366
commit: c47dacb69054f6fe1c2465df585985430f7fe366
branch: master
author: Jos? Melero Fern?ndez <jmelerofernandez at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-25T18:45:43-05:00
summary:

bpo-32574: Fix leaks in asyncio.Queue.put() and .get() (#5208)

files:
A Misc/NEWS.d/next/Library/2018-01-16-20-37-28.bpo-32574.ru8eZ9.rst
M Lib/asyncio/queues.py
M Lib/test/test_asyncio/test_queues.py

diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py
index 512ea6037f8..2a8d3e76044 100644
--- a/Lib/asyncio/queues.py
+++ b/Lib/asyncio/queues.py
@@ -121,6 +121,13 @@ def full(self):
                 await putter
             except:
                 putter.cancel()  # Just in case putter is not done yet.
+                try:
+                    # Clean self._putters from canceled putters.
+                    self._putters.remove(putter)
+                except ValueError:
+                    # The putter could be removed from self._putters by a
+                    # previous get_nowait call.
+                    pass
                 if not self.full() and not putter.cancelled():
                     # We were woken up by get_nowait(), but can't take
                     # the call.  Wake up the next in line.
@@ -152,12 +159,13 @@ def put_nowait(self, item):
                 await getter
             except:
                 getter.cancel()  # Just in case getter is not done yet.
-
                 try:
+                    # Clean self._getters from canceled getters.
                     self._getters.remove(getter)
                 except ValueError:
+                    # The getter could be removed from self._getters by a
+                    # previous put_nowait call.
                     pass
-
                 if not self.empty() and not getter.cancelled():
                     # We were woken up by put_nowait(), but can't take
                     # the call.  Wake up the next in line.
diff --git a/Lib/test/test_asyncio/test_queues.py b/Lib/test/test_asyncio/test_queues.py
index 8d78546318a..efe719ed39a 100644
--- a/Lib/test/test_asyncio/test_queues.py
+++ b/Lib/test/test_asyncio/test_queues.py
@@ -520,6 +520,56 @@ def test_why_are_putters_waiting(self):
         self.loop.run_until_complete(
             asyncio.gather(getter(), t0, t1, t2, t3, loop=self.loop))
 
+    def test_cancelled_puts_not_being_held_in_self_putters(self):
+        def a_generator():
+            yield 0.01
+            yield 0.1
+
+        loop = self.new_test_loop(a_generator)
+
+        # Full queue.
+        queue = asyncio.Queue(loop=loop, maxsize=1)
+        queue.put_nowait(1)
+
+        # Task waiting for space to put an item in the queue.
+        put_task = loop.create_task(queue.put(1))
+        loop.run_until_complete(asyncio.sleep(0.01, loop=loop))
+
+        # Check that the putter is correctly removed from queue._putters when
+        # the task is canceled.
+        self.assertEqual(len(queue._putters), 1)
+        put_task.cancel()
+        with self.assertRaises(asyncio.CancelledError):
+            loop.run_until_complete(put_task)
+        self.assertEqual(len(queue._putters), 0)
+
+    def test_cancelled_put_silence_value_error_exception(self):
+        def gen():
+            yield 0.01
+            yield 0.1
+
+        loop = self.new_test_loop(gen)
+
+        # Full Queue.
+        queue = asyncio.Queue(1, loop=loop)
+        queue.put_nowait(1)
+
+        # Task waiting for space to put a item in the queue.
+        put_task = loop.create_task(queue.put(1))
+        loop.run_until_complete(asyncio.sleep(0.01, loop=loop))
+
+        # get_nowait() remove the future of put_task from queue._putters.
+        queue.get_nowait()
+        # When canceled, queue.put is going to remove its future from
+        # self._putters but it was removed previously by queue.get_nowait().
+        put_task.cancel()
+
+        # The ValueError exception triggered by queue._putters.remove(putter)
+        # inside queue.put should be silenced.
+        # If the ValueError is silenced we should catch a CancelledError.
+        with self.assertRaises(asyncio.CancelledError):
+            loop.run_until_complete(put_task)
+
 
 class LifoQueueTests(_QueueTestBase):
 
diff --git a/Misc/NEWS.d/next/Library/2018-01-16-20-37-28.bpo-32574.ru8eZ9.rst b/Misc/NEWS.d/next/Library/2018-01-16-20-37-28.bpo-32574.ru8eZ9.rst
new file mode 100644
index 00000000000..00650d4d02e
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-16-20-37-28.bpo-32574.ru8eZ9.rst
@@ -0,0 +1,3 @@
+Fix memory leak in asyncio.Queue, when the queue has limited size and it is
+full, the cancelation of queue.put() can cause a memory leak. Patch by: Jos?
+Melero.


From webhook-mailer at python.org  Thu Jan 25 19:02:06 2018
From: webhook-mailer at python.org (=?utf-8?q?=C3=89ric?= Araujo)
Date: Fri, 26 Jan 2018 00:02:06 -0000
Subject: [Python-checkins] bpo-32304: Fix distutils upload for sdists ending
 with \x0d (GH-5264)
Message-ID: <mailman.57.1516924929.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/2fc98ae115e2a2095a0bcf388c27a878aafdb454
commit: 2fc98ae115e2a2095a0bcf388c27a878aafdb454
branch: master
author: Bo Bayles <bbayles at gmail.com>
committer: ?ric Araujo <merwok at netwok.org>
date: 2018-01-25T19:02:03-05:00
summary:

bpo-32304: Fix distutils upload for sdists ending with \x0d (GH-5264)

Patch by Bo Bayles.

files:
A Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
M Doc/whatsnew/3.7.rst
M Lib/distutils/command/upload.py
M Lib/distutils/tests/test_upload.py

diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index e9f057029eb..0e1714ea17e 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -416,6 +416,11 @@ therefore included in source distributions.
 and ``platforms`` fields are not specified as a list or a string.
 (Contributed by Berker Peksag in :issue:`19610`.)
 
+The ``upload`` command now longer tries to change CR end-of-line characters
+to CRLF.  This fixes a corruption issue with sdists that ended with a byte
+equivalent to CR.
+(Contributed by Bo Bayles in :issue:`32304`.)
+
 http.client
 -----------
 
diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
index 1fd574a9f15..f7752f9d12c 100644
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -159,8 +159,6 @@ def upload_file(self, command, pyversion, filename):
                 body.write(title.encode('utf-8'))
                 body.write(b"\r\n\r\n")
                 body.write(value)
-                if value and value[-1:] == b'\r':
-                    body.write(b'\n')  # write an extra newline (lurve Macs)
         body.write(end_boundary)
         body = body.getvalue()
 
diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py
index 2cb2f6ce938..c17d8e7d54e 100644
--- a/Lib/distutils/tests/test_upload.py
+++ b/Lib/distutils/tests/test_upload.py
@@ -143,6 +143,32 @@ def test_upload(self):
         results = self.get_logs(INFO)
         self.assertEqual(results[-1], 75 * '-' + '\nxyzzy\n' + 75 * '-')
 
+    # bpo-32304: archives whose last byte was b'\r' were corrupted due to
+    # normalization intended for Mac OS 9.
+    def test_upload_correct_cr(self):
+        # content that ends with \r should not be modified.
+        tmp = self.mkdtemp()
+        path = os.path.join(tmp, 'xxx')
+        self.write_file(path, content='yy\r')
+        command, pyversion, filename = 'xxx', '2.6', path
+        dist_files = [(command, pyversion, filename)]
+        self.write_file(self.rc, PYPIRC_LONG_PASSWORD)
+
+        # other fields that ended with \r used to be modified, now are
+        # preserved.
+        pkg_dir, dist = self.create_dist(
+            dist_files=dist_files,
+            description='long description\r'
+        )
+        cmd = upload(dist)
+        cmd.show_response = 1
+        cmd.ensure_finalized()
+        cmd.run()
+
+        headers = dict(self.last_open.req.headers)
+        self.assertEqual(headers['Content-length'], '2172')
+        self.assertIn(b'long description\r', self.last_open.req.data)
+
     def test_upload_fails(self):
         self.next_msg = "Not Found"
         self.next_code = 404
diff --git a/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst b/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
new file mode 100644
index 00000000000..c199a644ad4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
@@ -0,0 +1,2 @@
+distutils' upload command no longer corrupts tar files ending with a CR byte,
+and no longer tries to convert CR to CRLF in any of the upload text fields.


From webhook-mailer at python.org  Thu Jan 25 20:53:35 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Fri, 26 Jan 2018 01:53:35 -0000
Subject: [Python-checkins] bpo-32596: Make lazy-load portable (GH-5316)
Message-ID: <mailman.58.1516931616.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/4666ec597c38eea06a22bcfb4157d92a0abf891c
commit: 4666ec597c38eea06a22bcfb4157d92a0abf891c
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-26T10:53:31+09:00
summary:

bpo-32596: Make lazy-load portable (GH-5316)

Global variables should not used as import target.
Use temporary variable instead.

files:
M Lib/concurrent/futures/__init__.py

diff --git a/Lib/concurrent/futures/__init__.py b/Lib/concurrent/futures/__init__.py
index 72aca818d3e..8434fcf4b5e 100644
--- a/Lib/concurrent/futures/__init__.py
+++ b/Lib/concurrent/futures/__init__.py
@@ -40,11 +40,13 @@ def __getattr__(name):
     global ProcessPoolExecutor, ThreadPoolExecutor
 
     if name == 'ProcessPoolExecutor':
-        from .process import ProcessPoolExecutor
-        return ProcessPoolExecutor
+        from .process import ProcessPoolExecutor as pe
+        ProcessPoolExecutor = pe
+        return pe
 
     if name == 'ThreadPoolExecutor':
-        from .thread import ThreadPoolExecutor
-        return ThreadPoolExecutor
+        from .thread import ThreadPoolExecutor as te
+        ThreadPoolExecutor = te
+        return te
 
     raise AttributeError(f"module {__name__} has no attribute {name}")


From lp_benchmark_robot at intel.com  Thu Jan 25 23:33:11 2018
From: lp_benchmark_robot at intel.com (lp_benchmark_robot at intel.com)
Date: Thu, 25 Jan 2018 20:33:11 -0800
Subject: [Python-checkins] [1 down, 3 up,
 61 flat] Results for Python (master branch) 2018-01-25
Message-ID: <182df90b-ab52-4f82-a29d-dc05893a9e03@orsmsx107.amr.corp.intel.com>

Results for project python/master, build date: 2018-01-25 03:04:29-08:00.
  - commit:		2812d3d
  - previous commit:	018e1b7
  - revision date:	2018-01-25 19:52:58+09:00
  - environment:	Broadwell-EP
      - cpu:		Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz  2x22 cores,
        stepping 1, LLC 55 MB
      - mem:		128 GB
      - os:		Ubuntu 16.04.2 LTS
      - kernel:	4.4.0-62-generic  x86_64 GNU/Linux

Baseline results were generated using release v3.6.0, with hash 5c4568a from
2016-12-22 23:38:47+00:00.

+-----+------------------------+--------+------------+------------+------------+
|     |                        |relative|change since|change since|current rev |
|     |               benchmark|std_dev*|  last run  |  baseline  |run with PGO|
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    2to3| 0.909% |  +1.266%   |  +7.653%   |  +6.918%   |
+-----+------------------------+--------+------------+------------+------------+
| :-( |             call_method| 1.205% |  -5.352%   |  +18.725%  |  +13.712%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       call_method_slots| 1.179% |  -2.564%   |  +21.966%  |  +11.269%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     call_method_unknown| 0.798% |  -0.202%   |  +18.176%  |  +13.604%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             call_simple| 1.144% |  -1.777%   |  +6.396%   |  +15.945%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               chameleon| 1.785% |  -0.178%   |  +10.646%  |  +11.478%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   chaos| 1.309% |  +2.170%   |  +7.097%   |  +10.319%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            crypto_pyaes| 0.549% |  +1.395%   |  +0.274%   |  +8.156%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               deltablue| 1.766% |  +2.794%   |  +11.276%  |  +17.516%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         django_template| 1.420% |  +2.884%   |  +21.297%  |  +11.038%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             dulwich_log| 1.443% |  +0.599%   |  +5.829%   |  +7.108%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                fannkuch| 1.400% |  +0.781%   |  +4.880%   |  +6.794%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   float| 0.679% |  -0.024%   |  +2.508%   |  +8.084%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             genshi_text| 0.870% |  -0.735%   |  +12.258%  |  +9.144%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              genshi_xml| 1.323% |  -1.148%   |  +8.880%   |  +10.291%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                      go| 0.567% |  -0.292%   |  +6.217%   |  +12.818%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  hexiom| 1.288% |  +1.478%   |  +10.906%  |  +11.460%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                html5lib| 2.502% |  +2.390%   |  +10.418%  |  +10.339%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_dumps| 2.105% |  -1.221%   |  +1.362%   |  +9.738%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              json_loads| 0.964% |  -1.770%   |  -0.825%   |  +13.989%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_format| 1.562% |  +3.779%   |  +10.192%  |  +11.145%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          logging_silent| 2.791% |  +0.314%   |  +46.844%  |  +13.766%  |
+-----+------------------------+--------+------------+------------+------------+
| :-) |          logging_simple| 1.616% |  +5.656%   |  +12.094%  |  +11.746%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                    mako| 0.497% |  +0.333%   |  +17.304%  |  +13.132%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                     mdp| 3.344% |  +0.237%   |  +7.561%   |  +12.159%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |          meteor_contest| 0.835% |  -2.750%   |  +2.355%   |  +6.877%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   nbody| 0.709% |  +2.090%   |  -2.443%   |  +3.320%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 nqueens| 1.438% |  +1.081%   |  +5.585%   |  +8.033%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                 pathlib| 1.105% |  +1.164%   |  +4.148%   |  +9.688%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                  pickle| 1.216% |  +0.654%   |  -0.548%   |  +20.607%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_dict| 0.309% |  -0.196%   |  +0.384%   |  +24.068%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             pickle_list| 0.766% |  +0.494%   |  +2.413%   |  +20.813%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      pickle_pure_python| 1.871% |  +1.058%   |  +11.576%  |  +10.430%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                pidigits| 0.352% |  +0.195%   |  +0.256%   |  +10.409%  |
+-----+------------------------+--------+------------+------------+------------+
| :-) |          python_startup| 0.115% |  +2.355%   |  +13.141%  |  +4.877%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  python_startup_no_site| 0.082% |  +0.518%   |  +2.790%   |  +4.925%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                raytrace| 1.074% |  +0.241%   |  +9.572%   |  +13.849%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           regex_compile| 5.658% |  +0.535%   |  +3.089%   |  +9.387%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               regex_dna| 0.600% |  -0.659%   |  -0.806%   |  +10.095%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            regex_effbot| 2.232% |  +5.009%   |  -4.747%   |  +6.237%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                regex_v8| 0.745% |  -3.335%   |  +4.181%   |  +10.573%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                richards| 3.040% |  -2.370%   |  +6.510%   |  +17.544%  |
+-----+------------------------+--------+------------+------------+------------+
| :-) |             scimark_fft| 0.469% |  +2.717%   |  -1.831%   |  +3.417%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |              scimark_lu| 3.600% |  -3.273%   |  +23.918%  |  +12.779%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     scimark_monte_carlo| 2.362% |  +0.002%   |  +4.293%   |  +7.040%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |             scimark_sor| 1.756% |  +1.543%   |  +14.478%  |  +9.748%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| | scimark_sparse_mat_mult| 0.890% |  +2.040%   |  -7.355%   |  +2.536%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           spectral_norm| 2.811% |  +2.255%   |  +4.558%   |  +5.159%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |  sqlalchemy_declarative| 1.398% |  +3.077%   |  +9.419%   |  +4.599%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |   sqlalchemy_imperative| 2.825% |  +2.923%   |  +9.090%   |  +3.552%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sqlite_synth| 5.325% |  -1.726%   |  +17.132%  |  +9.329%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            sympy_expand| 2.608% |  +0.175%   |  +15.303%  |  +6.936%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         sympy_integrate| 1.982% |  +0.382%   |  +10.330%  |  +7.181%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_str| 3.775% |  -0.234%   |  +15.709%  |  +7.763%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |               sympy_sum| 5.215% |  +4.452%   |  +12.408%  |  +8.495%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                   telco| 4.783% |  -6.235%   |  +16.958%  |  +13.607%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |            tornado_http| 1.078% |  +0.774%   |  +6.995%   |  +6.864%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         unpack_sequence| 1.079% |  -0.287%   |  +1.700%   |  -1.215%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |                unpickle| 2.545% |  +0.308%   |  +8.296%   |  +21.197%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |           unpickle_list| 1.832% |  +0.170%   |  -1.735%   |  +13.007%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |    unpickle_pure_python| 0.813% |  +1.074%   |  +8.154%   |  +7.179%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |      xml_etree_generate| 0.847% |  -0.100%   |  +4.691%   |  +9.252%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |     xml_etree_iterparse| 2.162% |  -1.276%   |  +2.109%   |  +9.026%   |
+-----+------------------------+--------+------------+------------+------------+
| :-| |         xml_etree_parse| 0.614% |  +1.388%   |  -4.483%   |  +10.898%  |
+-----+------------------------+--------+------------+------------+------------+
| :-| |       xml_etree_process| 1.314% |  +0.212%   |  +5.889%   |  +10.029%  |
+-----+------------------------+--------+------------+------------+------------+

* Relative Standard Deviation (Standard Deviation/Average)

If this is not displayed properly please visit our results page here:
http://languagesperformance.intel.com/1-down-3-up-61-flat-results-for-python-master-branch-2018-01-25

Our lab does a nightly source pull and build of the Python project and measures
performance changes against the previous stable version and the previous nightly
measurement. This is provided as a service to the community so that quality
issues with current hardware can be identified quickly.

Intel technologies' features and benefits depend on system configuration and may
require enabled hardware, software or service activation. Performance varies
depending on system configuration.

From webhook-mailer at python.org  Thu Jan 25 23:49:59 2018
From: webhook-mailer at python.org (Ethan Furman)
Date: Fri, 26 Jan 2018 04:49:59 -0000
Subject: [Python-checkins] bpo-29237: Create enum for pstats sorting options
 (GH-5103)
Message-ID: <mailman.59.1516942204.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/863b1e4d0e95036bca4e97c1b8b2ca72c19790fb
commit: 863b1e4d0e95036bca4e97c1b8b2ca72c19790fb
branch: master
author: mwidjaja <mwidj at yahoo.com>
committer: Ethan Furman <ethan at stoneleaf.us>
date: 2018-01-25T20:49:56-08:00
summary:

bpo-29237: Create enum for pstats sorting options (GH-5103)

files:
A Misc/NEWS.d/next/Library/2018-01-04-14-45-33.bpo-29237.zenYA6.rst
M Doc/library/profile.rst
M Lib/pstats.py
M Lib/test/test_pstats.py
M Misc/ACKS

diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst
index 48426a00c9a..a6dc56f43cb 100644
--- a/Doc/library/profile.rst
+++ b/Doc/library/profile.rst
@@ -139,6 +139,7 @@ The :mod:`pstats` module's :class:`~pstats.Stats` class has a variety of methods
 for manipulating and printing the data saved into a profile results file::
 
    import pstats
+   from pstats import SortKey
    p = pstats.Stats('restats')
    p.strip_dirs().sort_stats(-1).print_stats()
 
@@ -148,14 +149,14 @@ entries according to the standard module/line/name string that is printed. The
 :meth:`~pstats.Stats.print_stats` method printed out all the statistics.  You
 might try the following sort calls::
 
-   p.sort_stats('name')
+   p.sort_stats(SortKey.NAME)
    p.print_stats()
 
 The first call will actually sort the list by function name, and the second call
 will print out the statistics.  The following are some interesting calls to
 experiment with::
 
-   p.sort_stats('cumulative').print_stats(10)
+   p.sort_stats(SortKey.CUMULATIVE).print_stats(10)
 
 This sorts the profile by cumulative time in a function, and then only prints
 the ten most significant lines.  If you want to understand what algorithms are
@@ -164,20 +165,20 @@ taking time, the above line is what you would use.
 If you were looking to see what functions were looping a lot, and taking a lot
 of time, you would do::
 
-   p.sort_stats('time').print_stats(10)
+   p.sort_stats(SortKey.TIME).print_stats(10)
 
 to sort according to time spent within each function, and then print the
 statistics for the top ten functions.
 
 You might also try::
 
-   p.sort_stats('file').print_stats('__init__')
+   p.sort_stats(SortKey.FILENAME).print_stats('__init__')
 
 This will sort all the statistics by file name, and then print out statistics
 for only the class init methods (since they are spelled with ``__init__`` in
 them).  As one final example, you could try::
 
-   p.sort_stats('time', 'cumulative').print_stats(.5, 'init')
+   p.sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(.5, 'init')
 
 This line sorts statistics with a primary key of time, and a secondary key of
 cumulative time, and then prints out some of the statistics. To be specific, the
@@ -250,12 +251,13 @@ functions:
    without writing the profile data to a file::
 
       import cProfile, pstats, io
+      from pstats import SortKey
       pr = cProfile.Profile()
       pr.enable()
       # ... do something ...
       pr.disable()
       s = io.StringIO()
-      sortby = 'cumulative'
+      sortby = SortKey.CUMULATIVE
       ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
       ps.print_stats()
       print(s.getvalue())
@@ -361,60 +363,65 @@ Analysis of the profiler data is done using the :class:`~pstats.Stats` class.
    .. method:: sort_stats(*keys)
 
       This method modifies the :class:`Stats` object by sorting it according to
-      the supplied criteria.  The argument is typically a string identifying the
-      basis of a sort (example: ``'time'`` or ``'name'``).
+      the supplied criteria.  The argument can be either a string or a SortKey
+      enum identifying the basis of a sort (example: ``'time'``, ``'name'``,
+      ``SortKey.TIME`` or ``SortKey.NAME``). The SortKey enums argument have
+      advantage over the string argument in that it is more robust and less
+      error prone.
 
       When more than one key is provided, then additional keys are used as
       secondary criteria when there is equality in all keys selected before
-      them.  For example, ``sort_stats('name', 'file')`` will sort all the
-      entries according to their function name, and resolve all ties (identical
-      function names) by sorting by file name.
-
-      Abbreviations can be used for any key names, as long as the abbreviation
-      is unambiguous.  The following are the keys currently defined:
-
-      +------------------+----------------------+
-      | Valid Arg        | Meaning              |
-      +==================+======================+
-      | ``'calls'``      | call count           |
-      +------------------+----------------------+
-      | ``'cumulative'`` | cumulative time      |
-      +------------------+----------------------+
-      | ``'cumtime'``    | cumulative time      |
-      +------------------+----------------------+
-      | ``'file'``       | file name            |
-      +------------------+----------------------+
-      | ``'filename'``   | file name            |
-      +------------------+----------------------+
-      | ``'module'``     | file name            |
-      +------------------+----------------------+
-      | ``'ncalls'``     | call count           |
-      +------------------+----------------------+
-      | ``'pcalls'``     | primitive call count |
-      +------------------+----------------------+
-      | ``'line'``       | line number          |
-      +------------------+----------------------+
-      | ``'name'``       | function name        |
-      +------------------+----------------------+
-      | ``'nfl'``        | name/file/line       |
-      +------------------+----------------------+
-      | ``'stdname'``    | standard name        |
-      +------------------+----------------------+
-      | ``'time'``       | internal time        |
-      +------------------+----------------------+
-      | ``'tottime'``    | internal time        |
-      +------------------+----------------------+
+      them.  For example, ``sort_stats(SortKey.NAME, SortKey.FILE)`` will sort
+      all the entries according to their function name, and resolve all ties
+      (identical function names) by sorting by file name.
+
+      For the string argument, abbreviations can be used for any key names, as
+      long as the abbreviation is unambiguous.
+
+      The following are the valid string and SortKey:
+
+      +------------------+---------------------+----------------------+
+      | Valid String Arg | Valid enum Arg      | Meaning              |
+      +==================+=====================+======================+
+      | ``'calls'``      | SortKey.CALLS       | call count           |
+      +------------------+---------------------+----------------------+
+      | ``'cumulative'`` | SortKey.CUMULATIVE  | cumulative time      |
+      +------------------+---------------------+----------------------+
+      | ``'cumtime'``    | N/A                 | cumulative time      |
+      +------------------+---------------------+----------------------+
+      | ``'file'``       | N/A                 | file name            |
+      +------------------+---------------------+----------------------+
+      | ``'filename'``   | SortKey.FILENAME    | file name            |
+      +------------------+---------------------+----------------------+
+      | ``'module'``     | N/A                 | file name            |
+      +------------------+---------------------+----------------------+
+      | ``'ncalls'``     | N/A                 | call count           |
+      +------------------+---------------------+----------------------+
+      | ``'pcalls'``     | SortKey.PCALLS      | primitive call count |
+      +------------------+---------------------+----------------------+
+      | ``'line'``       | SortKey.LINE        | line number          |
+      +------------------+---------------------+----------------------+
+      | ``'name'``       | SortKey.NAME        | function name        |
+      +------------------+---------------------+----------------------+
+      | ``'nfl'``        | SortKey.NFL         | name/file/line       |
+      +------------------+---------------------+----------------------+
+      | ``'stdname'``    | SortKey.STDNAME     | standard name        |
+      +------------------+---------------------+----------------------+
+      | ``'time'``       | SortKey.TIME        | internal time        |
+      +------------------+---------------------+----------------------+
+      | ``'tottime'``    | N/A                 | internal time        |
+      +------------------+---------------------+----------------------+
 
       Note that all sorts on statistics are in descending order (placing most
       time consuming items first), where as name, file, and line number searches
       are in ascending order (alphabetical). The subtle distinction between
-      ``'nfl'`` and ``'stdname'`` is that the standard name is a sort of the
-      name as printed, which means that the embedded line numbers get compared
-      in an odd way.  For example, lines 3, 20, and 40 would (if the file names
-      were the same) appear in the string order 20, 3 and 40.  In contrast,
-      ``'nfl'`` does a numeric compare of the line numbers.  In fact,
-      ``sort_stats('nfl')`` is the same as ``sort_stats('name', 'file',
-      'line')``.
+      ``SortKey.NFL`` and ``SortKey.STDNAME`` is that the standard name is a
+      sort of the name as printed, which means that the embedded line numbers
+      get compared in an odd way.  For example, lines 3, 20, and 40 would (if
+      the file names were the same) appear in the string order 20, 3 and 40.
+      In contrast, ``SortKey.NFL`` does a numeric compare of the line numbers.
+      In fact, ``sort_stats(SortKey.NFL)`` is the same as
+      ``sort_stats(SortKey.NAME, SortKey.FILENAME, SortKey.LINE)``.
 
       For backward-compatibility reasons, the numeric arguments ``-1``, ``0``,
       ``1``, and ``2`` are permitted.  They are interpreted as ``'stdname'``,
@@ -424,6 +431,8 @@ Analysis of the profiler data is done using the :class:`~pstats.Stats` class.
 
       .. For compatibility with the old profiler.
 
+      .. versionadded:: 3.7
+         Added the SortKey enum.
 
    .. method:: reverse_order()
 
diff --git a/Lib/pstats.py b/Lib/pstats.py
index b7a20542a39..1b57d26b5a5 100644
--- a/Lib/pstats.py
+++ b/Lib/pstats.py
@@ -25,9 +25,32 @@
 import time
 import marshal
 import re
+from enum import Enum
 from functools import cmp_to_key
 
-__all__ = ["Stats"]
+__all__ = ["Stats", "SortKey"]
+
+
+class SortKey(str, Enum):
+    CALLS = 'calls', 'ncalls'
+    CUMULATIVE = 'cumulative', 'cumtime'
+    FILENAME = 'filename', 'module'
+    LINE = 'line'
+    NAME = 'name'
+    NFL = 'nfl'
+    PCALLS = 'pcalls'
+    STDNAME = 'stdname'
+    TIME = 'time', 'tottime'
+
+    def __new__(cls, *values):
+        obj = str.__new__(cls)
+
+        obj._value_ = values[0]
+        for other_value in values[1:]:
+            cls._value2member_map_[other_value] = obj
+        obj._all_values = values
+        return obj
+
 
 class Stats:
     """This class is used for creating reports from data generated by the
@@ -49,13 +72,14 @@ class Stats:
 
     The sort_stats() method now processes some additional options (i.e., in
     addition to the old -1, 0, 1, or 2 that are respectively interpreted as
-    'stdname', 'calls', 'time', and 'cumulative').  It takes an arbitrary number
-    of quoted strings to select the sort order.
+    'stdname', 'calls', 'time', and 'cumulative').  It takes either an
+    arbitrary number of quoted strings or SortKey enum to select the sort
+    order.
 
-    For example sort_stats('time', 'name') sorts on the major key of 'internal
-    function time', and on the minor key of 'the name of the function'.  Look at
-    the two tables in sort_stats() and get_sort_arg_defs(self) for more
-    examples.
+    For example sort_stats('time', 'name') or sort_stats(SortKey.TIME,
+    SortKey.NAME) sorts on the major key of 'internal function time', and on
+    the minor key of 'the name of the function'.  Look at the two tables in
+    sort_stats() and get_sort_arg_defs(self) for more examples.
 
     All methods return self, so you can string together commands like:
         Stats('foo', 'goo').strip_dirs().sort_stats('calls').\
@@ -161,7 +185,6 @@ def dump_stats(self, filename):
               "ncalls"    : (((1,-1),              ), "call count"),
               "cumtime"   : (((3,-1),              ), "cumulative time"),
               "cumulative": (((3,-1),              ), "cumulative time"),
-              "file"      : (((4, 1),              ), "file name"),
               "filename"  : (((4, 1),              ), "file name"),
               "line"      : (((5, 1),              ), "line number"),
               "module"    : (((4, 1),              ), "file name"),
@@ -202,12 +225,19 @@ def sort_stats(self, *field):
                        0:  "calls",
                        1:  "time",
                        2:  "cumulative"}[field[0]] ]
+        elif len(field) >= 2:
+            for arg in field[1:]:
+                if type(arg) != type(field[0]):
+                    raise TypeError("Can't have mixed argument type")
 
         sort_arg_defs = self.get_sort_arg_defs()
+
         sort_tuple = ()
         self.sort_type = ""
         connector = ""
         for word in field:
+            if isinstance(word, SortKey):
+                word = word.value
             sort_tuple = sort_tuple + sort_arg_defs[word][0]
             self.sort_type += connector + sort_arg_defs[word][1]
             connector = ", "
diff --git a/Lib/test/test_pstats.py b/Lib/test/test_pstats.py
index 566b3eab771..f835ce309a6 100644
--- a/Lib/test/test_pstats.py
+++ b/Lib/test/test_pstats.py
@@ -2,6 +2,7 @@
 from test import support
 from io import StringIO
 import pstats
+from pstats import SortKey
 
 
 
@@ -33,6 +34,47 @@ def test_add(self):
         stats = pstats.Stats(stream=stream)
         stats.add(self.stats, self.stats)
 
+    def test_sort_stats_int(self):
+        valid_args = {-1: 'stdname',
+                      0: 'calls',
+                      1: 'time',
+                      2: 'cumulative'}
+        for arg_int, arg_str in valid_args.items():
+            self.stats.sort_stats(arg_int)
+            self.assertEqual(self.stats.sort_type,
+                             self.stats.sort_arg_dict_default[arg_str][-1])
+
+    def test_sort_stats_string(self):
+        for sort_name in ['calls', 'ncalls', 'cumtime', 'cumulative',
+                    'filename', 'line', 'module', 'name', 'nfl', 'pcalls',
+                    'stdname', 'time', 'tottime']:
+            self.stats.sort_stats(sort_name)
+            self.assertEqual(self.stats.sort_type,
+                             self.stats.sort_arg_dict_default[sort_name][-1])
+
+    def test_sort_stats_partial(self):
+        sortkey = 'filename'
+        for sort_name in ['f', 'fi', 'fil', 'file', 'filen', 'filena',
+                           'filenam', 'filename']:
+            self.stats.sort_stats(sort_name)
+            self.assertEqual(self.stats.sort_type,
+                             self.stats.sort_arg_dict_default[sortkey][-1])
+
+    def test_sort_stats_enum(self):
+        for member in SortKey:
+            self.stats.sort_stats(member)
+            self.assertEqual(
+                    self.stats.sort_type,
+                    self.stats.sort_arg_dict_default[member.value][-1])
+
+    def test_sort_starts_mix(self):
+        self.assertRaises(TypeError, self.stats.sort_stats,
+                          'calls',
+                          SortKey.TIME)
+        self.assertRaises(TypeError, self.stats.sort_stats,
+                          SortKey.TIME,
+                          'calls')
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Misc/ACKS b/Misc/ACKS
index 900604ca4c6..20210e8089b 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1706,6 +1706,7 @@ Jeff Wheeler
 Christopher White
 David White
 Mats Wichmann
+Marcel Widjaja
 Truida Wiedijk
 Felix Wiemann
 Gerry Wiener
diff --git a/Misc/NEWS.d/next/Library/2018-01-04-14-45-33.bpo-29237.zenYA6.rst b/Misc/NEWS.d/next/Library/2018-01-04-14-45-33.bpo-29237.zenYA6.rst
new file mode 100644
index 00000000000..f903aa7001b
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-04-14-45-33.bpo-29237.zenYA6.rst
@@ -0,0 +1 @@
+Create enum for pstats sorting options


From webhook-mailer at python.org  Fri Jan 26 01:31:06 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Fri, 26 Jan 2018 06:31:06 -0000
Subject: [Python-checkins] bpo-32662: Try making test_asyncio.test_server
 more reliable (#5338)
Message-ID: <mailman.60.1516948267.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/4112c5b97d9c1c7b034653d0e017ffa894a45c74
commit: 4112c5b97d9c1c7b034653d0e017ffa894a45c74
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-26T01:30:57-05:00
summary:

bpo-32662: Try making test_asyncio.test_server more reliable (#5338)

files:
M Lib/test/test_asyncio/test_server.py

diff --git a/Lib/test/test_asyncio/test_server.py b/Lib/test/test_asyncio/test_server.py
index 44d135db105..034293cb3f5 100644
--- a/Lib/test/test_asyncio/test_server.py
+++ b/Lib/test/test_asyncio/test_server.py
@@ -1,8 +1,10 @@
 import asyncio
 import socket
+import time
 import threading
 import unittest
 
+from test import support
 from test.test_asyncio import utils as test_utils
 from test.test_asyncio import functional as func_tests
 
@@ -16,6 +18,14 @@ def test_start_server_1(self):
         HELLO_MSG = b'1' * 1024 * 5 + b'\n'
 
         def client(sock, addr):
+            for i in range(10):
+                time.sleep(0.2)
+                if srv.is_serving():
+                    break
+            else:
+                raise RuntimeError
+
+            sock.settimeout(2)
             sock.connect(addr)
             sock.send(HELLO_MSG)
             sock.recv_all(1)
@@ -33,7 +43,7 @@ def client(sock, addr):
                 await srv.serve_forever()
 
         srv = self.loop.run_until_complete(asyncio.start_server(
-            serve, '127.0.0.1', 0, loop=self.loop, start_serving=False))
+            serve, support.HOSTv4, 0, loop=self.loop, start_serving=False))
 
         self.assertFalse(srv.is_serving())
 
@@ -65,6 +75,7 @@ def test_start_unix_server_1(self):
         started = threading.Event()
 
         def client(sock, addr):
+            sock.settimeout(2)
             started.wait(5)
             sock.connect(addr)
             sock.send(HELLO_MSG)


From webhook-mailer at python.org  Fri Jan 26 02:22:54 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Fri, 26 Jan 2018 07:22:54 -0000
Subject: [Python-checkins] bpo-32571: Fix reading uninitialized memory
 (GH-5332)
Message-ID: <mailman.61.1516951376.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/e76daebc0c8afa3981a4c5a8b54537f756e805de
commit: e76daebc0c8afa3981a4c5a8b54537f756e805de
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-26T16:22:51+09:00
summary:

bpo-32571: Fix reading uninitialized memory (GH-5332)

Reported by Coverity Scan.

files:
M Objects/object.c

diff --git a/Objects/object.c b/Objects/object.c
index 7b2adbea1a1..fef57fce7ba 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -917,6 +917,11 @@ _PyObject_LookupAttr(PyObject *v, PyObject *name, PyObject **result)
         }
         *result = (*tp->tp_getattr)(v, (char *)name_str);
     }
+    else {
+        *result = NULL;
+        return 0;
+    }
+
     if (*result != NULL) {
         return 1;
     }


From solipsis at pitrou.net  Fri Jan 26 04:20:00 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 26 Jan 2018 09:20:00 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=12
Message-ID: <20180126092000.1.9C361C7E54254F7C@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, 7] memory blocks, sum=7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [-1, 2, 0] memory blocks, sum=1


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogRXc4cJ', '--timeout', '7200']

From webhook-mailer at python.org  Fri Jan 26 10:46:04 2018
From: webhook-mailer at python.org (Senthil Kumaran)
Date: Fri, 26 Jan 2018 15:46:04 -0000
Subject: [Python-checkins] bpo-18533: Avoid RecursionError from repr() of
 recursive dictview (#4823)
Message-ID: <mailman.62.1516981565.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d7773d92bd11640a8c950d6c36a9cef1cee36f96
commit: d7773d92bd11640a8c950d6c36a9cef1cee36f96
branch: master
author: bennorth <ben at redfrontdoor.org>
committer: Senthil Kumaran <skumaran at gatech.edu>
date: 2018-01-26T07:46:01-08:00
summary:

bpo-18533: Avoid RecursionError from repr() of recursive dictview (#4823)

dictview_repr(): Use a Py_ReprEnter() / Py_ReprLeave() pair to check
for recursion, and produce "..." if so.

test_recursive_repr(): Check for the string rather than a
RecursionError.  (Test cannot be any tighter as contents are
implementation-dependent.)

test_deeply_nested_repr(): Add new test, replacing the original
test_recursive_repr().  It checks that a RecursionError is raised in
the case of a non-recursive but deeply nested structure.  (Very
similar to what test_repr_deep() in test/test_dict.py does for a
normal dict.)

OrderedDictTests: Add new test case, to test behavior on OrderedDict
instances containing their own values() or items().

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-13-16-46-23.bpo-18533.Dlk8d7.rst
M Lib/test/test_dictviews.py
M Lib/test/test_ordered_dict.py
M Objects/dictobject.c

diff --git a/Lib/test/test_dictviews.py b/Lib/test/test_dictviews.py
index 51ad9b381c5..2763cbfb4bb 100644
--- a/Lib/test/test_dictviews.py
+++ b/Lib/test/test_dictviews.py
@@ -1,6 +1,7 @@
 import collections.abc
 import copy
 import pickle
+import sys
 import unittest
 
 class DictSetTest(unittest.TestCase):
@@ -202,6 +203,20 @@ def test_items_set_operations(self):
     def test_recursive_repr(self):
         d = {}
         d[42] = d.values()
+        r = repr(d)
+        # Cannot perform a stronger test, as the contents of the repr
+        # are implementation-dependent.  All we can say is that we
+        # want a str result, not an exception of any sort.
+        self.assertIsInstance(r, str)
+        d[42] = d.items()
+        r = repr(d)
+        # Again.
+        self.assertIsInstance(r, str)
+
+    def test_deeply_nested_repr(self):
+        d = {}
+        for i in range(sys.getrecursionlimit() + 100):
+            d = {42: d.values()}
         self.assertRaises(RecursionError, repr, d)
 
     def test_copy(self):
diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py
index 6bafb8be307..20efe3729de 100644
--- a/Lib/test/test_ordered_dict.py
+++ b/Lib/test/test_ordered_dict.py
@@ -355,6 +355,20 @@ def test_repr_recursive(self):
         self.assertEqual(repr(od),
             "OrderedDict([('a', None), ('b', None), ('c', None), ('x', ...)])")
 
+    def test_repr_recursive_values(self):
+        OrderedDict = self.OrderedDict
+        od = OrderedDict()
+        od[42] = od.values()
+        r = repr(od)
+        # Cannot perform a stronger test, as the contents of the repr
+        # are implementation-dependent.  All we can say is that we
+        # want a str result, not an exception of any sort.
+        self.assertIsInstance(r, str)
+        od[42] = od.items()
+        r = repr(od)
+        # Again.
+        self.assertIsInstance(r, str)
+
     def test_setdefault(self):
         OrderedDict = self.OrderedDict
         pairs = [('c', 1), ('b', 2), ('a', 3), ('d', 4), ('e', 5), ('f', 6)]
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-13-16-46-23.bpo-18533.Dlk8d7.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-13-16-46-23.bpo-18533.Dlk8d7.rst
new file mode 100644
index 00000000000..a33eff5c705
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-13-16-46-23.bpo-18533.Dlk8d7.rst	
@@ -0,0 +1,3 @@
+``repr()`` on a dict containing its own ``values()`` or ``items()`` no
+longer raises ``RecursionError``; OrderedDict similarly.  Instead, use
+``...``, as for other recursive structures.  Patch by Ben North.
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 0d25739f72c..bb4ea1f5f9c 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -3853,14 +3853,22 @@ static PyObject *
 dictview_repr(_PyDictViewObject *dv)
 {
     PyObject *seq;
-    PyObject *result;
+    PyObject *result = NULL;
+    Py_ssize_t rc;
 
+    rc = Py_ReprEnter((PyObject *)dv);
+    if (rc != 0) {
+        return rc > 0 ? PyUnicode_FromString("...") : NULL;
+    }
     seq = PySequence_List((PyObject *)dv);
-    if (seq == NULL)
-        return NULL;
-
+    if (seq == NULL) {
+        goto Done;
+    }
     result = PyUnicode_FromFormat("%s(%R)", Py_TYPE(dv)->tp_name, seq);
     Py_DECREF(seq);
+
+Done:
+    Py_ReprLeave((PyObject *)dv);
     return result;
 }
 


From webhook-mailer at python.org  Fri Jan 26 11:20:23 2018
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Fri, 26 Jan 2018 16:20:23 -0000
Subject: [Python-checkins] String annotations [PEP 563] (#4390)
Message-ID: <mailman.63.1516983623.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/95e4d589137260530e18ef98a2ed84ee3ec57e12
commit: 95e4d589137260530e18ef98a2ed84ee3ec57e12
branch: master
author: Guido van Rossum <guido at python.org>
committer: ?ukasz Langa <lukasz at langa.pl>
date: 2018-01-26T08:20:18-08:00
summary:

String annotations [PEP 563] (#4390)

* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form

This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.

files:
A Python/ast_unparse.c
M Doc/glossary.rst
M Doc/library/__future__.rst
M Doc/reference/compound_stmts.rst
M Doc/reference/simple_stmts.rst
M Doc/whatsnew/3.7.rst
M Include/ast.h
M Include/code.h
M Include/compile.h
M Lib/__future__.py
M Lib/test/test_future.py
M Makefile.pre.in
M PCbuild/pythoncore.vcxproj
M PCbuild/pythoncore.vcxproj.filters
M Python/compile.c
M Python/future.c

diff --git a/Doc/glossary.rst b/Doc/glossary.rst
index f0fcb6273e0..dcfe086b38b 100644
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -372,9 +372,11 @@ Glossary
       may be accessed via the :attr:`__annotations__` special attribute of a
       function object.
 
-      Python itself does not assign any particular meaning to function
-      annotations. They are intended to be interpreted by third-party libraries
-      or tools.  See :pep:`3107`, which describes some of their potential uses.
+      See also the :term:`variable annotation` glossary entry.
+
+      Annotations are meant to provide a standard way for programmers to
+      document types of functions they design.  See :pep:`484`, which
+      describes this functionality.
 
    __future__
       A pseudo-module which programmers can use to enable new language features
@@ -1021,10 +1023,11 @@ Glossary
       attribute of a class or module object and can be accessed using
       :func:`typing.get_type_hints`.
 
-      Python itself does not assign any particular meaning to variable
-      annotations. They are intended to be interpreted by third-party libraries
-      or type checking tools. See :pep:`526`, :pep:`484` which describe
-      some of their potential uses.
+      See also the :term:`function annotation` glossary entry.
+
+      Annotations are meant to provide a standard way for programmers to
+      document types of functions they design.  See :pep:`484` and :pep:`526`
+      which describe this functionality.
 
    virtual environment
       A cooperatively isolated runtime environment that allows Python users
diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst
index 73d8b6b7e8a..e3d749e6017 100644
--- a/Doc/library/__future__.rst
+++ b/Doc/library/__future__.rst
@@ -90,6 +90,11 @@ language using this mechanism:
 | generator_stop   | 3.5.0b1     | 3.7          | :pep:`479`:                                 |
 |                  |             |              | *StopIteration handling inside generators*  |
 +------------------+-------------+--------------+---------------------------------------------+
+| annotations      | 3.7.0b1     | 4.0          | :pep:`563`:                                 |
+|                  |             |              | *Postponed evaluation of annotations*       |
++------------------+-------------+--------------+---------------------------------------------+
+
+.. XXX Adding a new entry?  Remember to update simple_stmts.rst, too.
 
 
 .. seealso::
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
index dca93624004..d7792f1eaf8 100644
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -559,12 +559,14 @@ Parameters may have annotations of the form "``: expression``" following the
 parameter name.  Any parameter may have an annotation even those of the form
 ``*identifier`` or ``**identifier``.  Functions may have "return" annotation of
 the form "``-> expression``" after the parameter list.  These annotations can be
-any valid Python expression and are evaluated when the function definition is
-executed.  Annotations may be evaluated in a different order than they appear in
-the source code.  The presence of annotations does not change the semantics of a
-function.  The annotation values are available as values of a dictionary keyed
-by the parameters' names in the :attr:`__annotations__` attribute of the
-function object.
+any valid Python expression.  The presence of annotations does not change the
+semantics of a function.  The annotation values are available as values of
+a dictionary keyed by the parameters' names in the :attr:`__annotations__`
+attribute of the function object.  If the ``annotations`` import from
+:mod:`__future__` is used, annotations are preserved as strings at runtime which
+enables postponed evaluation.  Otherwise, they are evaluated when the function
+definition is executed.  In this case annotations may be evaluated in
+a different order than they appear in the source code.
 
 .. index:: pair: lambda; expression
 
@@ -587,6 +589,17 @@ access the local variables of the function containing the def.  See section
    :pep:`3107` - Function Annotations
       The original specification for function annotations.
 
+   :pep:`484` - Type Hints
+      Definition of a standard meaning for annotations: type hints.
+
+   :pep:`526` - Syntax for Variable Annotations
+      Ability to type hint variable declarations, including class
+      variables and instance variables
+
+   :pep:`563` - Postponed Evaluation of Annotations
+      Support for forward references within annotations by preserving
+      annotations in a string form at runtime instead of eager evaluation.
+
 
 .. _class:
 
diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst
index 8d17383853a..ef9a5f0dc85 100644
--- a/Doc/reference/simple_stmts.rst
+++ b/Doc/reference/simple_stmts.rst
@@ -853,12 +853,15 @@ can appear before a future statement are:
 * blank lines, and
 * other future statements.
 
-.. XXX change this if future is cleaned out
-
-The features recognized by Python 3.0 are ``absolute_import``, ``division``,
-``generators``, ``unicode_literals``, ``print_function``, ``nested_scopes`` and
-``with_statement``.  They are all redundant because they are always enabled, and
-only kept for backwards compatibility.
+The only feature in Python 3.7 that requires using the future statement is
+``annotations``.
+
+All historical features enabled by the future statement are still recognized
+by Python 3.  The list includes ``absolute_import``, ``division``,
+``generators``, ``generator_stop``, ``unicode_literals``,
+``print_function``, ``nested_scopes`` and ``with_statement``.  They are
+all redundant because they are always enabled, and only kept for
+backwards compatibility.
 
 A future statement is recognized and treated specially at compile time: Changes
 to the semantics of core constructs are often implemented by generating
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 0e1714ea17e..a350919bb34 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -179,6 +179,57 @@ a normal ``__getattr__`` method, except that it will be defined on module
        PEP written and implemented by Ivan Levkivskyi
 
 
+PEP 563: Postponed evaluation of annotations
+--------------------------------------------
+
+The advent of type hints in Python uncovered two glaring usability issues
+with the functionality of annotations added in :pep:`3107` and refined
+further in :pep:`526`:
+
+* annotations could only use names which were already available in the
+  current scope, in other words they didn't support forward references
+  of any kind; and
+
+* annotating source code had adverse effects on startup time of Python
+  programs.
+
+Both of these issues are fixed by postponing the evaluation of
+annotations.  Instead of compiling code which executes expressions in
+annotations at their definition time, the compiler stores the annotation
+in a string form equivalent to the AST of the expression in question.
+If needed, annotations can be resolved at runtime using
+``typing.get_type_hints()``.  In the common case where this is not
+required, the annotations are cheaper to store (since short strings
+are interned by the interpreter) and make startup time faster.
+
+Usability-wise, annotations now support forward references, making the
+following syntax valid::
+
+    class C:
+        @classmethod
+        def from_string(cls, source: str) -> C:
+            ...
+
+        def validate_b(self, obj: B) -> bool:
+            ...
+
+    class B:
+        ...
+
+Since this change breaks compatibility, the new behavior can be enabled
+on a per-module basis in Python 3.7 using a ``__future__`` import, like
+this::
+
+    from __future__ import annotations
+
+It will become the default in Python 4.0.
+
+.. seealso::
+
+    :pep:`563` -- Postponed evaluation of annotations
+       PEP written and implemented by ?ukasz Langa.
+
+
 PEP 564: Add new time functions with nanosecond resolution
 ----------------------------------------------------------
 
diff --git a/Include/ast.h b/Include/ast.h
index 6a8c8165c05..639c4f82325 100644
--- a/Include/ast.h
+++ b/Include/ast.h
@@ -16,6 +16,15 @@ PyAPI_FUNC(mod_ty) PyAST_FromNodeObject(
     PyObject *filename,
     PyArena *arena);
 
+#ifndef Py_LIMITED_API
+
+/* _PyAST_ExprAsUnicode is defined in ast_unparse.c */
+PyAPI_FUNC(PyObject *) _PyAST_ExprAsUnicode(
+    expr_ty e,
+    int omit_parens);
+
+#endif /* !Py_LIMITED_API */
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/Include/code.h b/Include/code.h
index 8b0f8404237..2e661e8b36b 100644
--- a/Include/code.h
+++ b/Include/code.h
@@ -82,6 +82,7 @@ typedef struct {
 
 #define CO_FUTURE_BARRY_AS_BDFL  0x40000
 #define CO_FUTURE_GENERATOR_STOP  0x80000
+#define CO_FUTURE_ANNOTATIONS    0x100000
 
 /* This value is found in the co_cell2arg array when the associated cell
    variable does not correspond to an argument. */
diff --git a/Include/compile.h b/Include/compile.h
index edfd8bbfac6..edb961f4d72 100644
--- a/Include/compile.h
+++ b/Include/compile.h
@@ -16,7 +16,7 @@ PyAPI_FUNC(PyCodeObject *) PyNode_Compile(struct _node *, const char *);
 #define PyCF_MASK (CO_FUTURE_DIVISION | CO_FUTURE_ABSOLUTE_IMPORT | \
                    CO_FUTURE_WITH_STATEMENT | CO_FUTURE_PRINT_FUNCTION | \
                    CO_FUTURE_UNICODE_LITERALS | CO_FUTURE_BARRY_AS_BDFL | \
-                   CO_FUTURE_GENERATOR_STOP)
+                   CO_FUTURE_GENERATOR_STOP | CO_FUTURE_ANNOTATIONS)
 #define PyCF_MASK_OBSOLETE (CO_NESTED)
 #define PyCF_SOURCE_IS_UTF8  0x0100
 #define PyCF_DONT_IMPLY_DEDENT 0x0200
@@ -45,6 +45,7 @@ typedef struct {
 #define FUTURE_UNICODE_LITERALS "unicode_literals"
 #define FUTURE_BARRY_AS_BDFL "barry_as_FLUFL"
 #define FUTURE_GENERATOR_STOP "generator_stop"
+#define FUTURE_ANNOTATIONS "annotations"
 
 struct _mod; /* Declare the existence of this type */
 #define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)
diff --git a/Lib/__future__.py b/Lib/__future__.py
index 63b2be3524f..ce8bed7a643 100644
--- a/Lib/__future__.py
+++ b/Lib/__future__.py
@@ -57,13 +57,14 @@
     "unicode_literals",
     "barry_as_FLUFL",
     "generator_stop",
+    "annotations",
 ]
 
 __all__ = ["all_feature_names"] + all_feature_names
 
-# The CO_xxx symbols are defined here under the same names used by
-# compile.h, so that an editor search will find them here.  However,
-# they're not exported in __all__, because they don't really belong to
+# The CO_xxx symbols are defined here under the same names defined in
+# code.h and used by compile.h, so that an editor search will find them here.
+# However, they're not exported in __all__, because they don't really belong to
 # this module.
 CO_NESTED            = 0x0010   # nested_scopes
 CO_GENERATOR_ALLOWED = 0        # generators (obsolete, was 0x1000)
@@ -74,6 +75,7 @@
 CO_FUTURE_UNICODE_LITERALS = 0x20000 # unicode string literals
 CO_FUTURE_BARRY_AS_BDFL = 0x40000
 CO_FUTURE_GENERATOR_STOP  = 0x80000 # StopIteration becomes RuntimeError in generators
+CO_FUTURE_ANNOTATIONS     = 0x100000  # annotations become strings at runtime
 
 class _Feature:
     def __init__(self, optionalRelease, mandatoryRelease, compiler_flag):
@@ -132,9 +134,13 @@ def __repr__(self):
                             CO_FUTURE_UNICODE_LITERALS)
 
 barry_as_FLUFL = _Feature((3, 1, 0, "alpha", 2),
-                         (3, 9, 0, "alpha", 0),
-                         CO_FUTURE_BARRY_AS_BDFL)
+                          (3, 9, 0, "alpha", 0),
+                          CO_FUTURE_BARRY_AS_BDFL)
 
 generator_stop = _Feature((3, 5, 0, "beta", 1),
-                         (3, 7, 0, "alpha", 0),
-                         CO_FUTURE_GENERATOR_STOP)
+                          (3, 7, 0, "alpha", 0),
+                          CO_FUTURE_GENERATOR_STOP)
+
+annotations = _Feature((3, 7, 0, "beta", 1),
+                       (4, 0, 0, "alpha", 0),
+                       CO_FUTURE_ANNOTATIONS)
diff --git a/Lib/test/test_future.py b/Lib/test/test_future.py
index 2f1c410b1bf..29c4632e7e5 100644
--- a/Lib/test/test_future.py
+++ b/Lib/test/test_future.py
@@ -1,7 +1,9 @@
 # Test various flavors of legal and illegal future statements
 
+from functools import partial
 import unittest
 from test import support
+from textwrap import dedent
 import os
 import re
 
@@ -102,6 +104,167 @@ def test_unicode_literals_exec(self):
         exec("from __future__ import unicode_literals; x = ''", {}, scope)
         self.assertIsInstance(scope["x"], str)
 
+class AnnotationsFutureTestCase(unittest.TestCase):
+    template = dedent(
+        """
+        from __future__ import annotations
+        def f() -> {ann}:
+            ...
+        def g(arg: {ann}) -> None:
+            ...
+        var: {ann}
+        var2: {ann} = None
+        """
+    )
+
+    def getActual(self, annotation):
+        scope = {}
+        exec(self.template.format(ann=annotation), {}, scope)
+        func_ret_ann = scope['f'].__annotations__['return']
+        func_arg_ann = scope['g'].__annotations__['arg']
+        var_ann1 = scope['__annotations__']['var']
+        var_ann2 = scope['__annotations__']['var2']
+        self.assertEqual(func_ret_ann, func_arg_ann)
+        self.assertEqual(func_ret_ann, var_ann1)
+        self.assertEqual(func_ret_ann, var_ann2)
+        return func_ret_ann
+
+    def assertAnnotationEqual(
+        self, annotation, expected=None, drop_parens=False, is_tuple=False,
+    ):
+        actual = self.getActual(annotation)
+        if expected is None:
+            expected = annotation if not is_tuple else annotation[1:-1]
+        if drop_parens:
+            self.assertNotEqual(actual, expected)
+            actual = actual.replace("(", "").replace(")", "")
+
+        self.assertEqual(actual, expected)
+
+    def test_annotations(self):
+        eq = self.assertAnnotationEqual
+        eq('...')
+        eq("'some_string'")
+        eq("b'\\xa3'")
+        eq('Name')
+        eq('None')
+        eq('True')
+        eq('False')
+        eq('1')
+        eq('1.0')
+        eq('1j')
+        eq('True or False')
+        eq('True or False or None')
+        eq('True and False')
+        eq('True and False and None')
+        eq('(Name1 and Name2) or Name3')
+        eq('Name1 or (Name2 and Name3)')
+        eq('(Name1 and Name2) or (Name3 and Name4)')
+        eq('Name1 or (Name2 and Name3) or Name4')
+        eq('v1 << 2')
+        eq('1 >> v2')
+        eq(r'1 % finished')
+        eq('((1 + v2) - (v3 * 4)) ^ (((5 ** v6) / 7) // 8)')
+        eq('not great')
+        eq('~great')
+        eq('+value')
+        eq('-1')
+        eq('(~int) and (not ((v1 ^ (123 + v2)) | True))')
+        eq('lambda arg: None')
+        eq('lambda a=True: a')
+        eq('lambda a, b, c=True: a')
+        eq("lambda a, b, c=True, *, d=(1 << v2), e='str': a")
+        eq("lambda a, b, c=True, *vararg, d=(v1 << 2), e='str', **kwargs: a + b")
+        eq('1 if True else 2')
+        eq('(str or None) if True else (str or bytes or None)')
+        eq('(str or None) if (1 if True else 2) else (str or bytes or None)')
+        eq("{'2.7': dead, '3.7': (long_live or die_hard)}")
+        eq("{'2.7': dead, '3.7': (long_live or die_hard), **{'3.6': verygood}}")
+        eq("{**a, **b, **c}")
+        eq("{'2.7', '3.6', '3.7', '3.8', '3.9', ('4.0' if gilectomy else '3.10')}")
+        eq("({'a': 'b'}, (True or False), (+value), 'string', b'bytes') or None")
+        eq("()")
+        eq("(1,)")
+        eq("(1, 2)")
+        eq("(1, 2, 3)")
+        eq("[]")
+        eq("[1, 2, 3, 4, 5, 6, 7, 8, 9, (10 or A), (11 or B), (12 or C)]")
+        eq("{i for i in (1, 2, 3)}")
+        eq("{(i ** 2) for i in (1, 2, 3)}")
+        eq("{(i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c'))}")
+        eq("{((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3)}")
+        eq("[i for i in (1, 2, 3)]")
+        eq("[(i ** 2) for i in (1, 2, 3)]")
+        eq("[(i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c'))]")
+        eq("[((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3)]")
+        eq(r"{i: 0 for i in (1, 2, 3)}")
+        eq("{i: j for i, j in ((1, 'a'), (2, 'b'), (3, 'c'))}")
+        eq("Python3 > Python2 > COBOL")
+        eq("Life is Life")
+        eq("call()")
+        eq("call(arg)")
+        eq("call(kwarg='hey')")
+        eq("call(arg, kwarg='hey')")
+        eq("call(arg, another, kwarg='hey', **kwargs)")
+        eq("lukasz.langa.pl")
+        eq("call.me(maybe)")
+        eq("1 .real")
+        eq("1.0 .real")
+        eq("....__class__")
+        eq("list[str]")
+        eq("dict[str, int]")
+        eq("tuple[str, ...]")
+        eq("tuple[str, int, float, dict[str, int]]")
+        eq("slice[0]")
+        eq("slice[0:1]")
+        eq("slice[0:1:2]")
+        eq("slice[:]")
+        eq("slice[:-1]")
+        eq("slice[1:]")
+        eq("slice[::-1]")
+        eq('(str or None) if (sys.version_info[0] > (3,)) else (str or bytes or None)')
+        eq("f'f-string without formatted values is just a string'")
+        eq("f'{{NOT a formatted value}}'")
+        eq("f'some f-string with {a} {few():.2f} {formatted.values!r}'")
+        eq('''f"{f'{nested} inner'} outer"''')
+        eq("f'space between opening braces: { {a for a in (1, 2, 3)}}'")
+
+    def test_annotations_inexact(self):
+        """Source formatting is not always preserved
+
+        This is due to reconstruction from AST.  We *need to* put the parens
+        in nested expressions because we don't know if the source code
+        had them in the first place or not.
+        """
+        eq = partial(self.assertAnnotationEqual, drop_parens=True)
+        eq('Name1 and Name2 or Name3')
+        eq('Name1 or Name2 and Name3')
+        eq('Name1 and Name2 or Name3 and Name4')
+        eq('Name1 or Name2 and Name3 or Name4')
+        eq('1 + v2 - v3 * 4 ^ v5 ** 6 / 7 // 8')
+        eq('~int and not v1 ^ 123 + v2 | True')
+        eq('str or None if True else str or bytes or None')
+        eq("{'2.7': dead, '3.7': long_live or die_hard}")
+        eq("{'2.7', '3.6', '3.7', '3.8', '3.9', '4.0' if gilectomy else '3.10'}")
+        eq("[1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or A, 11 or B, 12 or C]")
+        # Consequently, we always drop unnecessary parens if they were given in
+        # the outer scope:
+        some_name = self.getActual("(SomeName)")
+        self.assertEqual(some_name, 'SomeName')
+        # Interestingly, in the case of tuples (and generator expressions) the
+        # parens are *required* by the Python syntax in the annotation context.
+        # But there's no point storing that detail in __annotations__ so we're
+        # fine with the parens-less form.
+        eq = partial(self.assertAnnotationEqual, is_tuple=True)
+        eq("(Good, Bad, Ugly)")
+        eq("(i for i in (1, 2, 3))")
+        eq("((i ** 2) for i in (1, 2, 3))")
+        eq("((i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c')))")
+        eq("(((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3))")
+        eq("(*starred)")
+        eq('(yield from outside_of_generator)')
+        eq('(yield)')
+        eq('(await some.complicated[0].call(with_args=(True or (1 is not 1))))')
 
 
 if __name__ == "__main__":
diff --git a/Makefile.pre.in b/Makefile.pre.in
index 162006a2f9c..f58b7153788 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -325,6 +325,7 @@ PYTHON_OBJS=	\
 		Python/asdl.o \
 		Python/ast.o \
 		Python/ast_opt.o \
+		Python/ast_unparse.o \
 		Python/bltinmodule.o \
 		Python/ceval.o \
 		Python/compile.o \
@@ -840,7 +841,7 @@ regen-opcode:
 		$(srcdir)/Include/opcode.h.new
 	$(UPDATE_FILE) $(srcdir)/Include/opcode.h $(srcdir)/Include/opcode.h.new
 
-Python/compile.o Python/symtable.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
+Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
 
 Python/getplatform.o: $(srcdir)/Python/getplatform.c
 		$(CC) -c $(PY_CORE_CFLAGS) -DPLATFORM='"$(MACHDEP)"' -o $@ $(srcdir)/Python/getplatform.c
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
index fbcd0512139..f3094550869 100644
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -358,6 +358,7 @@
     <ClCompile Include="..\Python\asdl.c" />
     <ClCompile Include="..\Python\ast.c" />
     <ClCompile Include="..\Python\ast_opt.c" />
+    <ClCompile Include="..\Python\ast_unparse.c" />
     <ClCompile Include="..\Python\bltinmodule.c" />
     <ClCompile Include="..\Python\bootstrap_hash.c" />
     <ClCompile Include="..\Python\ceval.c" />
diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters
index a10686c194b..5f980388e5f 100644
--- a/PCbuild/pythoncore.vcxproj.filters
+++ b/PCbuild/pythoncore.vcxproj.filters
@@ -839,6 +839,9 @@
     <ClCompile Include="..\Python\ast_opt.c">
       <Filter>Python</Filter>
     </ClCompile>
+    <ClCompile Include="..\Python\ast_unparse.c">
+      <Filter>Python</Filter>
+    </ClCompile>
     <ClCompile Include="..\Python\bltinmodule.c">
       <Filter>Python</Filter>
     </ClCompile>
diff --git a/Python/ast_unparse.c b/Python/ast_unparse.c
new file mode 100644
index 00000000000..ef9e948dc82
--- /dev/null
+++ b/Python/ast_unparse.c
@@ -0,0 +1,1163 @@
+#include <stdbool.h>
+#include "Python.h"
+#include "Python-ast.h"
+
+static PyObject *_str_open_br;
+static PyObject *_str_dbl_open_br;
+static PyObject *_str_close_br;
+static PyObject *_str_dbl_close_br;
+
+/* Forward declarations for recursion via helper functions. */
+static PyObject *
+expr_as_unicode(expr_ty e, bool omit_parens);
+static int
+append_ast_expr(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens);
+static int
+append_joinedstr(_PyUnicodeWriter *writer, expr_ty e, bool is_format_spec);
+static int
+append_formattedvalue(_PyUnicodeWriter *writer, expr_ty e, bool is_format_spec);
+
+static int
+append_charp(_PyUnicodeWriter *writer, const char *charp)
+{
+        return _PyUnicodeWriter_WriteASCIIString(writer, charp, -1);
+}
+
+static int
+append_repr(_PyUnicodeWriter *writer, PyObject *obj)
+{
+    int ret;
+    PyObject *repr;
+    repr = PyObject_Repr(obj);
+    if (!repr) {
+        return -1;
+    }
+    ret = _PyUnicodeWriter_WriteStr(writer, repr);
+    Py_DECREF(repr);
+    return ret;
+}
+
+static int
+append_ast_boolop(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    Py_ssize_t i, value_count;
+    asdl_seq *values;
+
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    values = e->v.BoolOp.values;
+    value_count = asdl_seq_LEN(values) - 1;
+    assert(value_count >= 0);
+
+    if (-1 == append_ast_expr(writer,
+                              (expr_ty)asdl_seq_GET(values, 0),
+                              false)) {
+        return -1;
+    }
+
+    const char *op = (e->v.BoolOp.op == And) ? " and " : " or ";
+    for (i = 1; i <= value_count; ++i) {
+        if (-1 == append_charp(writer, op)) {
+            return -1;
+        }
+
+        if (-1 == append_ast_expr(writer,
+                                  (expr_ty)asdl_seq_GET(values, i),
+                                  false)) {
+            return -1;
+        }
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_binop(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    const char *op;
+
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.BinOp.left, false)) {
+        return -1;
+    }
+
+    switch(e->v.BinOp.op) {
+    case Add: op = " + "; break;
+    case Sub: op = " - "; break;
+    case Mult: op = " * "; break;
+    case MatMult: op = " @ "; break;
+    case Div: op = " / "; break;
+    case Mod: op = " % "; break;
+    case LShift: op = " << "; break;
+    case RShift: op = " >> "; break;
+    case BitOr: op = " | "; break;
+    case BitXor: op = " ^ "; break;
+    case BitAnd: op = " & "; break;
+    case FloorDiv: op = " // "; break;
+    case Pow: op = " ** "; break;
+    }
+
+    if (-1 == append_charp(writer, op)) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.BinOp.right, false)) {
+        return -1;
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_unaryop(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    const char *op;
+
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    switch(e->v.UnaryOp.op) {
+    case Invert: op = "~"; break;
+    case Not: op = "not "; break;
+    case UAdd: op = "+"; break;
+    case USub: op = "-"; break;
+    }
+
+    if (-1 == append_charp(writer, op)) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.UnaryOp.operand, false)) {
+        return -1;
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_arg(_PyUnicodeWriter *writer, arg_ty arg)
+{
+    if (-1 == _PyUnicodeWriter_WriteStr(writer, arg->arg)) {
+        return -1;
+    }
+    if (arg->annotation) {
+        if (-1 == append_charp(writer, ": ")) {
+            return -1;
+        }
+        if (-1 == append_ast_expr(writer, arg->annotation, true)) {
+            return -1;
+        }
+    }
+    return 0;
+}
+
+static int
+append_ast_args(_PyUnicodeWriter *writer, arguments_ty args)
+{
+    bool first;
+    Py_ssize_t i, di, arg_count, default_count;
+    arg_ty arg;
+    expr_ty default_;
+
+    first = true;
+
+    /* positional arguments with defaults */
+    arg_count = asdl_seq_LEN(args->args);
+    default_count = asdl_seq_LEN(args->defaults);
+    for (i = 0; i < arg_count; i++) {
+        if (first) {
+            first = false;
+        }
+        else if (-1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        arg = (arg_ty)asdl_seq_GET(args->args, i);
+        if (-1 == append_ast_arg(writer, arg)) {
+            return -1;
+        }
+
+        di = i - arg_count + default_count;
+        if (di >= 0) {
+            if (-1 == append_charp(writer, "=")) {
+                return -1;
+            }
+            default_ = (expr_ty)asdl_seq_GET(args->defaults, di);
+            if (-1 == append_ast_expr(writer, default_, false)) {
+                return -1;
+            }
+        }
+    }
+
+    /* vararg, or bare '*' if no varargs but keyword-only arguments present */
+    if (args->vararg || args->kwonlyargs) {
+        if (first) {
+            first = false;
+        }
+        else if (-1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        if (-1 == append_charp(writer, "*")) {
+            return -1;
+        }
+
+        if (args->vararg) {
+            if (-1 == append_ast_arg(writer, args->vararg)) {
+                return -1;
+            }
+        }
+    }
+
+    /* keyword-only arguments */
+    arg_count = asdl_seq_LEN(args->kwonlyargs);
+    default_count = asdl_seq_LEN(args->kw_defaults);
+    for (i = 0; i < arg_count; i++) {
+        if (first) {
+            first = false;
+        }
+        else if (-1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        arg = (arg_ty)asdl_seq_GET(args->kwonlyargs, i);
+        if (-1 == append_ast_arg(writer, arg)) {
+            return -1;
+        }
+
+        di = i - arg_count + default_count;
+        if (di >= 0) {
+            if (-1 == append_charp(writer, "=")) {
+                return -1;
+            }
+            default_ = (expr_ty)asdl_seq_GET(args->kw_defaults, di);
+            if (-1 == append_ast_expr(writer, default_, false)) {
+                return -1;
+            }
+        }
+    }
+
+    /* **kwargs */
+    if (args->kwarg) {
+        if (first) {
+            first = false;
+        }
+        else if (-1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        if (-1 == append_charp(writer, "**")) {
+            return -1;
+        }
+
+        if (-1 == append_ast_arg(writer, args->kwarg)) {
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+static int
+append_ast_lambda(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, "lambda ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_args(writer, e->v.Lambda.args)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, ": ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.Lambda.body, true)) {
+        return -1;
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_ifexp(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.IfExp.body, false)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, " if ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.IfExp.test, false)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, " else ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.IfExp.orelse, false)) {
+        return -1;
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_dict(_PyUnicodeWriter *writer, expr_ty e)
+{
+    Py_ssize_t i, value_count;
+    expr_ty key_node, value_node;
+
+    if (-1 == append_charp(writer, "{")) {
+        return -1;
+    }
+
+    value_count = asdl_seq_LEN(e->v.Dict.values);
+
+    for (i = 0; i < value_count; i++) {
+        if (i > 0 && -1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+        key_node = (expr_ty)asdl_seq_GET(e->v.Dict.keys, i);
+        if (key_node != NULL) {
+            if (-1 == append_ast_expr(writer, key_node, false)) {
+                return -1;
+            }
+
+            if (-1 == append_charp(writer, ": ")) {
+                return -1;
+            }
+        }
+        else if (-1 == append_charp(writer, "**")) {
+            return -1;
+        }
+
+        value_node = (expr_ty)asdl_seq_GET(e->v.Dict.values, i);
+        if (-1 == append_ast_expr(writer, value_node, false)) {
+            return -1;
+        }
+    }
+
+    return append_charp(writer, "}");
+}
+
+static int
+append_ast_set(_PyUnicodeWriter *writer, expr_ty e)
+{
+    Py_ssize_t i, elem_count;
+    expr_ty elem_node;
+
+    if (-1 == append_charp(writer, "{")) {
+        return -1;
+    }
+
+    elem_count = asdl_seq_LEN(e->v.Set.elts);
+    for (i = 0; i < elem_count; i++) {
+        if (i > 0 && -1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        elem_node = (expr_ty)asdl_seq_GET(e->v.Set.elts, i);
+        if (-1 == append_ast_expr(writer, elem_node, false)) {
+            return -1;
+        }
+    }
+
+    return append_charp(writer, "}");
+}
+
+static int
+append_ast_list(_PyUnicodeWriter *writer, expr_ty e)
+{
+    Py_ssize_t i, elem_count;
+    expr_ty elem_node;
+
+    if (-1 == append_charp(writer, "[")) {
+        return -1;
+    }
+
+    elem_count = asdl_seq_LEN(e->v.List.elts);
+    for (i = 0; i < elem_count; i++) {
+        if (i > 0 && -1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+        elem_node = (expr_ty)asdl_seq_GET(e->v.List.elts, i);
+        if (-1 == append_ast_expr(writer, elem_node, false)) {
+            return -1;
+        }
+    }
+
+    return append_charp(writer, "]");
+}
+
+static int
+append_ast_tuple(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    Py_ssize_t i, elem_count;
+    expr_ty elem_node;
+
+    elem_count = asdl_seq_LEN(e->v.Tuple.elts);
+
+    if (!omit_parens || elem_count < 2) {
+        if (-1 == append_charp(writer, "(")) {
+            return -1;
+        }
+    }
+
+    for (i = 0; i < elem_count; i++) {
+        if ((i > 0 || elem_count == 1) && -1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+        elem_node = (expr_ty)asdl_seq_GET(e->v.Tuple.elts, i);
+        if (-1 == append_ast_expr(writer, elem_node, false)) {
+            return -1;
+        }
+    }
+
+    if (!omit_parens || elem_count < 2) {
+        return append_charp(writer, ")");
+    }
+
+    return 0;
+}
+
+static int
+append_ast_comprehension(_PyUnicodeWriter *writer, comprehension_ty gen)
+{
+    Py_ssize_t i, if_count;
+
+    if (-1 == append_charp(writer, gen->is_async ? " async for " : " for ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, gen->target, true)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, " in ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, gen->iter, false)) {
+        return -1;
+    }
+
+    if_count = asdl_seq_LEN(gen->ifs);
+    for (i = 0; i < if_count; i++) {
+        if (-1 == append_charp(writer, " if ")) {
+            return -1;
+        }
+
+        if (-1 == append_ast_expr(writer,
+                                  (expr_ty)asdl_seq_GET(gen->ifs, i),
+                                  false)) {
+            return -1;
+        }
+    }
+    return 0;
+}
+
+static int
+append_ast_comprehensions(_PyUnicodeWriter *writer, asdl_seq *comprehensions)
+{
+    Py_ssize_t i, gen_count;
+    comprehension_ty comp_node;
+    gen_count = asdl_seq_LEN(comprehensions);
+
+    for (i = 0; i < gen_count; i++) {
+        comp_node = (comprehension_ty)asdl_seq_GET(comprehensions, i);
+        if (-1 == append_ast_comprehension(writer, comp_node)) {
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+static int
+append_ast_genexp(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.GeneratorExp.elt, false)) {
+        return -1;
+    }
+
+    if (-1 == append_ast_comprehensions(writer, e->v.GeneratorExp.generators)) {
+        return -1;
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_listcomp(_PyUnicodeWriter *writer, expr_ty e)
+{
+    if (-1 == append_charp(writer, "[")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.ListComp.elt, false)) {
+        return -1;
+    }
+
+    if (-1 == append_ast_comprehensions(writer, e->v.ListComp.generators)) {
+        return -1;
+    }
+
+    return append_charp(writer, "]");
+}
+
+static int
+append_ast_setcomp(_PyUnicodeWriter *writer, expr_ty e)
+{
+    if (-1 == append_charp(writer, "{")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.SetComp.elt, false)) {
+        return -1;
+    }
+
+    if (-1 == append_ast_comprehensions(writer, e->v.SetComp.generators)) {
+        return -1;
+    }
+
+    return append_charp(writer, "}");
+}
+
+static int
+append_ast_dictcomp(_PyUnicodeWriter *writer, expr_ty e)
+{
+    if (-1 == append_charp(writer, "{")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.DictComp.key, false)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, ": ")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_expr(writer, e->v.DictComp.value, false)) {
+        return -1;
+    }
+
+    if (-1 == append_ast_comprehensions(writer, e->v.DictComp.generators)) {
+        return -1;
+    }
+
+    return append_charp(writer, "}");
+}
+
+static int
+append_ast_compare(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    const char *op;
+    Py_ssize_t i, comparator_count;
+    asdl_seq *comparators;
+    asdl_int_seq *ops;
+
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    comparators = e->v.Compare.comparators;
+    ops = e->v.Compare.ops;
+    comparator_count = asdl_seq_LEN(comparators);
+    assert(comparator_count > 0);
+    assert(comparator_count == asdl_seq_LEN(ops));
+
+    if (-1 == append_ast_expr(writer, e->v.Compare.left, false)) {
+        return -1;
+    }
+
+    for (i = 0; i < comparator_count; i++) {
+        switch ((cmpop_ty)asdl_seq_GET(ops, i)) {
+        case Eq:
+            op = " == ";
+            break;
+        case NotEq:
+            op = " != ";
+            break;
+        case Lt:
+            op = " < ";
+            break;
+        case LtE:
+            op = " <= ";
+            break;
+        case Gt:
+            op = " > ";
+            break;
+        case GtE:
+            op = " >= ";
+            break;
+        case Is:
+            op = " is ";
+            break;
+        case IsNot:
+            op = " is not ";
+            break;
+        case In:
+            op = " in ";
+            break;
+        case NotIn:
+            op = " not in ";
+            break;
+        default:
+            PyErr_SetString(PyExc_SystemError,
+                            "unexpected comparison kind");
+            return -1;
+        }
+
+        if (-1 == append_charp(writer, op)) {
+            return -1;
+        }
+
+        if (-1 == append_ast_expr(writer,
+                                  (expr_ty)asdl_seq_GET(comparators, i),
+                                  false)) {
+            return -1;
+        }
+    }
+
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_keyword(_PyUnicodeWriter *writer, keyword_ty kw)
+{
+    if (kw->arg == NULL) {
+        if (-1 == append_charp(writer, "**")) {
+            return -1;
+        }
+    }
+    else {
+        if (-1 == _PyUnicodeWriter_WriteStr(writer, kw->arg)) {
+            return -1;
+        }
+
+        if (-1 == append_charp(writer, "=")) {
+            return -1;
+        }
+    }
+
+    return append_ast_expr(writer, kw->value, false);
+}
+
+static int
+append_ast_call(_PyUnicodeWriter *writer, expr_ty e)
+{
+    bool first;
+    Py_ssize_t i, arg_count, kw_count;
+    expr_ty expr;
+    keyword_ty kw;
+
+    if (-1 == append_ast_expr(writer, e->v.Call.func, false)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    first = true;
+    arg_count = asdl_seq_LEN(e->v.Call.args);
+    for (i = 0; i < arg_count; i++) {
+        if (first) {
+            first = false;
+        }
+        else if (-1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        expr = (expr_ty)asdl_seq_GET(e->v.Call.args, i);
+        if (-1 == append_ast_expr(writer, expr, false)) {
+            return -1;
+        }
+    }
+
+    kw_count = asdl_seq_LEN(e->v.Call.keywords);
+    for (i = 0; i < kw_count; i++) {
+        if (first) {
+            first = false;
+        }
+        else if (-1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+
+        kw = (keyword_ty)asdl_seq_GET(e->v.Call.keywords, i);
+        if (-1 == append_ast_keyword(writer, kw)) {
+            return -1;
+        }
+    }
+
+    return append_charp(writer, ")");
+}
+
+static PyObject *
+escape_braces(PyObject *orig)
+{
+    PyObject *temp;
+    PyObject *result;
+    temp = PyUnicode_Replace(orig, _str_open_br, _str_dbl_open_br, -1);
+    if (!temp) {
+        return NULL;
+    }
+    result = PyUnicode_Replace(temp, _str_close_br, _str_dbl_close_br, -1);
+    Py_DECREF(temp);
+    return result;
+}
+
+static int
+append_fstring_unicode(_PyUnicodeWriter *writer, PyObject *unicode)
+{
+    PyObject *escaped;
+    int result = -1;
+    escaped = escape_braces(unicode);
+    if (escaped) {
+        result = _PyUnicodeWriter_WriteStr(writer, escaped);
+        Py_DECREF(escaped);
+    }
+    return result;
+}
+
+static int
+append_fstring_element(_PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
+{
+    switch (e->kind) {
+    case Constant_kind:
+        return append_fstring_unicode(writer, e->v.Constant.value);
+    case Str_kind:
+        return append_fstring_unicode(writer, e->v.Str.s);
+    case JoinedStr_kind:
+        return append_joinedstr(writer, e, is_format_spec);
+    case FormattedValue_kind:
+        return append_formattedvalue(writer, e, is_format_spec);
+    default:
+        PyErr_SetString(PyExc_SystemError,
+                        "unknown expression kind inside f-string");
+        return -1;
+    }
+}
+
+/* Build body separately to enable wrapping the entire stream of Strs,
+   Constants and FormattedValues in one opening and one closing quote. */
+static PyObject *
+build_fstring_body(asdl_seq *values, bool is_format_spec)
+{
+    Py_ssize_t i, value_count;
+    _PyUnicodeWriter body_writer;
+    _PyUnicodeWriter_Init(&body_writer);
+    body_writer.min_length = 256;
+    body_writer.overallocate = 1;
+
+    value_count = asdl_seq_LEN(values) - 1;
+    assert(value_count >= 0);
+    for (i = 0; i <= value_count; ++i) {
+        if (-1 == append_fstring_element(&body_writer,
+                                         (expr_ty)asdl_seq_GET(values, i),
+                                         is_format_spec
+                                         )) {
+            _PyUnicodeWriter_Dealloc(&body_writer);
+            return NULL;
+        }
+    }
+
+    return _PyUnicodeWriter_Finish(&body_writer);
+}
+
+static int
+append_joinedstr(_PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
+{
+    int result = -1;
+    PyObject *body = build_fstring_body(e->v.JoinedStr.values, is_format_spec);
+    if (!body) {
+        return -1;
+    }
+
+    if (!is_format_spec) {
+        if (-1 != append_charp(writer, "f") &&
+            -1 != append_repr(writer, body))
+        {
+            result = 0;
+        }
+    }
+    else {
+        result = _PyUnicodeWriter_WriteStr(writer, body);
+    }
+    Py_DECREF(body);
+    return result;
+}
+
+static int
+append_formattedvalue(_PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
+{
+    char *conversion;
+    char *outer_brace = "{";
+    PyObject *temp_fv_str = expr_as_unicode(e->v.FormattedValue.value, true);
+    if (!temp_fv_str) {
+        return -1;
+    }
+    if (PyUnicode_Find(temp_fv_str, _str_open_br, 0, 1, 1) == 0) {
+        /* Expression starts with a brace, split it with a space from the outer
+           one. */
+        outer_brace = "{ ";
+    }
+    if (-1 == append_charp(writer, outer_brace)) {
+        Py_DECREF(temp_fv_str);
+        return -1;
+    }
+    if (-1 == _PyUnicodeWriter_WriteStr(writer, temp_fv_str)) {
+        Py_DECREF(temp_fv_str);
+        return -1;
+    }
+    Py_DECREF(temp_fv_str);
+
+    if (e->v.FormattedValue.conversion > 0) {
+        switch (e->v.FormattedValue.conversion) {
+        case 97:
+            conversion = "!a";
+            break;
+        case 114:
+            conversion = "!r";
+            break;
+        case 115:
+            conversion = "!s";
+            break;
+        default:
+            PyErr_SetString(PyExc_SystemError,
+                            "unknown f-value conversion kind");
+            return -1;
+        }
+        if (-1 == append_charp(writer, conversion)) {
+            return -1;
+        }
+    }
+    if (e->v.FormattedValue.format_spec > 0) {
+        if (-1 == _PyUnicodeWriter_WriteASCIIString(writer, ":", 1) ||
+            -1 == append_fstring_element(writer,
+                                         e->v.FormattedValue.format_spec,
+                                         true
+                                        ))
+        {
+            return -1;
+        }
+    }
+    return append_charp(writer, "}");
+}
+
+static int
+append_ast_attribute(_PyUnicodeWriter *writer, expr_ty e)
+{
+    const char *period;
+    if (-1 == append_ast_expr(writer, e->v.Attribute.value, false)) {
+        return -1;
+    }
+
+    /* Special case: integers require a space for attribute access to be
+       unambiguous.  Floats and complex numbers don't but work with it, too. */
+    if (e->v.Attribute.value->kind == Num_kind ||
+        e->v.Attribute.value->kind == Constant_kind)
+    {
+        period = " .";
+    }
+    else {
+        period = ".";
+    }
+    if (-1 == append_charp(writer, period)) {
+        return -1;
+    }
+
+    return _PyUnicodeWriter_WriteStr(writer, e->v.Attribute.attr);
+}
+
+static int
+append_ast_simple_slice(_PyUnicodeWriter *writer, slice_ty slice)
+{
+    if (slice->v.Slice.lower) {
+        if (-1 == append_ast_expr(writer, slice->v.Slice.lower, false)) {
+            return -1;
+        }
+    }
+
+    if (-1 == append_charp(writer, ":")) {
+        return -1;
+    }
+
+    if (slice->v.Slice.upper) {
+        if (-1 == append_ast_expr(writer, slice->v.Slice.upper, false)) {
+            return -1;
+        }
+    }
+
+    if (slice->v.Slice.step) {
+        if (-1 == append_charp(writer, ":")) {
+            return -1;
+        }
+        if (-1 == append_ast_expr(writer, slice->v.Slice.step, false)) {
+            return -1;
+        }
+    }
+    return 0;
+}
+
+static int
+append_ast_ext_slice(_PyUnicodeWriter *writer, slice_ty slice)
+{
+    Py_ssize_t i, dims_count;
+    dims_count = asdl_seq_LEN(slice->v.ExtSlice.dims);
+    for (i = 0; i < dims_count; i++) {
+        if (i > 0 && -1 == append_charp(writer, ", ")) {
+            return -1;
+        }
+        if (-1 == append_ast_expr(writer,
+                                  (expr_ty)asdl_seq_GET(slice->v.ExtSlice.dims, i),
+                                  false)) {
+            return -1;
+        }
+    }
+    return 0;
+}
+
+static int
+append_ast_slice(_PyUnicodeWriter *writer, slice_ty slice, bool omit_parens)
+{
+    switch(slice->kind) {
+    case Slice_kind:
+        return append_ast_simple_slice(writer, slice);
+    case ExtSlice_kind:
+        return append_ast_ext_slice(writer, slice);
+    case Index_kind:
+        return append_ast_expr(writer, slice->v.Index.value, omit_parens);
+    default:
+        PyErr_SetString(PyExc_SystemError,
+                        "unexpected slice kind");
+        return -1;
+    }
+}
+
+static int
+append_ast_subscript(_PyUnicodeWriter *writer, expr_ty e)
+{
+    if (-1 == append_ast_expr(writer, e->v.Subscript.value, false)) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, "[")) {
+        return -1;
+    }
+
+    if (-1 == append_ast_slice(writer, e->v.Subscript.slice, true)) {
+        return -1;
+    }
+
+    return append_charp(writer, "]");
+}
+
+static int
+append_ast_starred(_PyUnicodeWriter *writer, expr_ty e)
+{
+    if (-1 == append_charp(writer, "*")) {
+        return -1;
+    }
+
+    return append_ast_expr(writer, e->v.Starred.value, false);
+}
+
+static int
+append_ast_yield(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, e->v.Yield.value ? "yield " : "yield")) {
+        return -1;
+    }
+
+    if (e->v.Yield.value) {
+        if (-1 == append_ast_expr(writer, e->v.Yield.value, false)) {
+            return -1;
+        }
+    }
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_yield_from(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer,
+                           e->v.YieldFrom.value ? "yield from " : "yield from")) {
+        return -1;
+    }
+
+    if (e->v.YieldFrom.value) {
+        if (-1 == append_ast_expr(writer, e->v.YieldFrom.value, false)) {
+            return -1;
+        }
+    }
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_await(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    if (!omit_parens && -1 == append_charp(writer, "(")) {
+        return -1;
+    }
+
+    if (-1 == append_charp(writer, e->v.Await.value ? "await " : "await")) {
+        return -1;
+    }
+
+    if (e->v.Await.value) {
+        if (-1 == append_ast_expr(writer, e->v.Await.value, false)) {
+            return -1;
+        }
+    }
+    return omit_parens ? 0 : append_charp(writer, ")");
+}
+
+static int
+append_ast_expr(_PyUnicodeWriter *writer, expr_ty e, bool omit_parens)
+{
+    switch (e->kind) {
+    case BoolOp_kind:
+        return append_ast_boolop(writer, e, omit_parens);
+    case BinOp_kind:
+        return append_ast_binop(writer, e, omit_parens);
+    case UnaryOp_kind:
+        return append_ast_unaryop(writer, e, omit_parens);
+    case Lambda_kind:
+        return append_ast_lambda(writer, e, omit_parens);
+    case IfExp_kind:
+        return append_ast_ifexp(writer, e, omit_parens);
+    case Dict_kind:
+        return append_ast_dict(writer, e);
+    case Set_kind:
+        return append_ast_set(writer, e);
+    case GeneratorExp_kind:
+        return append_ast_genexp(writer, e, omit_parens);
+    case ListComp_kind:
+        return append_ast_listcomp(writer, e);
+    case SetComp_kind:
+        return append_ast_setcomp(writer, e);
+    case DictComp_kind:
+        return append_ast_dictcomp(writer, e);
+    case Yield_kind:
+        return append_ast_yield(writer, e, omit_parens);
+    case YieldFrom_kind:
+        return append_ast_yield_from(writer, e, omit_parens);
+    case Await_kind:
+        return append_ast_await(writer, e, omit_parens);
+    case Compare_kind:
+        return append_ast_compare(writer, e, omit_parens);
+    case Call_kind:
+        return append_ast_call(writer, e);
+    case Constant_kind:
+        return append_repr(writer, e->v.Constant.value);
+    case Num_kind:
+        return append_repr(writer, e->v.Num.n);
+    case Str_kind:
+        return append_repr(writer, e->v.Str.s);
+    case JoinedStr_kind:
+        return append_joinedstr(writer, e, false);
+    case FormattedValue_kind:
+        return append_formattedvalue(writer, e, false);
+    case Bytes_kind:
+        return append_repr(writer, e->v.Bytes.s);
+    case Ellipsis_kind:
+        return append_charp(writer, "...");
+    case NameConstant_kind:
+        return append_repr(writer, e->v.NameConstant.value);
+    /* The following exprs can be assignment targets. */
+    case Attribute_kind:
+        return append_ast_attribute(writer, e);
+    case Subscript_kind:
+        return append_ast_subscript(writer, e);
+    case Starred_kind:
+        return append_ast_starred(writer, e);
+    case Name_kind:
+        return _PyUnicodeWriter_WriteStr(writer, e->v.Name.id);
+    /* child nodes of List and Tuple will have expr_context set */
+    case List_kind:
+        return append_ast_list(writer, e);
+    case Tuple_kind:
+        return append_ast_tuple(writer, e, omit_parens);
+    default:
+        PyErr_SetString(PyExc_SystemError,
+                        "unknown expression kind");
+        return -1;
+    }
+}
+
+static int
+maybe_init_static_strings()
+{
+    if (!_str_open_br &&
+        !(_str_open_br = PyUnicode_InternFromString("{"))) {
+        return -1;
+    }
+    if (!_str_dbl_open_br &&
+        !(_str_dbl_open_br = PyUnicode_InternFromString("{{"))) {
+        return -1;
+    }
+    if (!_str_close_br &&
+        !(_str_close_br = PyUnicode_InternFromString("}"))) {
+        return -1;
+    }
+    if (!_str_dbl_close_br &&
+        !(_str_dbl_close_br = PyUnicode_InternFromString("}}"))) {
+        return -1;
+    }
+    return 0;
+}
+
+static PyObject *
+expr_as_unicode(expr_ty e, bool omit_parens)
+{
+    _PyUnicodeWriter writer;
+    _PyUnicodeWriter_Init(&writer);
+    writer.min_length = 256;
+    writer.overallocate = 1;
+    if (-1 == maybe_init_static_strings() ||
+        -1 == append_ast_expr(&writer, e, omit_parens))
+    {
+        _PyUnicodeWriter_Dealloc(&writer);
+        return NULL;
+    }
+    return _PyUnicodeWriter_Finish(&writer);
+}
+
+PyObject *
+_PyAST_ExprAsUnicode(expr_ty e, bool omit_parens)
+{
+    return expr_as_unicode(e, omit_parens);
+}
diff --git a/Python/compile.c b/Python/compile.c
index 31efc28da43..3e8323b933f 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -1699,13 +1699,30 @@ compiler_visit_kwonlydefaults(struct compiler *c, asdl_seq *kwonlyargs,
     return 0;
 }
 
+static int
+compiler_visit_annexpr(struct compiler *c, expr_ty annotation)
+{
+    PyObject *ann_as_str;
+    ann_as_str = _PyAST_ExprAsUnicode(annotation, 1);
+    if (!ann_as_str) {
+        return 0;
+    }
+    ADDOP_N(c, LOAD_CONST, ann_as_str, consts);
+    return 1;
+}
+
 static int
 compiler_visit_argannotation(struct compiler *c, identifier id,
     expr_ty annotation, PyObject *names)
 {
     if (annotation) {
         PyObject *mangled;
-        VISIT(c, expr, annotation);
+        if (c->c_future->ff_features & CO_FUTURE_ANNOTATIONS) {
+            VISIT(c, annexpr, annotation)
+        }
+        else {
+            VISIT(c, expr, annotation);
+        }
         mangled = _Py_Mangle(c->u->u_private, id);
         if (!mangled)
             return 0;
@@ -4688,7 +4705,12 @@ compiler_annassign(struct compiler *c, stmt_ty s)
             if (!mangled) {
                 return 0;
             }
-            VISIT(c, expr, s->v.AnnAssign.annotation);
+            if (c->c_future->ff_features & CO_FUTURE_ANNOTATIONS) {
+                VISIT(c, annexpr, s->v.AnnAssign.annotation)
+            }
+            else {
+                VISIT(c, expr, s->v.AnnAssign.annotation);
+            }
             /* ADDOP_N decrefs its argument */
             ADDOP_N(c, STORE_ANNOTATION, mangled, names);
         }
diff --git a/Python/future.c b/Python/future.c
index 5cb21ac61c2..53faa6b9016 100644
--- a/Python/future.c
+++ b/Python/future.c
@@ -42,6 +42,8 @@ future_check_features(PyFutureFeatures *ff, stmt_ty s, PyObject *filename)
             ff->ff_features |= CO_FUTURE_BARRY_AS_BDFL;
         } else if (strcmp(feature, FUTURE_GENERATOR_STOP) == 0) {
             ff->ff_features |= CO_FUTURE_GENERATOR_STOP;
+        } else if (strcmp(feature, FUTURE_ANNOTATIONS) == 0) {
+            ff->ff_features |= CO_FUTURE_ANNOTATIONS;
         } else if (strcmp(feature, "braces") == 0) {
             PyErr_SetString(PyExc_SyntaxError,
                             "not a chance");


From webhook-mailer at python.org  Fri Jan 26 12:55:18 2018
From: webhook-mailer at python.org (Mariatta)
Date: Fri, 26 Jan 2018 17:55:18 -0000
Subject: [Python-checkins] [2.7] bpo-32640: Clarify the behavior of str.join
 and unicode object (GH-5333)
Message-ID: <mailman.64.1516989321.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/f5e8f71fe339fe5e47c4d24ac2e5177fa3c02922
commit: f5e8f71fe339fe5e47c4d24ac2e5177fa3c02922
branch: 2.7
author: Mariatta <Mariatta at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-26T09:55:15-08:00
summary:

[2.7] bpo-32640: Clarify the behavior of str.join and unicode object (GH-5333)

In str.join, if any of the iterable contains a Unicode object,
str.join will return a Unicode object.

A Type error will be raised if iterable contains values other
than a string or Unicode object.

files:
M Doc/library/stdtypes.rst

diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index bc7411c809a..e16bd77188f 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1071,9 +1071,10 @@ string functions based on regular expressions.
 .. method:: str.join(iterable)
 
    Return a string which is the concatenation of the strings in *iterable*.
-   A :exc:`TypeError` will be raised if there are any non-string values in
-   *iterable*, including :class:`bytes` objects.  The separator between
-   elements is the string providing this method.
+   If there is any Unicode object in *iterable*, return a Unicode instead.
+   A :exc:`TypeError` will be raised if there are any non-string or non Unicode
+   object values in *iterable*.  The separator between elements is the string
+   providing this method.
 
 
 .. method:: str.ljust(width[, fillchar])


From webhook-mailer at python.org  Fri Jan 26 14:28:39 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Fri, 26 Jan 2018 19:28:39 -0000
Subject: [Python-checkins] bpo-32591: fix abort in
 _PyErr_WarnUnawaitedCoroutine during shutdown (#5337)
Message-ID: <mailman.65.1516994922.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/dba976b8a28d6e5daa66ef31a6a7c694a9193f6a
commit: dba976b8a28d6e5daa66ef31a6a7c694a9193f6a
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-26T14:28:31-05:00
summary:

bpo-32591: fix abort in _PyErr_WarnUnawaitedCoroutine during shutdown (#5337)

When an unawaited coroutine is collected very late in shutdown --
like, during the final GC at the end of PyImport_Cleanup -- then it
was triggering an interpreter abort, because we'd try to look up the
"warnings" module and not only was it missing (we were prepared for
that), but the entire module system was missing (which we were not
prepared for).

I've tried to fix this at the source, by making the utility function
get_warnings_attr robust against this in general. Note that it already
has the convention that it can return NULL without setting an error,
which is how it signals that the attribute it was asked to fetch is
missing, and that all callers already check for NULL returns.

There's a similar check for being late in shutdown at the top of
warn_explicit, which might be unnecessary after this fix, but I'm not
sure so I'm going to leave it.

files:
M Lib/test/test_coroutines.py
M Python/_warnings.c

diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py
index 8a531b88892..a97535a12a8 100644
--- a/Lib/test/test_coroutines.py
+++ b/Lib/test/test_coroutines.py
@@ -8,6 +8,7 @@
 import unittest
 import warnings
 from test import support
+from test.support.script_helper import assert_python_ok
 
 
 class AsyncYieldFrom:
@@ -2168,6 +2169,27 @@ def test_unawaited_warning_when_module_broken(self):
         finally:
             warnings._warn_unawaited_coroutine = orig_wuc
 
+
+class UnawaitedWarningDuringShutdownTest(unittest.TestCase):
+    # https://bugs.python.org/issue32591#msg310726
+    def test_unawaited_warning_during_shutdown(self):
+        code = ("import asyncio\n"
+                "async def f(): pass\n"
+                "asyncio.gather(f())\n")
+        assert_python_ok("-c", code)
+
+        code = ("import sys\n"
+                "async def f(): pass\n"
+                "sys.coro = f()\n")
+        assert_python_ok("-c", code)
+
+        code = ("import sys\n"
+                "async def f(): pass\n"
+                "sys.corocycle = [f()]\n"
+                "sys.corocycle.append(sys.corocycle)\n")
+        assert_python_ok("-c", code)
+
+
 @support.cpython_only
 class CAPITest(unittest.TestCase):
 
diff --git a/Python/_warnings.c b/Python/_warnings.c
index 16ae932450c..0568af4df5a 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -75,6 +75,13 @@ get_warnings_attr(_Py_Identifier *attr_id, int try_import)
         }
     }
     else {
+        /* if we're so late into Python finalization that the module dict is
+           gone, then we can't even use PyImport_GetModule without triggering
+           an interpreter abort.
+        */
+        if (!PyThreadState_GET()->interp->modules) {
+            return NULL;
+        }
         warnings_module = PyImport_GetModule(warnings_str);
         if (warnings_module == NULL)
             return NULL;


From webhook-mailer at python.org  Fri Jan 26 15:24:32 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Fri, 26 Jan 2018 20:24:32 -0000
Subject: [Python-checkins] bpo-32670: Enforce PEP 479. (#5327)
Message-ID: <mailman.66.1516998273.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/43c47fe09640c579462978ec16f81295f5857cde
commit: 43c47fe09640c579462978ec16f81295f5857cde
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-26T15:24:24-05:00
summary:

bpo-32670: Enforce PEP 479. (#5327)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-25-17-03-46.bpo-32670.YsqJUC.rst
M Doc/library/exceptions.rst
M Doc/whatsnew/3.7.rst
M Lib/test/test_generators.py
M Lib/test/test_with.py
M Objects/genobject.c
M Python/future.c

diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
index aa3141053b3..42b99cc47e0 100644
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -367,17 +367,21 @@ The following exceptions are the exceptions that are usually raised.
    raised, and the value returned by the function is used as the
    :attr:`value` parameter to the constructor of the exception.
 
-   If a generator function defined in the presence of a ``from __future__
-   import generator_stop`` directive raises :exc:`StopIteration`, it will be
-   converted into a :exc:`RuntimeError` (retaining the :exc:`StopIteration`
-   as the new exception's cause).
+   If a generator code directly or indirectly raises :exc:`StopIteration`,
+   it is converted into a :exc:`RuntimeError` (retaining the
+   :exc:`StopIteration` as the new exception's cause).
 
    .. versionchanged:: 3.3
       Added ``value`` attribute and the ability for generator functions to
       use it to return a value.
 
    .. versionchanged:: 3.5
-      Introduced the RuntimeError transformation.
+      Introduced the RuntimeError transformation via
+      ``from __future__ import generator_stop``, see :pep:`479`.
+
+   .. versionchanged:: 3.7
+      Enable :pep:`479` for all code by default: a :exc:`StopIteration`
+      error raised in a generator is transformed into a :exc:`RuntimeError`.
 
 .. exception:: StopAsyncIteration
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index a350919bb34..43fbd013856 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -935,6 +935,12 @@ that may require changes to your code.
 Changes in Python behavior
 --------------------------
 
+* :pep:`479` is enabled for all code in Python 3.7, meaning that
+  :exc:`StopIteration` exceptions raised directly or indirectly in
+  coroutines and generators are transformed into :exc:`RuntimeError`
+  exceptions.
+  (Contributed by Yury Selivanov in :issue:`32670`.)
+
 * Due to an oversight, earlier Python versions erroneously accepted the
   following syntax::
 
diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py
index bd17ad4a21b..3461f203dee 100644
--- a/Lib/test/test_generators.py
+++ b/Lib/test/test_generators.py
@@ -265,26 +265,16 @@ def gen():
         self.assertEqual(next(g), "done")
         self.assertEqual(sys.exc_info(), (None, None, None))
 
-    def test_stopiteration_warning(self):
+    def test_stopiteration_error(self):
         # See also PEP 479.
 
         def gen():
             raise StopIteration
             yield
 
-        with self.assertRaises(StopIteration), \
-             self.assertWarnsRegex(DeprecationWarning, "StopIteration"):
-
-            next(gen())
-
-        with self.assertRaisesRegex(DeprecationWarning,
-                                    "generator .* raised StopIteration"), \
-             warnings.catch_warnings():
-
-            warnings.simplefilter('error')
+        with self.assertRaisesRegex(RuntimeError, 'raised StopIteration'):
             next(gen())
 
-
     def test_tutorial_stopiteration(self):
         # Raise StopIteration" stops the generator too:
 
@@ -296,13 +286,7 @@ def f():
         g = f()
         self.assertEqual(next(g), 1)
 
-        with self.assertWarnsRegex(DeprecationWarning, "StopIteration"):
-            with self.assertRaises(StopIteration):
-                next(g)
-
-        with self.assertRaises(StopIteration):
-            # This time StopIteration isn't raised from the generator's body,
-            # hence no warning.
+        with self.assertRaisesRegex(RuntimeError, 'raised StopIteration'):
             next(g)
 
     def test_return_tuple(self):
diff --git a/Lib/test/test_with.py b/Lib/test/test_with.py
index c70f6859b4b..b1d7a15b5e4 100644
--- a/Lib/test/test_with.py
+++ b/Lib/test/test_with.py
@@ -458,8 +458,8 @@ def shouldThrow():
             with cm():
                 raise StopIteration("from with")
 
-        with self.assertWarnsRegex(DeprecationWarning, "StopIteration"):
-            self.assertRaises(StopIteration, shouldThrow)
+        with self.assertRaisesRegex(StopIteration, 'from with'):
+            shouldThrow()
 
     def testRaisedStopIteration2(self):
         # From bug 1462485
@@ -473,7 +473,8 @@ def shouldThrow():
             with cm():
                 raise StopIteration("from with")
 
-        self.assertRaises(StopIteration, shouldThrow)
+        with self.assertRaisesRegex(StopIteration, 'from with'):
+            shouldThrow()
 
     def testRaisedStopIteration3(self):
         # Another variant where the exception hasn't been instantiated
@@ -486,8 +487,8 @@ def shouldThrow():
             with cm():
                 raise next(iter([]))
 
-        with self.assertWarnsRegex(DeprecationWarning, "StopIteration"):
-            self.assertRaises(StopIteration, shouldThrow)
+        with self.assertRaises(StopIteration):
+            shouldThrow()
 
     def testRaisedGeneratorExit1(self):
         # From bug 1462485
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-25-17-03-46.bpo-32670.YsqJUC.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-25-17-03-46.bpo-32670.YsqJUC.rst
new file mode 100644
index 00000000000..b22249b37e5
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-25-17-03-46.bpo-32670.YsqJUC.rst	
@@ -0,0 +1,5 @@
+Enforce PEP 479 for all code.
+
+This means that manually raising a StopIteration exception from a generator
+is prohibited for all code, regardless of whether 'from __future__ import
+generator_stop' was used or not.
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 7baffa7fc02..1fdb57c8ce8 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -248,59 +248,17 @@ gen_send_ex(PyGenObject *gen, PyObject *arg, int exc, int closing)
         Py_CLEAR(result);
     }
     else if (!result && PyErr_ExceptionMatches(PyExc_StopIteration)) {
-        /* Check for __future__ generator_stop and conditionally turn
-         * a leaking StopIteration into RuntimeError (with its cause
-         * set appropriately). */
-
-        const int check_stop_iter_error_flags = CO_FUTURE_GENERATOR_STOP |
-                                                CO_COROUTINE |
-                                                CO_ITERABLE_COROUTINE |
-                                                CO_ASYNC_GENERATOR;
-
-        if (gen->gi_code != NULL &&
-            ((PyCodeObject *)gen->gi_code)->co_flags &
-                check_stop_iter_error_flags)
-        {
-            /* `gen` is either:
-                  * a generator with CO_FUTURE_GENERATOR_STOP flag;
-                  * a coroutine;
-                  * a generator with CO_ITERABLE_COROUTINE flag
-                    (decorated with types.coroutine decorator);
-                  * an async generator.
-            */
-            const char *msg = "generator raised StopIteration";
-            if (PyCoro_CheckExact(gen)) {
-                msg = "coroutine raised StopIteration";
-            }
-            else if PyAsyncGen_CheckExact(gen) {
-                msg = "async generator raised StopIteration";
-            }
-            _PyErr_FormatFromCause(PyExc_RuntimeError, "%s", msg);
+        const char *msg = "generator raised StopIteration";
+        if (PyCoro_CheckExact(gen)) {
+            msg = "coroutine raised StopIteration";
         }
-        else {
-            /* `gen` is an ordinary generator without
-               CO_FUTURE_GENERATOR_STOP flag.
-            */
-
-            PyObject *exc, *val, *tb;
-
-            /* Pop the exception before issuing a warning. */
-            PyErr_Fetch(&exc, &val, &tb);
-
-            if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
-                                 "generator '%.50S' raised StopIteration",
-                                 gen->gi_qualname)) {
-                /* Warning was converted to an error. */
-                Py_XDECREF(exc);
-                Py_XDECREF(val);
-                Py_XDECREF(tb);
-            }
-            else {
-                PyErr_Restore(exc, val, tb);
-            }
+        else if PyAsyncGen_CheckExact(gen) {
+            msg = "async generator raised StopIteration";
         }
+        _PyErr_FormatFromCause(PyExc_RuntimeError, "%s", msg);
+
     }
-    else if (PyAsyncGen_CheckExact(gen) && !result &&
+    else if (!result && PyAsyncGen_CheckExact(gen) &&
              PyErr_ExceptionMatches(PyExc_StopAsyncIteration))
     {
         /* code in `gen` raised a StopAsyncIteration error:
diff --git a/Python/future.c b/Python/future.c
index 53faa6b9016..ade647f25ba 100644
--- a/Python/future.c
+++ b/Python/future.c
@@ -41,7 +41,7 @@ future_check_features(PyFutureFeatures *ff, stmt_ty s, PyObject *filename)
         } else if (strcmp(feature, FUTURE_BARRY_AS_BDFL) == 0) {
             ff->ff_features |= CO_FUTURE_BARRY_AS_BDFL;
         } else if (strcmp(feature, FUTURE_GENERATOR_STOP) == 0) {
-            ff->ff_features |= CO_FUTURE_GENERATOR_STOP;
+            continue;
         } else if (strcmp(feature, FUTURE_ANNOTATIONS) == 0) {
             ff->ff_features |= CO_FUTURE_ANNOTATIONS;
         } else if (strcmp(feature, "braces") == 0) {


From webhook-mailer at python.org  Fri Jan 26 17:24:55 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Fri, 26 Jan 2018 22:24:55 -0000
Subject: [Python-checkins] bpo-32436: Make PyContextVar_Get a little bit
 faster (#5350)
Message-ID: <mailman.67.1517005498.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/226e50049da43097d89e9e99f9c55f212d1f7bd5
commit: 226e50049da43097d89e9e99f9c55f212d1f7bd5
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-26T17:24:52-05:00
summary:

bpo-32436: Make PyContextVar_Get a little bit faster (#5350)

Since context.c is compiled with Py_BUILD_CORE, using a macro
will result in a slightly more optimal code.

files:
M Python/context.c

diff --git a/Python/context.c b/Python/context.c
index d90ed25c4d4..3928e94d064 100644
--- a/Python/context.c
+++ b/Python/context.c
@@ -145,7 +145,8 @@ PyContextVar_Get(PyContextVar *var, PyObject *def, PyObject **val)
 {
     assert(PyContextVar_CheckExact(var));
 
-    PyThreadState *ts = PyThreadState_Get();
+    PyThreadState *ts = PyThreadState_GET();
+    assert(ts != NULL);
     if (ts->context == NULL) {
         goto not_found;
     }


From webhook-mailer at python.org  Fri Jan 26 17:28:48 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Fri, 26 Jan 2018 22:28:48 -0000
Subject: [Python-checkins] bpo-32521: nis libnsl (#5190)
Message-ID: <mailman.68.1517005732.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/29a7df78277447cf6b898dfa0b1b42f8da7abc0c
commit: 29a7df78277447cf6b898dfa0b1b42f8da7abc0c
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-26T23:28:46+01:00
summary:

bpo-32521: nis libnsl (#5190)

The nismodule is now compatible with new libnsl and headers location

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
M setup.py

diff --git a/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst b/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
new file mode 100644
index 00000000000..ee704667c3c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
@@ -0,0 +1 @@
+The nis module is now compatible with new libnsl and headers location.
diff --git a/setup.py b/setup.py
index 258094e3ada..a6f4488cc99 100644
--- a/setup.py
+++ b/setup.py
@@ -1315,27 +1315,14 @@ class db_found(Exception): pass
             exts.append( Extension('termios', ['termios.c']) )
             # Jeremy Hylton's rlimit interface
             exts.append( Extension('resource', ['resource.c']) )
+        else:
+            missing.extend(['resource', 'termios'])
 
-            # Sun yellow pages. Some systems have the functions in libc.
-            if (host_platform not in ['cygwin', 'qnx6'] and
-                find_file('rpcsvc/yp_prot.h', inc_dirs, []) is not None):
-                nis_libs = []
-                nis_includes = []
-                if self.compiler.find_library_file(lib_dirs, 'nsl'):
-                    nis_libs.append('nsl')
-                if self.compiler.find_library_file(lib_dirs, 'tirpc'):
-                    # Sun RPC has been moved from glibc to libtirpc
-                    # rpcsvc/yp_prot.h is still in /usr/include, but
-                    # rpc/rpc.h has been moved into tirpc/ subdir.
-                    nis_libs.append('tirpc')
-                    nis_includes.append('/usr/include/tirpc')
-                exts.append( Extension('nis', ['nismodule.c'],
-                                       libraries = nis_libs,
-                                       include_dirs=nis_includes) )
-            else:
-                missing.append('nis')
+        nis = self._detect_nis(inc_dirs, lib_dirs)
+        if nis is not None:
+            exts.append(nis)
         else:
-            missing.extend(['nis', 'resource', 'termios'])
+            missing.append('nis')
 
         # Curses support, requiring the System V version of curses, often
         # provided by the ncurses library.
@@ -2175,6 +2162,51 @@ def split_var(name, sep):
 
         return ssl_ext, hashlib_ext
 
+    def _detect_nis(self, inc_dirs, lib_dirs):
+        if host_platform in {'win32', 'cygwin', 'qnx6'}:
+            return None
+
+        libs = []
+        library_dirs = []
+        includes_dirs = []
+
+        # bpo-32521: glibc has deprecated Sun RPC for some time. Fedora 28
+        # moved headers and libraries to libtirpc and libnsl. The headers
+        # are in tircp and nsl sub directories.
+        rpcsvc_inc = find_file(
+            'rpcsvc/yp_prot.h', inc_dirs,
+            [os.path.join(inc_dir, 'nsl') for inc_dir in inc_dirs]
+        )
+        rpc_inc = find_file(
+            'rpc/rpc.h', inc_dirs,
+            [os.path.join(inc_dir, 'tirpc') for inc_dir in inc_dirs]
+        )
+        if rpcsvc_inc is None or rpc_inc is None:
+            # not found
+            return None
+        includes_dirs.extend(rpcsvc_inc)
+        includes_dirs.extend(rpc_inc)
+
+        if self.compiler.find_library_file(lib_dirs, 'nsl'):
+            libs.append('nsl')
+        else:
+            # libnsl-devel: check for libnsl in nsl/ subdirectory
+            nsl_dirs = [os.path.join(lib_dir, 'nsl') for lib_dir in lib_dirs]
+            libnsl = self.compiler.find_library_file(nsl_dirs, 'nsl')
+            if libnsl is not None:
+                library_dirs.append(os.path.dirname(libnsl))
+                libs.append('nsl')
+
+        if self.compiler.find_library_file(lib_dirs, 'tirpc'):
+            libs.append('tirpc')
+
+        return Extension(
+            'nis', ['nismodule.c'],
+            libraries=libs,
+            library_dirs=library_dirs,
+            include_dirs=includes_dirs
+        )
+
 
 class PyBuildInstall(install):
     # Suppress the warning about installation into the lib_dynload


From webhook-mailer at python.org  Fri Jan 26 20:10:10 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Sat, 27 Jan 2018 01:10:10 -0000
Subject: [Python-checkins] bpo-32678: inspect: Import ast lazily (GH-5344)
Message-ID: <mailman.69.1517015411.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/37420deb80dcf0fc41a728838b0340b93ca01d90
commit: 37420deb80dcf0fc41a728838b0340b93ca01d90
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-27T10:10:06+09:00
summary:

bpo-32678: inspect: Import ast lazily (GH-5344)

files:
M Lib/inspect.py

diff --git a/Lib/inspect.py b/Lib/inspect.py
index b7551878b74..bc97efe179c 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -32,7 +32,6 @@
               'Yury Selivanov <yselivanov at sprymix.com>')
 
 import abc
-import ast
 import dis
 import collections.abc
 import enum
@@ -1940,6 +1939,9 @@ def _signature_fromstr(cls, obj, s, skip_bound_arg=True):
     """Private helper to parse content of '__text_signature__'
     and return a Signature based on it.
     """
+    # Lazy import ast because it's relatively heavy and
+    # it's not used for other than this function.
+    import ast
 
     Parameter = cls._parameter_cls
 


From webhook-mailer at python.org  Fri Jan 26 21:19:01 2018
From: webhook-mailer at python.org (=?utf-8?q?=C3=89ric?= Araujo)
Date: Sat, 27 Jan 2018 02:19:01 -0000
Subject: [Python-checkins] [3.6] bpo-32304: Fix distutils upload for tar
 files ending with b'\r' (GH-5264) (GH-5330)
Message-ID: <mailman.70.1517019542.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/995c60d2656c022359aac3fe713d8464c8db5716
commit: 995c60d2656c022359aac3fe713d8464c8db5716
branch: 3.6
author: Bo Bayles <bbayles at gmail.com>
committer: ?ric Araujo <merwok at netwok.org>
date: 2018-01-26T21:18:58-05:00
summary:

[3.6] bpo-32304: Fix distutils upload for tar files ending with b'\r' (GH-5264) (GH-5330)

Patch by Bo Bayles.

files:
A Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
M Doc/whatsnew/3.6.rst
M Lib/distutils/command/upload.py
M Lib/distutils/tests/test_upload.py

diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst
index f83508c9250..59329fa7069 100644
--- a/Doc/whatsnew/3.6.rst
+++ b/Doc/whatsnew/3.6.rst
@@ -1016,6 +1016,11 @@ attribute defaults to ``['gztar']``. Although not anticipated,
 any code relying on the presence of ``default_format`` may
 need to be adapted. See :issue:`27819` for more details.
 
+The ``upload`` command now longer tries to change CR end-of-line characters
+to CRLF.  This fixes a corruption issue with sdists that ended with a byte
+equivalent to CR.
+(Contributed by Bo Bayles in :issue:`32304`.)
+
 
 email
 -----
diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
index 1fd574a9f15..f7752f9d12c 100644
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -159,8 +159,6 @@ def upload_file(self, command, pyversion, filename):
                 body.write(title.encode('utf-8'))
                 body.write(b"\r\n\r\n")
                 body.write(value)
-                if value and value[-1:] == b'\r':
-                    body.write(b'\n')  # write an extra newline (lurve Macs)
         body.write(end_boundary)
         body = body.getvalue()
 
diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py
index 2cb2f6ce938..c17d8e7d54e 100644
--- a/Lib/distutils/tests/test_upload.py
+++ b/Lib/distutils/tests/test_upload.py
@@ -143,6 +143,32 @@ def test_upload(self):
         results = self.get_logs(INFO)
         self.assertEqual(results[-1], 75 * '-' + '\nxyzzy\n' + 75 * '-')
 
+    # bpo-32304: archives whose last byte was b'\r' were corrupted due to
+    # normalization intended for Mac OS 9.
+    def test_upload_correct_cr(self):
+        # content that ends with \r should not be modified.
+        tmp = self.mkdtemp()
+        path = os.path.join(tmp, 'xxx')
+        self.write_file(path, content='yy\r')
+        command, pyversion, filename = 'xxx', '2.6', path
+        dist_files = [(command, pyversion, filename)]
+        self.write_file(self.rc, PYPIRC_LONG_PASSWORD)
+
+        # other fields that ended with \r used to be modified, now are
+        # preserved.
+        pkg_dir, dist = self.create_dist(
+            dist_files=dist_files,
+            description='long description\r'
+        )
+        cmd = upload(dist)
+        cmd.show_response = 1
+        cmd.ensure_finalized()
+        cmd.run()
+
+        headers = dict(self.last_open.req.headers)
+        self.assertEqual(headers['Content-length'], '2172')
+        self.assertIn(b'long description\r', self.last_open.req.data)
+
     def test_upload_fails(self):
         self.next_msg = "Not Found"
         self.next_code = 404
diff --git a/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst b/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
new file mode 100644
index 00000000000..c199a644ad4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
@@ -0,0 +1,2 @@
+distutils' upload command no longer corrupts tar files ending with a CR byte,
+and no longer tries to convert CR to CRLF in any of the upload text fields.


From webhook-mailer at python.org  Fri Jan 26 21:40:54 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sat, 27 Jan 2018 02:40:54 -0000
Subject: [Python-checkins] bpo-27505: Add change notes in module attribute
 docs (GH-5320)
Message-ID: <mailman.71.1517020857.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/85527cf50a9a4eecaca4022f7c6cb9e0bae1fa5f
commit: 85527cf50a9a4eecaca4022f7c6cb9e0bae1fa5f
branch: master
author: Cheryl Sabella <cheryl.sabella at gmail.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-27T12:40:52+10:00
summary:

bpo-27505: Add change notes in module attribute docs (GH-5320)

Make it clear that setting __class__ on a module has worked since 3.5,
but support for __getattr__ and __dir__ on module instances requires 3.7+

Patch by Cheryl Sabella.

files:
M Doc/reference/datamodel.rst

diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index 8d47002065f..25b95c11543 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -1564,6 +1564,17 @@ a module object to a subclass of :class:`types.ModuleType`. For example::
    the module globals (whether by code within the module, or via a reference
    to the module's globals dictionary) is unaffected.
 
+.. versionchanged:: 3.5
+   ``__class__`` module attribute is now writable.
+
+.. versionadded:: 3.7
+   ``__getattr__`` and ``__dir__`` module attributes.
+
+.. seealso::
+
+   :pep:`562` - Module __getattr__ and __dir__
+      Describes the ``__getattr__`` and ``__dir__`` functions on modules.
+
 
 .. _descriptors:
 


From webhook-mailer at python.org  Fri Jan 26 21:45:27 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sat, 27 Jan 2018 02:45:27 -0000
Subject: [Python-checkins] [3.6] bpo-27505: Retrofit module __class__
 documentation from 3.7 (GH-5321)
Message-ID: <mailman.72.1517021130.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/8f68cb7db37322cfeeb8338e78474e5f25d930c5
commit: 8f68cb7db37322cfeeb8338e78474e5f25d930c5
branch: 3.6
author: Cheryl Sabella <cheryl.sabella at gmail.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-27T12:45:24+10:00
summary:

[3.6] bpo-27505: Retrofit module __class__ documentation from 3.7 (GH-5321)

The module `__class__` attribute documentation added to 3.7 for PEP 562
(dynamic module attributes) also applies to earlier versions. This backports
that subset of the new docs to the 3.6 branch so that it will appear in the
main online documentation and in the final 3.6 binary release.

Patch by Cheryl Sabella.

files:
M Doc/reference/datamodel.rst

diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index cd11020e0db..9752494972f 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -1500,6 +1500,39 @@ access (use of, assignment to, or deletion of ``x.name``) for class instances.
    returned. :func:`dir` converts the returned sequence to a list and sorts it.
 
 
+Customizing module attribute access
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. index::
+   single: __class__ (module attribute)
+
+For a more fine grained customization of the module behavior (setting
+attributes, properties, etc.), one can set the ``__class__`` attribute of
+a module object to a subclass of :class:`types.ModuleType`. For example::
+
+   import sys
+   from types import ModuleType
+
+   class VerboseModule(ModuleType):
+       def __repr__(self):
+           return f'Verbose {self.__name__}'
+
+       def __setattr__(self, attr, value):
+           print(f'Setting {attr}...')
+           setattr(self, attr, value)
+
+   sys.modules[__name__].__class__ = VerboseModule
+
+.. note::
+   Setting module ``__class__`` only affects lookups made using the attribute
+   access syntax -- directly accessing the module globals (whether by code
+   within the module, or via a reference to the module's globals dictionary)
+   is unaffected.
+
+.. versionchanged:: 3.5
+   ``__class__`` module attribute is now writable.
+
+
 .. _descriptors:
 
 Implementing Descriptors


From webhook-mailer at python.org  Sat Jan 27 00:06:24 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Sat, 27 Jan 2018 05:06:24 -0000
Subject: [Python-checkins] bpo-32677: Add .isascii() to str,
 bytes and bytearray (GH-5342)
Message-ID: <mailman.73.1517029587.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/a49ac9902903a798fab4970ccf563c531199c3f8
commit: a49ac9902903a798fab4970ccf563c531199c3f8
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-27T14:06:21+09:00
summary:

bpo-32677: Add .isascii() to str, bytes and bytearray (GH-5342)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-26-20-11-09.bpo-32677.xTGfCq.rst
M Doc/library/stdtypes.rst
M Include/bytes_methods.h
M Lib/collections/__init__.py
M Lib/test/string_tests.py
M Lib/test/test_doctest.py
M Lib/test/test_unicode.py
M Objects/bytearrayobject.c
M Objects/bytes_methods.c
M Objects/bytesobject.c
M Objects/clinic/unicodeobject.c.h
M Objects/stringlib/ctype.h
M Objects/unicodeobject.c

diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index 120b0d3399c..ad7f578e086 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1653,6 +1653,15 @@ expression support in the :mod:`re` module).
    from the "Alphabetic" property defined in the Unicode Standard.
 
 
+.. method:: str.isascii()
+
+   Return true if the string is empty or all characters in the string are ASCII,
+   false otherwise.
+   ASCII characters have code points in the range U+0000-U+007F.
+
+   .. versionadded:: 3.7
+
+
 .. method:: str.isdecimal()
 
    Return true if all characters in the string are decimal
@@ -2941,6 +2950,16 @@ place, and instead produce new objects.
       False
 
 
+.. method:: bytes.isascii()
+            bytearray.isascii()
+
+   Return true if the sequence is empty or all bytes in the sequence are ASCII,
+   false otherwise.
+   ASCII bytes are in the range 0-0x7F.
+
+   .. versionadded:: 3.7
+
+
 .. method:: bytes.isdigit()
             bytearray.isdigit()
 
diff --git a/Include/bytes_methods.h b/Include/bytes_methods.h
index 7fa7540c38b..8434a50a4bb 100644
--- a/Include/bytes_methods.h
+++ b/Include/bytes_methods.h
@@ -9,6 +9,7 @@
 extern PyObject* _Py_bytes_isspace(const char *cptr, Py_ssize_t len);
 extern PyObject* _Py_bytes_isalpha(const char *cptr, Py_ssize_t len);
 extern PyObject* _Py_bytes_isalnum(const char *cptr, Py_ssize_t len);
+extern PyObject* _Py_bytes_isascii(const char *cptr, Py_ssize_t len);
 extern PyObject* _Py_bytes_isdigit(const char *cptr, Py_ssize_t len);
 extern PyObject* _Py_bytes_islower(const char *cptr, Py_ssize_t len);
 extern PyObject* _Py_bytes_isupper(const char *cptr, Py_ssize_t len);
@@ -37,6 +38,7 @@ extern PyObject* _Py_bytes_maketrans(Py_buffer *frm, Py_buffer *to);
 extern const char _Py_isspace__doc__[];
 extern const char _Py_isalpha__doc__[];
 extern const char _Py_isalnum__doc__[];
+extern const char _Py_isascii__doc__[];
 extern const char _Py_isdigit__doc__[];
 extern const char _Py_islower__doc__[];
 extern const char _Py_isupper__doc__[];
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 7088b88e04a..21d91fd61d6 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -1214,6 +1214,7 @@ def index(self, sub, start=0, end=_sys.maxsize):
         return self.data.index(sub, start, end)
     def isalpha(self): return self.data.isalpha()
     def isalnum(self): return self.data.isalnum()
+    def isascii(self): return self.data.isascii()
     def isdecimal(self): return self.data.isdecimal()
     def isdigit(self): return self.data.isdigit()
     def isidentifier(self): return self.data.isidentifier()
diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py
index cd3ee48a92b..4be1d211897 100644
--- a/Lib/test/string_tests.py
+++ b/Lib/test/string_tests.py
@@ -909,6 +909,14 @@ def test_isalnum(self):
         self.checkequal(False, 'abc\n', 'isalnum')
         self.checkraises(TypeError, 'abc', 'isalnum', 42)
 
+    def test_isascii(self):
+        self.checkequal(True, '', 'isascii')
+        self.checkequal(True, '\x00', 'isascii')
+        self.checkequal(True, '\x7f', 'isascii')
+        self.checkequal(True, '\x00\x7f', 'isascii')
+        self.checkequal(False, '\x80', 'isascii')
+        self.checkequal(False, '\xe9', 'isascii')
+
     def test_isdigit(self):
         self.checkequal(False, '', 'isdigit')
         self.checkequal(False, 'a', 'isdigit')
diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py
index 2258c6b1bae..5ad94aba649 100644
--- a/Lib/test/test_doctest.py
+++ b/Lib/test/test_doctest.py
@@ -659,7 +659,7 @@ def non_Python_modules(): r"""
 
     >>> import builtins
     >>> tests = doctest.DocTestFinder().find(builtins)
-    >>> 790 < len(tests) < 810 # approximate number of objects with docstrings
+    >>> 800 < len(tests) < 820 # approximate number of objects with docstrings
     True
     >>> real_tests = [t for t in tests if len(t.examples) > 0]
     >>> len(real_tests) # objects that actually have doctests
diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
index 2b77863e52e..3cc018c0cc2 100644
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -638,6 +638,11 @@ def test_isalpha(self):
         self.assertFalse('\U0001F40D'.isalpha())
         self.assertFalse('\U0001F46F'.isalpha())
 
+    def test_isascii(self):
+        super().test_isascii()
+        self.assertFalse("\u20ac".isascii())
+        self.assertFalse("\U0010ffff".isascii())
+
     def test_isdecimal(self):
         self.checkequalnofix(False, '', 'isdecimal')
         self.checkequalnofix(False, 'a', 'isdecimal')
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-26-20-11-09.bpo-32677.xTGfCq.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-26-20-11-09.bpo-32677.xTGfCq.rst
new file mode 100644
index 00000000000..947c74f130b
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-26-20-11-09.bpo-32677.xTGfCq.rst	
@@ -0,0 +1,2 @@
+Add ``.isascii()`` method to ``str``, ``bytes`` and ``bytearray``.
+It can be used to test that string contains only ASCII characters.
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
index dc1515a059c..692b7be7392 100644
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -2159,6 +2159,8 @@ bytearray_methods[] = {
      _Py_isalnum__doc__},
     {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS,
      _Py_isalpha__doc__},
+    {"isascii", (PyCFunction)stringlib_isascii, METH_NOARGS,
+     _Py_isascii__doc__},
     {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS,
      _Py_isdigit__doc__},
     {"islower", (PyCFunction)stringlib_islower, METH_NOARGS,
diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
index bd79773a54e..149650f018f 100644
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -92,6 +92,26 @@ _Py_bytes_isalnum(const char *cptr, Py_ssize_t len)
 }
 
 
+PyDoc_STRVAR_shared(_Py_isascii__doc__,
+"B.isascii() -> bool\n\
+\n\
+Return True if B is empty or all characters in B are ASCII,\n\
+False otherwise.");
+
+PyObject*
+_Py_bytes_isascii(const char *cptr, Py_ssize_t len)
+{
+    const unsigned char *p = (unsigned char *) cptr;
+    const unsigned char *e = p + len;
+    for (; p < e; p++) {
+        if (*p >= 128) {
+            Py_RETURN_FALSE;
+        }
+    }
+    Py_RETURN_TRUE;
+}
+
+
 PyDoc_STRVAR_shared(_Py_isdigit__doc__,
 "B.isdigit() -> bool\n\
 \n\
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index a921d9cb8c5..c358756bfea 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -2459,6 +2459,8 @@ bytes_methods[] = {
      _Py_isalnum__doc__},
     {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS,
      _Py_isalpha__doc__},
+    {"isascii", (PyCFunction)stringlib_isascii, METH_NOARGS,
+     _Py_isascii__doc__},
     {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS,
      _Py_isdigit__doc__},
     {"islower", (PyCFunction)stringlib_islower, METH_NOARGS,
diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h
index 643ef0491b3..8072516a8a3 100644
--- a/Objects/clinic/unicodeobject.c.h
+++ b/Objects/clinic/unicodeobject.c.h
@@ -165,6 +165,27 @@ unicode_expandtabs(PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyOb
     return return_value;
 }
 
+PyDoc_STRVAR(unicode_isascii__doc__,
+"isascii($self, /)\n"
+"--\n"
+"\n"
+"Return True if all characters in the string are ASCII, False otherwise.\n"
+"\n"
+"ASCII characters have code points in the range U+0000-U+007F.\n"
+"Empty string is ASCII too.");
+
+#define UNICODE_ISASCII_METHODDEF    \
+    {"isascii", (PyCFunction)unicode_isascii, METH_NOARGS, unicode_isascii__doc__},
+
+static PyObject *
+unicode_isascii_impl(PyObject *self);
+
+static PyObject *
+unicode_isascii(PyObject *self, PyObject *Py_UNUSED(ignored))
+{
+    return unicode_isascii_impl(self);
+}
+
 PyDoc_STRVAR(unicode_islower__doc__,
 "islower($self, /)\n"
 "--\n"
@@ -930,4 +951,4 @@ unicode_sizeof(PyObject *self, PyObject *Py_UNUSED(ignored))
 {
     return unicode_sizeof_impl(self);
 }
-/*[clinic end generated code: output=1ad4e81b68194264 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=561c88c912b8fe3b input=a9049054013a1b77]*/
diff --git a/Objects/stringlib/ctype.h b/Objects/stringlib/ctype.h
index f0546256ed9..fd7b1bd49e5 100644
--- a/Objects/stringlib/ctype.h
+++ b/Objects/stringlib/ctype.h
@@ -22,6 +22,12 @@ stringlib_isalnum(PyObject *self)
     return _Py_bytes_isalnum(STRINGLIB_STR(self), STRINGLIB_LEN(self));
 }
 
+static PyObject*
+stringlib_isascii(PyObject *self)
+{
+    return _Py_bytes_isascii(STRINGLIB_STR(self), STRINGLIB_LEN(self));
+}
+
 static PyObject*
 stringlib_isdigit(PyObject *self)
 {
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 07330119dc3..4b90cc364b8 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -11611,6 +11611,25 @@ unicode_index(PyObject *self, PyObject *args)
     return PyLong_FromSsize_t(result);
 }
 
+/*[clinic input]
+str.isascii as unicode_isascii
+
+Return True if all characters in the string are ASCII, False otherwise.
+
+ASCII characters have code points in the range U+0000-U+007F.
+Empty string is ASCII too.
+[clinic start generated code]*/
+
+static PyObject *
+unicode_isascii_impl(PyObject *self)
+/*[clinic end generated code: output=c5910d64b5a8003f input=5a43cbc6399621d5]*/
+{
+    if (PyUnicode_READY(self) == -1) {
+        return NULL;
+    }
+    return PyBool_FromLong(PyUnicode_IS_ASCII(self));
+}
+
 /*[clinic input]
 str.islower as unicode_islower
 
@@ -13801,6 +13820,7 @@ static PyMethodDef unicode_methods[] = {
     UNICODE_UPPER_METHODDEF
     {"startswith", (PyCFunction) unicode_startswith, METH_VARARGS, startswith__doc__},
     {"endswith", (PyCFunction) unicode_endswith, METH_VARARGS, endswith__doc__},
+    UNICODE_ISASCII_METHODDEF
     UNICODE_ISLOWER_METHODDEF
     UNICODE_ISUPPER_METHODDEF
     UNICODE_ISTITLE_METHODDEF


From webhook-mailer at python.org  Sat Jan 27 03:39:19 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 27 Jan 2018 08:39:19 -0000
Subject: [Python-checkins] [3.6] bpo-32521: nis libnsl (GH-5190) (#5352)
Message-ID: <mailman.74.1517042363.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/12ae40766ea39f09c2ddf6842017ee80f7f3d00b
commit: 12ae40766ea39f09c2ddf6842017ee80f7f3d00b
branch: 3.6
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-27T09:39:16+01:00
summary:

[3.6] bpo-32521: nis libnsl (GH-5190) (#5352)

* bpo-32521: nis libnsl (#5190)

The nismodule is now compatible with new libnsl and headers location

Signed-off-by: Christian Heimes <christian at python.org>
(cherry picked from commit 29a7df78277447cf6b898dfa0b1b42f8da7abc0c)

files:
A Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
M setup.py

diff --git a/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst b/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
new file mode 100644
index 00000000000..ee704667c3c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
@@ -0,0 +1 @@
+The nis module is now compatible with new libnsl and headers location.
diff --git a/setup.py b/setup.py
index e8891e6b79e..1ec58aa3792 100644
--- a/setup.py
+++ b/setup.py
@@ -1361,27 +1361,14 @@ class db_found(Exception): pass
             exts.append( Extension('termios', ['termios.c']) )
             # Jeremy Hylton's rlimit interface
             exts.append( Extension('resource', ['resource.c']) )
+        else:
+            missing.extend(['resource', 'termios'])
 
-            # Sun yellow pages. Some systems have the functions in libc.
-            if (host_platform not in ['cygwin', 'qnx6'] and
-                find_file('rpcsvc/yp_prot.h', inc_dirs, []) is not None):
-                nis_libs = []
-                nis_includes = []
-                if self.compiler.find_library_file(lib_dirs, 'nsl'):
-                    nis_libs.append('nsl')
-                if self.compiler.find_library_file(lib_dirs, 'tirpc'):
-                    # Sun RPC has been moved from glibc to libtirpc
-                    # rpcsvc/yp_prot.h is still in /usr/include, but
-                    # rpc/rpc.h has been moved into tirpc/ subdir.
-                    nis_libs.append('tirpc')
-                    nis_includes.append('/usr/include/tirpc')
-                exts.append( Extension('nis', ['nismodule.c'],
-                                       libraries = nis_libs,
-                                       include_dirs=nis_includes) )
-            else:
-                missing.append('nis')
+        nis = self._detect_nis(inc_dirs, lib_dirs)
+        if nis is not None:
+            exts.append(nis)
         else:
-            missing.extend(['nis', 'resource', 'termios'])
+            missing.append('nis')
 
         # Curses support, requiring the System V version of curses, often
         # provided by the ncurses library.
@@ -2225,6 +2212,52 @@ def _decimal_ext(self):
         )
         return ext
 
+    def _detect_nis(self, inc_dirs, lib_dirs):
+        if host_platform in {'win32', 'cygwin', 'qnx6'}:
+            return None
+
+        libs = []
+        library_dirs = []
+        includes_dirs = []
+
+        # bpo-32521: glibc has deprecated Sun RPC for some time. Fedora 28
+        # moved headers and libraries to libtirpc and libnsl. The headers
+        # are in tircp and nsl sub directories.
+        rpcsvc_inc = find_file(
+            'rpcsvc/yp_prot.h', inc_dirs,
+            [os.path.join(inc_dir, 'nsl') for inc_dir in inc_dirs]
+        )
+        rpc_inc = find_file(
+            'rpc/rpc.h', inc_dirs,
+            [os.path.join(inc_dir, 'tirpc') for inc_dir in inc_dirs]
+        )
+        if rpcsvc_inc is None or rpc_inc is None:
+            # not found
+            return None
+        includes_dirs.extend(rpcsvc_inc)
+        includes_dirs.extend(rpc_inc)
+
+        if self.compiler.find_library_file(lib_dirs, 'nsl'):
+            libs.append('nsl')
+        else:
+            # libnsl-devel: check for libnsl in nsl/ subdirectory
+            nsl_dirs = [os.path.join(lib_dir, 'nsl') for lib_dir in lib_dirs]
+            libnsl = self.compiler.find_library_file(nsl_dirs, 'nsl')
+            if libnsl is not None:
+                library_dirs.append(os.path.dirname(libnsl))
+                libs.append('nsl')
+
+        if self.compiler.find_library_file(lib_dirs, 'tirpc'):
+            libs.append('tirpc')
+
+        return Extension(
+            'nis', ['nismodule.c'],
+            libraries=libs,
+            library_dirs=library_dirs,
+            include_dirs=includes_dirs
+        )
+
+
 class PyBuildInstall(install):
     # Suppress the warning about installation into the lib_dynload
     # directory, which is not in sys.path when running Python during


From webhook-mailer at python.org  Sat Jan 27 03:39:41 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 27 Jan 2018 08:39:41 -0000
Subject: [Python-checkins] [2.7] bpo-32521: nis libnsl (GH-5190) (#5353)
Message-ID: <mailman.75.1517042384.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/38487a05cce32536b49dd6c89df7dc85b96eaef4
commit: 38487a05cce32536b49dd6c89df7dc85b96eaef4
branch: 2.7
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-27T09:39:39+01:00
summary:

[2.7] bpo-32521: nis libnsl (GH-5190) (#5353)

The nismodule is now compatible with new libnsl and headers location

Signed-off-by: Christian Heimes <christian at python.org>.
(cherry picked from commit 29a7df78277447cf6b898dfa0b1b42f8da7abc0c)

files:
A Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
M setup.py

diff --git a/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst b/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
new file mode 100644
index 00000000000..ee704667c3c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-15-12-53-13.bpo-32521.IxX4Ba.rst
@@ -0,0 +1 @@
+The nis module is now compatible with new libnsl and headers location.
diff --git a/setup.py b/setup.py
index 784a8d21f1f..c1366c8436a 100644
--- a/setup.py
+++ b/setup.py
@@ -1346,26 +1346,11 @@ class db_found(Exception): pass
             else:
                 missing.append('resource')
 
-            # Sun yellow pages. Some systems have the functions in libc.
-            if (host_platform not in ['cygwin', 'atheos', 'qnx6'] and
-                find_file('rpcsvc/yp_prot.h', inc_dirs, []) is not None):
-                nis_libs = []
-                nis_includes = []
-                if self.compiler.find_library_file(lib_dirs, 'nsl'):
-                    nis_libs.append('nsl')
-                if self.compiler.find_library_file(lib_dirs, 'tirpc'):
-                    # Sun RPC has been moved from glibc to libtirpc
-                    # rpcsvc/yp_prot.h is still in /usr/include, but
-                    # rpc/rpc.h has been moved into tirpc/ subdir.
-                    nis_libs.append('tirpc')
-                    nis_includes.append('/usr/include/tirpc')
-                exts.append( Extension('nis', ['nismodule.c'],
-                                       libraries = nis_libs,
-                                       include_dirs=nis_includes) )
+            nis = self._detect_nis(inc_dirs, lib_dirs)
+            if nis is not None:
+                exts.append(nis)
             else:
                 missing.append('nis')
-        else:
-            missing.extend(['nis', 'resource', 'termios'])
 
         # Curses support, requiring the System V version of curses, often
         # provided by the ncurses library.
@@ -2164,6 +2149,51 @@ def detect_ctypes(self, inc_dirs, lib_dirs):
             ext.libraries.append(ffi_lib)
             self.use_system_libffi = True
 
+    def _detect_nis(self, inc_dirs, lib_dirs):
+        if host_platform in {'win32', 'cygwin', 'qnx6'}:
+            return None
+
+        libs = []
+        library_dirs = []
+        includes_dirs = []
+
+        # bpo-32521: glibc has deprecated Sun RPC for some time. Fedora 28
+        # moved headers and libraries to libtirpc and libnsl. The headers
+        # are in tircp and nsl sub directories.
+        rpcsvc_inc = find_file(
+            'rpcsvc/yp_prot.h', inc_dirs,
+            [os.path.join(inc_dir, 'nsl') for inc_dir in inc_dirs]
+        )
+        rpc_inc = find_file(
+            'rpc/rpc.h', inc_dirs,
+            [os.path.join(inc_dir, 'tirpc') for inc_dir in inc_dirs]
+        )
+        if rpcsvc_inc is None or rpc_inc is None:
+            # not found
+            return None
+        includes_dirs.extend(rpcsvc_inc)
+        includes_dirs.extend(rpc_inc)
+
+        if self.compiler.find_library_file(lib_dirs, 'nsl'):
+            libs.append('nsl')
+        else:
+            # libnsl-devel: check for libnsl in nsl/ subdirectory
+            nsl_dirs = [os.path.join(lib_dir, 'nsl') for lib_dir in lib_dirs]
+            libnsl = self.compiler.find_library_file(nsl_dirs, 'nsl')
+            if libnsl is not None:
+                library_dirs.append(os.path.dirname(libnsl))
+                libs.append('nsl')
+
+        if self.compiler.find_library_file(lib_dirs, 'tirpc'):
+            libs.append('tirpc')
+
+        return Extension(
+            'nis', ['nismodule.c'],
+            libraries=libs,
+            library_dirs=library_dirs,
+            include_dirs=includes_dirs
+        )
+
 
 class PyBuildInstall(install):
     # Suppress the warning about installation into the lib_dynload


From webhook-mailer at python.org  Sat Jan 27 03:53:48 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 27 Jan 2018 08:53:48 -0000
Subject: [Python-checkins] bpo-32433: Optimized HMAC digest (#5023)
Message-ID: <mailman.76.1517043229.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/2f050c7e1b36bf641e7023f7b28b451454c6b98a
commit: 2f050c7e1b36bf641e7023f7b28b451454c6b98a
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-27T09:53:43+01:00
summary:

bpo-32433: Optimized HMAC digest (#5023)

The hmac module now has hmac.digest(), which provides an optimized HMAC
digest for short messages. hmac.digest() is up to three times faster
than hmac.HMAC().digest().

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2017-12-27-20-09-27.bpo-32433.vmxsVI.rst
M Doc/library/hmac.rst
M Doc/whatsnew/3.7.rst
M Lib/hmac.py
M Lib/test/test_hmac.py
M Modules/_hashopenssl.c
M Modules/clinic/_hashopenssl.c.h

diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst
index adbf78a7b46..fcda86cf797 100644
--- a/Doc/library/hmac.rst
+++ b/Doc/library/hmac.rst
@@ -31,6 +31,21 @@ This module implements the HMAC algorithm as described by :rfc:`2104`.
       MD5 as implicit default digest for *digestmod* is deprecated.
 
 
+.. function:: digest(key, msg, digest)
+
+   Return digest of *msg* for given secret *key* and *digest*. The
+   function is equivalent to ``HMAC(key, msg, digest).digest()``, but
+   uses an optimized C or inline implementation, which is faster for messages
+   that fit into memory. The parameters *key*, *msg*, and *digest* have
+   the same meaning as in :func:`~hmac.new`.
+
+   CPython implementation detail, the optimized C implementation is only used
+   when *digest* is a string and name of a digest algorithm, which is
+   supported by OpenSSL.
+
+   .. versionadded:: 3.7
+
+
 An HMAC object has the following methods:
 
 .. method:: HMAC.update(msg)
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 43fbd013856..133975a04dd 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -492,6 +492,13 @@ and the ``--directory`` to the command line of the module :mod:`~http.server`.
 With this parameter, the server serves the specified directory, by default it uses the current working directory.
 (Contributed by St?phane Wirtel and Julien Palard in :issue:`28707`.)
 
+hmac
+----
+
+The hmac module now has an optimized one-shot :func:`~hmac.digest` function,
+which is up to three times faster than :func:`~hmac.HMAC`.
+(Contributed by Christian Heimes in :issue:`32433`.)
+
 importlib
 ---------
 
diff --git a/Lib/hmac.py b/Lib/hmac.py
index 121029aa670..93c084e4ae8 100644
--- a/Lib/hmac.py
+++ b/Lib/hmac.py
@@ -5,6 +5,13 @@
 
 import warnings as _warnings
 from _operator import _compare_digest as compare_digest
+try:
+    import _hashlib as _hashopenssl
+except ImportError:
+    _hashopenssl = None
+    _openssl_md_meths = None
+else:
+    _openssl_md_meths = frozenset(_hashopenssl.openssl_md_meth_names)
 import hashlib as _hashlib
 
 trans_5C = bytes((x ^ 0x5C) for x in range(256))
@@ -142,3 +149,38 @@ def new(key, msg = None, digestmod = None):
     method.
     """
     return HMAC(key, msg, digestmod)
+
+
+def digest(key, msg, digest):
+    """Fast inline implementation of HMAC
+
+    key:    key for the keyed hash object.
+    msg:    input message
+    digest: A hash name suitable for hashlib.new() for best performance. *OR*
+            A hashlib constructor returning a new hash object. *OR*
+            A module supporting PEP 247.
+
+    Note: key and msg must be a bytes or bytearray objects.
+    """
+    if (_hashopenssl is not None and
+            isinstance(digest, str) and digest in _openssl_md_meths):
+        return _hashopenssl.hmac_digest(key, msg, digest)
+
+    if callable(digest):
+        digest_cons = digest
+    elif isinstance(digest, str):
+        digest_cons = lambda d=b'': _hashlib.new(digest, d)
+    else:
+        digest_cons = lambda d=b'': digest.new(d)
+
+    inner = digest_cons()
+    outer = digest_cons()
+    blocksize = getattr(inner, 'block_size', 64)
+    if len(key) > blocksize:
+        key = digest_cons(key).digest()
+    key = key + b'\x00' * (blocksize - len(key))
+    inner.update(key.translate(trans_36))
+    outer.update(key.translate(trans_5C))
+    inner.update(msg)
+    outer.update(inner.digest())
+    return outer.digest()
diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py
index 067e13f1079..4e4ef0ec0e8 100644
--- a/Lib/test/test_hmac.py
+++ b/Lib/test/test_hmac.py
@@ -1,7 +1,9 @@
+import binascii
 import functools
 import hmac
 import hashlib
 import unittest
+import unittest.mock
 import warnings
 
 
@@ -23,16 +25,27 @@ def test_md5_vectors(self):
         def md5test(key, data, digest):
             h = hmac.HMAC(key, data, digestmod=hashlib.md5)
             self.assertEqual(h.hexdigest().upper(), digest.upper())
+            self.assertEqual(h.digest(), binascii.unhexlify(digest))
             self.assertEqual(h.name, "hmac-md5")
             self.assertEqual(h.digest_size, 16)
             self.assertEqual(h.block_size, 64)
 
             h = hmac.HMAC(key, data, digestmod='md5')
             self.assertEqual(h.hexdigest().upper(), digest.upper())
+            self.assertEqual(h.digest(), binascii.unhexlify(digest))
             self.assertEqual(h.name, "hmac-md5")
             self.assertEqual(h.digest_size, 16)
             self.assertEqual(h.block_size, 64)
 
+            self.assertEqual(
+                hmac.digest(key, data, digest='md5'),
+                binascii.unhexlify(digest)
+            )
+            with unittest.mock.patch('hmac._openssl_md_meths', {}):
+                self.assertEqual(
+                    hmac.digest(key, data, digest='md5'),
+                    binascii.unhexlify(digest)
+                )
 
         md5test(b"\x0b" * 16,
                 b"Hi There",
@@ -67,16 +80,23 @@ def test_sha_vectors(self):
         def shatest(key, data, digest):
             h = hmac.HMAC(key, data, digestmod=hashlib.sha1)
             self.assertEqual(h.hexdigest().upper(), digest.upper())
+            self.assertEqual(h.digest(), binascii.unhexlify(digest))
             self.assertEqual(h.name, "hmac-sha1")
             self.assertEqual(h.digest_size, 20)
             self.assertEqual(h.block_size, 64)
 
             h = hmac.HMAC(key, data, digestmod='sha1')
             self.assertEqual(h.hexdigest().upper(), digest.upper())
+            self.assertEqual(h.digest(), binascii.unhexlify(digest))
             self.assertEqual(h.name, "hmac-sha1")
             self.assertEqual(h.digest_size, 20)
             self.assertEqual(h.block_size, 64)
 
+            self.assertEqual(
+                hmac.digest(key, data, digest='sha1'),
+                binascii.unhexlify(digest)
+            )
+
 
         shatest(b"\x0b" * 20,
                 b"Hi There",
@@ -122,6 +142,24 @@ def hmactest(key, data, hexdigests):
             self.assertEqual(h.digest_size, digest_size)
             self.assertEqual(h.block_size, block_size)
 
+            self.assertEqual(
+                hmac.digest(key, data, digest=hashfunc),
+                binascii.unhexlify(hexdigests[hashfunc])
+            )
+            self.assertEqual(
+                hmac.digest(key, data, digest=hash_name),
+                binascii.unhexlify(hexdigests[hashfunc])
+            )
+
+            with unittest.mock.patch('hmac._openssl_md_meths', {}):
+                self.assertEqual(
+                    hmac.digest(key, data, digest=hashfunc),
+                    binascii.unhexlify(hexdigests[hashfunc])
+                )
+                self.assertEqual(
+                    hmac.digest(key, data, digest=hash_name),
+                    binascii.unhexlify(hexdigests[hashfunc])
+                )
 
         # 4.2.  Test Case 1
         hmactest(key = b'\x0b'*20,
diff --git a/Misc/NEWS.d/next/Library/2017-12-27-20-09-27.bpo-32433.vmxsVI.rst b/Misc/NEWS.d/next/Library/2017-12-27-20-09-27.bpo-32433.vmxsVI.rst
new file mode 100644
index 00000000000..d9b326e128e
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-27-20-09-27.bpo-32433.vmxsVI.rst
@@ -0,0 +1,2 @@
+The hmac module now has hmac.digest(), which provides an optimized HMAC
+digest.
diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c
index c8d175860b7..50fe9d5a10b 100644
--- a/Modules/_hashopenssl.c
+++ b/Modules/_hashopenssl.c
@@ -21,6 +21,7 @@
 
 /* EVP is the preferred interface to hashing in OpenSSL */
 #include <openssl/evp.h>
+#include <openssl/hmac.h>
 /* We use the object interface to discover what hashes OpenSSL supports. */
 #include <openssl/objects.h>
 #include "openssl/err.h"
@@ -528,8 +529,6 @@ EVP_new(PyObject *self, PyObject *args, PyObject *kwdict)
     return ret_obj;
 }
 
-
-
 #if (OPENSSL_VERSION_NUMBER >= 0x10000000 && !defined(OPENSSL_NO_HMAC) \
      && !defined(OPENSSL_NO_SHA))
 
@@ -849,6 +848,61 @@ _hashlib_scrypt_impl(PyObject *module, Py_buffer *password, Py_buffer *salt,
 }
 #endif
 
+/* Fast HMAC for hmac.digest()
+ */
+
+/*[clinic input]
+_hashlib.hmac_digest
+
+    key: Py_buffer
+    msg: Py_buffer
+    digest: str
+
+Single-shot HMAC
+[clinic start generated code]*/
+
+static PyObject *
+_hashlib_hmac_digest_impl(PyObject *module, Py_buffer *key, Py_buffer *msg,
+                          const char *digest)
+/*[clinic end generated code: output=75630e684cdd8762 input=10e964917921e2f2]*/
+{
+    unsigned char md[EVP_MAX_MD_SIZE] = {0};
+    unsigned int md_len = 0;
+    unsigned char *result;
+    const EVP_MD *evp;
+
+    evp = EVP_get_digestbyname(digest);
+    if (evp == NULL) {
+        PyErr_SetString(PyExc_ValueError, "unsupported hash type");
+        return NULL;
+    }
+    if (key->len > INT_MAX) {
+        PyErr_SetString(PyExc_OverflowError,
+                        "key is too long.");
+        return NULL;
+    }
+    if (msg->len > INT_MAX) {
+        PyErr_SetString(PyExc_OverflowError,
+                        "msg is too long.");
+        return NULL;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    result = HMAC(
+        evp,
+        (const void*)key->buf, (int)key->len,
+        (const unsigned char*)msg->buf, (int)msg->len,
+        md, &md_len
+    );
+    Py_END_ALLOW_THREADS
+
+    if (result == NULL) {
+        _setException(PyExc_ValueError);
+        return NULL;
+    }
+    return PyBytes_FromStringAndSize((const char*)md, md_len);
+}
+
 /* State for our callback function so that it can accumulate a result. */
 typedef struct _internal_name_mapper_state {
     PyObject *set;
@@ -982,6 +1036,7 @@ static struct PyMethodDef EVP_functions[] = {
      pbkdf2_hmac__doc__},
 #endif
     _HASHLIB_SCRYPT_METHODDEF
+    _HASHLIB_HMAC_DIGEST_METHODDEF
     CONSTRUCTOR_METH_DEF(md5),
     CONSTRUCTOR_METH_DEF(sha1),
     CONSTRUCTOR_METH_DEF(sha224),
diff --git a/Modules/clinic/_hashopenssl.c.h b/Modules/clinic/_hashopenssl.c.h
index f08d7f3afd4..cbc8638c946 100644
--- a/Modules/clinic/_hashopenssl.c.h
+++ b/Modules/clinic/_hashopenssl.c.h
@@ -54,7 +54,49 @@ _hashlib_scrypt(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObj
 
 #endif /* (OPENSSL_VERSION_NUMBER > 0x10100000L && !defined(OPENSSL_NO_SCRYPT) && !defined(LIBRESSL_VERSION_NUMBER)) */
 
+PyDoc_STRVAR(_hashlib_hmac_digest__doc__,
+"hmac_digest($module, /, key, msg, digest)\n"
+"--\n"
+"\n"
+"Single-shot HMAC");
+
+#define _HASHLIB_HMAC_DIGEST_METHODDEF    \
+    {"hmac_digest", (PyCFunction)_hashlib_hmac_digest, METH_FASTCALL|METH_KEYWORDS, _hashlib_hmac_digest__doc__},
+
+static PyObject *
+_hashlib_hmac_digest_impl(PyObject *module, Py_buffer *key, Py_buffer *msg,
+                          const char *digest);
+
+static PyObject *
+_hashlib_hmac_digest(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    static const char * const _keywords[] = {"key", "msg", "digest", NULL};
+    static _PyArg_Parser _parser = {"y*y*s:hmac_digest", _keywords, 0};
+    Py_buffer key = {NULL, NULL};
+    Py_buffer msg = {NULL, NULL};
+    const char *digest;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &key, &msg, &digest)) {
+        goto exit;
+    }
+    return_value = _hashlib_hmac_digest_impl(module, &key, &msg, digest);
+
+exit:
+    /* Cleanup for key */
+    if (key.obj) {
+       PyBuffer_Release(&key);
+    }
+    /* Cleanup for msg */
+    if (msg.obj) {
+       PyBuffer_Release(&msg);
+    }
+
+    return return_value;
+}
+
 #ifndef _HASHLIB_SCRYPT_METHODDEF
     #define _HASHLIB_SCRYPT_METHODDEF
 #endif /* !defined(_HASHLIB_SCRYPT_METHODDEF) */
-/*[clinic end generated code: output=1ea7d0397f38e2c2 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=b5b90821caf05391 input=a9049054013a1b77]*/


From webhook-mailer at python.org  Sat Jan 27 03:54:16 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 27 Jan 2018 08:54:16 -0000
Subject: [Python-checkins] bpo-32454: socket closefd (#5048)
Message-ID: <mailman.77.1517043259.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d0e31b980f18101738d0ec518cb991a5fb73fe93
commit: d0e31b980f18101738d0ec518cb991a5fb73fe93
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-27T09:54:13+01:00
summary:

bpo-32454: socket closefd (#5048)

Add close(fd) function to the socket module

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2017-12-30-10-38-05.bpo-32454.wsZnl-.rst
M Doc/library/socket.rst
M Lib/test/test_socket.py
M Modules/socketmodule.c

diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index db032ca7f1f..7edd4ba3604 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -578,6 +578,14 @@ Other functions
 The :mod:`socket` module also offers various network-related services:
 
 
+.. function:: close(fd)
+
+   Close a socket file descriptor. This is like :func:`os.close`, but for
+   sockets. On some platforms (most noticeable Windows) :func:`os.close`
+   does not work for socket file descriptors.
+
+   .. versionadded:: 3.7
+
 .. function:: getaddrinfo(host, port, family=0, type=0, proto=0, flags=0)
 
    Translate the *host*/*port* argument into a sequence of 5-tuples that contain
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 22d142c8f80..275384c28fb 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1519,6 +1519,22 @@ def test_unusable_closed_socketio(self):
             self.assertRaises(ValueError, fp.writable)
             self.assertRaises(ValueError, fp.seekable)
 
+    def test_socket_close(self):
+        sock = socket.socket()
+        try:
+            sock.bind((HOST, 0))
+            socket.close(sock.fileno())
+            with self.assertRaises(OSError):
+                sock.listen(1)
+        finally:
+            with self.assertRaises(OSError):
+                # sock.close() fails with EBADF
+                sock.close()
+        with self.assertRaises(TypeError):
+            socket.close(None)
+        with self.assertRaises(OSError):
+            socket.close(-1)
+
     def test_makefile_mode(self):
         for mode in 'r', 'rb', 'rw', 'w', 'wb':
             with self.subTest(mode=mode):
diff --git a/Misc/NEWS.d/next/Library/2017-12-30-10-38-05.bpo-32454.wsZnl-.rst b/Misc/NEWS.d/next/Library/2017-12-30-10-38-05.bpo-32454.wsZnl-.rst
new file mode 100644
index 00000000000..b8eaa859fa5
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-30-10-38-05.bpo-32454.wsZnl-.rst
@@ -0,0 +1 @@
+Add close(fd) function to the socket module.
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index d75a51af9a6..5fe2431bee8 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -2836,7 +2836,7 @@ sock_close(PySocketSockObject *s)
     Py_RETURN_NONE;
 }
 
-PyDoc_STRVAR(close_doc,
+PyDoc_STRVAR(sock_close_doc,
 "close()\n\
 \n\
 Close the socket.  It cannot be used after this call.");
@@ -4558,7 +4558,7 @@ static PyMethodDef sock_methods[] = {
     {"bind",              (PyCFunction)sock_bind, METH_O,
                       bind_doc},
     {"close",             (PyCFunction)sock_close, METH_NOARGS,
-                      close_doc},
+                      sock_close_doc},
     {"connect",           (PyCFunction)sock_connect, METH_O,
                       connect_doc},
     {"connect_ex",        (PyCFunction)sock_connect_ex, METH_O,
@@ -5456,6 +5456,31 @@ PyDoc_STRVAR(getprotobyname_doc,
 \n\
 Return the protocol number for the named protocol.  (Rarely used.)");
 
+static PyObject *
+socket_close(PyObject *self, PyObject *fdobj)
+{
+    SOCKET_T fd;
+    int res;
+
+    fd = PyLong_AsSocket_t(fdobj);
+    if (fd == (SOCKET_T)(-1) && PyErr_Occurred())
+        return NULL;
+    Py_BEGIN_ALLOW_THREADS
+    res = SOCKETCLOSE(fd);
+    Py_END_ALLOW_THREADS
+    /* bpo-30319: The peer can already have closed the connection.
+       Python ignores ECONNRESET on close(). */
+    if (res < 0 && !CHECK_ERRNO(ECONNRESET)) {
+        return set_error();
+    }
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(close_doc,
+"close(integer) -> None\n\
+\n\
+Close an integer socket file descriptor.  This is like os.close(), but for\n\
+sockets; on some platforms os.close() won't work for socket file descriptors.");
 
 #ifndef NO_DUP
 /* dup() function for socket fds */
@@ -6397,6 +6422,8 @@ static PyMethodDef socket_methods[] = {
      METH_VARARGS, getservbyport_doc},
     {"getprotobyname",          socket_getprotobyname,
      METH_VARARGS, getprotobyname_doc},
+    {"close",                   socket_close,
+     METH_O, close_doc},
 #ifndef NO_DUP
     {"dup",                     socket_dup,
      METH_O, dup_doc},


From solipsis at pitrou.net  Sat Jan 27 04:18:20 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 27 Jan 2018 09:18:20 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=4
Message-ID: <20180127091820.1.15CE34FBCA5A1769@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 7, -7] memory blocks, sum=0
test_functools leaked [0, 3, 1] memory blocks, sum=4


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflog7jWQJu', '--timeout', '7200']

From webhook-mailer at python.org  Sat Jan 27 07:34:34 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 27 Jan 2018 12:34:34 -0000
Subject: [Python-checkins] bpo-31853: Replaced socket.method calls with
 super() in SSLSocket. (#4048)
Message-ID: <mailman.78.1517056476.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/746cc75541f31278864a10b995e7d009bd2ff053
commit: 746cc75541f31278864a10b995e7d009bd2ff053
branch: master
author: Mads Jensen <mje at inducks.org>
committer: Christian Heimes <christian at python.org>
date: 2018-01-27T13:34:28+01:00
summary:

bpo-31853: Replaced socket.method calls with super() in SSLSocket. (#4048)

files:
A Misc/NEWS.d/next/Library/2017-10-23-22-55-51.bpo-31853.h5fjrP.rst
M Lib/ssl.py

diff --git a/Lib/ssl.py b/Lib/ssl.py
index fa83606e7cd..7c4cccf4c0c 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -786,17 +786,16 @@ def __init__(self, sock=None, keyfile=None, certfile=None,
         self.do_handshake_on_connect = do_handshake_on_connect
         self.suppress_ragged_eofs = suppress_ragged_eofs
         if sock is not None:
-            socket.__init__(self,
-                            family=sock.family,
-                            type=sock.type,
-                            proto=sock.proto,
-                            fileno=sock.fileno())
+            super().__init__(family=sock.family,
+                             type=sock.type,
+                             proto=sock.proto,
+                             fileno=sock.fileno())
             self.settimeout(sock.gettimeout())
             sock.detach()
         elif fileno is not None:
-            socket.__init__(self, fileno=fileno)
+            super().__init__(fileno=fileno)
         else:
-            socket.__init__(self, family=family, type=type, proto=proto)
+            super().__init__(family=family, type=type, proto=proto)
 
         # See if we are connected
         try:
@@ -952,7 +951,7 @@ def send(self, data, flags=0):
                     self.__class__)
             return self._sslobj.write(data)
         else:
-            return socket.send(self, data, flags)
+            return super().send(data, flags)
 
     def sendto(self, data, flags_or_addr, addr=None):
         self._checkClosed()
@@ -960,9 +959,9 @@ def sendto(self, data, flags_or_addr, addr=None):
             raise ValueError("sendto not allowed on instances of %s" %
                              self.__class__)
         elif addr is None:
-            return socket.sendto(self, data, flags_or_addr)
+            return super().sendto(data, flags_or_addr)
         else:
-            return socket.sendto(self, data, flags_or_addr, addr)
+            return super().sendto(data, flags_or_addr, addr)
 
     def sendmsg(self, *args, **kwargs):
         # Ensure programs don't send data unencrypted if they try to
@@ -984,7 +983,7 @@ def sendall(self, data, flags=0):
                     v = self.send(byte_view[count:])
                     count += v
         else:
-            return socket.sendall(self, data, flags)
+            return super().sendall(data, flags)
 
     def sendfile(self, file, offset=0, count=None):
         """Send a file, possibly by using os.sendfile() if this is a
@@ -1005,7 +1004,7 @@ def recv(self, buflen=1024, flags=0):
                     self.__class__)
             return self.read(buflen)
         else:
-            return socket.recv(self, buflen, flags)
+            return super().recv(buflen, flags)
 
     def recv_into(self, buffer, nbytes=None, flags=0):
         self._checkClosed()
@@ -1020,7 +1019,7 @@ def recv_into(self, buffer, nbytes=None, flags=0):
                   self.__class__)
             return self.read(nbytes, buffer)
         else:
-            return socket.recv_into(self, buffer, nbytes, flags)
+            return super().recv_into(buffer, nbytes, flags)
 
     def recvfrom(self, buflen=1024, flags=0):
         self._checkClosed()
@@ -1028,7 +1027,7 @@ def recvfrom(self, buflen=1024, flags=0):
             raise ValueError("recvfrom not allowed on instances of %s" %
                              self.__class__)
         else:
-            return socket.recvfrom(self, buflen, flags)
+            return super().recvfrom(buflen, flags)
 
     def recvfrom_into(self, buffer, nbytes=None, flags=0):
         self._checkClosed()
@@ -1036,7 +1035,7 @@ def recvfrom_into(self, buffer, nbytes=None, flags=0):
             raise ValueError("recvfrom_into not allowed on instances of %s" %
                              self.__class__)
         else:
-            return socket.recvfrom_into(self, buffer, nbytes, flags)
+            return super().recvfrom_into(buffer, nbytes, flags)
 
     def recvmsg(self, *args, **kwargs):
         raise NotImplementedError("recvmsg not allowed on instances of %s" %
@@ -1056,7 +1055,7 @@ def pending(self):
     def shutdown(self, how):
         self._checkClosed()
         self._sslobj = None
-        socket.shutdown(self, how)
+        super().shutdown(how)
 
     def unwrap(self):
         if self._sslobj:
@@ -1068,7 +1067,7 @@ def unwrap(self):
 
     def _real_close(self):
         self._sslobj = None
-        socket._real_close(self)
+        super()._real_close()
 
     def do_handshake(self, block=False):
         """Perform a TLS/SSL handshake."""
@@ -1093,10 +1092,10 @@ def _real_connect(self, addr, connect_ex):
                                  session=self._session)
         try:
             if connect_ex:
-                rc = socket.connect_ex(self, addr)
+                rc = super().connect_ex(addr)
             else:
                 rc = None
-                socket.connect(self, addr)
+                super().connect(addr)
             if not rc:
                 self._connected = True
                 if self.do_handshake_on_connect:
@@ -1121,7 +1120,7 @@ def accept(self):
         a tuple containing that new connection wrapped with a server-side
         SSL channel, and the address of the remote client."""
 
-        newsock, addr = socket.accept(self)
+        newsock, addr = super().accept()
         newsock = self.context.wrap_socket(newsock,
                     do_handshake_on_connect=self.do_handshake_on_connect,
                     suppress_ragged_eofs=self.suppress_ragged_eofs,
diff --git a/Misc/NEWS.d/next/Library/2017-10-23-22-55-51.bpo-31853.h5fjrP.rst b/Misc/NEWS.d/next/Library/2017-10-23-22-55-51.bpo-31853.h5fjrP.rst
new file mode 100644
index 00000000000..516214d0a19
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-10-23-22-55-51.bpo-31853.h5fjrP.rst
@@ -0,0 +1,2 @@
+Use super().method instead of socket.method in SSLSocket.  They were
+there most likely for legacy reasons.


From webhook-mailer at python.org  Sat Jan 27 09:51:41 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Sat, 27 Jan 2018 14:51:41 -0000
Subject: [Python-checkins] bpo-31399: Let OpenSSL verify hostname and IP
 address (#3462)
Message-ID: <mailman.79.1517064705.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/61d478c71c5341cdc54e6bfb4ace4252852fd972
commit: 61d478c71c5341cdc54e6bfb4ace4252852fd972
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-27T15:51:38+01:00
summary:

bpo-31399: Let OpenSSL verify hostname and IP address (#3462)

bpo-31399: Let OpenSSL verify hostname and IP

The ssl module now uses OpenSSL's X509_VERIFY_PARAM_set1_host() and
X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses.

* Remove match_hostname calls
* Check for libssl with set1_host, libssl must provide X509_VERIFY_PARAM_set1_host()
* Add documentation for OpenSSL 1.0.2 requirement
* Don't support OpenSSL special mode with a leading dot, e.g. ".example.org" matches "www.example.org". It's not standard conform.
* Add hostname_checks_common_name

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2017-09-08-14-05-33.bpo-31399.FtBrrt.rst
M Doc/library/ssl.rst
M Doc/whatsnew/3.7.rst
M Lib/asyncio/sslproto.py
M Lib/http/client.py
M Lib/ssl.py
M Lib/test/test_asyncio/test_events.py
M Lib/test/test_ftplib.py
M Lib/test/test_imaplib.py
M Lib/test/test_poplib.py
M Lib/test/test_ssl.py
M Lib/test/test_urllib2_localnet.py
M Modules/_ssl.c
M PC/pyconfig.h
M setup.py

diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst
index 4c44ffa23ac..aa1075d4b02 100644
--- a/Doc/library/ssl.rst
+++ b/Doc/library/ssl.rst
@@ -146,9 +146,10 @@ Functions, Constants, and Exceptions
 
 .. exception:: CertificateError
 
-   Raised to signal an error with a certificate (such as mismatching
-   hostname).  Certificate errors detected by OpenSSL, though, raise
-   an :exc:`SSLCertVerificationError`.
+   An alias for :exc:`SSLCertVerificationError`.
+
+   .. versionchanged:: 3.7
+      The exception is now an alias for :exc:`SSLCertVerificationError`.
 
 
 Socket creation
@@ -430,8 +431,14 @@ Certificate handling
       of the certificate, is now supported.
 
    .. versionchanged:: 3.7
+      The function is no longer used to TLS connections. Hostname matching
+      is now performed by OpenSSL.
+
       Allow wildcard when it is the leftmost and the only character
-      in that segment.
+      in that segment. Partial wildcards like ``www*.example.com`` are no
+      longer supported.
+
+   .. deprecated:: 3.7
 
 .. function:: cert_time_to_seconds(cert_time)
 
@@ -850,6 +857,14 @@ Constants
 
    .. versionadded:: 3.5
 
+.. data:: HAS_NEVER_CHECK_COMMON_NAME
+
+   Whether the OpenSSL library has built-in support not checking subject
+   common name and :attr:`SSLContext.hostname_checks_common_name` is
+   writeable.
+
+   .. versionadded:: 3.7
+
 .. data:: HAS_ECDH
 
    Whether the OpenSSL library has built-in support for Elliptic Curve-based
@@ -1075,6 +1090,12 @@ SSL sockets also have the following additional methods and attributes:
       The socket timeout is no more reset each time bytes are received or sent.
       The socket timeout is now to maximum total duration of the handshake.
 
+   .. versionchanged:: 3.7
+      Hostname or IP address is matched by OpenSSL during handshake. The
+      function :func:`match_hostname` is no longer used. In case OpenSSL
+      refuses a hostname or IP address, the handshake is aborted early and
+      a TLS alert message is send to the peer.
+
 .. method:: SSLSocket.getpeercert(binary_form=False)
 
    If there is no certificate for the peer on the other end of the connection,
@@ -1730,6 +1751,17 @@ to speed up repeated connections from the same clients.
    The protocol version chosen when constructing the context.  This attribute
    is read-only.
 
+.. attribute:: SSLContext.hostname_checks_common_name
+
+   Whether :attr:`~SSLContext.check_hostname` falls back to verify the cert's
+   subject common name in the absence of a subject alternative name
+   extension (default: true).
+
+   .. versionadded:: 3.7
+
+   .. note::
+      Only writeable with OpenSSL 1.1.0 or higher.
+
 .. attribute:: SSLContext.verify_flags
 
    The flags for certificate verification operations. You can set flags like
@@ -2324,6 +2356,10 @@ in this case, the :func:`match_hostname` function can be used.  This common
 check is automatically performed when :attr:`SSLContext.check_hostname` is
 enabled.
 
+.. versionchanged:: 3.7
+   Hostname matchings is now performed by OpenSSL. Python no longer uses
+   :func:`match_hostname`.
+
 In server mode, if you want to authenticate your clients using the SSL layer
 (rather than using a higher-level authentication mechanism), you'll also have
 to specify :const:`CERT_REQUIRED` and similarly check the client certificate.
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 133975a04dd..1ece6a3befd 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -568,6 +568,32 @@ can be set within the scope of a group.
 ``'^$'`` or ``(?=-)`` that matches an empty string.
 (Contributed by Serhiy Storchaka in :issue:`25054`.)
 
+ssl
+---
+
+The ssl module now uses OpenSSL's builtin API instead of
+:func:`~ssl.match_hostname` to check host name or IP address. Values
+are validated during TLS handshake. Any cert validation error including
+a failing host name match now raises :exc:`~ssl.SSLCertVerificationError` and
+aborts the handshake with a proper TLS Alert message. The new exception
+contains additional information. Host name validation can be customized
+with :attr:`~ssl.SSLContext.host_flags`.
+(Contributed by Christian Heimes in :issue:`31399`.)
+
+.. note::
+   The improved host name check requires an OpenSSL 1.0.2 or 1.1 compatible
+   libssl. OpenSSL 0.9.8 and 1.0.1 are no longer supported. LibreSSL is
+   temporarily not supported until it gains the necessary OpenSSL 1.0.2 APIs.
+
+The ssl module no longer sends IP addresses in SNI TLS extension.
+(Contributed by Christian Heimes in :issue:`32185`.)
+
+:func:`~ssl.match_hostname` no longer supports partial wildcards like
+``www*.example.org``. :attr:`~ssl.SSLContext.host_flags` has partial
+wildcard matching disabled by default.
+(Contributed by Mandeep Singh in :issue:`23033` and Christian Heimes in
+:issue:`31399`.)
+
 string
 ------
 
@@ -1120,6 +1146,12 @@ Other CPython implementation changes
   emitted in the first place), and an explicit ``error::BytesWarning``
   warnings filter added to convert them to exceptions.
 
+* CPython' :mod:`ssl` module requires OpenSSL 1.0.2 or 1.1 compatible libssl.
+  OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer
+  supported. LibreSSL is temporarily not supported as well. LibreSSL releases
+  up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs.
+
+
 Documentation
 =============
 
diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py
index 2d377c4ae39..1130bced8ae 100644
--- a/Lib/asyncio/sslproto.py
+++ b/Lib/asyncio/sslproto.py
@@ -590,12 +590,6 @@ def _on_handshake_complete(self, handshake_exc):
                 raise handshake_exc
 
             peercert = sslobj.getpeercert()
-            if not hasattr(self._sslcontext, 'check_hostname'):
-                # Verify hostname if requested, Python 3.4+ uses check_hostname
-                # and checks the hostname in do_handshake()
-                if (self._server_hostname and
-                        self._sslcontext.verify_mode != ssl.CERT_NONE):
-                    ssl.match_hostname(peercert, self._server_hostname)
         except BaseException as exc:
             if self._loop.get_debug():
                 if isinstance(exc, ssl.CertificateError):
diff --git a/Lib/http/client.py b/Lib/http/client.py
index 1a852cd76e4..1292db74784 100644
--- a/Lib/http/client.py
+++ b/Lib/http/client.py
@@ -1375,7 +1375,8 @@ def __init__(self, host, port=None, key_file=None, cert_file=None,
             if key_file or cert_file:
                 context.load_cert_chain(cert_file, key_file)
             self._context = context
-            self._check_hostname = check_hostname
+            if check_hostname is not None:
+                self._context.check_hostname = check_hostname
 
         def connect(self):
             "Connect to a host on a given (SSL) port."
@@ -1389,13 +1390,6 @@ def connect(self):
 
             self.sock = self._context.wrap_socket(self.sock,
                                                   server_hostname=server_hostname)
-            if not self._context.check_hostname and self._check_hostname:
-                try:
-                    ssl.match_hostname(self.sock.getpeercert(), server_hostname)
-                except Exception:
-                    self.sock.shutdown(socket.SHUT_RDWR)
-                    self.sock.close()
-                    raise
 
     __all__.append("HTTPSConnection")
 
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 7c4cccf4c0c..5f972e1d3d8 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -148,7 +148,6 @@
     lambda name: name.startswith('CERT_'),
     source=_ssl)
 
-
 PROTOCOL_SSLv23 = _SSLMethod.PROTOCOL_SSLv23 = _SSLMethod.PROTOCOL_TLS
 _PROTOCOL_NAMES = {value: name for name, value in _SSLMethod.__members__.items()}
 
@@ -172,6 +171,8 @@
 else:
     CHANNEL_BINDING_TYPES = []
 
+HAS_NEVER_CHECK_COMMON_NAME = hasattr(_ssl, 'HOSTFLAG_NEVER_CHECK_SUBJECT')
+
 
 # Disable weak or insecure ciphers by default
 # (OpenSSL's default setting is 'DEFAULT:!aNULL:!eNULL')
@@ -216,9 +217,7 @@
     '!aNULL:!eNULL:!MD5:!DSS:!RC4:!3DES'
 )
 
-
-class CertificateError(ValueError):
-    pass
+CertificateError = SSLCertVerificationError
 
 
 def _dnsname_match(dn, hostname):
@@ -473,6 +472,23 @@ def options(self):
     def options(self, value):
         super(SSLContext, SSLContext).options.__set__(self, value)
 
+    if hasattr(_ssl, 'HOSTFLAG_NEVER_CHECK_SUBJECT'):
+        @property
+        def hostname_checks_common_name(self):
+            ncs = self._host_flags & _ssl.HOSTFLAG_NEVER_CHECK_SUBJECT
+            return ncs != _ssl.HOSTFLAG_NEVER_CHECK_SUBJECT
+
+        @hostname_checks_common_name.setter
+        def hostname_checks_common_name(self, value):
+            if value:
+                self._host_flags &= ~_ssl.HOSTFLAG_NEVER_CHECK_SUBJECT
+            else:
+                self._host_flags |= _ssl.HOSTFLAG_NEVER_CHECK_SUBJECT
+    else:
+        @property
+        def hostname_checks_common_name(self):
+            return True
+
     @property
     def verify_flags(self):
         return VerifyFlags(super().verify_flags)
@@ -699,11 +715,6 @@ def pending(self):
     def do_handshake(self):
         """Start the SSL/TLS handshake."""
         self._sslobj.do_handshake()
-        if self.context.check_hostname:
-            if not self.server_hostname:
-                raise ValueError("check_hostname needs server_hostname "
-                                 "argument")
-            match_hostname(self.getpeercert(), self.server_hostname)
 
     def unwrap(self):
         """Start the SSL shutdown handshake."""
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index e4b053681b4..cf217538a06 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -1148,11 +1148,13 @@ def test_create_server_ssl_match_failed(self):
             with test_utils.disable_logger():
                 with self.assertRaisesRegex(
                         ssl.CertificateError,
-                        "hostname '127.0.0.1' doesn't match 'localhost'"):
+                        "IP address mismatch, certificate is not valid for "
+                        "'127.0.0.1'"):
                     self.loop.run_until_complete(f_c)
 
         # close connection
-        proto.transport.close()
+        # transport is None because TLS ALERT aborted the handshake
+        self.assertIsNone(proto.transport)
         server.close()
 
     @support.skip_unless_bind_unix_socket
diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py
index f1b0185b2bf..bb5a67f27e3 100644
--- a/Lib/test/test_ftplib.py
+++ b/Lib/test/test_ftplib.py
@@ -330,6 +330,9 @@ def _do_ssl_handshake(self):
                     return
                 elif err.args[0] == ssl.SSL_ERROR_EOF:
                     return self.handle_close()
+                # TODO: SSLError does not expose alert information
+                elif "SSLV3_ALERT_BAD_CERTIFICATE" in err.args[1]:
+                    return self.handle_close()
                 raise
             except OSError as err:
                 if err.args[0] == errno.ECONNABORTED:
diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py
index 4a45be65724..f16bacd0006 100644
--- a/Lib/test/test_imaplib.py
+++ b/Lib/test/test_imaplib.py
@@ -485,7 +485,8 @@ def test_ssl_raises(self):
         ssl_context.load_verify_locations(CAFILE)
 
         with self.assertRaisesRegex(ssl.CertificateError,
-                "hostname '127.0.0.1' doesn't match 'localhost'"):
+                "IP address mismatch, certificate is not valid for "
+                "'127.0.0.1'"):
             _, server = self._setup(SimpleIMAPHandler)
             client = self.imap_class(*server.server_address,
                                      ssl_context=ssl_context)
@@ -874,7 +875,8 @@ def test_ssl_verified(self):
 
         with self.assertRaisesRegex(
                 ssl.CertificateError,
-                "hostname '127.0.0.1' doesn't match 'localhost'"):
+                "IP address mismatch, certificate is not valid for "
+                "'127.0.0.1'"):
             with self.reaped_server(SimpleIMAPHandler) as server:
                 client = self.imap_class(*server.server_address,
                                          ssl_context=ssl_context)
diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py
index 9ba678f2039..4d7a4394086 100644
--- a/Lib/test/test_poplib.py
+++ b/Lib/test/test_poplib.py
@@ -176,6 +176,9 @@ def _do_tls_handshake(self):
                     return
                 elif err.args[0] == ssl.SSL_ERROR_EOF:
                     return self.handle_close()
+                # TODO: SSLError does not expose alert information
+                elif "SSLV3_ALERT_BAD_CERTIFICATE" in err.args[1]:
+                    return self.handle_close()
                 raise
             except OSError as err:
                 if err.args[0] == errno.ECONNABORTED:
diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index e3fa4233ed6..fdf727f788c 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -988,6 +988,19 @@ def test_verify_mode_protocol(self):
         self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED)
         self.assertTrue(ctx.check_hostname)
 
+    def test_hostname_checks_common_name(self):
+        ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+        self.assertTrue(ctx.hostname_checks_common_name)
+        if ssl.HAS_NEVER_CHECK_COMMON_NAME:
+            ctx.hostname_checks_common_name = True
+            self.assertTrue(ctx.hostname_checks_common_name)
+            ctx.hostname_checks_common_name = False
+            self.assertFalse(ctx.hostname_checks_common_name)
+            ctx.hostname_checks_common_name = True
+            self.assertTrue(ctx.hostname_checks_common_name)
+        else:
+            with self.assertRaises(AttributeError):
+                ctx.hostname_checks_common_name = True
 
     @unittest.skipUnless(have_verify_flags(),
                          "verify_flags need OpenSSL > 0.9.8")
@@ -1511,6 +1524,16 @@ def test_bad_idna_in_server_hostname(self):
             ctx.wrap_bio(ssl.MemoryBIO(), ssl.MemoryBIO(),
                          server_hostname="xn--.com")
 
+    def test_bad_server_hostname(self):
+        ctx = ssl.create_default_context()
+        with self.assertRaises(ValueError):
+            ctx.wrap_bio(ssl.MemoryBIO(), ssl.MemoryBIO(),
+                         server_hostname="")
+        with self.assertRaises(ValueError):
+            ctx.wrap_bio(ssl.MemoryBIO(), ssl.MemoryBIO(),
+                         server_hostname=".example.org")
+
+
 class MemoryBIOTests(unittest.TestCase):
 
     def test_read_write(self):
@@ -2536,8 +2559,9 @@ def test_check_hostname(self):
         with server:
             with client_context.wrap_socket(socket.socket(),
                                             server_hostname="invalid") as s:
-                with self.assertRaisesRegex(ssl.CertificateError,
-                                            "hostname 'invalid' doesn't match 'localhost'"):
+                with self.assertRaisesRegex(
+                        ssl.CertificateError,
+                        "Hostname mismatch, certificate is not valid for 'invalid'."):
                     s.connect((HOST, server.port))
 
         # missing server_hostname arg should cause an exception, too
diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py
index b2d1e5f9804..52c897af433 100644
--- a/Lib/test/test_urllib2_localnet.py
+++ b/Lib/test/test_urllib2_localnet.py
@@ -573,7 +573,7 @@ def test_https_with_cafile(self):
                              cafile=CERT_fakehostname)
             # Good cert, but mismatching hostname
             handler = self.start_https_server(certfile=CERT_fakehostname)
-            with self.assertRaises(ssl.CertificateError) as cm:
+            with self.assertRaises(urllib.error.URLError) as cm:
                 self.urlopen("https://localhost:%s/bizarre" % handler.port,
                              cafile=CERT_fakehostname)
 
diff --git a/Misc/NEWS.d/next/Library/2017-09-08-14-05-33.bpo-31399.FtBrrt.rst b/Misc/NEWS.d/next/Library/2017-09-08-14-05-33.bpo-31399.FtBrrt.rst
new file mode 100644
index 00000000000..e50ce2a1a93
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-09-08-14-05-33.bpo-31399.FtBrrt.rst
@@ -0,0 +1,4 @@
+The ssl module now uses OpenSSL's X509_VERIFY_PARAM_set1_host() and
+X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses. Subject
+common name fallback can be disabled with
+SSLContext.hostname_checks_common_name.
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index c5eec7eded6..ec8c8af54bd 100644
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -64,10 +64,13 @@ static PySocketModule_APIObject PySocketModule;
 #include "openssl/rand.h"
 #include "openssl/bio.h"
 
-/* Set HAVE_X509_VERIFY_PARAM_SET1_HOST for non-autoconf builds */
 #ifndef HAVE_X509_VERIFY_PARAM_SET1_HOST
-#  if !defined(LIBRESSL_VERSION_NUMBER) && OPENSSL_VERSION_NUMBER > 0x1000200fL
+#  ifdef LIBRESSL_VERSION_NUMBER
+#    error "LibreSSL is missing X509_VERIFY_PARAM_set1_host(), see https://github.com/libressl-portable/portable/issues/381"
+#  elif OPENSSL_VERSION_NUMBER > 0x1000200fL
 #    define HAVE_X509_VERIFY_PARAM_SET1_HOST
+#  else
+#    error "libssl is too old and does not support X509_VERIFY_PARAM_set1_host()"
 #  endif
 #endif
 
@@ -217,11 +220,6 @@ static STACK_OF(X509_OBJECT) *X509_STORE_get0_objects(X509_STORE *store) {
     return store->objs;
 }
 
-static X509_VERIFY_PARAM *X509_STORE_get0_param(X509_STORE *store)
-{
-    return store->param;
-}
-
 static int
 SSL_SESSION_has_ticket(const SSL_SESSION *s)
 {
@@ -317,6 +315,10 @@ typedef struct {
     PyObject *set_hostname;
 #endif
     int check_hostname;
+    /* OpenSSL has no API to get hostflags from X509_VERIFY_PARAM* struct.
+     * We have to maintain our own copy. OpenSSL's hostflags default to 0.
+     */
+    unsigned int hostflags;
 } PySSLContext;
 
 typedef struct {
@@ -701,6 +703,74 @@ _setSSLError (const char *errstr, int errcode, const char *filename, int lineno)
     return NULL;
 }
 
+/*
+ * SSL objects
+ */
+
+static int
+_ssl_configure_hostname(PySSLSocket *self, const char* server_hostname)
+{
+    int retval = -1;
+    ASN1_OCTET_STRING *ip;
+    PyObject *hostname;
+    size_t len;
+
+    assert(server_hostname);
+
+    /* Disable OpenSSL's special mode with leading dot in hostname:
+     * When name starts with a dot (e.g ".example.com"), it will be
+     * matched by a certificate valid for any sub-domain of name.
+     */
+    len = strlen(server_hostname);
+    if (len == 0 || *server_hostname == '.') {
+        PyErr_SetString(
+            PyExc_ValueError,
+            "server_hostname cannot be an empty string or start with a "
+            "leading dot.");
+        return retval;
+    }
+
+    /* inet_pton is not available on all platforms. */
+    ip = a2i_IPADDRESS(server_hostname);
+    if (ip == NULL) {
+        ERR_clear_error();
+    }
+
+    hostname = PyUnicode_Decode(server_hostname, len, "idna", "strict");
+    if (hostname == NULL) {
+        goto error;
+    }
+    self->server_hostname = hostname;
+
+    /* Only send SNI extension for non-IP hostnames */
+    if (ip == NULL) {
+        if (!SSL_set_tlsext_host_name(self->ssl, server_hostname)) {
+            _setSSLError(NULL, 0, __FILE__, __LINE__);
+        }
+    }
+    if (self->ctx->check_hostname) {
+        X509_VERIFY_PARAM *param = SSL_get0_param(self->ssl);
+        if (ip == NULL) {
+            if (!X509_VERIFY_PARAM_set1_host(param, server_hostname, 0)) {
+                _setSSLError(NULL, 0, __FILE__, __LINE__);
+                goto error;
+            }
+        } else {
+            if (!X509_VERIFY_PARAM_set1_ip(param, ASN1_STRING_data(ip),
+                                           ASN1_STRING_length(ip))) {
+                _setSSLError(NULL, 0, __FILE__, __LINE__);
+                goto error;
+            }
+        }
+    }
+    retval = 0;
+  error:
+    if (ip != NULL) {
+        ASN1_OCTET_STRING_free(ip);
+    }
+    return retval;
+}
+
 static PySSLSocket *
 newPySSLSocket(PySSLContext *sslctx, PySocketSockObject *sock,
                enum py_ssl_server_or_client socket_type,
@@ -722,15 +792,6 @@ newPySSLSocket(PySSLContext *sslctx, PySocketSockObject *sock,
     self->shutdown_seen_zero = 0;
     self->owner = NULL;
     self->server_hostname = NULL;
-    if (server_hostname != NULL) {
-        PyObject *hostname = PyUnicode_Decode(server_hostname, strlen(server_hostname),
-                                              "idna", "strict");
-        if (hostname == NULL) {
-            Py_DECREF(self);
-            return NULL;
-        }
-        self->server_hostname = hostname;
-    }
     self->ssl_errno = 0;
     self->c_errno = 0;
 #ifdef MS_WINDOWS
@@ -761,10 +822,12 @@ newPySSLSocket(PySSLContext *sslctx, PySocketSockObject *sock,
 #endif
     SSL_set_mode(self->ssl, mode);
 
-#if HAVE_SNI
-    if (server_hostname != NULL)
-        SSL_set_tlsext_host_name(self->ssl, server_hostname);
-#endif
+    if (server_hostname != NULL) {
+        if (_ssl_configure_hostname(self, server_hostname) < 0) {
+            Py_DECREF(self);
+            return NULL;
+        }
+    }
     /* If the socket is in non-blocking mode or timeout mode, set the BIO
      * to non-blocking mode (blocking is the default)
      */
@@ -2711,6 +2774,7 @@ _ssl__SSLContext_impl(PyTypeObject *type, int proto_version)
     PySSLContext *self;
     long options;
     SSL_CTX *ctx = NULL;
+    X509_VERIFY_PARAM *params;
     int result;
 #if defined(SSL_MODE_RELEASE_BUFFERS)
     unsigned long libver;
@@ -2760,6 +2824,7 @@ _ssl__SSLContext_impl(PyTypeObject *type, int proto_version)
         return NULL;
     }
     self->ctx = ctx;
+    self->hostflags = X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS;
 #if defined(OPENSSL_NPN_NEGOTIATED) && !defined(OPENSSL_NO_NEXTPROTONEG)
     self->npn_protocols = NULL;
 #endif
@@ -2858,14 +2923,13 @@ _ssl__SSLContext_impl(PyTypeObject *type, int proto_version)
                                    sizeof(SID_CTX));
 #undef SID_CTX
 
+    params = SSL_CTX_get0_param(self->ctx);
 #ifdef X509_V_FLAG_TRUSTED_FIRST
-    {
-        /* Improve trust chain building when cross-signed intermediate
-           certificates are present. See https://bugs.python.org/issue23476. */
-        X509_STORE *store = SSL_CTX_get_cert_store(self->ctx);
-        X509_STORE_set_flags(store, X509_V_FLAG_TRUSTED_FIRST);
-    }
+    /* Improve trust chain building when cross-signed intermediate
+       certificates are present. See https://bugs.python.org/issue23476. */
+    X509_VERIFY_PARAM_set_flags(params, X509_V_FLAG_TRUSTED_FIRST);
 #endif
+    X509_VERIFY_PARAM_set_hostflags(params, self->hostflags);
 
     return (PyObject *)self;
 }
@@ -3152,12 +3216,10 @@ set_verify_mode(PySSLContext *self, PyObject *arg, void *c)
 static PyObject *
 get_verify_flags(PySSLContext *self, void *c)
 {
-    X509_STORE *store;
     X509_VERIFY_PARAM *param;
     unsigned long flags;
 
-    store = SSL_CTX_get_cert_store(self->ctx);
-    param = X509_STORE_get0_param(store);
+    param = SSL_CTX_get0_param(self->ctx);
     flags = X509_VERIFY_PARAM_get_flags(param);
     return PyLong_FromUnsignedLong(flags);
 }
@@ -3165,14 +3227,12 @@ get_verify_flags(PySSLContext *self, void *c)
 static int
 set_verify_flags(PySSLContext *self, PyObject *arg, void *c)
 {
-    X509_STORE *store;
     X509_VERIFY_PARAM *param;
     unsigned long new_flags, flags, set, clear;
 
     if (!PyArg_Parse(arg, "k", &new_flags))
         return -1;
-    store = SSL_CTX_get_cert_store(self->ctx);
-    param = X509_STORE_get0_param(store);
+    param = SSL_CTX_get0_param(self->ctx);
     flags = X509_VERIFY_PARAM_get_flags(param);
     clear = flags & ~new_flags;
     set = ~flags & new_flags;
@@ -3220,6 +3280,27 @@ set_options(PySSLContext *self, PyObject *arg, void *c)
     return 0;
 }
 
+static PyObject *
+get_host_flags(PySSLContext *self, void *c)
+{
+    return PyLong_FromUnsignedLong(self->hostflags);
+}
+
+static int
+set_host_flags(PySSLContext *self, PyObject *arg, void *c)
+{
+    X509_VERIFY_PARAM *param;
+    unsigned int new_flags = 0;
+
+    if (!PyArg_Parse(arg, "I", &new_flags))
+        return -1;
+
+    param = SSL_CTX_get0_param(self->ctx);
+    self->hostflags = new_flags;
+    X509_VERIFY_PARAM_set_hostflags(param, new_flags);
+    return 0;
+}
+
 static PyObject *
 get_check_hostname(PySSLContext *self, void *c)
 {
@@ -4104,6 +4185,8 @@ _ssl__SSLContext_get_ca_certs_impl(PySSLContext *self, int binary_form)
 static PyGetSetDef context_getsetlist[] = {
     {"check_hostname", (getter) get_check_hostname,
                        (setter) set_check_hostname, NULL},
+    {"_host_flags", (getter) get_host_flags,
+                    (setter) set_host_flags, NULL},
     {"options", (getter) get_options,
                 (setter) set_options, NULL},
     {"verify_flags", (getter) get_verify_flags,
@@ -5491,6 +5574,31 @@ PyInit__ssl(void)
                             SSL_OP_NO_COMPRESSION);
 #endif
 
+#ifdef X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT
+    PyModule_AddIntConstant(m, "HOSTFLAG_ALWAYS_CHECK_SUBJECT",
+                            X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT);
+#endif
+#ifdef X509_CHECK_FLAG_NEVER_CHECK_SUBJECT
+    PyModule_AddIntConstant(m, "HOSTFLAG_NEVER_CHECK_SUBJECT",
+                            X509_CHECK_FLAG_NEVER_CHECK_SUBJECT);
+#endif
+#ifdef X509_CHECK_FLAG_NO_WILDCARDS
+    PyModule_AddIntConstant(m, "HOSTFLAG_NO_WILDCARDS",
+                            X509_CHECK_FLAG_NO_WILDCARDS);
+#endif
+#ifdef X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS
+    PyModule_AddIntConstant(m, "HOSTFLAG_NO_PARTIAL_WILDCARDS",
+                            X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS);
+#endif
+#ifdef X509_CHECK_FLAG_MULTI_LABEL_WILDCARDS
+    PyModule_AddIntConstant(m, "HOSTFLAG_MULTI_LABEL_WILDCARDS",
+                            X509_CHECK_FLAG_MULTI_LABEL_WILDCARDS);
+#endif
+#ifdef X509_CHECK_FLAG_SINGLE_LABEL_SUBDOMAINS
+    PyModule_AddIntConstant(m, "HOSTFLAG_SINGLE_LABEL_SUBDOMAINS",
+                            X509_CHECK_FLAG_SINGLE_LABEL_SUBDOMAINS);
+#endif
+
 #if HAVE_SNI
     r = Py_True;
 #else
diff --git a/PC/pyconfig.h b/PC/pyconfig.h
index db745dee761..d2a3f5dd39b 100644
--- a/PC/pyconfig.h
+++ b/PC/pyconfig.h
@@ -687,4 +687,7 @@ Py_NO_ENABLE_SHARED to find out.  Also support MS_NO_COREDLL for b/w compat */
 /* framework name */
 #define _PYTHONFRAMEWORK ""
 
+/* Define if libssl has X509_VERIFY_PARAM_set1_host and related function */
+#define HAVE_X509_VERIFY_PARAM_SET1_HOST 1
+
 #endif /* !Py_CONFIG_H */
diff --git a/setup.py b/setup.py
index a6f4488cc99..ba0a7624cfc 100644
--- a/setup.py
+++ b/setup.py
@@ -363,6 +363,16 @@ def print_three_column(lst):
             print_three_column(failed)
             print()
 
+        if any('_ssl' in l
+               for l in (missing, self.failed, self.failed_on_import)):
+            print()
+            print("Could not build the ssl module!")
+            print("Python requires an OpenSSL 1.0.2 or 1.1 compatible "
+                  "libssl with X509_VERIFY_PARAM_set1_host().")
+            print("LibreSSL 2.6.4 and earlier do not provide the necessary "
+                  "APIs, https://github.com/libressl-portable/portable/issues/381")
+            print()
+
     def build_extension(self, ext):
 
         if ext.name == '_ctypes':
@@ -2144,13 +2154,16 @@ def split_var(name, sep):
         if krb5_h:
             ssl_incs.extend(krb5_h)
 
-        ssl_ext = Extension(
-            '_ssl', ['_ssl.c'],
-            include_dirs=openssl_includes,
-            library_dirs=openssl_libdirs,
-            libraries=openssl_libs,
-            depends=['socketmodule.h']
-        )
+        if config_vars.get("HAVE_X509_VERIFY_PARAM_SET1_HOST"):
+            ssl_ext = Extension(
+                '_ssl', ['_ssl.c'],
+                include_dirs=openssl_includes,
+                library_dirs=openssl_libdirs,
+                libraries=openssl_libs,
+                depends=['socketmodule.h']
+            )
+        else:
+            ssl_ext = None
 
         hashlib_ext = Extension(
             '_hashlib', ['_hashopenssl.c'],


From webhook-mailer at python.org  Sat Jan 27 10:04:49 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sat, 27 Jan 2018 15:04:49 -0000
Subject: [Python-checkins] bpo-32532: Improve documentation of settrace and
 setprofile (#5359)
Message-ID: <mailman.80.1517065490.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/60da99b8e2f7bf497569ae4d6c218866575729bf
commit: 60da99b8e2f7bf497569ae4d6c218866575729bf
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-27T23:04:47+08:00
summary:

bpo-32532: Improve documentation of settrace and setprofile (#5359)

Mention in the documentation of settrace and setprofile that errors in the
registered handlers will cause themselves unset.

files:
M Doc/library/sys.rst

diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index ab08f61f6c0..67925e426bb 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1089,7 +1089,8 @@ always available.
    but the return event is reported even when an exception has been set). The function is
    thread-specific, but there is no way for the profiler to know about context switches between
    threads, so it does not make sense to use this in the presence of multiple threads. Also,
-   its return value is not used, so it can simply return ``None``.
+   its return value is not used, so it can simply return ``None``.  Error in the profile
+   function will cause itself unset.
 
    Profile functions should have three arguments: *frame*, *event*, and
    *arg*. *frame* is the current stack frame.  *event* is a string: ``'call'``,
@@ -1173,6 +1174,9 @@ always available.
    function for further tracing in that scope), or ``None`` to turn off tracing
    in that scope.
 
+   If there is any error occurred in the trace function, it will be unset, just
+   like ``settrace(None)`` is called.
+
    The events have the following meaning:
 
    ``'call'``


From webhook-mailer at python.org  Sat Jan 27 11:16:40 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Sat, 27 Jan 2018 16:16:40 -0000
Subject: [Python-checkins] bpo-31368: Expose preadv and pwritev in the os
 module (#5239)
Message-ID: <mailman.81.1517069801.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/4defba3b95ec0f52ce75b8466831d30fb5d333f3
commit: 4defba3b95ec0f52ce75b8466831d30fb5d333f3
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-27T17:16:37+01:00
summary:

bpo-31368: Expose preadv and pwritev in the os module (#5239)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-19-01-54-22.bpo-31368.kzKqUR.rst
M Doc/library/os.rst
M Lib/test/test_posix.py
M Modules/clinic/posixmodule.c.h
M Modules/posixmodule.c
M aclocal.m4
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index 9cb34deb042..b8f57f55eb4 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -1102,6 +1102,45 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo
    .. versionadded:: 3.3
 
 
+.. function:: pwritev(fd, buffers, offset, flags=0)
+
+   Combines the functionality of :func:`os.writev` and :func:`os.pwrite`. It
+   writes the contents of *buffers* to file descriptor *fd* at offset *offset*.
+   *buffers* must be a sequence of :term:`bytes-like objects <bytes-like object>`.
+   Buffers are processed in array order. Entire contents of first buffer is written
+   before proceeding to second, and so on. The operating system may set a limit
+   (sysconf() value SC_IOV_MAX) on the number of buffers that can be used.
+   :func:`~os.pwritev` writes the contents of each object to the file descriptor
+   and returns the total number of bytes written.
+
+   The *flags* argument contains a bitwise OR of zero or more of the following
+   flags:
+
+   - RWF_DSYNC
+   - RWF_SYNC
+
+   Using non-zero flags requires Linux 4.7 or newer.
+
+   Availability: Linux (version 2.6.30), FreeBSD 6.0 and newer,
+   OpenBSD (version 2.7 and newer).
+
+   .. versionadded:: 3.7
+
+.. data:: RWF_DSYNC (since Linux 4.7)
+   Provide a per-write equivalent of the O_DSYNC open(2) flag. This flag
+   is meaningful only for pwritev2(), and its effect applies only to the
+   data range written by the system call.
+
+   .. versionadded:: 3.7
+
+.. data:: RWF_SYNC (since Linux 4.7)
+   Provide a per-write equivalent of the O_SYNC open(2) flag. This flag is
+   meaningful only for pwritev2(), and its effect applies only to the data
+   range written by the system call.
+
+   .. versionadded:: 3.7
+
+
 .. function:: read(fd, n)
 
    Read at most *n* bytes from file descriptor *fd*. Return a bytestring containing the
@@ -1196,6 +1235,51 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo
    .. versionadded:: 3.3
 
 
+.. function:: preadv(fd, buffers, offset, flags=0)
+
+   Combines the functionality of :func:`os.readv` and :func:`os.pread`. It
+   reads from a file descriptor *fd* into a number of mutable :term:`bytes-like
+   objects <bytes-like object>` *buffers*. As :func:`os.readv`, it will transfer
+   data into each buffer until it is full and then move on to the next buffer in
+   the sequence to hold the rest of the data. Its fourth argument, *offset*,
+   specifies the file offset at which the input operation is to be performed.
+   :func:`~os.preadv` return the total number of bytes read (which can be less than
+   the total capacity of all the objects).
+
+   The flags argument contains a bitwise OR of zero or more of the following
+   flags:
+
+   - RWF_HIPRI
+   - RWF_NOWAIT
+
+   Using non-zero flags requires Linux 4.6 or newer.
+
+   Availability: Linux (version 2.6.30), FreeBSD 6.0 and newer,
+   OpenBSD (version 2.7 and newer).
+
+   .. versionadded:: 3.7
+
+
+.. data:: RWF_HIPRI (since Linux 4.6)
+   High priority read/write. Allows block-based filesystems to use polling
+   of the device, which provides lower latency, but may use additional
+   resources. (Currently, this feature is usable only on a file descriptor
+   opened using the O_DIRECT flag.)
+
+   .. versionadded:: 3.7
+
+
+.. data:: RWF_NOWAIT (since Linux 4.14)
+   Do not wait for data which is not immediately available. If this flag
+   is  specified, the preadv2() system call will return instantly
+   if it would have to read data from the backing storage or wait for a lock.
+   If some data was successfully read, it will return the number of bytes
+   read. If no bytes were read, it will return -1 and set errno to EAGAIN.
+   Currently, this flag is meaningful only for preadv2().
+
+   .. versionadded:: 3.7
+
+
 .. function:: tcgetpgrp(fd)
 
    Return the process group associated with the terminal given by *fd* (an open
diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
index 6c504063179..a7f3d34f66d 100644
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -272,6 +272,28 @@ def test_pread(self):
         finally:
             os.close(fd)
 
+    @unittest.skipUnless(hasattr(posix, 'preadv'), "test needs posix.preadv()")
+    def test_preadv(self):
+        fd = os.open(support.TESTFN, os.O_RDWR | os.O_CREAT)
+        try:
+            os.write(fd, b'test1tt2t3t5t6t6t8')
+            buf = [bytearray(i) for i in [5, 3, 2]]
+            self.assertEqual(posix.preadv(fd, buf, 3), 10)
+            self.assertEqual([b't1tt2', b't3t', b'5t'], list(buf))
+        finally:
+            os.close(fd)
+
+    @unittest.skipUnless(hasattr(posix, 'RWF_HIPRI'), "test needs posix.RWF_HIPRI")
+    def test_preadv_flags(self):
+        fd = os.open(support.TESTFN, os.O_RDWR | os.O_CREAT)
+        try:
+            os.write(fd, b'test1tt2t3t5t6t6t8')
+            buf = [bytearray(i) for i in [5, 3, 2]]
+            self.assertEqual(posix.preadv(fd, buf, 3, os.RWF_HIPRI), 10)
+            self.assertEqual([b't1tt2', b't3t', b'5t'], list(buf))
+        finally:
+            os.close(fd)
+
     @unittest.skipUnless(hasattr(posix, 'pwrite'), "test needs posix.pwrite()")
     def test_pwrite(self):
         fd = os.open(support.TESTFN, os.O_RDWR | os.O_CREAT)
@@ -283,6 +305,34 @@ def test_pwrite(self):
         finally:
             os.close(fd)
 
+    @unittest.skipUnless(hasattr(posix, 'pwritev'), "test needs posix.pwritev()")
+    def test_pwritev(self):
+        fd = os.open(support.TESTFN, os.O_RDWR | os.O_CREAT)
+        try:
+            os.write(fd, b"xx")
+            os.lseek(fd, 0, os.SEEK_SET)
+            n = os.pwritev(fd, [b'test1', b'tt2', b't3'], 2)
+            self.assertEqual(n, 10)
+
+            os.lseek(fd, 0, os.SEEK_SET)
+            self.assertEqual(b'xxtest1tt2t3', posix.read(fd, 100))
+        finally:
+            os.close(fd)
+
+    @unittest.skipUnless(hasattr(posix, 'os.RWF_SYNC'), "test needs os.RWF_SYNC")
+    def test_pwritev_flags(self):
+        fd = os.open(support.TESTFN, os.O_RDWR | os.O_CREAT)
+        try:
+            os.write(fd,b"xx")
+            os.lseek(fd, 0, os.SEEK_SET)
+            n = os.pwritev(fd, [b'test1', b'tt2', b't3'], 2, os.RWF_SYNC)
+            self.assertEqual(n, 10)
+
+            os.lseek(fd, 0, os.SEEK_SET)
+            self.assertEqual(b'xxtest1tt2', posix.read(fd, 100))
+        finally:
+            os.close(fd)
+
     @unittest.skipUnless(hasattr(posix, 'posix_fallocate'),
         "test needs posix.posix_fallocate()")
     def test_posix_fallocate(self):
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-19-01-54-22.bpo-31368.kzKqUR.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-19-01-54-22.bpo-31368.kzKqUR.rst
new file mode 100644
index 00000000000..e13a58bb2c2
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-19-01-54-22.bpo-31368.kzKqUR.rst	
@@ -0,0 +1 @@
+Expose preadv and pwritev system calls in the os module. Patch by Pablo Galindo
diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h
index 6f4c028e7ef..f432437e99e 100644
--- a/Modules/clinic/posixmodule.c.h
+++ b/Modules/clinic/posixmodule.c.h
@@ -3705,6 +3705,61 @@ os_pread(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
 
 #endif /* defined(HAVE_PREAD) */
 
+#if (defined(HAVE_PREADV) || defined (HAVE_PREADV2))
+
+PyDoc_STRVAR(os_preadv__doc__,
+"preadv($module, fd, buffers, offset, flags=0, /)\n"
+"--\n"
+"\n"
+"Reads from a file descriptor into a number of mutable bytes-like objects.\n"
+"\n"
+"Combines the functionality of readv() and pread(). As readv(), it will\n"
+"transfer data into each buffer until it is full and then move on to the next\n"
+"buffer in the sequence to hold the rest of the data. Its fourth argument,\n"
+"specifies the file offset at which the input operation is to be performed. It\n"
+"will return the total number of bytes read (which can be less than the total\n"
+"capacity of all the objects).\n"
+"\n"
+"The flags argument contains a bitwise OR of zero or more of the following flags:\n"
+"\n"
+"- RWF_HIPRI\n"
+"- RWF_NOWAIT\n"
+"\n"
+"Using non-zero flags requires Linux 4.6 or newer.");
+
+#define OS_PREADV_METHODDEF    \
+    {"preadv", (PyCFunction)os_preadv, METH_FASTCALL, os_preadv__doc__},
+
+static Py_ssize_t
+os_preadv_impl(PyObject *module, int fd, PyObject *buffers, Py_off_t offset,
+               int flags);
+
+static PyObject *
+os_preadv(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
+{
+    PyObject *return_value = NULL;
+    int fd;
+    PyObject *buffers;
+    Py_off_t offset;
+    int flags = 0;
+    Py_ssize_t _return_value;
+
+    if (!_PyArg_ParseStack(args, nargs, "iOO&|i:preadv",
+        &fd, &buffers, Py_off_t_converter, &offset, &flags)) {
+        goto exit;
+    }
+    _return_value = os_preadv_impl(module, fd, buffers, offset, flags);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyLong_FromSsize_t(_return_value);
+
+exit:
+    return return_value;
+}
+
+#endif /* (defined(HAVE_PREADV) || defined (HAVE_PREADV2)) */
+
 PyDoc_STRVAR(os_write__doc__,
 "write($module, fd, data, /)\n"
 "--\n"
@@ -3963,6 +4018,61 @@ os_pwrite(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
 
 #endif /* defined(HAVE_PWRITE) */
 
+#if (defined(HAVE_PWRITEV) || defined (HAVE_PWRITEV2))
+
+PyDoc_STRVAR(os_pwritev__doc__,
+"pwritev($module, fd, buffers, offset, flags=0, /)\n"
+"--\n"
+"\n"
+"Writes the contents of bytes-like objects to a file descriptor at a given offset.\n"
+"\n"
+"Combines the functionality of writev() and pwrite(). All buffers must be a sequence\n"
+"of bytes-like objects. Buffers are processed in array order. Entire contents of first\n"
+"buffer is written before proceeding to second, and so on. The operating system may\n"
+"set a limit (sysconf() value SC_IOV_MAX) on the number of buffers that can be used.\n"
+"This function writes the contents of each object to the file descriptor and returns\n"
+"the total number of bytes written.\n"
+"\n"
+"The flags argument contains a bitwise OR of zero or more of the following flags:\n"
+"\n"
+"- RWF_DSYNC\n"
+"- RWF_SYNC\n"
+"\n"
+"Using non-zero flags requires Linux 4.7 or newer.");
+
+#define OS_PWRITEV_METHODDEF    \
+    {"pwritev", (PyCFunction)os_pwritev, METH_FASTCALL, os_pwritev__doc__},
+
+static Py_ssize_t
+os_pwritev_impl(PyObject *module, int fd, PyObject *buffers, Py_off_t offset,
+                int flags);
+
+static PyObject *
+os_pwritev(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
+{
+    PyObject *return_value = NULL;
+    int fd;
+    PyObject *buffers;
+    Py_off_t offset;
+    int flags = 0;
+    Py_ssize_t _return_value;
+
+    if (!_PyArg_ParseStack(args, nargs, "iOO&|i:pwritev",
+        &fd, &buffers, Py_off_t_converter, &offset, &flags)) {
+        goto exit;
+    }
+    _return_value = os_pwritev_impl(module, fd, buffers, offset, flags);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyLong_FromSsize_t(_return_value);
+
+exit:
+    return return_value;
+}
+
+#endif /* (defined(HAVE_PWRITEV) || defined (HAVE_PWRITEV2)) */
+
 #if defined(HAVE_MKFIFO)
 
 PyDoc_STRVAR(os_mkfifo__doc__,
@@ -6239,6 +6349,10 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject
     #define OS_PREAD_METHODDEF
 #endif /* !defined(OS_PREAD_METHODDEF) */
 
+#ifndef OS_PREADV_METHODDEF
+    #define OS_PREADV_METHODDEF
+#endif /* !defined(OS_PREADV_METHODDEF) */
+
 #ifndef OS_PIPE_METHODDEF
     #define OS_PIPE_METHODDEF
 #endif /* !defined(OS_PIPE_METHODDEF) */
@@ -6255,6 +6369,10 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject
     #define OS_PWRITE_METHODDEF
 #endif /* !defined(OS_PWRITE_METHODDEF) */
 
+#ifndef OS_PWRITEV_METHODDEF
+    #define OS_PWRITEV_METHODDEF
+#endif /* !defined(OS_PWRITEV_METHODDEF) */
+
 #ifndef OS_MKFIFO_METHODDEF
     #define OS_MKFIFO_METHODDEF
 #endif /* !defined(OS_MKFIFO_METHODDEF) */
@@ -6410,4 +6528,4 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject
 #ifndef OS_GETRANDOM_METHODDEF
     #define OS_GETRANDOM_METHODDEF
 #endif /* !defined(OS_GETRANDOM_METHODDEF) */
-/*[clinic end generated code: output=6345053cd5992caf input=a9049054013a1b77]*/
+/*[clinic end generated code: output=06ace805893aa10c input=a9049054013a1b77]*/
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index b0e48dabbd5..a1a38ebda38 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -8163,6 +8163,94 @@ os_pread_impl(PyObject *module, int fd, int length, Py_off_t offset)
 }
 #endif /* HAVE_PREAD */
 
+#if defined(HAVE_PREADV) || defined (HAVE_PREADV2)
+/*[clinic input]
+os.preadv -> Py_ssize_t
+
+    fd: int
+    buffers: object
+    offset: Py_off_t
+    flags: int = 0
+    /
+
+Reads from a file descriptor into a number of mutable bytes-like objects.
+
+Combines the functionality of readv() and pread(). As readv(), it will
+transfer data into each buffer until it is full and then move on to the next
+buffer in the sequence to hold the rest of the data. Its fourth argument,
+specifies the file offset at which the input operation is to be performed. It
+will return the total number of bytes read (which can be less than the total
+capacity of all the objects).
+
+The flags argument contains a bitwise OR of zero or more of the following flags:
+
+- RWF_HIPRI
+- RWF_NOWAIT
+
+Using non-zero flags requires Linux 4.6 or newer.
+[clinic start generated code]*/
+
+static Py_ssize_t
+os_preadv_impl(PyObject *module, int fd, PyObject *buffers, Py_off_t offset,
+               int flags)
+/*[clinic end generated code: output=26fc9c6e58e7ada5 input=4173919dc1f7ed99]*/
+{
+    Py_ssize_t cnt, n;
+    int async_err = 0;
+    struct iovec *iov;
+    Py_buffer *buf;
+
+    if (!PySequence_Check(buffers)) {
+        PyErr_SetString(PyExc_TypeError,
+            "preadv2() arg 2 must be a sequence");
+        return -1;
+    }
+
+    cnt = PySequence_Size(buffers);
+    if (cnt < 0) {
+        return -1;
+    }
+
+#ifndef HAVE_PREADV2
+    if(flags != 0) {
+        argument_unavailable_error("preadv2", "flags");
+        return -1;
+    }
+#endif
+
+    if (iov_setup(&iov, &buf, buffers, cnt, PyBUF_WRITABLE) < 0) {
+        return -1;
+    }
+#ifdef HAVE_PREADV2
+    do {
+        Py_BEGIN_ALLOW_THREADS
+        _Py_BEGIN_SUPPRESS_IPH
+        n = preadv2(fd, iov, cnt, offset, flags);
+        _Py_END_SUPPRESS_IPH
+        Py_END_ALLOW_THREADS
+    } while (n < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
+#else
+    do {
+        Py_BEGIN_ALLOW_THREADS
+        _Py_BEGIN_SUPPRESS_IPH
+        n = preadv(fd, iov, cnt, offset);
+        _Py_END_SUPPRESS_IPH
+        Py_END_ALLOW_THREADS
+    } while (n < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
+#endif
+
+    iov_cleanup(iov, buf, cnt);
+    if (n < 0) {
+        if (!async_err) {
+            posix_error();
+        }
+        return -1;
+    }
+
+    return n;
+}
+#endif /* HAVE_PREADV */
+
 
 /*[clinic input]
 os.write -> Py_ssize_t
@@ -8613,6 +8701,97 @@ os_pwrite_impl(PyObject *module, int fd, Py_buffer *buffer, Py_off_t offset)
 }
 #endif /* HAVE_PWRITE */
 
+#if defined(HAVE_PWRITEV) || defined (HAVE_PWRITEV2)
+/*[clinic input]
+os.pwritev -> Py_ssize_t
+
+    fd: int
+    buffers: object
+    offset: Py_off_t
+    flags: int = 0
+    /
+
+Writes the contents of bytes-like objects to a file descriptor at a given offset.
+
+Combines the functionality of writev() and pwrite(). All buffers must be a sequence
+of bytes-like objects. Buffers are processed in array order. Entire contents of first
+buffer is written before proceeding to second, and so on. The operating system may
+set a limit (sysconf() value SC_IOV_MAX) on the number of buffers that can be used.
+This function writes the contents of each object to the file descriptor and returns
+the total number of bytes written.
+
+The flags argument contains a bitwise OR of zero or more of the following flags:
+
+- RWF_DSYNC
+- RWF_SYNC
+
+Using non-zero flags requires Linux 4.7 or newer.
+[clinic start generated code]*/
+
+static Py_ssize_t
+os_pwritev_impl(PyObject *module, int fd, PyObject *buffers, Py_off_t offset,
+                int flags)
+/*[clinic end generated code: output=e3dd3e9d11a6a5c7 input=803dc5ddbf0cfd3b]*/
+{
+    Py_ssize_t cnt;
+    Py_ssize_t result;
+    int async_err = 0;
+    struct iovec *iov;
+    Py_buffer *buf;
+
+    if (!PySequence_Check(buffers)) {
+        PyErr_SetString(PyExc_TypeError,
+            "pwritev() arg 2 must be a sequence");
+        return -1;
+    }
+
+    cnt = PySequence_Size(buffers);
+    if (cnt < 0) {
+        return -1;
+    }
+
+#ifndef HAVE_PWRITEV2
+    if(flags != 0) {
+        argument_unavailable_error("pwritev2", "flags");
+        return -1;
+    }
+#endif
+
+    if (iov_setup(&iov, &buf, buffers, cnt, PyBUF_SIMPLE) < 0) {
+        return -1;
+    }
+#ifdef HAVE_PWRITEV2
+    do {
+        Py_BEGIN_ALLOW_THREADS
+        _Py_BEGIN_SUPPRESS_IPH
+        result = pwritev2(fd, iov, cnt, offset, flags);
+        _Py_END_SUPPRESS_IPH
+        Py_END_ALLOW_THREADS
+    } while (result < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
+#else
+    do {
+        Py_BEGIN_ALLOW_THREADS
+        _Py_BEGIN_SUPPRESS_IPH
+        result = pwritev(fd, iov, cnt, offset);
+        _Py_END_SUPPRESS_IPH
+        Py_END_ALLOW_THREADS
+    } while (result < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
+#endif
+
+    iov_cleanup(iov, buf, cnt);
+    if (result < 0) {
+        if (!async_err) {
+            posix_error();
+        }
+        return -1;
+    }
+
+    return result;
+}
+#endif /* HAVE_PWRITEV */
+
+
+
 
 #ifdef HAVE_MKFIFO
 /*[clinic input]
@@ -12506,9 +12685,11 @@ static PyMethodDef posix_methods[] = {
     OS_READ_METHODDEF
     OS_READV_METHODDEF
     OS_PREAD_METHODDEF
+    OS_PREADV_METHODDEF
     OS_WRITE_METHODDEF
     OS_WRITEV_METHODDEF
     OS_PWRITE_METHODDEF
+    OS_PWRITEV_METHODDEF
 #ifdef HAVE_SENDFILE
     {"sendfile",        (PyCFunction)posix_sendfile, METH_VARARGS | METH_KEYWORDS,
                             posix_sendfile__doc__},
@@ -12953,6 +13134,19 @@ all_ins(PyObject *m)
     if (PyModule_AddIntMacro(m, F_TEST)) return -1;
 #endif
 
+#ifdef RWF_DSYNC
+    if (PyModule_AddIntConstant(m, "RWF_DSYNC", RWF_DSYNC)) return -1;
+#endif
+#ifdef RWF_HIPRI
+    if (PyModule_AddIntConstant(m, "RWF_HIPRI", RWF_HIPRI)) return -1;
+#endif
+#ifdef RWF_SYNC
+    if (PyModule_AddIntConstant(m, "RWF_SYNC", RWF_SYNC)) return -1;
+#endif
+#ifdef RWF_NOWAIT
+    if (PyModule_AddIntConstant(m, "RWF_NOWAIT", RWF_NOWAIT)) return -1;
+#endif
+
 #ifdef HAVE_SPAWNV
     if (PyModule_AddIntConstant(m, "P_WAIT", _P_WAIT)) return -1;
     if (PyModule_AddIntConstant(m, "P_NOWAIT", _P_NOWAIT)) return -1;
diff --git a/aclocal.m4 b/aclocal.m4
index 09694962914..8ed232fb515 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -13,7 +13,7 @@
 
 m4_ifndef([AC_CONFIG_MACRO_DIRS], [m4_defun([_AM_CONFIG_MACRO_DIRS], [])m4_defun([AC_CONFIG_MACRO_DIRS], [_AM_CONFIG_MACRO_DIRS($@)])])
 # pkg.m4 - Macros to locate and utilise pkg-config.   -*- Autoconf -*-
-# serial 11 (pkg-config-0.29.1)
+# serial 12 (pkg-config-0.29.2)
 
 dnl Copyright ? 2004 Scott James Remnant <scott at netsplit.com>.
 dnl Copyright ? 2012-2015 Dan Nicholson <dbn.lists at gmail.com>
@@ -55,7 +55,7 @@ dnl
 dnl See the "Since" comment for each macro you use to see what version
 dnl of the macros you require.
 m4_defun([PKG_PREREQ],
-[m4_define([PKG_MACROS_VERSION], [0.29.1])
+[m4_define([PKG_MACROS_VERSION], [0.29.2])
 m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1,
     [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])])
 ])dnl PKG_PREREQ
@@ -156,7 +156,7 @@ AC_ARG_VAR([$1][_CFLAGS], [C compiler flags for $1, overriding pkg-config])dnl
 AC_ARG_VAR([$1][_LIBS], [linker flags for $1, overriding pkg-config])dnl
 
 pkg_failed=no
-AC_MSG_CHECKING([for $1])
+AC_MSG_CHECKING([for $2])
 
 _PKG_CONFIG([$1][_CFLAGS], [cflags], [$2])
 _PKG_CONFIG([$1][_LIBS], [libs], [$2])
@@ -166,11 +166,11 @@ and $1[]_LIBS to avoid the need to call pkg-config.
 See the pkg-config man page for more details.])
 
 if test $pkg_failed = yes; then
-   	AC_MSG_RESULT([no])
+        AC_MSG_RESULT([no])
         _PKG_SHORT_ERRORS_SUPPORTED
         if test $_pkg_short_errors_supported = yes; then
 	        $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1`
-        else 
+        else
 	        $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1`
         fi
 	# Put the nasty error message in config.log where it belongs
@@ -187,7 +187,7 @@ installed software in a non-standard prefix.
 _PKG_TEXT])[]dnl
         ])
 elif test $pkg_failed = untried; then
-     	AC_MSG_RESULT([no])
+        AC_MSG_RESULT([no])
 	m4_default([$4], [AC_MSG_FAILURE(
 [The pkg-config script could not be found or is too old.  Make sure it
 is in your PATH or set the PKG_CONFIG environment variable to the full
@@ -288,72 +288,4 @@ AS_VAR_COPY([$1], [pkg_cv_][$1])
 AS_VAR_IF([$1], [""], [$5], [$4])dnl
 ])dnl PKG_CHECK_VAR
 
-dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES,
-dnl   [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND],
-dnl   [DESCRIPTION], [DEFAULT])
-dnl ------------------------------------------
-dnl
-dnl Prepare a "--with-" configure option using the lowercase
-dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and
-dnl PKG_CHECK_MODULES in a single macro.
-AC_DEFUN([PKG_WITH_MODULES],
-[
-m4_pushdef([with_arg], m4_tolower([$1]))
-
-m4_pushdef([description],
-           [m4_default([$5], [build with ]with_arg[ support])])
-
-m4_pushdef([def_arg], [m4_default([$6], [auto])])
-m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes])
-m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no])
-
-m4_case(def_arg,
-            [yes],[m4_pushdef([with_without], [--without-]with_arg)],
-            [m4_pushdef([with_without],[--with-]with_arg)])
-
-AC_ARG_WITH(with_arg,
-     AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),,
-    [AS_TR_SH([with_]with_arg)=def_arg])
-
-AS_CASE([$AS_TR_SH([with_]with_arg)],
-            [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)],
-            [auto],[PKG_CHECK_MODULES([$1],[$2],
-                                        [m4_n([def_action_if_found]) $3],
-                                        [m4_n([def_action_if_not_found]) $4])])
-
-m4_popdef([with_arg])
-m4_popdef([description])
-m4_popdef([def_arg])
-
-])dnl PKG_WITH_MODULES
-
-dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES,
-dnl   [DESCRIPTION], [DEFAULT])
-dnl -----------------------------------------------
-dnl
-dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES
-dnl check._[VARIABLE-PREFIX] is exported as make variable.
-AC_DEFUN([PKG_HAVE_WITH_MODULES],
-[
-PKG_WITH_MODULES([$1],[$2],,,[$3],[$4])
-
-AM_CONDITIONAL([HAVE_][$1],
-               [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"])
-])dnl PKG_HAVE_WITH_MODULES
-
-dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES,
-dnl   [DESCRIPTION], [DEFAULT])
-dnl ------------------------------------------------------
-dnl
-dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after
-dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make
-dnl and preprocessor variable.
-AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES],
-[
-PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4])
-
-AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"],
-        [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])])
-])dnl PKG_HAVE_DEFINE_WITH_MODULES
-
 m4_include([m4/ax_check_openssl.m4])
diff --git a/configure b/configure
index 5904b54c3ed..6ce0728af11 100755
--- a/configure
+++ b/configure
@@ -11197,8 +11197,8 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \
  initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
  memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
- posix_fallocate posix_fadvise pread \
- pthread_init pthread_kill putenv pwrite readlink readlinkat readv realpath renameat \
+ posix_fallocate posix_fadvise pread preadv preadv2 \
+ pthread_init pthread_kill putenv pwrite pwritev pwritev2 readlink readlinkat readv realpath renameat \
  sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \
  setgid sethostname \
  setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \
diff --git a/configure.ac b/configure.ac
index 15ef872a53a..b901f807657 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3431,8 +3431,8 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \
  initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
  memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
- posix_fallocate posix_fadvise pread \
- pthread_init pthread_kill putenv pwrite readlink readlinkat readv realpath renameat \
+ posix_fallocate posix_fadvise pread preadv preadv2 \
+ pthread_init pthread_kill putenv pwrite pwritev pwritev2 readlink readlinkat readv realpath renameat \
  sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \
  setgid sethostname \
  setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \
diff --git a/pyconfig.h.in b/pyconfig.h.in
index fd5a0c38d89..27fe136a976 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -710,6 +710,12 @@
 /* Define to 1 if you have the `pread' function. */
 #undef HAVE_PREAD
 
+/* Define to 1 if you have the `preadv' function. */
+#undef HAVE_PREADV
+
+/* Define to 1 if you have the `preadv2' function. */
+#undef HAVE_PREADV2
+
 /* Define if you have the 'prlimit' functions. */
 #undef HAVE_PRLIMIT
 
@@ -746,6 +752,12 @@
 /* Define to 1 if you have the `pwrite' function. */
 #undef HAVE_PWRITE
 
+/* Define to 1 if you have the `pwritev' function. */
+#undef HAVE_PWRITEV
+
+/* Define to 1 if you have the `pwritev2' function. */
+#undef HAVE_PWRITEV2
+
 /* Define to 1 if you have the `readlink' function. */
 #undef HAVE_READLINK
 


From webhook-mailer at python.org  Sat Jan 27 12:07:12 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Sat, 27 Jan 2018 17:07:12 -0000
Subject: [Python-checkins] Fix wrong assert in unicodeobject (GH-5340)
Message-ID: <mailman.82.1517072833.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/7cc95f5069c8983afb8a55928db6956abe2d5afa
commit: 7cc95f5069c8983afb8a55928db6956abe2d5afa
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T02:07:09+09:00
summary:

Fix wrong assert in unicodeobject (GH-5340)

files:
M Objects/unicodeobject.c

diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 4b90cc364b8..775bd15cf6d 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -11038,7 +11038,7 @@ _PyUnicode_EqualToASCIIId(PyObject *left, _Py_Identifier *right)
     if (PyUnicode_CHECK_INTERNED(left))
         return 0;
 
-    assert(_PyUnicode_HASH(right_uni) != 1);
+    assert(_PyUnicode_HASH(right_uni) != -1);
     hash = _PyUnicode_HASH(left);
     if (hash != -1 && hash != _PyUnicode_HASH(right_uni))
         return 0;


From webhook-mailer at python.org  Sat Jan 27 12:41:27 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Sat, 27 Jan 2018 17:41:27 -0000
Subject: [Python-checkins] Fix wrong assert in unicodeobject (GH-5340)
Message-ID: <mailman.83.1517074888.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d7274c6638fc88d72b73f8cdc6b888270a5e7a50
commit: d7274c6638fc88d72b73f8cdc6b888270a5e7a50
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: INADA Naoki <methane at users.noreply.github.com>
date: 2018-01-28T02:41:24+09:00
summary:

Fix wrong assert in unicodeobject (GH-5340)

(cherry picked from commit 7cc95f5069c8983afb8a55928db6956abe2d5afa)

files:
M Objects/unicodeobject.c

diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 86cac96cd62..9ccd06eebb4 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -11217,7 +11217,7 @@ _PyUnicode_EqualToASCIIId(PyObject *left, _Py_Identifier *right)
     if (PyUnicode_CHECK_INTERNED(left))
         return 0;
 
-    assert(_PyUnicode_HASH(right_uni) != 1);
+    assert(_PyUnicode_HASH(right_uni) != -1);
     hash = _PyUnicode_HASH(left);
     if (hash != -1 && hash != _PyUnicode_HASH(right_uni))
         return 0;


From webhook-mailer at python.org  Sat Jan 27 13:24:23 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 27 Jan 2018 18:24:23 -0000
Subject: [Python-checkins] bpo-32436: Use PyThreadState_GET() in all hot
 paths (GH-5363)
Message-ID: <mailman.84.1517077465.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/bc4123b0b380edda774b8bff2fa1bcc96453b440
commit: bc4123b0b380edda774b8bff2fa1bcc96453b440
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-27T13:24:20-05:00
summary:

bpo-32436: Use PyThreadState_GET() in all hot paths (GH-5363)

files:
M Python/context.c

diff --git a/Python/context.c b/Python/context.c
index 3928e94d064..2034a204121 100644
--- a/Python/context.c
+++ b/Python/context.c
@@ -85,7 +85,8 @@ PyContext_Enter(PyContext *ctx)
         return -1;
     }
 
-    PyThreadState *ts = PyThreadState_Get();
+    PyThreadState *ts = PyThreadState_GET();
+    assert(ts != NULL);
 
     ctx->ctx_prev = (PyContext *)ts->context;  /* borrow */
     ctx->ctx_entered = 1;
@@ -107,7 +108,8 @@ PyContext_Exit(PyContext *ctx)
         return -1;
     }
 
-    PyThreadState *ts = PyThreadState_Get();
+    PyThreadState *ts = PyThreadState_GET();
+    assert(ts != NULL);
 
     if (ts->context != (PyObject *)ctx) {
         /* Can only happen if someone misuses the C API */
@@ -341,7 +343,8 @@ context_new_from_vars(PyHamtObject *vars)
 static inline PyContext *
 context_get(void)
 {
-    PyThreadState *ts = PyThreadState_Get();
+    PyThreadState *ts = PyThreadState_GET();
+    assert(ts != NULL);
     PyContext *current_ctx = (PyContext *)ts->context;
     if (current_ctx == NULL) {
         current_ctx = context_new_empty();


From webhook-mailer at python.org  Sat Jan 27 13:46:49 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 27 Jan 2018 18:46:49 -0000
Subject: [Python-checkins] bpo-32630: Use contextvars in decimal (GH-5278)
Message-ID: <mailman.85.1517078810.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/f13f12d8daa587b5fcc66fe3ed1090a5dadab289
commit: f13f12d8daa587b5fcc66fe3ed1090a5dadab289
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-27T13:46:46-05:00
summary:

bpo-32630: Use contextvars in decimal (GH-5278)

files:
A Lib/test/test_asyncio/test_context.py
A Misc/NEWS.d/next/Library/2018-01-23-01-57-36.bpo-32630.6KRHBs.rst
M Lib/_pydecimal.py
M Modules/_decimal/_decimal.c

diff --git a/Lib/_pydecimal.py b/Lib/_pydecimal.py
index a1662bbd671..359690003fe 100644
--- a/Lib/_pydecimal.py
+++ b/Lib/_pydecimal.py
@@ -433,13 +433,11 @@ class FloatOperation(DecimalException, TypeError):
 # The getcontext() and setcontext() function manage access to a thread-local
 # current context.
 
-import threading
+import contextvars
 
-local = threading.local()
-if hasattr(local, '__decimal_context__'):
-    del local.__decimal_context__
+_current_context_var = contextvars.ContextVar('decimal_context')
 
-def getcontext(_local=local):
+def getcontext():
     """Returns this thread's context.
 
     If this thread does not yet have a context, returns
@@ -447,20 +445,20 @@ def getcontext(_local=local):
     New contexts are copies of DefaultContext.
     """
     try:
-        return _local.__decimal_context__
-    except AttributeError:
+        return _current_context_var.get()
+    except LookupError:
         context = Context()
-        _local.__decimal_context__ = context
+        _current_context_var.set(context)
         return context
 
-def setcontext(context, _local=local):
+def setcontext(context):
     """Set this thread's context to context."""
     if context in (DefaultContext, BasicContext, ExtendedContext):
         context = context.copy()
         context.clear_flags()
-    _local.__decimal_context__ = context
+    _current_context_var.set(context)
 
-del threading, local        # Don't contaminate the namespace
+del contextvars        # Don't contaminate the namespace
 
 def localcontext(ctx=None):
     """Return a context manager for a copy of the supplied context
diff --git a/Lib/test/test_asyncio/test_context.py b/Lib/test/test_asyncio/test_context.py
new file mode 100644
index 00000000000..6abddd9f251
--- /dev/null
+++ b/Lib/test/test_asyncio/test_context.py
@@ -0,0 +1,29 @@
+import asyncio
+import decimal
+import unittest
+
+
+class DecimalContextTest(unittest.TestCase):
+
+    def test_asyncio_task_decimal_context(self):
+        async def fractions(t, precision, x, y):
+            with decimal.localcontext() as ctx:
+                ctx.prec = precision
+                a = decimal.Decimal(x) / decimal.Decimal(y)
+                await asyncio.sleep(t)
+                b = decimal.Decimal(x) / decimal.Decimal(y ** 2)
+                return a, b
+
+        async def main():
+            r1, r2 = await asyncio.gather(
+                fractions(0.1, 3, 1, 3), fractions(0.2, 6, 1, 3))
+
+            return r1, r2
+
+        r1, r2 = asyncio.run(main())
+
+        self.assertEqual(str(r1[0]), '0.333')
+        self.assertEqual(str(r1[1]), '0.111')
+
+        self.assertEqual(str(r2[0]), '0.333333')
+        self.assertEqual(str(r2[1]), '0.111111')
diff --git a/Misc/NEWS.d/next/Library/2018-01-23-01-57-36.bpo-32630.6KRHBs.rst b/Misc/NEWS.d/next/Library/2018-01-23-01-57-36.bpo-32630.6KRHBs.rst
new file mode 100644
index 00000000000..1bbcbb173eb
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-23-01-57-36.bpo-32630.6KRHBs.rst
@@ -0,0 +1 @@
+Refactor decimal module to use contextvars to store decimal context.
diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c
index 18fa2e4fa5e..fddb39ef652 100644
--- a/Modules/_decimal/_decimal.c
+++ b/Modules/_decimal/_decimal.c
@@ -122,10 +122,7 @@ incr_false(void)
 }
 
 
-/* Key for thread state dictionary */
-static PyObject *tls_context_key = NULL;
-/* Invariant: NULL or the most recently accessed thread local context */
-static PyDecContextObject *cached_context = NULL;
+static PyContextVar *current_context_var;
 
 /* Template for creating new thread contexts, calling Context() without
  * arguments and initializing the module_context on first access. */
@@ -1220,10 +1217,6 @@ context_new(PyTypeObject *type, PyObject *args UNUSED, PyObject *kwds UNUSED)
 static void
 context_dealloc(PyDecContextObject *self)
 {
-    if (self == cached_context) {
-        cached_context = NULL;
-    }
-
     Py_XDECREF(self->traps);
     Py_XDECREF(self->flags);
     Py_TYPE(self)->tp_free(self);
@@ -1498,69 +1491,38 @@ static PyGetSetDef context_getsets [] =
  * operation.
  */
 
-/* Get the context from the thread state dictionary. */
 static PyObject *
-current_context_from_dict(void)
+init_current_context(void)
 {
-    PyObject *dict;
-    PyObject *tl_context;
-    PyThreadState *tstate;
-
-    dict = PyThreadState_GetDict();
-    if (dict == NULL) {
-        PyErr_SetString(PyExc_RuntimeError,
-            "cannot get thread state");
+    PyObject *tl_context = context_copy(default_context_template, NULL);
+    if (tl_context == NULL) {
         return NULL;
     }
+    CTX(tl_context)->status = 0;
 
-    tl_context = PyDict_GetItemWithError(dict, tls_context_key);
-    if (tl_context != NULL) {
-        /* We already have a thread local context. */
-        CONTEXT_CHECK(tl_context);
-    }
-    else {
-        if (PyErr_Occurred()) {
-            return NULL;
-        }
-
-        /* Set up a new thread local context. */
-        tl_context = context_copy(default_context_template, NULL);
-        if (tl_context == NULL) {
-            return NULL;
-        }
-        CTX(tl_context)->status = 0;
-
-        if (PyDict_SetItem(dict, tls_context_key, tl_context) < 0) {
-            Py_DECREF(tl_context);
-            return NULL;
-        }
+    PyContextToken *tok = PyContextVar_Set(current_context_var, tl_context);
+    if (tok == NULL) {
         Py_DECREF(tl_context);
+        return NULL;
     }
+    Py_DECREF(tok);
 
-    /* Cache the context of the current thread, assuming that it
-     * will be accessed several times before a thread switch. */
-    tstate = PyThreadState_GET();
-    if (tstate) {
-        cached_context = (PyDecContextObject *)tl_context;
-        cached_context->tstate = tstate;
-    }
-
-    /* Borrowed reference with refcount==1 */
     return tl_context;
 }
 
-/* Return borrowed reference to thread local context. */
-static PyObject *
+static inline PyObject *
 current_context(void)
 {
-    PyThreadState *tstate;
+    PyObject *tl_context;
+    if (PyContextVar_Get(current_context_var, NULL, &tl_context) < 0) {
+        return NULL;
+    }
 
-    tstate = PyThreadState_GET();
-    if (cached_context && cached_context->tstate == tstate) {
-        return (PyObject *)cached_context;
+    if (tl_context != NULL) {
+        return tl_context;
     }
 
-    return current_context_from_dict();
+    return init_current_context();
 }
 
 /* ctxobj := borrowed reference to the current context */
@@ -1568,47 +1530,22 @@ current_context(void)
     ctxobj = current_context(); \
     if (ctxobj == NULL) {       \
         return NULL;            \
-    }
-
-/* ctx := pointer to the mpd_context_t struct of the current context */
-#define CURRENT_CONTEXT_ADDR(ctx) { \
-    PyObject *_c_t_x_o_b_j = current_context(); \
-    if (_c_t_x_o_b_j == NULL) {                 \
-        return NULL;                            \
-    }                                           \
-    ctx = CTX(_c_t_x_o_b_j);                    \
-}
+    }                           \
+    Py_DECREF(ctxobj);
 
 /* Return a new reference to the current context */
 static PyObject *
 PyDec_GetCurrentContext(PyObject *self UNUSED, PyObject *args UNUSED)
 {
-    PyObject *context;
-
-    context = current_context();
-    if (context == NULL) {
-        return NULL;
-    }
-
-    Py_INCREF(context);
-    return context;
+    return current_context();
 }
 
 /* Set the thread local context to a new context, decrement old reference */
 static PyObject *
 PyDec_SetCurrentContext(PyObject *self UNUSED, PyObject *v)
 {
-    PyObject *dict;
-
     CONTEXT_CHECK(v);
 
-    dict = PyThreadState_GetDict();
-    if (dict == NULL) {
-        PyErr_SetString(PyExc_RuntimeError,
-            "cannot get thread state");
-        return NULL;
-    }
-
     /* If the new context is one of the templates, make a copy.
      * This is the current behavior of decimal.py. */
     if (v == default_context_template ||
@@ -1624,13 +1561,13 @@ PyDec_SetCurrentContext(PyObject *self UNUSED, PyObject *v)
         Py_INCREF(v);
     }
 
-    cached_context = NULL;
-    if (PyDict_SetItem(dict, tls_context_key, v) < 0) {
-        Py_DECREF(v);
+    PyContextToken *tok = PyContextVar_Set(current_context_var, v);
+    Py_DECREF(v);
+    if (tok == NULL) {
         return NULL;
     }
+    Py_DECREF(tok);
 
-    Py_DECREF(v);
     Py_RETURN_NONE;
 }
 
@@ -4458,6 +4395,7 @@ _dec_hash(PyDecObject *v)
     if (context == NULL) {
         return -1;
     }
+    Py_DECREF(context);
 
     if (mpd_isspecial(MPD(v))) {
         if (mpd_issnan(MPD(v))) {
@@ -5599,6 +5537,11 @@ PyInit__decimal(void)
     mpd_free = PyMem_Free;
     mpd_setminalloc(_Py_DEC_MINALLOC);
 
+    /* Init context variable */
+    current_context_var = PyContextVar_New("decimal_context", NULL);
+    if (current_context_var == NULL) {
+        goto error;
+    }
 
     /* Init external C-API functions */
     _py_long_multiply = PyLong_Type.tp_as_number->nb_multiply;
@@ -5768,7 +5711,6 @@ PyInit__decimal(void)
     CHECK_INT(PyModule_AddObject(m, "DefaultContext",
                                  default_context_template));
 
-    ASSIGN_PTR(tls_context_key, PyUnicode_FromString("___DECIMAL_CTX__"));
     Py_INCREF(Py_True);
     CHECK_INT(PyModule_AddObject(m, "HAVE_THREADS", Py_True));
 
@@ -5827,9 +5769,9 @@ PyInit__decimal(void)
     Py_CLEAR(SignalTuple); /* GCOV_NOT_REACHED */
     Py_CLEAR(DecimalTuple); /* GCOV_NOT_REACHED */
     Py_CLEAR(default_context_template); /* GCOV_NOT_REACHED */
-    Py_CLEAR(tls_context_key); /* GCOV_NOT_REACHED */
     Py_CLEAR(basic_context_template); /* GCOV_NOT_REACHED */
     Py_CLEAR(extended_context_template); /* GCOV_NOT_REACHED */
+    Py_CLEAR(current_context_var); /* GCOV_NOT_REACHED */
     Py_CLEAR(m); /* GCOV_NOT_REACHED */
 
     return NULL; /* GCOV_NOT_REACHED */


From webhook-mailer at python.org  Sat Jan 27 14:22:50 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Sat, 27 Jan 2018 19:22:50 -0000
Subject: [Python-checkins] bpo-32622: Implement loop.sendfile() (#5271)
Message-ID: <mailman.86.1517080974.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/7c684073f951dd891021676ecfd86ffc18b8895e
commit: 7c684073f951dd891021676ecfd86ffc18b8895e
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-27T21:22:47+02:00
summary:

bpo-32622: Implement loop.sendfile() (#5271)

files:
A Misc/NEWS.d/next/Library/2018-01-22-18-18-44.bpo-32622.A1D6FP.rst
M Doc/library/asyncio-eventloop.rst
M Lib/asyncio/base_events.py
M Lib/asyncio/constants.py
M Lib/asyncio/events.py
M Lib/asyncio/proactor_events.py
M Lib/asyncio/selector_events.py
M Lib/asyncio/sslproto.py
M Lib/asyncio/windows_events.py
M Lib/test/test_asyncio/test_base_events.py
M Lib/test/test_asyncio/test_events.py
M Modules/overlapped.c

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 834a4e85c2f..fe162236e01 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -543,6 +543,37 @@ Creating listening connections
    .. versionadded:: 3.5.3
 
 
+File Transferring
+-----------------
+
+.. coroutinemethod:: AbstractEventLoop.sendfile(sock, transport, \
+                                                offset=0, count=None, \
+                                                *, fallback=True)
+
+   Send a *file* to *transport*, return the total number of bytes
+   which were sent.
+
+   The method uses high-performance :meth:`os.sendfile` if available.
+
+   *file* must be a regular file object opened in binary mode.
+
+   *offset* tells from where to start reading the file. If specified,
+   *count* is the total number of bytes to transmit as opposed to
+   sending the file until EOF is reached. File position is updated on
+   return or also in case of error in which case :meth:`file.tell()
+   <io.IOBase.tell>` can be used to figure out the number of bytes
+   which were sent.
+
+   *fallback* set to ``True`` makes asyncio to manually read and send
+   the file when the platform does not support the sendfile syscall
+   (e.g. Windows or SSL socket on Unix).
+
+   Raise :exc:`SendfileNotAvailableError` if the system does not support
+   *sendfile* syscall and *fallback* is ``False``.
+
+   .. versionadded:: 3.7
+
+
 TLS Upgrade
 -----------
 
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index 94eb3089e93..f532dc42132 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -38,8 +38,10 @@
 from . import coroutines
 from . import events
 from . import futures
+from . import protocols
 from . import sslproto
 from . import tasks
+from . import transports
 from .log import logger
 
 
@@ -155,6 +157,75 @@ def _run_until_complete_cb(fut):
     futures._get_loop(fut).stop()
 
 
+
+class _SendfileFallbackProtocol(protocols.Protocol):
+    def __init__(self, transp):
+        if not isinstance(transp, transports._FlowControlMixin):
+            raise TypeError("transport should be _FlowControlMixin instance")
+        self._transport = transp
+        self._proto = transp.get_protocol()
+        self._should_resume_reading = transp.is_reading()
+        self._should_resume_writing = transp._protocol_paused
+        transp.pause_reading()
+        transp.set_protocol(self)
+        if self._should_resume_writing:
+            self._write_ready_fut = self._transport._loop.create_future()
+        else:
+            self._write_ready_fut = None
+
+    async def drain(self):
+        if self._transport.is_closing():
+            raise ConnectionError("Connection closed by peer")
+        fut = self._write_ready_fut
+        if fut is None:
+            return
+        await fut
+
+    def connection_made(self, transport):
+        raise RuntimeError("Invalid state: "
+                           "connection should have been established already.")
+
+    def connection_lost(self, exc):
+        if self._write_ready_fut is not None:
+            # Never happens if peer disconnects after sending the whole content
+            # Thus disconnection is always an exception from user perspective
+            if exc is None:
+                self._write_ready_fut.set_exception(
+                    ConnectionError("Connection is closed by peer"))
+            else:
+                self._write_ready_fut.set_exception(exc)
+        self._proto.connection_lost(exc)
+
+    def pause_writing(self):
+        if self._write_ready_fut is not None:
+            return
+        self._write_ready_fut = self._transport._loop.create_future()
+
+    def resume_writing(self):
+        if self._write_ready_fut is None:
+            return
+        self._write_ready_fut.set_result(False)
+        self._write_ready_fut = None
+
+    def data_received(self, data):
+        raise RuntimeError("Invalid state: reading should be paused")
+
+    def eof_received(self):
+        raise RuntimeError("Invalid state: reading should be paused")
+
+    async def restore(self):
+        self._transport.set_protocol(self._proto)
+        if self._should_resume_reading:
+            self._transport.resume_reading()
+        if self._write_ready_fut is not None:
+            # Cancel the future.
+            # Basically it has no effect because protocol is switched back,
+            # no code should wait for it anymore.
+            self._write_ready_fut.cancel()
+        if self._should_resume_writing:
+            self._proto.resume_writing()
+
+
 class Server(events.AbstractServer):
 
     def __init__(self, loop, sockets, protocol_factory, ssl_context, backlog,
@@ -926,6 +997,77 @@ def _check_sendfile_params(self, sock, file, offset, count):
 
         return transport, protocol
 
+    async def sendfile(self, transport, file, offset=0, count=None,
+                       *, fallback=True):
+        """Send a file to transport.
+
+        Return the total number of bytes which were sent.
+
+        The method uses high-performance os.sendfile if available.
+
+        file must be a regular file object opened in binary mode.
+
+        offset tells from where to start reading the file. If specified,
+        count is the total number of bytes to transmit as opposed to
+        sending the file until EOF is reached. File position is updated on
+        return or also in case of error in which case file.tell()
+        can be used to figure out the number of bytes
+        which were sent.
+
+        fallback set to True makes asyncio to manually read and send
+        the file when the platform does not support the sendfile syscall
+        (e.g. Windows or SSL socket on Unix).
+
+        Raise SendfileNotAvailableError if the system does not support
+        sendfile syscall and fallback is False.
+        """
+        if transport.is_closing():
+            raise RuntimeError("Transport is closing")
+        mode = getattr(transport, '_sendfile_compatible',
+                       constants._SendfileMode.UNSUPPORTED)
+        if mode is constants._SendfileMode.UNSUPPORTED:
+            raise RuntimeError(
+                f"sendfile is not supported for transport {transport!r}")
+        if mode is constants._SendfileMode.TRY_NATIVE:
+            try:
+                return await self._sendfile_native(transport, file,
+                                                   offset, count)
+            except events.SendfileNotAvailableError as exc:
+                if not fallback:
+                    raise
+        # the mode is FALLBACK or fallback is True
+        return await self._sendfile_fallback(transport, file,
+                                             offset, count)
+
+    async def _sendfile_native(self, transp, file, offset, count):
+        raise events.SendfileNotAvailableError(
+            "sendfile syscall is not supported")
+
+    async def _sendfile_fallback(self, transp, file, offset, count):
+        if offset:
+            file.seek(offset)
+        blocksize = min(count, 16384) if count else 16384
+        buf = bytearray(blocksize)
+        total_sent = 0
+        proto = _SendfileFallbackProtocol(transp)
+        try:
+            while True:
+                if count:
+                    blocksize = min(count - total_sent, blocksize)
+                    if blocksize <= 0:
+                        return total_sent
+                view = memoryview(buf)[:blocksize]
+                read = file.readinto(view)
+                if not read:
+                    return total_sent  # EOF
+                await proto.drain()
+                transp.write(view)
+                total_sent += read
+        finally:
+            if total_sent > 0 and hasattr(file, 'seek'):
+                file.seek(offset + total_sent)
+            await proto.restore()
+
     async def start_tls(self, transport, protocol, sslcontext, *,
                         server_side=False,
                         server_hostname=None,
diff --git a/Lib/asyncio/constants.py b/Lib/asyncio/constants.py
index 0ad974ff2fb..739b0a70c13 100644
--- a/Lib/asyncio/constants.py
+++ b/Lib/asyncio/constants.py
@@ -1,3 +1,5 @@
+import enum
+
 # After the connection is lost, log warnings after this many write()s.
 LOG_THRESHOLD_FOR_CONNLOST_WRITES = 5
 
@@ -11,3 +13,10 @@
 
 # Number of seconds to wait for SSL handshake to complete
 SSL_HANDSHAKE_TIMEOUT = 10.0
+
+# The enum should be here to break circular dependencies between
+# base_events and sslproto
+class _SendfileMode(enum.Enum):
+    UNSUPPORTED = enum.auto()
+    TRY_NATIVE = enum.auto()
+    FALLBACK = enum.auto()
diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index 7aa3de02c95..bdefcf62a05 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -354,6 +354,14 @@ def set_default_executor(self, executor):
         """
         raise NotImplementedError
 
+    async def sendfile(self, transport, file, offset=0, count=None,
+                       *, fallback=True):
+        """Send a file through a transport.
+
+        Return an amount of sent bytes.
+        """
+        raise NotImplementedError
+
     async def start_tls(self, transport, protocol, sslcontext, *,
                         server_side=False,
                         server_hostname=None,
diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
index ab1285b7999..6d27e532387 100644
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -180,7 +180,12 @@ def _loop_reading(self, fut=None):
                 assert self._read_fut is fut or (self._read_fut is None and
                                                  self._closing)
                 self._read_fut = None
-                data = fut.result()  # deliver data later in "finally" clause
+                if fut.done():
+                    # deliver data later in "finally" clause
+                    data = fut.result()
+                else:
+                    # the future will be replaced by next proactor.recv call
+                    fut.cancel()
 
             if self._closing:
                 # since close() has been called we ignore any read data
@@ -345,6 +350,8 @@ class _ProactorSocketTransport(_ProactorReadPipeTransport,
                                transports.Transport):
     """Transport for connected sockets."""
 
+    _sendfile_compatible = constants._SendfileMode.FALLBACK
+
     def _set_extra(self, sock):
         self._extra['socket'] = sock
 
diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
index 9446ae6a3bc..5956f2d993e 100644
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -540,6 +540,20 @@ def _sock_accept(self, fut, registered, sock):
         else:
             fut.set_result((conn, address))
 
+    async def _sendfile_native(self, transp, file, offset, count):
+        del self._transports[transp._sock_fd]
+        resume_reading = transp.is_reading()
+        transp.pause_reading()
+        await transp._make_empty_waiter()
+        try:
+            return await self.sock_sendfile(transp._sock, file, offset, count,
+                                            fallback=False)
+        finally:
+            transp._reset_empty_waiter()
+            if resume_reading:
+                transp.resume_reading()
+            self._transports[transp._sock_fd] = transp
+
     def _process_events(self, event_list):
         for key, mask in event_list:
             fileobj, (reader, writer) = key.fileobj, key.data
@@ -695,12 +709,14 @@ def get_write_buffer_size(self):
 class _SelectorSocketTransport(_SelectorTransport):
 
     _start_tls_compatible = True
+    _sendfile_compatible = constants._SendfileMode.TRY_NATIVE
 
     def __init__(self, loop, sock, protocol, waiter=None,
                  extra=None, server=None):
         super().__init__(loop, sock, protocol, extra, server)
         self._eof = False
         self._paused = False
+        self._empty_waiter = None
 
         # Disable the Nagle algorithm -- small writes will be
         # sent without waiting for the TCP ACK.  This generally
@@ -765,6 +781,8 @@ def write(self, data):
                             f'not {type(data).__name__!r}')
         if self._eof:
             raise RuntimeError('Cannot call write() after write_eof()')
+        if self._empty_waiter is not None:
+            raise RuntimeError('unable to write; sendfile is in progress')
         if not data:
             return
 
@@ -807,12 +825,16 @@ def _write_ready(self):
             self._loop._remove_writer(self._sock_fd)
             self._buffer.clear()
             self._fatal_error(exc, 'Fatal write error on socket transport')
+            if self._empty_waiter is not None:
+                self._empty_waiter.set_exception(exc)
         else:
             if n:
                 del self._buffer[:n]
             self._maybe_resume_protocol()  # May append to buffer.
             if not self._buffer:
                 self._loop._remove_writer(self._sock_fd)
+                if self._empty_waiter is not None:
+                    self._empty_waiter.set_result(None)
                 if self._closing:
                     self._call_connection_lost(None)
                 elif self._eof:
@@ -828,6 +850,23 @@ def write_eof(self):
     def can_write_eof(self):
         return True
 
+    def _call_connection_lost(self, exc):
+        super()._call_connection_lost(exc)
+        if self._empty_waiter is not None:
+            self._empty_waiter.set_exception(
+                ConnectionError("Connection is closed by peer"))
+
+    def _make_empty_waiter(self):
+        if self._empty_waiter is not None:
+            raise RuntimeError("Empty waiter is already set")
+        self._empty_waiter = self._loop.create_future()
+        if not self._buffer:
+            self._empty_waiter.set_result(None)
+        return self._empty_waiter
+
+    def _reset_empty_waiter(self):
+        self._empty_waiter = None
+
 
 class _SelectorDatagramTransport(_SelectorTransport):
 
diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py
index 1130bced8ae..863b54313cc 100644
--- a/Lib/asyncio/sslproto.py
+++ b/Lib/asyncio/sslproto.py
@@ -282,6 +282,8 @@ def feed_appdata(self, data, offset=0):
 class _SSLProtocolTransport(transports._FlowControlMixin,
                             transports.Transport):
 
+    _sendfile_compatible = constants._SendfileMode.FALLBACK
+
     def __init__(self, loop, ssl_protocol):
         self._loop = loop
         # SSLProtocol instance
@@ -365,6 +367,11 @@ def get_write_buffer_size(self):
         """Return the current size of the write buffer."""
         return self._ssl_protocol._transport.get_write_buffer_size()
 
+    @property
+    def _protocol_paused(self):
+        # Required for sendfile fallback pause_writing/resume_writing logic
+        return self._ssl_protocol._transport._protocol_paused
+
     def write(self, data):
         """Write some data bytes to the transport.
 
diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py
index 890fce8b405..f91fcddb2aa 100644
--- a/Lib/asyncio/windows_events.py
+++ b/Lib/asyncio/windows_events.py
@@ -425,7 +425,8 @@ def finish_recv(trans, key, ov):
             try:
                 return ov.getresult()
             except OSError as exc:
-                if exc.winerror == _overlapped.ERROR_NETNAME_DELETED:
+                if exc.winerror in (_overlapped.ERROR_NETNAME_DELETED,
+                                    _overlapped.ERROR_OPERATION_ABORTED):
                     raise ConnectionResetError(*exc.args)
                 else:
                     raise
@@ -447,7 +448,8 @@ def finish_recv(trans, key, ov):
             try:
                 return ov.getresult()
             except OSError as exc:
-                if exc.winerror == _overlapped.ERROR_NETNAME_DELETED:
+                if exc.winerror in (_overlapped.ERROR_NETNAME_DELETED,
+                                    _overlapped.ERROR_OPERATION_ABORTED):
                     raise ConnectionResetError(*exc.args)
                 else:
                     raise
@@ -466,7 +468,8 @@ def finish_send(trans, key, ov):
             try:
                 return ov.getresult()
             except OSError as exc:
-                if exc.winerror == _overlapped.ERROR_NETNAME_DELETED:
+                if exc.winerror in (_overlapped.ERROR_NETNAME_DELETED,
+                                    _overlapped.ERROR_OPERATION_ABORTED):
                     raise ConnectionResetError(*exc.args)
                 else:
                     raise
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index 6489f50f272..ab6560c70b9 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -1788,7 +1788,7 @@ def runner(loop):
             outer_loop.close()
 
 
-class BaseLoopSendfileTests(test_utils.TestCase):
+class BaseLoopSockSendfileTests(test_utils.TestCase):
 
     DATA = b"12345abcde" * 16 * 1024  # 160 KiB
 
@@ -1799,9 +1799,11 @@ def __init__(self, loop):
             self.closed = False
             self.data = bytearray()
             self.fut = loop.create_future()
+            self.transport = None
 
         def connection_made(self, transport):
             self.started = True
+            self.transport = transport
 
         def data_received(self, data):
             self.data.extend(data)
@@ -1809,6 +1811,7 @@ def data_received(self, data):
         def connection_lost(self, exc):
             self.closed = True
             self.fut.set_result(None)
+            self.transport = None
 
         async def wait_closed(self):
             await self.fut
@@ -1853,6 +1856,10 @@ def prepare(self):
         def cleanup():
             server.close()
             self.run_loop(server.wait_closed())
+            sock.close()
+            if proto.transport is not None:
+                proto.transport.close()
+                self.run_loop(proto.wait_closed())
 
         self.addCleanup(cleanup)
 
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index cf217538a06..0981bd6ac91 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -26,6 +26,7 @@
     import tty
 
 import asyncio
+from asyncio import base_events
 from asyncio import coroutines
 from asyncio import events
 from asyncio import proactor_events
@@ -2090,14 +2091,308 @@ def test_subprocess_shell_invalid_args(self):
             self.loop.run_until_complete(connect(shell=False))
 
 
+class MySendfileProto(MyBaseProto):
+
+    def __init__(self, loop=None, close_after=0):
+        super().__init__(loop)
+        self.data = bytearray()
+        self.close_after = close_after
+
+    def data_received(self, data):
+        self.data.extend(data)
+        super().data_received(data)
+        if self.close_after and self.nbytes >= self.close_after:
+            self.transport.close()
+
+
+class SendfileMixin:
+    # Note: sendfile via SSL transport is equal to sendfile fallback
+
+    DATA = b"12345abcde" * 160 * 1024  # 160 KiB
+
+    @classmethod
+    def setUpClass(cls):
+        with open(support.TESTFN, 'wb') as fp:
+            fp.write(cls.DATA)
+        super().setUpClass()
+
+    @classmethod
+    def tearDownClass(cls):
+        support.unlink(support.TESTFN)
+        super().tearDownClass()
+
+    def setUp(self):
+        self.file = open(support.TESTFN, 'rb')
+        self.addCleanup(self.file.close)
+        super().setUp()
+
+    def run_loop(self, coro):
+        return self.loop.run_until_complete(coro)
+
+    def prepare(self, *, is_ssl=False, close_after=0):
+        port = support.find_unused_port()
+        srv_proto = MySendfileProto(loop=self.loop, close_after=close_after)
+        if is_ssl:
+            srv_ctx = test_utils.simple_server_sslcontext()
+            cli_ctx = test_utils.simple_client_sslcontext()
+        else:
+            srv_ctx = None
+            cli_ctx = None
+        srv_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        # reduce recv socket buffer size to test on relative small data sets
+        srv_sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024)
+        srv_sock.bind((support.HOST, port))
+        server = self.run_loop(self.loop.create_server(
+            lambda: srv_proto, sock=srv_sock, ssl=srv_ctx))
+
+        if is_ssl:
+            server_hostname = support.HOST
+        else:
+            server_hostname = None
+        cli_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        # reduce send socket buffer size to test on relative small data sets
+        cli_sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
+        cli_sock.connect((support.HOST, port))
+        cli_proto = MySendfileProto(loop=self.loop)
+        tr, pr = self.run_loop(self.loop.create_connection(
+            lambda: cli_proto, sock=cli_sock,
+            ssl=cli_ctx, server_hostname=server_hostname))
+
+        def cleanup():
+            srv_proto.transport.close()
+            cli_proto.transport.close()
+            self.run_loop(srv_proto.done)
+            self.run_loop(cli_proto.done)
+
+            server.close()
+            self.run_loop(server.wait_closed())
+
+        self.addCleanup(cleanup)
+        return srv_proto, cli_proto
+
+    @unittest.skipIf(sys.platform == 'win32', "UDP sockets are not supported")
+    def test_sendfile_not_supported(self):
+        tr, pr = self.run_loop(
+            self.loop.create_datagram_endpoint(
+                lambda: MyDatagramProto(loop=self.loop),
+                family=socket.AF_INET))
+        try:
+            with self.assertRaisesRegex(RuntimeError, "not supported"):
+                self.run_loop(
+                    self.loop.sendfile(tr, self.file))
+            self.assertEqual(0, self.file.tell())
+        finally:
+            # don't use self.addCleanup because it produces resource warning
+            tr.close()
+
+    def test_sendfile(self):
+        srv_proto, cli_proto = self.prepare()
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.nbytes, len(self.DATA))
+        self.assertEqual(srv_proto.data, self.DATA)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_force_fallback(self):
+        srv_proto, cli_proto = self.prepare()
+
+        def sendfile_native(transp, file, offset, count):
+            # to raise SendfileNotAvailableError
+            return base_events.BaseEventLoop._sendfile_native(
+                self.loop, transp, file, offset, count)
+
+        self.loop._sendfile_native = sendfile_native
+
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.nbytes, len(self.DATA))
+        self.assertEqual(srv_proto.data, self.DATA)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_force_unsupported_native(self):
+        if sys.platform == 'win32':
+            if isinstance(self.loop, asyncio.ProactorEventLoop):
+                self.skipTest("Fails on proactor event loop")
+        srv_proto, cli_proto = self.prepare()
+
+        def sendfile_native(transp, file, offset, count):
+            # to raise SendfileNotAvailableError
+            return base_events.BaseEventLoop._sendfile_native(
+                self.loop, transp, file, offset, count)
+
+        self.loop._sendfile_native = sendfile_native
+
+        with self.assertRaisesRegex(events.SendfileNotAvailableError,
+                                    "not supported"):
+            self.run_loop(
+                self.loop.sendfile(cli_proto.transport, self.file,
+                                   fallback=False))
+
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(srv_proto.nbytes, 0)
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sendfile_ssl(self):
+        srv_proto, cli_proto = self.prepare(is_ssl=True)
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.nbytes, len(self.DATA))
+        self.assertEqual(srv_proto.data, self.DATA)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_for_closing_transp(self):
+        srv_proto, cli_proto = self.prepare()
+        cli_proto.transport.close()
+        with self.assertRaisesRegex(RuntimeError, "is closing"):
+            self.run_loop(self.loop.sendfile(cli_proto.transport, self.file))
+        self.run_loop(srv_proto.done)
+        self.assertEqual(srv_proto.nbytes, 0)
+        self.assertEqual(self.file.tell(), 0)
+
+    def test_sendfile_pre_and_post_data(self):
+        srv_proto, cli_proto = self.prepare()
+        PREFIX = b'zxcvbnm' * 1024
+        SUFFIX = b'0987654321' * 1024
+        cli_proto.transport.write(PREFIX)
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        cli_proto.transport.write(SUFFIX)
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.data, PREFIX + self.DATA + SUFFIX)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_ssl_pre_and_post_data(self):
+        srv_proto, cli_proto = self.prepare(is_ssl=True)
+        PREFIX = b'zxcvbnm' * 1024
+        SUFFIX = b'0987654321' * 1024
+        cli_proto.transport.write(PREFIX)
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        cli_proto.transport.write(SUFFIX)
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.data, PREFIX + self.DATA + SUFFIX)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_partial(self):
+        srv_proto, cli_proto = self.prepare()
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file, 1000, 100))
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, 100)
+        self.assertEqual(srv_proto.nbytes, 100)
+        self.assertEqual(srv_proto.data, self.DATA[1000:1100])
+        self.assertEqual(self.file.tell(), 1100)
+
+    def test_sendfile_ssl_partial(self):
+        srv_proto, cli_proto = self.prepare(is_ssl=True)
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file, 1000, 100))
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, 100)
+        self.assertEqual(srv_proto.nbytes, 100)
+        self.assertEqual(srv_proto.data, self.DATA[1000:1100])
+        self.assertEqual(self.file.tell(), 1100)
+
+    def test_sendfile_close_peer_after_receiving(self):
+        srv_proto, cli_proto = self.prepare(close_after=len(self.DATA))
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        cli_proto.transport.close()
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.nbytes, len(self.DATA))
+        self.assertEqual(srv_proto.data, self.DATA)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_ssl_close_peer_after_receiving(self):
+        srv_proto, cli_proto = self.prepare(is_ssl=True,
+                                            close_after=len(self.DATA))
+        ret = self.run_loop(
+            self.loop.sendfile(cli_proto.transport, self.file))
+        self.run_loop(srv_proto.done)
+        self.assertEqual(ret, len(self.DATA))
+        self.assertEqual(srv_proto.nbytes, len(self.DATA))
+        self.assertEqual(srv_proto.data, self.DATA)
+        self.assertEqual(self.file.tell(), len(self.DATA))
+
+    def test_sendfile_close_peer_in_middle_of_receiving(self):
+        srv_proto, cli_proto = self.prepare(close_after=1024)
+        with self.assertRaises(ConnectionError):
+            self.run_loop(
+                self.loop.sendfile(cli_proto.transport, self.file))
+        self.run_loop(srv_proto.done)
+
+        self.assertTrue(1024 <= srv_proto.nbytes < len(self.DATA),
+                        srv_proto.nbytes)
+        self.assertTrue(1024 <= self.file.tell() < len(self.DATA),
+                        self.file.tell())
+
+    def test_sendfile_fallback_close_peer_in_middle_of_receiving(self):
+
+        def sendfile_native(transp, file, offset, count):
+            # to raise SendfileNotAvailableError
+            return base_events.BaseEventLoop._sendfile_native(
+                self.loop, transp, file, offset, count)
+
+        self.loop._sendfile_native = sendfile_native
+
+        srv_proto, cli_proto = self.prepare(close_after=1024)
+        with self.assertRaises(ConnectionError):
+            self.run_loop(
+                self.loop.sendfile(cli_proto.transport, self.file))
+        self.run_loop(srv_proto.done)
+
+        self.assertTrue(1024 <= srv_proto.nbytes < len(self.DATA),
+                        srv_proto.nbytes)
+        self.assertTrue(1024 <= self.file.tell() < len(self.DATA),
+                        self.file.tell())
+
+    @unittest.skipIf(not hasattr(os, 'sendfile'),
+                     "Don't have native sendfile support")
+    def test_sendfile_prevents_bare_write(self):
+        srv_proto, cli_proto = self.prepare()
+        fut = self.loop.create_future()
+
+        async def coro():
+            fut.set_result(None)
+            return await self.loop.sendfile(cli_proto.transport, self.file)
+
+        t = self.loop.create_task(coro())
+        self.run_loop(fut)
+        with self.assertRaisesRegex(RuntimeError,
+                                    "sendfile is in progress"):
+            cli_proto.transport.write(b'data')
+        ret = self.run_loop(t)
+        self.assertEqual(ret, len(self.DATA))
+
+
 if sys.platform == 'win32':
 
-    class SelectEventLoopTests(EventLoopTestsMixin, test_utils.TestCase):
+    class SelectEventLoopTests(EventLoopTestsMixin,
+                               SendfileMixin,
+                               test_utils.TestCase):
 
         def create_event_loop(self):
             return asyncio.SelectorEventLoop()
 
     class ProactorEventLoopTests(EventLoopTestsMixin,
+                                 SendfileMixin,
                                  SubprocessTestsMixin,
                                  test_utils.TestCase):
 
@@ -2125,7 +2420,7 @@ def test_remove_fds_after_closing(self):
 else:
     import selectors
 
-    class UnixEventLoopTestsMixin(EventLoopTestsMixin):
+    class UnixEventLoopTestsMixin(EventLoopTestsMixin, SendfileMixin):
         def setUp(self):
             super().setUp()
             watcher = asyncio.SafeChildWatcher()
@@ -2556,7 +2851,9 @@ def test_not_implemented_async(self):
             with self.assertRaises(NotImplementedError):
                 await loop.sock_accept(f)
             with self.assertRaises(NotImplementedError):
-                await loop.sock_sendfile(f, mock.Mock())
+                await loop.sock_sendfile(f, f)
+            with self.assertRaises(NotImplementedError):
+                await loop.sendfile(f, f)
             with self.assertRaises(NotImplementedError):
                 await loop.connect_read_pipe(f, mock.sentinel.pipe)
             with self.assertRaises(NotImplementedError):
diff --git a/Misc/NEWS.d/next/Library/2018-01-22-18-18-44.bpo-32622.A1D6FP.rst b/Misc/NEWS.d/next/Library/2018-01-22-18-18-44.bpo-32622.A1D6FP.rst
new file mode 100644
index 00000000000..d7433fa3cb1
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-22-18-18-44.bpo-32622.A1D6FP.rst
@@ -0,0 +1 @@
+Add :meth:`asyncio.AbstractEventLoop.sendfile` method.
diff --git a/Modules/overlapped.c b/Modules/overlapped.c
index e66e8566840..447a337fdd1 100644
--- a/Modules/overlapped.c
+++ b/Modules/overlapped.c
@@ -1436,6 +1436,7 @@ PyInit__overlapped(void)
 
     WINAPI_CONSTANT(F_DWORD,  ERROR_IO_PENDING);
     WINAPI_CONSTANT(F_DWORD,  ERROR_NETNAME_DELETED);
+    WINAPI_CONSTANT(F_DWORD,  ERROR_OPERATION_ABORTED);
     WINAPI_CONSTANT(F_DWORD,  ERROR_SEM_TIMEOUT);
     WINAPI_CONSTANT(F_DWORD,  ERROR_PIPE_BUSY);
     WINAPI_CONSTANT(F_DWORD,  INFINITE);


From webhook-mailer at python.org  Sat Jan 27 15:52:54 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 27 Jan 2018 20:52:54 -0000
Subject: [Python-checkins] bpo-32622: Enforce sendfile fallback policy for
 FALLBACK transports (#5364)
Message-ID: <mailman.88.1517086377.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/b1a6ac4c4026d648b3d948945b734a4d0f175a3c
commit: b1a6ac4c4026d648b3d948945b734a4d0f175a3c
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-27T15:52:52-05:00
summary:

bpo-32622: Enforce sendfile fallback policy for FALLBACK transports (#5364)

files:
M Lib/asyncio/base_events.py
M Lib/test/test_asyncio/test_events.py

diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index f532dc42132..7442bf28b03 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -1035,7 +1035,12 @@ def _check_sendfile_params(self, sock, file, offset, count):
             except events.SendfileNotAvailableError as exc:
                 if not fallback:
                     raise
-        # the mode is FALLBACK or fallback is True
+
+        if not fallback:
+            raise RuntimeError(
+                f"fallback is disabled and native sendfile is not "
+                f"supported for transport {transport!r}")
+
         return await self._sendfile_fallback(transport, file,
                                              offset, count)
 
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index 0981bd6ac91..b66d91d415e 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -27,6 +27,7 @@
 
 import asyncio
 from asyncio import base_events
+from asyncio import constants
 from asyncio import coroutines
 from asyncio import events
 from asyncio import proactor_events
@@ -2381,6 +2382,14 @@ def test_sendfile_prevents_bare_write(self):
         ret = self.run_loop(t)
         self.assertEqual(ret, len(self.DATA))
 
+    def test_sendfile_no_fallback_for_fallback_transport(self):
+        transport = mock.Mock()
+        transport.is_closing.side_effect = lambda: False
+        transport._sendfile_compatible = constants._SendfileMode.FALLBACK
+        with self.assertRaisesRegex(RuntimeError, 'fallback is disabled'):
+            self.loop.run_until_complete(
+                self.loop.sendfile(transport, None, fallback=False))
+
 
 if sys.platform == 'win32':
 


From webhook-mailer at python.org  Sat Jan 27 15:53:59 2018
From: webhook-mailer at python.org (Berker Peksag)
Date: Sat, 27 Jan 2018 20:53:59 -0000
Subject: [Python-checkins] Fix changed file detection on Travis (GH-3129)
Message-ID: <mailman.89.1517086441.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/b2ec3615c81ca4f3c938245842a45956da8d5acb
commit: b2ec3615c81ca4f3c938245842a45956da8d5acb
branch: master
author: Ammar Askar <ammar_askar at hotmail.com>
committer: Berker Peksag <berker.peksag at gmail.com>
date: 2018-01-27T23:53:56+03:00
summary:

Fix changed file detection on Travis (GH-3129)

Travis when merging changes from a pull request onto the target branch
does not perform a rebase, instead it does a simple merge which causes
the PR commits to retain their commit dates. This means that the commit
log can potentially look like:

PR merge <-- HEAD
normal master commit <- master
more commits from normal workflow
PR commit 1
another master commit
PR commit 2

Performing a git diff from PR commit 2 to master will accidentally
include files that should not be there.

Closes python/core-workflow#14

files:
M .travis.yml

diff --git a/.travis.yml b/.travis.yml
index 687d0214ab8..d7387e5f983 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -83,7 +83,21 @@ matrix:
 before_script:
   - |
       set -e
-      if ! git diff --name-only $TRAVIS_COMMIT_RANGE | grep -qvE '(\.rst$)|(^Doc)|(^Misc)'
+      if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
+        files_changed=$(git diff --name-only $TRAVIS_COMMIT_RANGE)
+      else
+        # Pull requests are slightly complicated because merging the PR commit without
+        # rebasing causes it to retain its old commit date. Meaning in history if any
+        # commits have been made on master that post-date it, they will be accidentally
+        # included in the diff if we use the TRAVIS_COMMIT_RANGE variable.
+        files_changed=$(git diff --name-only HEAD $(git merge-base HEAD $TRAVIS_BRANCH))
+      fi
+
+      # Prints changed files in this commit to help debug doc-only build issues.
+      echo "Files changed: "
+      echo $files_changed
+
+      if ! echo $files_changed | grep -qvE '(\.rst$)|(^Doc)|(^Misc)'
       then
         echo "Only docs were updated, stopping build process."
         exit


From webhook-mailer at python.org  Sat Jan 27 17:00:40 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Sat, 27 Jan 2018 22:00:40 -0000
Subject: [Python-checkins] Fix tests if Python compiled without SSL (#5367)
Message-ID: <mailman.90.1517090440.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/0f54e00e9633638a8da7f35b2f65556396eedfd0
commit: 0f54e00e9633638a8da7f35b2f65556396eedfd0
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T00:00:37+02:00
summary:

Fix tests if Python compiled without SSL (#5367)

files:
M Lib/test/test_asyncio/test_events.py

diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index b66d91d415e8..f54802cf8af2 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -2134,6 +2134,8 @@ def prepare(self, *, is_ssl=False, close_after=0):
         port = support.find_unused_port()
         srv_proto = MySendfileProto(loop=self.loop, close_after=close_after)
         if is_ssl:
+            if not ssl:
+                self.skipTest("No ssl module")
             srv_ctx = test_utils.simple_server_sslcontext()
             cli_ctx = test_utils.simple_client_sslcontext()
         else:


From webhook-mailer at python.org  Sat Jan 27 17:11:13 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 27 Jan 2018 22:11:13 -0000
Subject: [Python-checkins] bpo-32622: Fix AbstractEventLoop.sendfile
 signature in documentation. (GH-5368)
Message-ID: <mailman.91.1517091075.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/ee72ac0683e685b134f67cb0c6612c664ecadb65
commit: ee72ac0683e685b134f67cb0c6612c664ecadb65
branch: master
author: Elvis Pranskevichus <elvis at magic.io>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-27T17:11:10-05:00
summary:

bpo-32622: Fix AbstractEventLoop.sendfile signature in documentation. (GH-5368)

files:
M Doc/library/asyncio-eventloop.rst

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index fe162236e010..78ae18391f73 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -546,7 +546,7 @@ Creating listening connections
 File Transferring
 -----------------
 
-.. coroutinemethod:: AbstractEventLoop.sendfile(sock, transport, \
+.. coroutinemethod:: AbstractEventLoop.sendfile(transport, file, \
                                                 offset=0, count=None, \
                                                 *, fallback=True)
 


From webhook-mailer at python.org  Sat Jan 27 17:20:53 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 27 Jan 2018 22:20:53 -0000
Subject: [Python-checkins] bpo-32221: makeipaddr(): remove interface part +
 speedup (GH-4724)
Message-ID: <mailman.92.1517091655.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/47c0b1f7d4115e6f15e6776c1f91d28e7d96fe0c
commit: 47c0b1f7d4115e6f15e6776c1f91d28e7d96fe0c
branch: master
author: ????????? ???? <socketpair at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-27T17:20:50-05:00
summary:

bpo-32221: makeipaddr(): remove interface part + speedup (GH-4724)

files:
A Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst
M Doc/library/socket.rst
M Lib/test/test_socket.py
M Modules/socketmodule.c

diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index 7edd4ba36043..777710f16f26 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -77,6 +77,11 @@ created.  Socket addresses are represented as follows:
   backward compatibility.  Note, however, omission of *scopeid* can cause problems
   in manipulating scoped IPv6 addresses.
 
+  .. versionchanged:: 3.7
+     For multicast addresses (with *scopeid* meaningful) *address* may not contain
+     ``%scope`` (or ``zone id``) part. This information is superfluous and may
+     be safely omitted (recommended).
+
 - :const:`AF_NETLINK` sockets are represented as pairs ``(pid, groups)``.
 
 - Linux-only support for TIPC is available using the :const:`AF_TIPC`
@@ -630,6 +635,10 @@ The :mod:`socket` module also offers various network-related services:
    .. versionchanged:: 3.2
       parameters can now be passed using keyword arguments.
 
+   .. versionchanged:: 3.7
+      for IPv6 multicast addresses, string representing an address will not
+      contain ``%scope`` part.
+
 .. function:: getfqdn([name])
 
    Return a fully qualified domain name for *name*. If *name* is omitted or empty,
@@ -688,6 +697,8 @@ The :mod:`socket` module also offers various network-related services:
    or numeric address representation in *host*.  Similarly, *port* can contain a
    string port name or a numeric port number.
 
+   For IPv6 addresses, ``%scope`` is appended to the host part if *sockaddr*
+   contains meaningful *scopeid*. Usually this happens for multicast addresses.
 
 .. function:: getprotobyname(protocolname)
 
@@ -1178,6 +1189,10 @@ to sockets.
       an exception, the method now retries the system call instead of raising
       an :exc:`InterruptedError` exception (see :pep:`475` for the rationale).
 
+   .. versionchanged:: 3.7
+      For multicast IPv6 address, first item of *address* does not contain
+      ``%scope`` part anymore. In order to get full IPv6 address use
+      :func:`getnameinfo`.
 
 .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]])
 
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 275384c28fb3..a971f921546f 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1586,6 +1586,50 @@ def test_flowinfo(self):
         with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as s:
             self.assertRaises(OverflowError, s.bind, (support.HOSTv6, 0, -10))
 
+    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
+    def test_getaddrinfo_ipv6_basic(self):
+        ((*_, sockaddr),) = socket.getaddrinfo(
+            'ff02::1de:c0:face:8D',  # Note capital letter `D`.
+            1234, socket.AF_INET6,
+            socket.SOCK_DGRAM,
+            socket.IPPROTO_UDP
+        )
+        self.assertEqual(sockaddr, ('ff02::1de:c0:face:8d', 1234, 0, 0))
+
+    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
+    @unittest.skipUnless(
+        hasattr(socket, 'if_nameindex'),
+        'IPv6 scope id by interface name is not supported'
+    )
+    def test_getaddrinfo_ipv6_scopeid(self):
+        test_interface = 'lo'
+        ifindex = socket.if_nametoindex(test_interface)
+        ((*_, sockaddr),) = socket.getaddrinfo(
+            'ff02::1de:c0:face:8D%' + test_interface,
+            1234, socket.AF_INET6,
+            socket.SOCK_DGRAM,
+            socket.IPPROTO_UDP
+        )
+        # Note missing interface name part in IPv6 address
+        self.assertEqual(sockaddr, ('ff02::1de:c0:face:8d', 1234, 0, ifindex))
+
+    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
+    def test_getaddrinfo_ipv6_scopeid_numeric(self):
+        ((*_, sockaddr),) = socket.getaddrinfo(
+            'ff02::1de:c0:face:8D%42',
+            1234, socket.AF_INET6,
+            socket.SOCK_DGRAM,
+            socket.IPPROTO_UDP
+        )
+        # Note missing interface name part in IPv6 address
+        self.assertEqual(sockaddr, ('ff02::1de:c0:face:8d', 1234, 0, 42))
+
+    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
+    def test_getnameinfo_ipv6_scopeid(self):
+        sockaddr = ('ff02::1de:c0:face:8D', 1234, 0, 100500)  # Note capital letter `D`.
+        nameinfo = socket.getnameinfo(sockaddr, socket.NI_NUMERICHOST | socket.NI_NUMERICSERV)
+        self.assertEqual(nameinfo, ('ff02::1de:c0:face:8d%100500', '1234'))
+
     def test_str_for_enums(self):
         # Make sure that the AF_* and SOCK_* constants have enum-like string
         # reprs.
diff --git a/Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst b/Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst
new file mode 100644
index 000000000000..a88dcf48e02b
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst
@@ -0,0 +1,4 @@
+Various functions returning tuple containig IPv6 addresses now omit ``%scope``
+part since the same information is already encoded in *scopeid* tuple item.
+Especially this speeds up :func:`socket.recvfrom` when it receives multicast
+packet since useless resolving of network interface name is omitted.
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index 5fe2431bee84..30afe81d6352 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -1094,25 +1094,33 @@ setipaddr(const char *name, struct sockaddr *addr_ret, size_t addr_ret_size, int
 }
 
 
-/* Create a string object representing an IP address.
-   This is always a string of the form 'dd.dd.dd.dd' (with variable
-   size numbers). */
+/* Convert IPv4 sockaddr to a Python str. */
 
 static PyObject *
-makeipaddr(struct sockaddr *addr, int addrlen)
+make_ipv4_addr(const struct sockaddr_in *addr)
 {
-    char buf[NI_MAXHOST];
-    int error;
-
-    error = getnameinfo(addr, addrlen, buf, sizeof(buf), NULL, 0,
-        NI_NUMERICHOST);
-    if (error) {
-        set_gaierror(error);
+    char buf[INET_ADDRSTRLEN];
+    if (inet_ntop(AF_INET, &addr->sin_addr, buf, sizeof(buf)) == NULL) {
+        PyErr_SetFromErrno(PyExc_OSError);
         return NULL;
     }
     return PyUnicode_FromString(buf);
 }
 
+#ifdef ENABLE_IPV6
+/* Convert IPv6 sockaddr to a Python str. */
+
+static PyObject *
+make_ipv6_addr(const struct sockaddr_in6 *addr)
+{
+    char buf[INET6_ADDRSTRLEN];
+    if (inet_ntop(AF_INET6, &addr->sin6_addr, buf, sizeof(buf)) == NULL) {
+        PyErr_SetFromErrno(PyExc_OSError);
+        return NULL;
+    }
+    return PyUnicode_FromString(buf);
+}
+#endif
 
 #ifdef USE_BLUETOOTH
 /* Convert a string representation of a Bluetooth address into a numeric
@@ -1177,11 +1185,10 @@ makesockaddr(SOCKET_T sockfd, struct sockaddr *addr, size_t addrlen, int proto)
 
     case AF_INET:
     {
-        struct sockaddr_in *a;
-        PyObject *addrobj = makeipaddr(addr, sizeof(*a));
+        const struct sockaddr_in *a = (const struct sockaddr_in *)addr;
+        PyObject *addrobj = make_ipv4_addr(a);
         PyObject *ret = NULL;
         if (addrobj) {
-            a = (struct sockaddr_in *)addr;
             ret = Py_BuildValue("Oi", addrobj, ntohs(a->sin_port));
             Py_DECREF(addrobj);
         }
@@ -1225,11 +1232,10 @@ makesockaddr(SOCKET_T sockfd, struct sockaddr *addr, size_t addrlen, int proto)
 #ifdef ENABLE_IPV6
     case AF_INET6:
     {
-        struct sockaddr_in6 *a;
-        PyObject *addrobj = makeipaddr(addr, sizeof(*a));
+        const struct sockaddr_in6 *a = (const struct sockaddr_in6 *)addr;
+        PyObject *addrobj = make_ipv6_addr(a);
         PyObject *ret = NULL;
         if (addrobj) {
-            a = (struct sockaddr_in6 *)addr;
             ret = Py_BuildValue("OiII",
                                 addrobj,
                                 ntohs(a->sin6_port),
@@ -5036,14 +5042,14 @@ static PyObject *
 socket_gethostbyname(PyObject *self, PyObject *args)
 {
     char *name;
-    sock_addr_t addrbuf;
+    struct sockaddr_in addrbuf;
     PyObject *ret = NULL;
 
     if (!PyArg_ParseTuple(args, "et:gethostbyname", "idna", &name))
         return NULL;
-    if (setipaddr(name, SAS2SA(&addrbuf),  sizeof(addrbuf), AF_INET) < 0)
+    if (setipaddr(name, (struct sockaddr *)&addrbuf,  sizeof(addrbuf), AF_INET) < 0)
         goto finally;
-    ret = makeipaddr(SAS2SA(&addrbuf), sizeof(struct sockaddr_in));
+    ret = make_ipv4_addr(&addrbuf);
 finally:
     PyMem_Free(name);
     return ret;
@@ -5145,7 +5151,7 @@ gethost_common(struct hostent *h, struct sockaddr *addr, size_t alen, int af)
             sin.sin_len = sizeof(sin);
 #endif
             memcpy(&sin.sin_addr, *pch, sizeof(sin.sin_addr));
-            tmp = makeipaddr((struct sockaddr *)&sin, sizeof(sin));
+            tmp = make_ipv4_addr(&sin);
 
             if (pch == h->h_addr_list && alen >= sizeof(sin))
                 memcpy((char *) addr, &sin, sizeof(sin));
@@ -5162,8 +5168,7 @@ gethost_common(struct hostent *h, struct sockaddr *addr, size_t alen, int af)
             sin6.sin6_len = sizeof(sin6);
 #endif
             memcpy(&sin6.sin6_addr, *pch, sizeof(sin6.sin6_addr));
-            tmp = makeipaddr((struct sockaddr *)&sin6,
-                sizeof(sin6));
+            tmp = make_ipv6_addr(&sin6);
 
             if (pch == h->h_addr_list && alen >= sizeof(sin6))
                 memcpy((char *) addr, &sin6, sizeof(sin6));
@@ -5934,14 +5939,11 @@ socket_inet_ntop(PyObject *self, PyObject *args)
     Py_buffer packed_ip;
     const char* retval;
 #ifdef ENABLE_IPV6
-    char ip[Py_MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1];
+    char ip[Py_MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN)];
 #else
-    char ip[INET_ADDRSTRLEN + 1];
+    char ip[INET_ADDRSTRLEN];
 #endif
 
-    /* Guarantee NUL-termination for PyUnicode_FromString() below */
-    memset((void *) &ip[0], '\0', sizeof(ip));
-
     if (!PyArg_ParseTuple(args, "iy*:inet_ntop", &af, &packed_ip)) {
         return NULL;
     }
@@ -5969,6 +5971,7 @@ socket_inet_ntop(PyObject *self, PyObject *args)
         return NULL;
     }
 
+    /* inet_ntop guarantee NUL-termination of resulting string. */
     retval = inet_ntop(af, packed_ip.buf, ip, sizeof(ip));
     PyBuffer_Release(&packed_ip);
     if (!retval) {


From webhook-mailer at python.org  Sat Jan 27 17:22:04 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 27 Jan 2018 22:22:04 -0000
Subject: [Python-checkins] bpo-32622: Normalize ENOTCONN to ConnectionError
 on macOS (GH-5369)
Message-ID: <mailman.93.1517091726.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/2a2247ce5e1984eb2f2c41b269b38dbb795a60cf
commit: 2a2247ce5e1984eb2f2c41b269b38dbb795a60cf
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-27T17:22:01-05:00
summary:

bpo-32622: Normalize ENOTCONN to ConnectionError on macOS (GH-5369)

On mac, sendfile throws ENOTCONN on a repeated sendfile call if
the connection is closed. Normalize it to behave like other systems.

files:
M Lib/asyncio/unix_events.py

diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
index a4d892acad05..6cac137cacb3 100644
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -362,6 +362,17 @@ def _sock_sendfile_native_impl(self, fut, registered_fd, sock, fileno,
                             fd, sock, fileno,
                             offset, count, blocksize, total_sent)
         except OSError as exc:
+            if (registered_fd is not None and
+                    exc.errno == errno.ENOTCONN and
+                    type(exc) is not ConnectionError):
+                # If we have an ENOTCONN and this isn't a first call to
+                # sendfile(), i.e. the connection was closed in the middle
+                # of the operation, normalize the error to ConnectionError
+                # to make it consistent across all Posix systems.
+                new_exc = ConnectionError(
+                    "socket is not connected", errno.ENOTCONN)
+                new_exc.__cause__ = exc
+                exc = new_exc
             if total_sent == 0:
                 # We can get here for different reasons, the main
                 # one being 'file' is not a regular mmap(2)-like


From webhook-mailer at python.org  Sat Jan 27 17:48:12 2018
From: webhook-mailer at python.org (Berker Peksag)
Date: Sat, 27 Jan 2018 22:48:12 -0000
Subject: [Python-checkins] Fix changed file detection on Travis (GH-3129)
Message-ID: <mailman.94.1517093293.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/4d2dd64471c5047fe4aa24fb407e9d855344388e
commit: 4d2dd64471c5047fe4aa24fb407e9d855344388e
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Berker Peksag <berker.peksag at gmail.com>
date: 2018-01-28T01:48:09+03:00
summary:

Fix changed file detection on Travis (GH-3129)

Travis when merging changes from a pull request onto the target branch
does not perform a rebase, instead it does a simple merge which causes
the PR commits to retain their commit dates. This means that the commit
log can potentially look like:

PR merge <-- HEAD
normal master commit <- master
more commits from normal workflow
PR commit 1
another master commit
PR commit 2

Performing a git diff from PR commit 2 to master will accidentally
include files that should not be there.

Closes python/core-workflow#14

(cherry picked from commit b2ec3615c81ca4f3c938245842a45956da8d5acb)

files:
M .travis.yml

diff --git a/.travis.yml b/.travis.yml
index e624092bba36..d568534929ad 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -66,7 +66,21 @@ matrix:
 before_script:
   - |
       set -e
-      if ! git diff --name-only $TRAVIS_COMMIT_RANGE | grep -qvE '(\.rst$)|(^Doc)|(^Misc)'
+      if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
+        files_changed=$(git diff --name-only $TRAVIS_COMMIT_RANGE)
+      else
+        # Pull requests are slightly complicated because merging the PR commit without
+        # rebasing causes it to retain its old commit date. Meaning in history if any
+        # commits have been made on master that post-date it, they will be accidentally
+        # included in the diff if we use the TRAVIS_COMMIT_RANGE variable.
+        files_changed=$(git diff --name-only HEAD $(git merge-base HEAD $TRAVIS_BRANCH))
+      fi
+
+      # Prints changed files in this commit to help debug doc-only build issues.
+      echo "Files changed: "
+      echo $files_changed
+
+      if ! echo $files_changed | grep -qvE '(\.rst$)|(^Doc)|(^Misc)'
       then
         echo "Only docs were updated, stopping build process."
         exit


From webhook-mailer at python.org  Sat Jan 27 19:07:43 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sun, 28 Jan 2018 00:07:43 -0000
Subject: [Python-checkins] bpo-32513: Make it easier to override dunders in
 dataclasses. (GH-5366)
Message-ID: <mailman.95.1517098067.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/ea8fc52e75363276db23c6a8d7a689f79efce4f9
commit: ea8fc52e75363276db23c6a8d7a689f79efce4f9
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-27T19:07:40-05:00
summary:

bpo-32513: Make it easier to override dunders in dataclasses. (GH-5366)

Class authors no longer need to specify repr=False if they want to provide a custom __repr__ for dataclasses. The same thing applies for the other dunder methods that the dataclass decorator adds. If dataclass finds that a dunder methods is defined in the class, it will not overwrite it.

files:
A Misc/NEWS.d/next/Library/2018-01-27-11-20-16.bpo-32513.ak-iD2.rst
M Lib/dataclasses.py
M Lib/test/test_dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 7d30da1aacff..fb279cd30517 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -18,6 +18,142 @@
            'is_dataclass',
            ]
 
+# Conditions for adding methods.  The boxes indicate what action the
+#  dataclass decorator takes.  For all of these tables, when I talk
+#  about init=, repr=, eq=, order=, hash=, or frozen=, I'm referring
+#  to the arguments to the @dataclass decorator.  When checking if a
+#  dunder method already exists, I mean check for an entry in the
+#  class's __dict__.  I never check to see if an attribute is defined
+#  in a base class.
+
+# Key:
+# +=========+=========================================+
+# + Value   | Meaning                                 |
+# +=========+=========================================+
+# | <blank> | No action: no method is added.          |
+# +---------+-----------------------------------------+
+# | add     | Generated method is added.              |
+# +---------+-----------------------------------------+
+# | add*    | Generated method is added only if the   |
+# |         |  existing attribute is None and if the  |
+# |         |  user supplied a __eq__ method in the   |
+# |         |  class definition.                      |
+# +---------+-----------------------------------------+
+# | raise   | TypeError is raised.                    |
+# +---------+-----------------------------------------+
+# | None    | Attribute is set to None.               |
+# +=========+=========================================+
+
+# __init__
+#
+#   +--- init= parameter
+#   |
+#   v     |       |       |
+#         |  no   |  yes  |  <--- class has __init__ in __dict__?
+# +=======+=======+=======+
+# | False |       |       |
+# +-------+-------+-------+
+# | True  | add   |       |  <- the default
+# +=======+=======+=======+
+
+# __repr__
+#
+#    +--- repr= parameter
+#    |
+#    v    |       |       |
+#         |  no   |  yes  |  <--- class has __repr__ in __dict__?
+# +=======+=======+=======+
+# | False |       |       |
+# +-------+-------+-------+
+# | True  | add   |       |  <- the default
+# +=======+=======+=======+
+
+
+# __setattr__
+# __delattr__
+#
+#    +--- frozen= parameter
+#    |
+#    v    |       |       |
+#         |  no   |  yes  |  <--- class has __setattr__ or __delattr__ in __dict__?
+# +=======+=======+=======+
+# | False |       |       |  <- the default
+# +-------+-------+-------+
+# | True  | add   | raise |
+# +=======+=======+=======+
+# Raise because not adding these methods would break the "frozen-ness"
+#  of the class.
+
+# __eq__
+#
+#    +--- eq= parameter
+#    |
+#    v    |       |       |
+#         |  no   |  yes  |  <--- class has __eq__ in __dict__?
+# +=======+=======+=======+
+# | False |       |       |
+# +-------+-------+-------+
+# | True  | add   |       |  <- the default
+# +=======+=======+=======+
+
+# __lt__
+# __le__
+# __gt__
+# __ge__
+#
+#    +--- order= parameter
+#    |
+#    v    |       |       |
+#         |  no   |  yes  |  <--- class has any comparison method in __dict__?
+# +=======+=======+=======+
+# | False |       |       |  <- the default
+# +-------+-------+-------+
+# | True  | add   | raise |
+# +=======+=======+=======+
+# Raise because to allow this case would interfere with using
+#  functools.total_ordering.
+
+# __hash__
+
+#      +------------------- hash= parameter
+#      |       +----------- eq= parameter
+#      |       |       +--- frozen= parameter
+#      |       |       |
+#      v       v       v    |        |        |
+#                           |   no   |  yes   |  <--- class has __hash__ in __dict__?
+# +=========+=======+=======+========+========+
+# | 1 None  | False | False |        |        | No __eq__, use the base class __hash__
+# +---------+-------+-------+--------+--------+
+# | 2 None  | False | True  |        |        | No __eq__, use the base class __hash__
+# +---------+-------+-------+--------+--------+
+# | 3 None  | True  | False | None   |        | <-- the default, not hashable
+# +---------+-------+-------+--------+--------+
+# | 4 None  | True  | True  | add    | add*   | Frozen, so hashable
+# +---------+-------+-------+--------+--------+
+# | 5 False | False | False |        |        |
+# +---------+-------+-------+--------+--------+
+# | 6 False | False | True  |        |        |
+# +---------+-------+-------+--------+--------+
+# | 7 False | True  | False |        |        |
+# +---------+-------+-------+--------+--------+
+# | 8 False | True  | True  |        |        |
+# +---------+-------+-------+--------+--------+
+# | 9 True  | False | False | add    | add*   | Has no __eq__, but hashable
+# +---------+-------+-------+--------+--------+
+# |10 True  | False | True  | add    | add*   | Has no __eq__, but hashable
+# +---------+-------+-------+--------+--------+
+# |11 True  | True  | False | add    | add*   | Not frozen, but hashable
+# +---------+-------+-------+--------+--------+
+# |12 True  | True  | True  | add    | add*   | Frozen, so hashable
+# +=========+=======+=======+========+========+
+# For boxes that are blank, __hash__ is untouched and therefore
+#  inherited from the base class.  If the base is object, then
+#  id-based hashing is used.
+# Note that a class may have already __hash__=None if it specified an
+#  __eq__ method in the class body (not one that was created by
+#  @dataclass).
+
+
 # Raised when an attempt is made to modify a frozen class.
 class FrozenInstanceError(AttributeError): pass
 
@@ -143,13 +279,13 @@ def _tuple_str(obj_name, fields):
     #  return "(self.x,self.y)".
 
     # Special case for the 0-tuple.
-    if len(fields) == 0:
+    if not fields:
         return '()'
     # Note the trailing comma, needed if this turns out to be a 1-tuple.
     return f'({",".join([f"{obj_name}.{f.name}" for f in fields])},)'
 
 
-def _create_fn(name, args, body, globals=None, locals=None,
+def _create_fn(name, args, body, *, globals=None, locals=None,
                return_type=MISSING):
     # Note that we mutate locals when exec() is called. Caller beware!
     if locals is None:
@@ -287,7 +423,7 @@ def _init_fn(fields, frozen, has_post_init, self_name):
         body_lines += [f'{self_name}.{_POST_INIT_NAME}({params_str})']
 
     # If no body lines, use 'pass'.
-    if len(body_lines) == 0:
+    if not body_lines:
         body_lines = ['pass']
 
     locals = {f'_type_{f.name}': f.type for f in fields}
@@ -329,32 +465,6 @@ def _cmp_fn(name, op, self_tuple, other_tuple):
                         'return NotImplemented'])
 
 
-def _set_eq_fns(cls, fields):
-    # Create and set the equality comparison methods on cls.
-    # Pre-compute self_tuple and other_tuple, then re-use them for
-    #  each function.
-    self_tuple = _tuple_str('self', fields)
-    other_tuple = _tuple_str('other', fields)
-    for name, op in [('__eq__', '=='),
-                     ('__ne__', '!='),
-                     ]:
-        _set_attribute(cls, name, _cmp_fn(name, op, self_tuple, other_tuple))
-
-
-def _set_order_fns(cls, fields):
-    # Create and set the ordering methods on cls.
-    # Pre-compute self_tuple and other_tuple, then re-use them for
-    #  each function.
-    self_tuple = _tuple_str('self', fields)
-    other_tuple = _tuple_str('other', fields)
-    for name, op in [('__lt__', '<'),
-                     ('__le__', '<='),
-                     ('__gt__', '>'),
-                     ('__ge__', '>='),
-                     ]:
-        _set_attribute(cls, name, _cmp_fn(name, op, self_tuple, other_tuple))
-
-
 def _hash_fn(fields):
     self_tuple = _tuple_str('self', fields)
     return _create_fn('__hash__',
@@ -431,20 +541,20 @@ def _find_fields(cls):
     #  a Field(), then it contains additional info beyond (and
     #  possibly including) the actual default value.  Pseudo-fields
     #  ClassVars and InitVars are included, despite the fact that
-    #  they're not real fields.  That's deal with later.
+    #  they're not real fields.  That's dealt with later.
 
     annotations = getattr(cls, '__annotations__', {})
-
     return [_get_field(cls, a_name, a_type)
             for a_name, a_type in annotations.items()]
 
 
-def _set_attribute(cls, name, value):
-    # Raise TypeError if an attribute by this name already exists.
+def _set_new_attribute(cls, name, value):
+    # Never overwrites an existing attribute.  Returns True if the
+    #  attribute already exists.
     if name in cls.__dict__:
-        raise TypeError(f'Cannot overwrite attribute {name} '
-                        f'in {cls.__name__}')
+        return True
     setattr(cls, name, value)
+    return False
 
 
 def _process_class(cls, repr, eq, order, hash, init, frozen):
@@ -495,6 +605,9 @@ def _process_class(cls, repr, eq, order, hash, init, frozen):
     #  be inherited down.
     is_frozen = frozen or cls.__setattr__ is _frozen_setattr
 
+    # Was this class defined with an __eq__?  Used in __hash__ logic.
+    auto_hash_test= '__eq__' in cls.__dict__ and getattr(cls.__dict__, '__hash__', MISSING) is None
+
     # If we're generating ordering methods, we must be generating
     #  the eq methods.
     if order and not eq:
@@ -505,62 +618,91 @@ def _process_class(cls, repr, eq, order, hash, init, frozen):
         has_post_init = hasattr(cls, _POST_INIT_NAME)
 
         # Include InitVars and regular fields (so, not ClassVars).
-        _set_attribute(cls, '__init__',
-                       _init_fn(list(filter(lambda f: f._field_type
-                                              in (_FIELD, _FIELD_INITVAR),
-                                            fields.values())),
-                                is_frozen,
-                                has_post_init,
-                                # The name to use for the "self" param
-                                #  in __init__.  Use "self" if possible.
-                                '__dataclass_self__' if 'self' in fields
-                                    else 'self',
-                                ))
+        flds = [f for f in fields.values()
+                if f._field_type in (_FIELD, _FIELD_INITVAR)]
+        _set_new_attribute(cls, '__init__',
+                           _init_fn(flds,
+                                    is_frozen,
+                                    has_post_init,
+                                    # The name to use for the "self" param
+                                    #  in __init__.  Use "self" if possible.
+                                    '__dataclass_self__' if 'self' in fields
+                                            else 'self',
+                          ))
 
     # Get the fields as a list, and include only real fields.  This is
     #  used in all of the following methods.
-    field_list = list(filter(lambda f: f._field_type is _FIELD,
-                             fields.values()))
+    field_list = [f for f in fields.values() if f._field_type is _FIELD]
 
     if repr:
-        _set_attribute(cls, '__repr__',
-                       _repr_fn(list(filter(lambda f: f.repr, field_list))))
-
-    if is_frozen:
-        _set_attribute(cls, '__setattr__', _frozen_setattr)
-        _set_attribute(cls, '__delattr__', _frozen_delattr)
-
-    generate_hash = False
-    if hash is None:
-        if eq and frozen:
-            # Generate a hash function.
-            generate_hash = True
-        elif eq and not frozen:
-            # Not hashable.
-            _set_attribute(cls, '__hash__', None)
-        elif not eq:
-            # Otherwise, use the base class definition of hash().  That is,
-            #  don't set anything on this class.
-            pass
-        else:
-            assert "can't get here"
-    else:
-        generate_hash = hash
-    if generate_hash:
-        _set_attribute(cls, '__hash__',
-                       _hash_fn(list(filter(lambda f: f.compare
-                                                      if f.hash is None
-                                                      else f.hash,
-                                            field_list))))
+        flds = [f for f in field_list if f.repr]
+        _set_new_attribute(cls, '__repr__', _repr_fn(flds))
 
     if eq:
-        # Create and __eq__ and __ne__ methods.
-        _set_eq_fns(cls, list(filter(lambda f: f.compare, field_list)))
+        # Create _eq__ method.  There's no need for a __ne__ method,
+        #  since python will call __eq__ and negate it.
+        flds = [f for f in field_list if f.compare]
+        self_tuple = _tuple_str('self', flds)
+        other_tuple = _tuple_str('other', flds)
+        _set_new_attribute(cls, '__eq__',
+                           _cmp_fn('__eq__', '==',
+                                   self_tuple, other_tuple))
 
     if order:
-        # Create and __lt__, __le__, __gt__, and __ge__ methods.
-        # Create and set the comparison functions.
-        _set_order_fns(cls, list(filter(lambda f: f.compare, field_list)))
+        # Create and set the ordering methods.
+        flds = [f for f in field_list if f.compare]
+        self_tuple = _tuple_str('self', flds)
+        other_tuple = _tuple_str('other', flds)
+        for name, op in [('__lt__', '<'),
+                         ('__le__', '<='),
+                         ('__gt__', '>'),
+                         ('__ge__', '>='),
+                         ]:
+            if _set_new_attribute(cls, name,
+                                  _cmp_fn(name, op, self_tuple, other_tuple)):
+                raise TypeError(f'Cannot overwrite attribute {name} '
+                                f'in {cls.__name__}. Consider using '
+                                'functools.total_ordering')
+
+    if is_frozen:
+        for name, fn in [('__setattr__', _frozen_setattr),
+                         ('__delattr__', _frozen_delattr)]:
+            if _set_new_attribute(cls, name, fn):
+                raise TypeError(f'Cannot overwrite attribute {name} '
+                                f'in {cls.__name__}')
+
+    # Decide if/how we're going to create a hash function.
+    # TODO: Move this table to module scope, so it's not recreated
+    #  all the time.
+    generate_hash = {(None,  False, False): ('',     ''),
+                     (None,  False, True):  ('',     ''),
+                     (None,  True,  False): ('none', ''),
+                     (None,  True,  True):  ('fn',   'fn-x'),
+                     (False, False, False): ('',     ''),
+                     (False, False, True):  ('',     ''),
+                     (False, True,  False): ('',     ''),
+                     (False, True,  True):  ('',     ''),
+                     (True,  False, False): ('fn',   'fn-x'),
+                     (True,  False, True):  ('fn',   'fn-x'),
+                     (True,  True,  False): ('fn',   'fn-x'),
+                     (True,  True,  True):  ('fn',   'fn-x'),
+                     }[None if hash is None else bool(hash),   # Force bool() if not None.
+                       bool(eq),
+                       bool(frozen)]['__hash__' in cls.__dict__]
+    # No need to call _set_new_attribute here, since we already know if
+    #  we're overwriting a __hash__ or not.
+    if generate_hash == '':
+        # Do nothing.
+        pass
+    elif generate_hash == 'none':
+        cls.__hash__ = None
+    elif generate_hash in ('fn', 'fn-x'):
+        if generate_hash == 'fn' or auto_hash_test:
+            flds = [f for f in field_list
+                    if (f.compare if f.hash is None else f.hash)]
+            cls.__hash__ = _hash_fn(flds)
+    else:
+        assert False, f"can't get here: {generate_hash}"
 
     if not getattr(cls, '__doc__'):
         # Create a class doc-string.
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index 69819ea45078..53281f9dd9de 100755
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -9,6 +9,7 @@
 from unittest.mock import Mock
 from typing import ClassVar, Any, List, Union, Tuple, Dict, Generic, TypeVar
 from collections import deque, OrderedDict, namedtuple
+from functools import total_ordering
 
 # Just any custom exception we can catch.
 class CustomError(Exception): pass
@@ -82,68 +83,12 @@ class B:
             class C(B):
                 x: int = 0
 
-    def test_overwriting_init(self):
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __init__ '
-                                    'in C'):
-            @dataclass
-            class C:
-                x: int
-                def __init__(self, x):
-                    self.x = 2 * x
-
-        @dataclass(init=False)
-        class C:
-            x: int
-            def __init__(self, x):
-                self.x = 2 * x
-        self.assertEqual(C(5).x, 10)
-
-    def test_overwriting_repr(self):
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __repr__ '
-                                    'in C'):
-            @dataclass
-            class C:
-                x: int
-                def __repr__(self):
-                    pass
-
-        @dataclass(repr=False)
-        class C:
-            x: int
-            def __repr__(self):
-                return 'x'
-        self.assertEqual(repr(C(0)), 'x')
-
-    def test_overwriting_cmp(self):
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __eq__ '
-                                    'in C'):
-            # This will generate the comparison functions, make sure we can't
-            #  overwrite them.
-            @dataclass(hash=False, frozen=False)
-            class C:
-                x: int
-                def __eq__(self):
-                    pass
-
-        @dataclass(order=False, eq=False)
+    def test_overwriting_hash(self):
+        @dataclass(frozen=True)
         class C:
             x: int
-            def __eq__(self, other):
-                return True
-        self.assertEqual(C(0), 'x')
-
-    def test_overwriting_hash(self):
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __hash__ '
-                                    'in C'):
-            @dataclass(frozen=True)
-            class C:
-                x: int
-                def __hash__(self):
-                    pass
+            def __hash__(self):
+                pass
 
         @dataclass(frozen=True,hash=False)
         class C:
@@ -152,14 +97,11 @@ def __hash__(self):
                 return 600
         self.assertEqual(hash(C(0)), 600)
 
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __hash__ '
-                                    'in C'):
-            @dataclass(frozen=True)
-            class C:
-                x: int
-                def __hash__(self):
-                    pass
+        @dataclass(frozen=True)
+        class C:
+            x: int
+            def __hash__(self):
+                pass
 
         @dataclass(frozen=True, hash=False)
         class C:
@@ -168,33 +110,6 @@ def __hash__(self):
                 return 600
         self.assertEqual(hash(C(0)), 600)
 
-    def test_overwriting_frozen(self):
-        # frozen uses __setattr__ and __delattr__
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __setattr__ '
-                                    'in C'):
-            @dataclass(frozen=True)
-            class C:
-                x: int
-                def __setattr__(self):
-                    pass
-
-        with self.assertRaisesRegex(TypeError,
-                                    'Cannot overwrite attribute __delattr__ '
-                                    'in C'):
-            @dataclass(frozen=True)
-            class C:
-                x: int
-                def __delattr__(self):
-                    pass
-
-        @dataclass(frozen=False)
-        class C:
-            x: int
-            def __setattr__(self, name, value):
-                self.__dict__['x'] = value * 2
-        self.assertEqual(C(10).x, 20)
-
     def test_overwrite_fields_in_derived_class(self):
         # Note that x from C1 replaces x in Base, but the order remains
         #  the same as defined in Base.
@@ -239,34 +154,6 @@ class C:
         first = next(iter(sig.parameters))
         self.assertEqual('self', first)
 
-    def test_repr(self):
-        @dataclass
-        class B:
-            x: int
-
-        @dataclass
-        class C(B):
-            y: int = 10
-
-        o = C(4)
-        self.assertEqual(repr(o), 'TestCase.test_repr.<locals>.C(x=4, y=10)')
-
-        @dataclass
-        class D(C):
-            x: int = 20
-        self.assertEqual(repr(D()), 'TestCase.test_repr.<locals>.D(x=20, y=10)')
-
-        @dataclass
-        class C:
-            @dataclass
-            class D:
-                i: int
-            @dataclass
-            class E:
-                pass
-        self.assertEqual(repr(C.D(0)), 'TestCase.test_repr.<locals>.C.D(i=0)')
-        self.assertEqual(repr(C.E()), 'TestCase.test_repr.<locals>.C.E()')
-
     def test_0_field_compare(self):
         # Ensure that order=False is the default.
         @dataclass
@@ -420,80 +307,8 @@ class C:
         self.assertEqual(hash(C(4)), hash((4,)))
         self.assertEqual(hash(C(42)), hash((42,)))
 
-    def test_hash(self):
-        @dataclass(hash=True)
-        class C:
-            x: int
-            y: str
-        self.assertEqual(hash(C(1, 'foo')), hash((1, 'foo')))
-
-    def test_no_hash(self):
-        @dataclass(hash=None)
-        class C:
-            x: int
-        with self.assertRaisesRegex(TypeError,
-                                    "unhashable type: 'C'"):
-            hash(C(1))
-
-    def test_hash_rules(self):
-        # There are 24 cases of:
-        #  hash=True/False/None
-        #  eq=True/False
-        #  order=True/False
-        #  frozen=True/False
-        for (hash,  eq,    order, frozen, result  ) in [
-            (False, False, False, False,  'absent'),
-            (False, False, False, True,   'absent'),
-            (False, False, True,  False,  'exception'),
-            (False, False, True,  True,   'exception'),
-            (False, True,  False, False,  'absent'),
-            (False, True,  False, True,   'absent'),
-            (False, True,  True,  False,  'absent'),
-            (False, True,  True,  True,   'absent'),
-            (True,  False, False, False,  'fn'),
-            (True,  False, False, True,   'fn'),
-            (True,  False, True,  False,  'exception'),
-            (True,  False, True,  True,   'exception'),
-            (True,  True,  False, False,  'fn'),
-            (True,  True,  False, True,   'fn'),
-            (True,  True,  True,  False,  'fn'),
-            (True,  True,  True,  True,   'fn'),
-            (None,  False, False, False,  'absent'),
-            (None,  False, False, True,   'absent'),
-            (None,  False, True,  False,  'exception'),
-            (None,  False, True,  True,   'exception'),
-            (None,  True,  False, False,  'none'),
-            (None,  True,  False, True,   'fn'),
-            (None,  True,  True,  False,  'none'),
-            (None,  True,  True,  True,   'fn'),
-        ]:
-            with self.subTest(hash=hash, eq=eq, order=order, frozen=frozen):
-                if result == 'exception':
-                    with self.assertRaisesRegex(ValueError, 'eq must be true if order is true'):
-                        @dataclass(hash=hash, eq=eq, order=order, frozen=frozen)
-                        class C:
-                            pass
-                else:
-                    @dataclass(hash=hash, eq=eq, order=order, frozen=frozen)
-                    class C:
-                        pass
-
-                    # See if the result matches what's expected.
-                    if result == 'fn':
-                        # __hash__ contains the function we generated.
-                        self.assertIn('__hash__', C.__dict__)
-                        self.assertIsNotNone(C.__dict__['__hash__'])
-                    elif result == 'absent':
-                        # __hash__ is not present in our class.
-                        self.assertNotIn('__hash__', C.__dict__)
-                    elif result == 'none':
-                        # __hash__ is set to None.
-                        self.assertIn('__hash__', C.__dict__)
-                        self.assertIsNone(C.__dict__['__hash__'])
-                    else:
-                        assert False, f'unknown result {result!r}'
-
     def test_eq_order(self):
+        # Test combining eq and order.
         for (eq,    order, result   ) in [
             (False, False, 'neither'),
             (False, True,  'exception'),
@@ -513,21 +328,18 @@ class C:
 
                     if result == 'neither':
                         self.assertNotIn('__eq__', C.__dict__)
-                        self.assertNotIn('__ne__', C.__dict__)
                         self.assertNotIn('__lt__', C.__dict__)
                         self.assertNotIn('__le__', C.__dict__)
                         self.assertNotIn('__gt__', C.__dict__)
                         self.assertNotIn('__ge__', C.__dict__)
                     elif result == 'both':
                         self.assertIn('__eq__', C.__dict__)
-                        self.assertIn('__ne__', C.__dict__)
                         self.assertIn('__lt__', C.__dict__)
                         self.assertIn('__le__', C.__dict__)
                         self.assertIn('__gt__', C.__dict__)
                         self.assertIn('__ge__', C.__dict__)
                     elif result == 'eq_only':
                         self.assertIn('__eq__', C.__dict__)
-                        self.assertIn('__ne__', C.__dict__)
                         self.assertNotIn('__lt__', C.__dict__)
                         self.assertNotIn('__le__', C.__dict__)
                         self.assertNotIn('__gt__', C.__dict__)
@@ -811,19 +623,6 @@ class C:
             y: int
         self.assertNotEqual(Point(1, 3), C(1, 3))
 
-    def test_base_has_init(self):
-        class B:
-            def __init__(self):
-                pass
-
-        # Make sure that declaring this class doesn't raise an error.
-        #  The issue is that we can't override __init__ in our class,
-        #  but it should be okay to add __init__ to us if our base has
-        #  an __init__.
-        @dataclass
-        class C(B):
-            x: int = 0
-
     def test_frozen(self):
         @dataclass(frozen=True)
         class C:
@@ -2065,6 +1864,7 @@ def test_helper_make_dataclass_no_types(self):
                                              'y': int,
                                              'z': 'typing.Any'})
 
+
 class TestDocString(unittest.TestCase):
     def assertDocStrEqual(self, a, b):
         # Because 3.6 and 3.7 differ in how inspect.signature work
@@ -2154,5 +1954,445 @@ class C:
         self.assertDocStrEqual(C.__doc__, "C(x:collections.deque=<factory>)")
 
 
+class TestInit(unittest.TestCase):
+    def test_base_has_init(self):
+        class B:
+            def __init__(self):
+                self.z = 100
+                pass
+
+        # Make sure that declaring this class doesn't raise an error.
+        #  The issue is that we can't override __init__ in our class,
+        #  but it should be okay to add __init__ to us if our base has
+        #  an __init__.
+        @dataclass
+        class C(B):
+            x: int = 0
+        c = C(10)
+        self.assertEqual(c.x, 10)
+        self.assertNotIn('z', vars(c))
+
+        # Make sure that if we don't add an init, the base __init__
+        #  gets called.
+        @dataclass(init=False)
+        class C(B):
+            x: int = 10
+        c = C()
+        self.assertEqual(c.x, 10)
+        self.assertEqual(c.z, 100)
+
+    def test_no_init(self):
+        dataclass(init=False)
+        class C:
+            i: int = 0
+        self.assertEqual(C().i, 0)
+
+        dataclass(init=False)
+        class C:
+            i: int = 2
+            def __init__(self):
+                self.i = 3
+        self.assertEqual(C().i, 3)
+
+    def test_overwriting_init(self):
+        # If the class has __init__, use it no matter the value of
+        #  init=.
+
+        @dataclass
+        class C:
+            x: int
+            def __init__(self, x):
+                self.x = 2 * x
+        self.assertEqual(C(3).x, 6)
+
+        @dataclass(init=True)
+        class C:
+            x: int
+            def __init__(self, x):
+                self.x = 2 * x
+        self.assertEqual(C(4).x, 8)
+
+        @dataclass(init=False)
+        class C:
+            x: int
+            def __init__(self, x):
+                self.x = 2 * x
+        self.assertEqual(C(5).x, 10)
+
+
+class TestRepr(unittest.TestCase):
+    def test_repr(self):
+        @dataclass
+        class B:
+            x: int
+
+        @dataclass
+        class C(B):
+            y: int = 10
+
+        o = C(4)
+        self.assertEqual(repr(o), 'TestRepr.test_repr.<locals>.C(x=4, y=10)')
+
+        @dataclass
+        class D(C):
+            x: int = 20
+        self.assertEqual(repr(D()), 'TestRepr.test_repr.<locals>.D(x=20, y=10)')
+
+        @dataclass
+        class C:
+            @dataclass
+            class D:
+                i: int
+            @dataclass
+            class E:
+                pass
+        self.assertEqual(repr(C.D(0)), 'TestRepr.test_repr.<locals>.C.D(i=0)')
+        self.assertEqual(repr(C.E()), 'TestRepr.test_repr.<locals>.C.E()')
+
+    def test_no_repr(self):
+        # Test a class with no __repr__ and repr=False.
+        @dataclass(repr=False)
+        class C:
+            x: int
+        self.assertIn('test_dataclasses.TestRepr.test_no_repr.<locals>.C object at',
+                      repr(C(3)))
+
+        # Test a class with a __repr__ and repr=False.
+        @dataclass(repr=False)
+        class C:
+            x: int
+            def __repr__(self):
+                return 'C-class'
+        self.assertEqual(repr(C(3)), 'C-class')
+
+    def test_overwriting_repr(self):
+        # If the class has __repr__, use it no matter the value of
+        #  repr=.
+
+        @dataclass
+        class C:
+            x: int
+            def __repr__(self):
+                return 'x'
+        self.assertEqual(repr(C(0)), 'x')
+
+        @dataclass(repr=True)
+        class C:
+            x: int
+            def __repr__(self):
+                return 'x'
+        self.assertEqual(repr(C(0)), 'x')
+
+        @dataclass(repr=False)
+        class C:
+            x: int
+            def __repr__(self):
+                return 'x'
+        self.assertEqual(repr(C(0)), 'x')
+
+
+class TestFrozen(unittest.TestCase):
+    def test_overwriting_frozen(self):
+        # frozen uses __setattr__ and __delattr__
+        with self.assertRaisesRegex(TypeError,
+                                    'Cannot overwrite attribute __setattr__'):
+            @dataclass(frozen=True)
+            class C:
+                x: int
+                def __setattr__(self):
+                    pass
+
+        with self.assertRaisesRegex(TypeError,
+                                    'Cannot overwrite attribute __delattr__'):
+            @dataclass(frozen=True)
+            class C:
+                x: int
+                def __delattr__(self):
+                    pass
+
+        @dataclass(frozen=False)
+        class C:
+            x: int
+            def __setattr__(self, name, value):
+                self.__dict__['x'] = value * 2
+        self.assertEqual(C(10).x, 20)
+
+
+class TestEq(unittest.TestCase):
+    def test_no_eq(self):
+        # Test a class with no __eq__ and eq=False.
+        @dataclass(eq=False)
+        class C:
+            x: int
+        self.assertNotEqual(C(0), C(0))
+        c = C(3)
+        self.assertEqual(c, c)
+
+        # Test a class with an __eq__ and eq=False.
+        @dataclass(eq=False)
+        class C:
+            x: int
+            def __eq__(self, other):
+                return other == 10
+        self.assertEqual(C(3), 10)
+
+    def test_overwriting_eq(self):
+        # If the class has __eq__, use it no matter the value of
+        #  eq=.
+
+        @dataclass
+        class C:
+            x: int
+            def __eq__(self, other):
+                return other == 3
+        self.assertEqual(C(1), 3)
+        self.assertNotEqual(C(1), 1)
+
+        @dataclass(eq=True)
+        class C:
+            x: int
+            def __eq__(self, other):
+                return other == 4
+        self.assertEqual(C(1), 4)
+        self.assertNotEqual(C(1), 1)
+
+        @dataclass(eq=False)
+        class C:
+            x: int
+            def __eq__(self, other):
+                return other == 5
+        self.assertEqual(C(1), 5)
+        self.assertNotEqual(C(1), 1)
+
+
+class TestOrdering(unittest.TestCase):
+    def test_functools_total_ordering(self):
+        # Test that functools.total_ordering works with this class.
+        @total_ordering
+        @dataclass
+        class C:
+            x: int
+            def __lt__(self, other):
+                # Perform the test "backward", just to make
+                #  sure this is being called.
+                return self.x >= other
+
+        self.assertLess(C(0), -1)
+        self.assertLessEqual(C(0), -1)
+        self.assertGreater(C(0), 1)
+        self.assertGreaterEqual(C(0), 1)
+
+    def test_no_order(self):
+        # Test that no ordering functions are added by default.
+        @dataclass(order=False)
+        class C:
+            x: int
+        # Make sure no order methods are added.
+        self.assertNotIn('__le__', C.__dict__)
+        self.assertNotIn('__lt__', C.__dict__)
+        self.assertNotIn('__ge__', C.__dict__)
+        self.assertNotIn('__gt__', C.__dict__)
+
+        # Test that __lt__ is still called
+        @dataclass(order=False)
+        class C:
+            x: int
+            def __lt__(self, other):
+                return False
+        # Make sure other methods aren't added.
+        self.assertNotIn('__le__', C.__dict__)
+        self.assertNotIn('__ge__', C.__dict__)
+        self.assertNotIn('__gt__', C.__dict__)
+
+    def test_overwriting_order(self):
+        with self.assertRaisesRegex(TypeError,
+                                    'Cannot overwrite attribute __lt__'
+                                    '.*using functools.total_ordering'):
+            @dataclass(order=True)
+            class C:
+                x: int
+                def __lt__(self):
+                    pass
+
+        with self.assertRaisesRegex(TypeError,
+                                    'Cannot overwrite attribute __le__'
+                                    '.*using functools.total_ordering'):
+            @dataclass(order=True)
+            class C:
+                x: int
+                def __le__(self):
+                    pass
+
+        with self.assertRaisesRegex(TypeError,
+                                    'Cannot overwrite attribute __gt__'
+                                    '.*using functools.total_ordering'):
+            @dataclass(order=True)
+            class C:
+                x: int
+                def __gt__(self):
+                    pass
+
+        with self.assertRaisesRegex(TypeError,
+                                    'Cannot overwrite attribute __ge__'
+                                    '.*using functools.total_ordering'):
+            @dataclass(order=True)
+            class C:
+                x: int
+                def __ge__(self):
+                    pass
+
+class TestHash(unittest.TestCase):
+    def test_hash(self):
+        @dataclass(hash=True)
+        class C:
+            x: int
+            y: str
+        self.assertEqual(hash(C(1, 'foo')), hash((1, 'foo')))
+
+    def test_hash_false(self):
+        @dataclass(hash=False)
+        class C:
+            x: int
+            y: str
+        self.assertNotEqual(hash(C(1, 'foo')), hash((1, 'foo')))
+
+    def test_hash_none(self):
+        @dataclass(hash=None)
+        class C:
+            x: int
+        with self.assertRaisesRegex(TypeError,
+                                    "unhashable type: 'C'"):
+            hash(C(1))
+
+    def test_hash_rules(self):
+        def non_bool(value):
+            # Map to something else that's True, but not a bool.
+            if value is None:
+                return None
+            if value:
+                return (3,)
+            return 0
+
+        def test(case, hash, eq, frozen, with_hash, result):
+            with self.subTest(case=case, hash=hash, eq=eq, frozen=frozen):
+                if with_hash:
+                    @dataclass(hash=hash, eq=eq, frozen=frozen)
+                    class C:
+                        def __hash__(self):
+                            return 0
+                else:
+                    @dataclass(hash=hash, eq=eq, frozen=frozen)
+                    class C:
+                        pass
+
+                # See if the result matches what's expected.
+                if result in ('fn', 'fn-x'):
+                    # __hash__ contains the function we generated.
+                    self.assertIn('__hash__', C.__dict__)
+                    self.assertIsNotNone(C.__dict__['__hash__'])
+
+                    if result == 'fn-x':
+                        # This is the "auto-hash test" case.  We
+                        #  should overwrite __hash__ iff there's an
+                        #  __eq__ and if __hash__=None.
+
+                        # There are two ways of getting __hash__=None:
+                        #  explicitely, and by defining __eq__.  If
+                        #  __eq__ is defined, python will add __hash__
+                        #  when the class is created.
+                        @dataclass(hash=hash, eq=eq, frozen=frozen)
+                        class C:
+                            def __eq__(self, other): pass
+                            __hash__ = None
+
+                        # Hash should be overwritten (non-None).
+                        self.assertIsNotNone(C.__dict__['__hash__'])
+
+                        # Same test as above, but we don't provide
+                        #  __hash__, it will implicitely set to None.
+                        @dataclass(hash=hash, eq=eq, frozen=frozen)
+                        class C:
+                            def __eq__(self, other): pass
+
+                        # Hash should be overwritten (non-None).
+                        self.assertIsNotNone(C.__dict__['__hash__'])
+
+                elif result == '':
+                    # __hash__ is not present in our class.
+                    if not with_hash:
+                        self.assertNotIn('__hash__', C.__dict__)
+                elif result == 'none':
+                    # __hash__ is set to None.
+                    self.assertIn('__hash__', C.__dict__)
+                    self.assertIsNone(C.__dict__['__hash__'])
+                else:
+                    assert False, f'unknown result {result!r}'
+
+        # There are 12 cases of:
+        #  hash=True/False/None
+        #  eq=True/False
+        #  frozen=True/False
+        # And for each of these, a different result if
+        #  __hash__ is defined or not.
+        for case, (hash,  eq,    frozen, result_no, result_yes) in enumerate([
+                  (None,  False, False,  '',        ''),
+                  (None,  False, True,   '',        ''),
+                  (None,  True,  False,  'none',    ''),
+                  (None,  True,  True,   'fn',      'fn-x'),
+                  (False, False, False,  '',        ''),
+                  (False, False, True,   '',        ''),
+                  (False, True,  False,  '',        ''),
+                  (False, True,  True,   '',        ''),
+                  (True,  False, False,  'fn',      'fn-x'),
+                  (True,  False, True,   'fn',      'fn-x'),
+                  (True,  True,  False,  'fn',      'fn-x'),
+                  (True,  True,  True,   'fn',      'fn-x'),
+        ], 1):
+            test(case, hash, eq, frozen, False, result_no)
+            test(case, hash, eq, frozen, True,  result_yes)
+
+            # Test non-bool truth values, too.  This is just to
+            #  make sure the data-driven table in the decorator
+            #  handles non-bool values.
+            test(case, non_bool(hash), non_bool(eq), non_bool(frozen), False, result_no)
+            test(case, non_bool(hash), non_bool(eq), non_bool(frozen), True,  result_yes)
+
+
+    def test_eq_only(self):
+        # If a class defines __eq__, __hash__ is automatically added
+        #  and set to None.  This is normal Python behavior, not
+        #  related to dataclasses.  Make sure we don't interfere with
+        #  that (see bpo=32546).
+
+        @dataclass
+        class C:
+            i: int
+            def __eq__(self, other):
+                return self.i == other.i
+        self.assertEqual(C(1), C(1))
+        self.assertNotEqual(C(1), C(4))
+
+        # And make sure things work in this case if we specify
+        #  hash=True.
+        @dataclass(hash=True)
+        class C:
+            i: int
+            def __eq__(self, other):
+                return self.i == other.i
+        self.assertEqual(C(1), C(1.0))
+        self.assertEqual(hash(C(1)), hash(C(1.0)))
+
+        # And check that the classes __eq__ is being used, despite
+        #  specifying eq=True.
+        @dataclass(hash=True, eq=True)
+        class C:
+            i: int
+            def __eq__(self, other):
+                return self.i == 3 and self.i == other.i
+        self.assertEqual(C(3), C(3))
+        self.assertNotEqual(C(1), C(1))
+        self.assertEqual(hash(C(1)), hash(C(1.0)))
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/Misc/NEWS.d/next/Library/2018-01-27-11-20-16.bpo-32513.ak-iD2.rst b/Misc/NEWS.d/next/Library/2018-01-27-11-20-16.bpo-32513.ak-iD2.rst
new file mode 100644
index 000000000000..48072417f771
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-27-11-20-16.bpo-32513.ak-iD2.rst
@@ -0,0 +1,2 @@
+In dataclasses, allow easier overriding of dunder methods without specifying
+decorator parameters.


From webhook-mailer at python.org  Sat Jan 27 19:59:15 2018
From: webhook-mailer at python.org (INADA Naoki)
Date: Sun, 28 Jan 2018 00:59:15 -0000
Subject: [Python-checkins] bpo-32677: Optimize str.isascii() (GH-5356)
Message-ID: <mailman.96.1517101158.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/bea57060c863d0c3474c79350bd9c557f2ff0e7c
commit: bea57060c863d0c3474c79350bd9c557f2ff0e7c
branch: master
author: INADA Naoki <methane at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T09:59:12+09:00
summary:

bpo-32677: Optimize str.isascii() (GH-5356)

files:
M Lib/test/string_tests.py
M Objects/bytes_methods.c

diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py
index 4be1d2118978..561b09a2d5ee 100644
--- a/Lib/test/string_tests.py
+++ b/Lib/test/string_tests.py
@@ -916,6 +916,13 @@ def test_isascii(self):
         self.checkequal(True, '\x00\x7f', 'isascii')
         self.checkequal(False, '\x80', 'isascii')
         self.checkequal(False, '\xe9', 'isascii')
+        # bytes.isascii() and bytearray.isascii() has optimization which
+        # check 4 or 8 bytes at once.  So check some alignments.
+        for p in range(8):
+            self.checkequal(True, ' '*p + '\x7f', 'isascii')
+            self.checkequal(False, ' '*p + '\x80', 'isascii')
+            self.checkequal(True, ' '*p + '\x7f' + ' '*8, 'isascii')
+            self.checkequal(False, ' '*p + '\x80' + ' '*8, 'isascii')
 
     def test_isdigit(self):
         self.checkequal(False, '', 'isdigit')
diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
index 149650f018fc..07842f746910 100644
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -98,19 +98,51 @@ PyDoc_STRVAR_shared(_Py_isascii__doc__,
 Return True if B is empty or all characters in B are ASCII,\n\
 False otherwise.");
 
+// Optimization is copied from ascii_decode in unicodeobject.c
+/* Mask to quickly check whether a C 'long' contains a
+   non-ASCII, UTF8-encoded char. */
+#if (SIZEOF_LONG == 8)
+# define ASCII_CHAR_MASK 0x8080808080808080UL
+#elif (SIZEOF_LONG == 4)
+# define ASCII_CHAR_MASK 0x80808080UL
+#else
+# error C 'long' size should be either 4 or 8!
+#endif
+
 PyObject*
 _Py_bytes_isascii(const char *cptr, Py_ssize_t len)
 {
-    const unsigned char *p = (unsigned char *) cptr;
-    const unsigned char *e = p + len;
-    for (; p < e; p++) {
-        if (*p >= 128) {
+    const char *p = cptr;
+    const char *end = p + len;
+    const char *aligned_end = (const char *) _Py_ALIGN_DOWN(end, SIZEOF_LONG);
+
+    while (p < end) {
+        /* Fast path, see in STRINGLIB(utf8_decode) in stringlib/codecs.h
+           for an explanation. */
+        if (_Py_IS_ALIGNED(p, SIZEOF_LONG)) {
+            /* Help allocation */
+            const char *_p = p;
+            while (_p < aligned_end) {
+                unsigned long value = *(unsigned long *) _p;
+                if (value & ASCII_CHAR_MASK) {
+                    Py_RETURN_FALSE;
+                }
+                _p += SIZEOF_LONG;
+            }
+            p = _p;
+            if (_p == end)
+                break;
+        }
+        if ((unsigned char)*p & 0x80) {
             Py_RETURN_FALSE;
         }
+        p++;
     }
     Py_RETURN_TRUE;
 }
 
+#undef ASCII_CHAR_MASK
+
 
 PyDoc_STRVAR_shared(_Py_isdigit__doc__,
 "B.isdigit() -> bool\n\


From webhook-mailer at python.org  Sat Jan 27 20:30:40 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sun, 28 Jan 2018 01:30:40 -0000
Subject: [Python-checkins] Pep 557 What's New (GH-5371)
Message-ID: <mailman.97.1517103041.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/c4b1248308f051146a50ecc5e31d85d794dc8d05
commit: c4b1248308f051146a50ecc5e31d85d794dc8d05
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-27T20:30:37-05:00
summary:

Pep 557 What's New (GH-5371)

Add PEP 557 Data CLasses to What's New.

files:
M Doc/whatsnew/3.7.rst

diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 1ece6a3befd..5e355345e31 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -311,6 +311,24 @@ environment variable are added to control the UTF-8 mode.
        PEP written and implemented by Victor Stinner
 
 
+.. _whatsnew37-pep557:
+
+PEP 557: Data Classes
+---------------------
+
+Adds a new module ``dataclasses``.  It provides a class decorator
+``dataclass`` which inspects the class's variable annotations (see
+:pep:`526`) and using them, adds methods such as ``__init__``,
+``__repr__``, and ``__eq__`` to the class.  It is similar to
+``typing.NamedTuple``, but also works on classes with mutable
+instances, among other features.
+
+.. seealso::
+
+    :pep:`557` -- Data Classes
+       PEP written and implemented by Eric V. Smith
+
+
 New Development Mode: -X dev
 ----------------------------
 


From webhook-mailer at python.org  Sat Jan 27 22:52:55 2018
From: webhook-mailer at python.org (Mariatta)
Date: Sun, 28 Jan 2018 03:52:55 -0000
Subject: [Python-checkins] [3.6] bpo-32664: Add missing "|" connector in
 Exceptions doc (GH-1173) (GH-5372)
Message-ID: <mailman.98.1517111578.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/d331515847f0a053b929e5a4795715cb111f42b3
commit: d331515847f0a053b929e5a4795715cb111f42b3
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-27T19:52:52-08:00
summary:

[3.6] bpo-32664: Add missing "|" connector in Exceptions doc (GH-1173) (GH-5372)

(cherry picked from commit 992ae6444cc745c23d3bcc604983cc8e39405bd2)

files:
M Doc/license.rst
M Lib/test/exception_hierarchy.txt

diff --git a/Doc/license.rst b/Doc/license.rst
index e8dbeb1ef28f..90190cd57e0c 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -571,7 +571,7 @@ The :mod:`test_epoll` contains the following notice::
 Select kqueue
 -------------
 
-The :mod:`select` and contains the following notice for the kqueue interface::
+The :mod:`select` contains the following notice for the kqueue interface::
 
   Copyright (c) 2000 Doug White, 2006 James Knight, 2007 Christian Heimes
   All rights reserved.
diff --git a/Lib/test/exception_hierarchy.txt b/Lib/test/exception_hierarchy.txt
index 7333b2aa6243..763a6c899b48 100644
--- a/Lib/test/exception_hierarchy.txt
+++ b/Lib/test/exception_hierarchy.txt
@@ -14,7 +14,7 @@ BaseException
       +-- BufferError
       +-- EOFError
       +-- ImportError
-           +-- ModuleNotFoundError
+      |    +-- ModuleNotFoundError
       +-- LookupError
       |    +-- IndexError
       |    +-- KeyError


From webhook-mailer at python.org  Sat Jan 27 23:17:49 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 28 Jan 2018 04:17:49 -0000
Subject: [Python-checkins] bpo-30306: release arguments of contextmanager
 (GH-1500)
Message-ID: <mailman.99.1517113072.2748.python-checkins@python.org>

https://github.com/python/cpython/commit/dd0e087edc8f1e4d2c0913236b1a62a77d9db6d8
commit: dd0e087edc8f1e4d2c0913236b1a62a77d9db6d8
branch: master
author: Martin Teichmann <lkb.teichmann at gmail.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-28T14:17:46+10:00
summary:

bpo-30306: release arguments of contextmanager (GH-1500)

The arguments to a generator function which is declared as a
contextmanager are stored inside the context manager, and
thus are kept alive, even when it is used as a regular context
manager, and not as a function decorator (where it needs
the original arguments to recreate the generator on each
call).

This is a possible unnecessary memory leak, so this changes
contextmanager.__enter__ to release the saved arguments,
as that method being called means that particular CM instance
isn't going to need to recreate the underlying generator.

Patch by Martin Teichmann.

files:
M Lib/contextlib.py
M Lib/test/test_contextlib.py

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index ef8f8c9f55bc..1ff8cdf1cecf 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -105,6 +105,9 @@ def _recreate_cm(self):
         return self.__class__(self.func, self.args, self.kwds)
 
     def __enter__(self):
+        # do not keep args and kwds alive unnecessarily
+        # they are only needed for recreation, which is not possible anymore
+        del self.args, self.kwds, self.func
         try:
             return next(self.gen)
         except StopIteration:
diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py
index 96ceaa7f0600..2a44404a603e 100644
--- a/Lib/test/test_contextlib.py
+++ b/Lib/test/test_contextlib.py
@@ -8,6 +8,7 @@
 import unittest
 from contextlib import *  # Tests __all__
 from test import support
+import weakref
 
 
 class TestAbstractContextManager(unittest.TestCase):
@@ -219,6 +220,52 @@ def woohoo(self, func, args, kwds):
         with woohoo(self=11, func=22, args=33, kwds=44) as target:
             self.assertEqual(target, (11, 22, 33, 44))
 
+    def test_nokeepref(self):
+        class A:
+            pass
+
+        @contextmanager
+        def woohoo(a, b):
+            a = weakref.ref(a)
+            b = weakref.ref(b)
+            self.assertIsNone(a())
+            self.assertIsNone(b())
+            yield
+
+        with woohoo(A(), b=A()):
+            pass
+
+    def test_param_errors(self):
+        @contextmanager
+        def woohoo(a, *, b):
+            yield
+
+        with self.assertRaises(TypeError):
+            woohoo()
+        with self.assertRaises(TypeError):
+            woohoo(3, 5)
+        with self.assertRaises(TypeError):
+            woohoo(b=3)
+
+    def test_recursive(self):
+        depth = 0
+        @contextmanager
+        def woohoo():
+            nonlocal depth
+            before = depth
+            depth += 1
+            yield
+            depth -= 1
+            self.assertEqual(depth, before)
+
+        @woohoo()
+        def recursive():
+            if depth < 10:
+                recursive()
+
+        recursive()
+        self.assertEqual(depth, 0)
+
 
 class ClosingTestCase(unittest.TestCase):
 


From webhook-mailer at python.org  Sat Jan 27 23:58:50 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 28 Jan 2018 04:58:50 -0000
Subject: [Python-checkins] bpo-32206: Update pdb usage to include new module
 option (GH-5111)
Message-ID: <mailman.0.1517115530.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/fcf8b4c1022bc2d2f84dcef5d72eabf40a25ea19
commit: fcf8b4c1022bc2d2f84dcef5d72eabf40a25ea19
branch: master
author: Mario Corchero <mariocj89 at gmail.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-28T14:58:47+10:00
summary:

bpo-32206: Update pdb usage to include new module option (GH-5111)

files:
M Lib/pdb.py
M Lib/test/test_pdb.py

diff --git a/Lib/pdb.py b/Lib/pdb.py
index d1a74bb7d7be..366a85b31960 100755
--- a/Lib/pdb.py
+++ b/Lib/pdb.py
@@ -1638,9 +1638,11 @@ def help():
     pydoc.pager(__doc__)
 
 _usage = """\
-usage: pdb.py [-c command] ... pyfile [arg] ...
+usage: pdb.py [-c command] ... [-m module | pyfile] [arg] ...
 
-Debug the Python program given by pyfile.
+Debug the Python program given by pyfile. Alternatively,
+an executable module or package to debug can be specified using
+the -m switch.
 
 Initial commands are read from .pdbrc files in your home directory
 and in the current directory, if they exist.  Commands supplied with
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 0cd235e98ae1..85b47d533491 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -1182,7 +1182,10 @@ def test_run_pdb_with_pdb(self):
             quit
         """
         stdout, stderr = self._run_pdb(["-m", "pdb"], commands)
-        self.assertIn("Debug the Python program given by pyfile.", stdout.splitlines())
+        self.assertIn(
+            pdb._usage,
+            stdout.replace('\r', '')  # remove \r for windows
+        )
 
     def test_module_without_a_main(self):
         module_name = 't_main'


From webhook-mailer at python.org  Sun Jan 28 00:06:14 2018
From: webhook-mailer at python.org (Mariatta)
Date: Sun, 28 Jan 2018 05:06:14 -0000
Subject: [Python-checkins] Minor grammar fixes to License.rst (GH-1174)
 (GH-5373)
Message-ID: <mailman.1.1517115974.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/0135dfdc1fb3bbbc5080f86c86b20a8961dbc11a
commit: 0135dfdc1fb3bbbc5080f86c86b20a8961dbc11a
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-27T21:06:12-08:00
summary:

Minor grammar fixes to License.rst (GH-1174) (GH-5373)

(cherry picked from commit 8312fba0a1ef8f0a1a6c1760c73a89e29cfca09a)

files:
M Doc/license.rst

diff --git a/Doc/license.rst b/Doc/license.rst
index 90190cd57e0c..aa4b1450e91a 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -545,7 +545,7 @@ The :mod:`xmlrpc.client` module contains the following notice::
 test_epoll
 ----------
 
-The :mod:`test_epoll` contains the following notice::
+The :mod:`test_epoll` module contains the following notice::
 
   Copyright (c) 2001-2006 Twisted Matrix Laboratories.
 
@@ -571,7 +571,8 @@ The :mod:`test_epoll` contains the following notice::
 Select kqueue
 -------------
 
-The :mod:`select` contains the following notice for the kqueue interface::
+The :mod:`select` module contains the following notice for the kqueue
+interface::
 
   Copyright (c) 2000 Doug White, 2006 James Knight, 2007 Christian Heimes
   All rights reserved.
@@ -927,7 +928,7 @@ on the cfuhash project::
 libmpdec
 --------
 
-The :mod:`_decimal` Module is built using an included copy of the libmpdec
+The :mod:`_decimal` module is built using an included copy of the libmpdec
 library unless the build is configured ``--with-system-libmpdec``::
 
    Copyright (c) 2008-2016 Stefan Krah. All rights reserved.


From webhook-mailer at python.org  Sun Jan 28 00:14:32 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 28 Jan 2018 05:14:32 -0000
Subject: [Python-checkins] bpo-30306: Add missing NEWS entry (GH-5374)
Message-ID: <mailman.2.1517116475.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/a278ad2faa76ae61569a58f36e06ba8745f4a9b7
commit: a278ad2faa76ae61569a58f36e06ba8745f4a9b7
branch: master
author: Nick Coghlan <ncoghlan at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T15:14:30+10:00
summary:

bpo-30306: Add missing NEWS entry (GH-5374)

files:
A Misc/NEWS.d/next/Library/2018-01-28-14-10-51.bpo-30306.TmKMXi.rst

diff --git a/Misc/NEWS.d/next/Library/2018-01-28-14-10-51.bpo-30306.TmKMXi.rst b/Misc/NEWS.d/next/Library/2018-01-28-14-10-51.bpo-30306.TmKMXi.rst
new file mode 100644
index 000000000000..65df2cdac022
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-28-14-10-51.bpo-30306.TmKMXi.rst
@@ -0,0 +1,4 @@
+contextlib.contextmanager now releases the arguments passed to the
+underlying generator as soon as the context manager is entered. Previously
+it would keep them alive for as long as the context manager was alive, even
+when not being used as a function decorator. Patch by Martin Teichmann.


From webhook-mailer at python.org  Sun Jan 28 02:38:23 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sun, 28 Jan 2018 07:38:23 -0000
Subject: [Python-checkins] bpo-32687: Fix wrong meaning of args for
 PyTrace_LINE/CALL in documentation (#5361)
Message-ID: <mailman.3.1517125106.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/9ed0aee27c249dada410a22fff4325a4a61df36d
commit: 9ed0aee27c249dada410a22fff4325a4a61df36d
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T15:38:21+08:00
summary:

bpo-32687: Fix wrong meaning of args for PyTrace_LINE/CALL in documentation (#5361)

files:
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 86faf47f53bc..9887d28b09d9 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1283,12 +1283,12 @@ Python-level trace functions in previous versions.
    +------------------------------+--------------------------------------+
    | Value of *what*              | Meaning of *arg*                     |
    +==============================+======================================+
-   | :const:`PyTrace_CALL`        | Always *NULL*.                       |
+   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.            |
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_EXCEPTION`   | Exception information as returned by |
    |                              | :func:`sys.exc_info`.                |
    +------------------------------+--------------------------------------+
-   | :const:`PyTrace_LINE`        | Always *NULL*.                       |
+   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.            |
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_RETURN`      | Value being returned to the caller,  |
    |                              | or *NULL* if caused by an exception. |


From webhook-mailer at python.org  Sun Jan 28 03:41:10 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sun, 28 Jan 2018 08:41:10 -0000
Subject: [Python-checkins] bpo-32687: Fix wrong meaning of args for
 PyTrace_LINE/CALL in documentation (GH-5361) (GH-5377)
Message-ID: <mailman.4.1517128874.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/854f0424de389f023b375dfd0b50c34dea29e8f8
commit: 854f0424de389f023b375dfd0b50c34dea29e8f8
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2018-01-28T16:41:05+08:00
summary:

bpo-32687: Fix wrong meaning of args for PyTrace_LINE/CALL in documentation (GH-5361) (GH-5377)

(cherry picked from commit 9ed0aee27c249dada410a22fff4325a4a61df36d)

files:
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index a2c55fea9f7b..bc92c3e16b89 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1063,12 +1063,12 @@ Python-level trace functions in previous versions.
    +------------------------------+--------------------------------------+
    | Value of *what*              | Meaning of *arg*                     |
    +==============================+======================================+
-   | :const:`PyTrace_CALL`        | Always *NULL*.                       |
+   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.            |
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_EXCEPTION`   | Exception information as returned by |
    |                              | :func:`sys.exc_info`.                |
    +------------------------------+--------------------------------------+
-   | :const:`PyTrace_LINE`        | Always *NULL*.                       |
+   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.            |
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_RETURN`      | Value being returned to the caller,  |
    |                              | or *NULL* if caused by an exception. |


From webhook-mailer at python.org  Sun Jan 28 03:41:32 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sun, 28 Jan 2018 08:41:32 -0000
Subject: [Python-checkins] bpo-32687: Fix wrong meaning of args for
 PyTrace_LINE/CALL in documentation (GH-5361) (GH-5378)
Message-ID: <mailman.5.1517128895.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/745e9de7dd9d237d54e92d6e4b469e916fb0352b
commit: 745e9de7dd9d237d54e92d6e4b469e916fb0352b
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2018-01-28T16:41:29+08:00
summary:

bpo-32687: Fix wrong meaning of args for PyTrace_LINE/CALL in documentation (GH-5361) (GH-5378)

(cherry picked from commit 9ed0aee27c249dada410a22fff4325a4a61df36d)

files:
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index eade4121a25f..f091ac81dae9 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -983,12 +983,12 @@ in previous versions.
    +------------------------------+--------------------------------------+
    | Value of *what*              | Meaning of *arg*                     |
    +==============================+======================================+
-   | :const:`PyTrace_CALL`        | Always *NULL*.                       |
+   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.            |
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_EXCEPTION`   | Exception information as returned by |
    |                              | :func:`sys.exc_info`.                |
    +------------------------------+--------------------------------------+
-   | :const:`PyTrace_LINE`        | Always *NULL*.                       |
+   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.            |
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_RETURN`      | Value being returned to the caller,  |
    |                              | or *NULL* if caused by an exception. |


From solipsis at pitrou.net  Sun Jan 28 04:13:29 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 28 Jan 2018 09:13:29 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=2
Message-ID: <20180128091329.1.C09B80EF1494EDF0@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_spawn leaked [0, 0, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogqhRLcz', '--timeout', '7200']

From webhook-mailer at python.org  Sun Jan 28 04:53:47 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 28 Jan 2018 09:53:47 -0000
Subject: [Python-checkins] bpo-32649: Add C API docs for per-opcode tracing
 & profiling (GH-5360)
Message-ID: <mailman.6.1517133230.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/255f7a26da47ca6b7bd1d375b8a04920f68c119c
commit: 255f7a26da47ca6b7bd1d375b8a04920f68c119c
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-28T19:53:38+10:00
summary:

bpo-32649: Add C API docs for per-opcode tracing & profiling (GH-5360)

Updating the C API docs was missed when the per-opcode
tracing & profiling support was initially added.

files:
A Misc/NEWS.d/next/Documentation/2018-01-27-23-36-31.bpo-32649.o7qOjF.rst
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 9887d28b09d9..33cb02d8c9cc 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1277,8 +1277,8 @@ Python-level trace functions in previous versions.
    registration function as *obj*, *frame* is the frame object to which the event
    pertains, *what* is one of the constants :const:`PyTrace_CALL`,
    :const:`PyTrace_EXCEPTION`, :const:`PyTrace_LINE`, :const:`PyTrace_RETURN`,
-   :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION`, or
-   :const:`PyTrace_C_RETURN`, and *arg* depends on the value of *what*:
+   :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION`, :const:`PyTrace_C_RETURN`,
+   or :const:`PyTrace_OPCODE`, and *arg* depends on the value of *what*:
 
    +------------------------------+--------------------------------------+
    | Value of *what*              | Meaning of *arg*                     |
@@ -1299,6 +1299,8 @@ Python-level trace functions in previous versions.
    +------------------------------+--------------------------------------+
    | :const:`PyTrace_C_RETURN`    | Function object being called.        |
    +------------------------------+--------------------------------------+
+   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.            |
+   +------------------------------+--------------------------------------+
 
 .. c:var:: int PyTrace_CALL
 
@@ -1322,8 +1324,9 @@ Python-level trace functions in previous versions.
 
 .. c:var:: int PyTrace_LINE
 
-   The value passed as the *what* parameter to a trace function (but not a
-   profiling function) when a line-number event is being reported.
+   The value passed as the *what* parameter to a :c:type:`Py_tracefunc` function
+   (but not a profiling function) when a line-number event is being reported.
+   It may be disabled for a frame by setting :attr:`f_trace_lines` to *0* on that frame.
 
 
 .. c:var:: int PyTrace_RETURN
@@ -1350,6 +1353,14 @@ Python-level trace functions in previous versions.
    function has returned.
 
 
+.. c:var:: int PyTrace_OPCODE
+
+   The value for the *what* parameter to :c:type:`Py_tracefunc` functions (but not
+   profiling functions) when a new opcode is about to be executed.  This event is
+   not emitted by default: it must be explicitly requested by setting
+   :attr:`f_trace_opcodes` to *1* on the frame.
+
+
 .. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj)
 
    Set the profiler function to *func*.  The *obj* parameter is passed to the
@@ -1357,17 +1368,17 @@ Python-level trace functions in previous versions.
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
    profile function is called for all monitored events except :const:`PyTrace_LINE`
-   and :const:`PyTrace_EXCEPTION`.
+   :const:`PyTrace_OPCODE` and :const:`PyTrace_EXCEPTION`.
 
 
 .. c:function:: void PyEval_SetTrace(Py_tracefunc func, PyObject *obj)
 
    Set the tracing function to *func*.  This is similar to
    :c:func:`PyEval_SetProfile`, except the tracing function does receive line-number
-   events and does not receive any event related to C function objects being called. Any
-   trace function registered using :c:func:`PyEval_SetTrace` will not receive
-   :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or :const:`PyTrace_C_RETURN`
-   as a value for the *what* parameter.
+   events and per-opcode events, but does not receive any event related to C function
+   objects being called.  Any trace function registered using :c:func:`PyEval_SetTrace`
+   will not receive :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or
+   :const:`PyTrace_C_RETURN` as a value for the *what* parameter.
 
 .. _advanced-debugging:
 
diff --git a/Misc/NEWS.d/next/Documentation/2018-01-27-23-36-31.bpo-32649.o7qOjF.rst b/Misc/NEWS.d/next/Documentation/2018-01-27-23-36-31.bpo-32649.o7qOjF.rst
new file mode 100644
index 000000000000..8e8045808fbd
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2018-01-27-23-36-31.bpo-32649.o7qOjF.rst
@@ -0,0 +1,2 @@
+Complete the C API documentation, profiling and tracing part with the newly
+added per-opcode events.


From webhook-mailer at python.org  Sun Jan 28 05:56:10 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 28 Jan 2018 10:56:10 -0000
Subject: [Python-checkins] bpo-32685: Improve suggestion for print statement
 (GH-5375)
Message-ID: <mailman.7.1517136973.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/43c0f1ac5ed8bc9c3bd048d2ce4de4c98a83de99
commit: 43c0f1ac5ed8bc9c3bd048d2ce4de4c98a83de99
branch: master
author: Nitish Chandra <nitishchandrachinta at gmail.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-28T20:56:02+10:00
summary:

bpo-32685: Improve suggestion for print statement (GH-5375)

Better account for single-line compound statements and
semi-colon separated statements when suggesting
Py3 replacements for Py2 print statements.

Initial patch by Nitish Chandra.

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst
M Lib/test/test_print.py
M Objects/exceptions.c

diff --git a/Lib/test/test_print.py b/Lib/test/test_print.py
index 7bc23cf9fc63..e8381122182e 100644
--- a/Lib/test/test_print.py
+++ b/Lib/test/test_print.py
@@ -165,6 +165,23 @@ def test_string_with_leading_whitespace(self):
 
         self.assertIn('print("Hello World")', str(context.exception))
 
+    # bpo-32685: Suggestions for print statement should be proper when
+    # it is in the same line as the header of a compound statement
+    # and/or followed by a semicolon
+    def test_string_with_semicolon(self):
+        python2_print_str = 'print p;'
+        with self.assertRaises(SyntaxError) as context:
+            exec(python2_print_str)
+
+        self.assertIn('print(p)', str(context.exception))
+
+    def test_string_in_loop_on_same_line(self):
+        python2_print_str = 'for i in s: print i'
+        with self.assertRaises(SyntaxError) as context:
+            exec(python2_print_str)
+
+        self.assertIn('print(i)', str(context.exception))
+
     def test_stream_redirection_hint_for_py2_migration(self):
         # Test correct hint produced for Py2 redirection syntax
         with self.assertRaises(TypeError) as context:
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst
new file mode 100644
index 000000000000..07d7a072f6e1
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst	
@@ -0,0 +1,3 @@
+Improve suggestion when the Python 2 form of print statement is either
+present on the same line as the header of a compound statement or else
+terminated by a semi-colon instead of a newline. Patch by Nitish Chandra.
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
index d59abd1a7f41..2cce40f88444 100644
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -2840,26 +2840,31 @@ _PyErr_TrySetFromCause(const char *format, ...)
 static int
 _set_legacy_print_statement_msg(PySyntaxErrorObject *self, Py_ssize_t start)
 {
-    PyObject *strip_sep_obj = PyUnicode_FromString(" \t\r\n");
-    if (strip_sep_obj == NULL)
-        return -1;
-
-    // PRINT_OFFSET is to remove `print ` word from the data.
+    // PRINT_OFFSET is to remove the `print ` prefix from the data.
     const int PRINT_OFFSET = 6;
     const int STRIP_BOTH = 2;
-    // Issue 32028: Handle case when whitespace is used with print call
-    PyObject *initial_data = _PyUnicode_XStrip(self->text, STRIP_BOTH, strip_sep_obj);
-    if (initial_data == NULL) {
-        Py_DECREF(strip_sep_obj);
-        return -1;
+    Py_ssize_t start_pos = start + PRINT_OFFSET;
+    Py_ssize_t text_len = PyUnicode_GET_LENGTH(self->text);
+    Py_UCS4 semicolon = ';';
+    Py_ssize_t end_pos = PyUnicode_FindChar(self->text, semicolon,
+                                            start_pos, text_len, 1);
+    if (end_pos < -1) {
+      return -1;
+    } else if (end_pos == -1) {
+      end_pos = text_len;
     }
-    Py_ssize_t text_len = PyUnicode_GET_LENGTH(initial_data);
-    PyObject *data = PyUnicode_Substring(initial_data, PRINT_OFFSET, text_len);
-    Py_DECREF(initial_data);
+
+    PyObject *data = PyUnicode_Substring(self->text, start_pos, end_pos);
     if (data == NULL) {
-        Py_DECREF(strip_sep_obj);
         return -1;
     }
+
+    PyObject *strip_sep_obj = PyUnicode_FromString(" \t\r\n");
+    if (strip_sep_obj == NULL) {
+        Py_DECREF(data);
+        return -1;
+    }
+
     PyObject *new_data = _PyUnicode_XStrip(data, STRIP_BOTH, strip_sep_obj);
     Py_DECREF(data);
     Py_DECREF(strip_sep_obj);


From webhook-mailer at python.org  Sun Jan 28 06:31:51 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Sun, 28 Jan 2018 11:31:51 -0000
Subject: [Python-checkins] bpo-32685: Improve suggestion for print statement
 (GH-5380)
Message-ID: <mailman.8.1517139112.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/b3b4b81d0147534151941105eba4af984acdc763
commit: b3b4b81d0147534151941105eba4af984acdc763
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Nick Coghlan <ncoghlan at gmail.com>
date: 2018-01-28T21:31:48+10:00
summary:

bpo-32685: Improve suggestion for print statement (GH-5380)

Better account for single-line compound statements and
semi-colon separated statements when suggesting
Py3 replacements for Py2 print statements.

Initial patch by Nitish Chandra.
(cherry picked from commit 43c0f1ac5ed8bc9c3bd048d2ce4de4c98a83de99)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst
M Lib/test/test_print.py
M Objects/exceptions.c

diff --git a/Lib/test/test_print.py b/Lib/test/test_print.py
index 7bc23cf9fc63..e8381122182e 100644
--- a/Lib/test/test_print.py
+++ b/Lib/test/test_print.py
@@ -165,6 +165,23 @@ def test_string_with_leading_whitespace(self):
 
         self.assertIn('print("Hello World")', str(context.exception))
 
+    # bpo-32685: Suggestions for print statement should be proper when
+    # it is in the same line as the header of a compound statement
+    # and/or followed by a semicolon
+    def test_string_with_semicolon(self):
+        python2_print_str = 'print p;'
+        with self.assertRaises(SyntaxError) as context:
+            exec(python2_print_str)
+
+        self.assertIn('print(p)', str(context.exception))
+
+    def test_string_in_loop_on_same_line(self):
+        python2_print_str = 'for i in s: print i'
+        with self.assertRaises(SyntaxError) as context:
+            exec(python2_print_str)
+
+        self.assertIn('print(i)', str(context.exception))
+
     def test_stream_redirection_hint_for_py2_migration(self):
         # Test correct hint produced for Py2 redirection syntax
         with self.assertRaises(TypeError) as context:
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst
new file mode 100644
index 000000000000..07d7a072f6e1
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-12-25-06.bpo-32685.nGctze.rst	
@@ -0,0 +1,3 @@
+Improve suggestion when the Python 2 form of print statement is either
+present on the same line as the header of a compound statement or else
+terminated by a semi-colon instead of a newline. Patch by Nitish Chandra.
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
index 0a75f1b0fe64..3bec50ea69e4 100644
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -2859,26 +2859,31 @@ _PyErr_TrySetFromCause(const char *format, ...)
 static int
 _set_legacy_print_statement_msg(PySyntaxErrorObject *self, Py_ssize_t start)
 {
-    PyObject *strip_sep_obj = PyUnicode_FromString(" \t\r\n");
-    if (strip_sep_obj == NULL)
-        return -1;
-
-    // PRINT_OFFSET is to remove `print ` word from the data.
+    // PRINT_OFFSET is to remove the `print ` prefix from the data.
     const int PRINT_OFFSET = 6;
     const int STRIP_BOTH = 2;
-    // Issue 32028: Handle case when whitespace is used with print call
-    PyObject *initial_data = _PyUnicode_XStrip(self->text, STRIP_BOTH, strip_sep_obj);
-    if (initial_data == NULL) {
-        Py_DECREF(strip_sep_obj);
-        return -1;
+    Py_ssize_t start_pos = start + PRINT_OFFSET;
+    Py_ssize_t text_len = PyUnicode_GET_LENGTH(self->text);
+    Py_UCS4 semicolon = ';';
+    Py_ssize_t end_pos = PyUnicode_FindChar(self->text, semicolon,
+                                            start_pos, text_len, 1);
+    if (end_pos < -1) {
+      return -1;
+    } else if (end_pos == -1) {
+      end_pos = text_len;
     }
-    Py_ssize_t text_len = PyUnicode_GET_LENGTH(initial_data);
-    PyObject *data = PyUnicode_Substring(initial_data, PRINT_OFFSET, text_len);
-    Py_DECREF(initial_data);
+
+    PyObject *data = PyUnicode_Substring(self->text, start_pos, end_pos);
     if (data == NULL) {
-        Py_DECREF(strip_sep_obj);
         return -1;
     }
+
+    PyObject *strip_sep_obj = PyUnicode_FromString(" \t\r\n");
+    if (strip_sep_obj == NULL) {
+        Py_DECREF(data);
+        return -1;
+    }
+
     PyObject *new_data = _PyUnicode_XStrip(data, STRIP_BOTH, strip_sep_obj);
     Py_DECREF(data);
     Py_DECREF(strip_sep_obj);


From webhook-mailer at python.org  Sun Jan 28 08:00:12 2018
From: webhook-mailer at python.org (jcea)
Date: Sun, 28 Jan 2018 13:00:12 -0000
Subject: [Python-checkins] bpo-32659: Solaris "stat" should support
 "st_fstype" (#5307)
Message-ID: <mailman.9.1517144414.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/6c51d518800cdda7ba16ae163be0d211d2c4fa12
commit: 6c51d518800cdda7ba16ae163be0d211d2c4fa12
branch: master
author: jcea <jcea at jcea.es>
committer: GitHub <noreply at github.com>
date: 2018-01-28T14:00:08+01:00
summary:

bpo-32659: Solaris "stat" should support "st_fstype" (#5307)

* bpo-32659: Solaris "stat" should support "st_fstype"

* Add 'versionadded'

files:
A Misc/NEWS.d/next/Library/2018-01-25-03-46-00.bpo-32659.VHYoON.rst
M Doc/library/os.rst
M Modules/posixmodule.c

diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index b8f57f55eb41..bae432d33b0b 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -2474,6 +2474,14 @@ features:
 
       Time of file creation.
 
+   On Solaris and derivatives, the following attributes may also be
+   available:
+
+   .. attribute:: st_fstype
+
+      String that uniquely identifies the type of the filesystem that
+      contains the file.
+
    On Mac OS systems, the following attributes may also be available:
 
    .. attribute:: st_rsize
@@ -2517,6 +2525,8 @@ features:
    .. versionadded:: 3.5
       Added the :attr:`st_file_attributes` member on Windows.
 
+   .. versionadded:: 3.7
+      Added the :attr:`st_fstype` member to Solaris/derivatives.
 
 .. function:: statvfs(path)
 
diff --git a/Misc/NEWS.d/next/Library/2018-01-25-03-46-00.bpo-32659.VHYoON.rst b/Misc/NEWS.d/next/Library/2018-01-25-03-46-00.bpo-32659.VHYoON.rst
new file mode 100644
index 000000000000..f29f08786f55
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-25-03-46-00.bpo-32659.VHYoON.rst
@@ -0,0 +1 @@
+Under Solaris and derivatives, :class:`os.stat_result` provides a st_fstype attribute.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index a1a38ebda387..ceea85559c85 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -394,6 +394,10 @@ static int win32_can_symlink = 0;
 #define MODNAME "posix"
 #endif
 
+#if defined(__sun)
+/* Something to implement in autoconf, not present in autoconf 2.69 */
+#define HAVE_STRUCT_STAT_ST_FSTYPE 1
+#endif
 
 #ifdef HAVE_FORK
 static void
@@ -1788,6 +1792,9 @@ static PyStructSequence_Field stat_result_fields[] = {
 #endif
 #ifdef HAVE_STRUCT_STAT_ST_FILE_ATTRIBUTES
     {"st_file_attributes", "Windows file attribute bits"},
+#endif
+#ifdef HAVE_STRUCT_STAT_ST_FSTYPE
+    {"st_fstype",  "Type of filesystem"},
 #endif
     {0}
 };
@@ -1834,6 +1841,12 @@ static PyStructSequence_Field stat_result_fields[] = {
 #define ST_FILE_ATTRIBUTES_IDX ST_BIRTHTIME_IDX
 #endif
 
+#ifdef HAVE_STRUCT_STAT_ST_FSTYPE
+#define ST_FSTYPE_IDX (ST_FILE_ATTRIBUTES_IDX+1)
+#else
+#define ST_FSTYPE_IDX ST_FILE_ATTRIBUTES_IDX
+#endif
+
 static PyStructSequence_Desc stat_result_desc = {
     "stat_result", /* name */
     stat_result__doc__, /* doc */
@@ -2057,6 +2070,10 @@ _pystat_fromstructstat(STRUCT_STAT *st)
     PyStructSequence_SET_ITEM(v, ST_FILE_ATTRIBUTES_IDX,
                               PyLong_FromUnsignedLong(st->st_file_attributes));
 #endif
+#ifdef HAVE_STRUCT_STAT_ST_FSTYPE
+   PyStructSequence_SET_ITEM(v, ST_FSTYPE_IDX,
+                              PyUnicode_FromString(st->st_fstype));
+#endif
 
     if (PyErr_Occurred()) {
         Py_DECREF(v);


From webhook-mailer at python.org  Sun Jan 28 08:00:51 2018
From: webhook-mailer at python.org (jcea)
Date: Sun, 28 Jan 2018 13:00:51 -0000
Subject: [Python-checkins] bpo-32660: Solaris should support constants like
 termios' FIONREAD (#5328)
Message-ID: <mailman.10.1517144453.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/f0a95f27c043e847a23940534fdfc53e1b3e31a1
commit: f0a95f27c043e847a23940534fdfc53e1b3e31a1
branch: master
author: jcea <jcea at jcea.es>
committer: GitHub <noreply at github.com>
date: 2018-01-28T14:00:48+01:00
summary:

bpo-32660: Solaris should support constants like termios' FIONREAD (#5328)

files:
A Misc/NEWS.d/next/Library/2018-01-26-01-26-00.bpo-32660.tVJIWV.rst
M Modules/termios.c

diff --git a/Misc/NEWS.d/next/Library/2018-01-26-01-26-00.bpo-32660.tVJIWV.rst b/Misc/NEWS.d/next/Library/2018-01-26-01-26-00.bpo-32660.tVJIWV.rst
new file mode 100644
index 000000000000..5b33e1a19f96
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-26-01-26-00.bpo-32660.tVJIWV.rst
@@ -0,0 +1,2 @@
+:mod:`termios` makes available ``FIONREAD``, ``FIONCLEX``, ``FIOCLEX``,
+``FIOASYNC`` and ``FIONBIO`` also under Solaris/derivatives.
diff --git a/Modules/termios.c b/Modules/termios.c
index 6ed4395ae4dd..b4aa77f89764 100644
--- a/Modules/termios.c
+++ b/Modules/termios.c
@@ -8,6 +8,12 @@
 #define CTRL(c) ((c)&037)
 #endif
 
+#if defined(__sun)
+/* We could do better. Check issue-32660 */
+#include <sys/filio.h>
+#include <sys/sockio.h>
+#endif
+
 #include <termios.h>
 #include <sys/ioctl.h>
 


From webhook-mailer at python.org  Sun Jan 28 09:25:47 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Sun, 28 Jan 2018 14:25:47 -0000
Subject: [Python-checkins] Add example for PEP 557. (GH-5383)
Message-ID: <mailman.11.1517149549.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/4d0296649a48ff9dbc290394ff656cf3dea86107
commit: 4d0296649a48ff9dbc290394ff656cf3dea86107
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T09:25:45-05:00
summary:

Add example for PEP 557. (GH-5383)

files:
M Doc/whatsnew/3.7.rst

diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 5e355345e31d..9979e6923990 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -323,6 +323,17 @@ Adds a new module ``dataclasses``.  It provides a class decorator
 ``typing.NamedTuple``, but also works on classes with mutable
 instances, among other features.
 
+For example::
+
+    @dataclass
+    class Point:
+        x: float
+        y: float
+        z: float = 0.0
+
+    p = Point(1.5, 2.5)
+    print(p)   # produces "Point(x=1.5, y=2.5, z=0.0)"
+
 .. seealso::
 
     :pep:`557` -- Data Classes


From webhook-mailer at python.org  Sun Jan 28 09:54:44 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sun, 28 Jan 2018 14:54:44 -0000
Subject: [Python-checkins] Fix PyTrace_RETURN documentation (GH-5384)
Message-ID: <mailman.12.1517151285.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/79db11ce99332d62917be9d03b31494b1ff2f96a
commit: 79db11ce99332d62917be9d03b31494b1ff2f96a
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T22:54:42+08:00
summary:

Fix PyTrace_RETURN documentation (GH-5384)

It will be triggered when propagating an exception.

files:
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 33cb02d8c9cc..02189a9e3591 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1332,7 +1332,7 @@ Python-level trace functions in previous versions.
 .. c:var:: int PyTrace_RETURN
 
    The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a
-   call is returning without propagating an exception.
+   call is about to return.
 
 
 .. c:var:: int PyTrace_C_CALL


From webhook-mailer at python.org  Sun Jan 28 10:33:26 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sun, 28 Jan 2018 15:33:26 -0000
Subject: [Python-checkins] Fix PyTrace_RETURN documentation (GH-5384)
 (GH-5386)
Message-ID: <mailman.13.1517153607.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/33febfb03998598a35f97b4ef66d33a2d43716cd
commit: 33febfb03998598a35f97b4ef66d33a2d43716cd
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2018-01-28T23:33:23+08:00
summary:

Fix PyTrace_RETURN documentation (GH-5384) (GH-5386)

It will be triggered when propagating an exception.
(cherry picked from commit 79db11ce99332d62917be9d03b31494b1ff2f96a)

files:
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index bc92c3e16b89..c8d429de51bb 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1110,7 +1110,7 @@ Python-level trace functions in previous versions.
 .. c:var:: int PyTrace_RETURN
 
    The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a
-   call is returning without propagating an exception.
+   call is about to return.
 
 
 .. c:var:: int PyTrace_C_CALL


From webhook-mailer at python.org  Sun Jan 28 10:49:16 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Sun, 28 Jan 2018 15:49:16 -0000
Subject: [Python-checkins] Fix PyTrace_RETURN documentation (GH-5384)
 (GH-5387)
Message-ID: <mailman.14.1517154558.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/5679277fdde2d001ffa439559e2078590332baf2
commit: 5679277fdde2d001ffa439559e2078590332baf2
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2018-01-28T23:49:14+08:00
summary:

Fix PyTrace_RETURN documentation (GH-5384) (GH-5387)

It will be triggered when propagating an exception.
(cherry picked from commit 79db11ce99332d62917be9d03b31494b1ff2f96a)

files:
M Doc/c-api/init.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index f091ac81dae9..229cae5244ab 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1030,7 +1030,7 @@ in previous versions.
 .. c:var:: int PyTrace_RETURN
 
    The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a
-   call is returning without propagating an exception.
+   call is about to return.
 
 
 .. c:var:: int PyTrace_C_CALL


From webhook-mailer at python.org  Sun Jan 28 11:00:12 2018
From: webhook-mailer at python.org (Antoine Pitrou)
Date: Sun, 28 Jan 2018 16:00:12 -0000
Subject: [Python-checkins] bpo-32228: Reset raw_pos after unwinding the raw
 stream (#4858)
Message-ID: <mailman.15.1517155214.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/059f58ce938d9c3f0286412a4efb1b9131339421
commit: 059f58ce938d9c3f0286412a4efb1b9131339421
branch: master
author: Nitish Chandra <nitishchandrachinta at gmail.com>
committer: Antoine Pitrou <pitrou at free.fr>
date: 2018-01-28T17:00:09+01:00
summary:

bpo-32228: Reset raw_pos after unwinding the raw stream (#4858)

Ensure that ``truncate()`` preserves the file position (as reported by ``tell()``) after writes longer than the buffer size.

files:
A Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst
M Lib/test/test_io.py
M Modules/_io/bufferedio.c

diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
index c6345e9199ca..4be5631ab446 100644
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -1723,6 +1723,23 @@ def test_truncate(self):
         with self.open(support.TESTFN, "rb", buffering=0) as f:
             self.assertEqual(f.read(), b"abc")
 
+    def test_truncate_after_write(self):
+        # Ensure that truncate preserves the file position after
+        # writes longer than the buffer size.
+        # Issue: https://bugs.python.org/issue32228
+        with self.open(support.TESTFN, "wb") as f:
+            # Fill with some buffer
+            f.write(b'\x00' * 10000)
+        buffer_sizes = [8192, 4096, 200]
+        for buffer_size in buffer_sizes:
+            with self.open(support.TESTFN, "r+b", buffering=buffer_size) as f:
+                f.write(b'\x00' * (buffer_size + 1))
+                # After write write_pos and write_end are set to 0
+                f.read(1)
+                # read operation makes sure that pos != raw_pos
+                f.truncate()
+                self.assertEqual(f.tell(), buffer_size + 2)
+
     @support.requires_resource('cpu')
     def test_threads(self):
         try:
diff --git a/Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst b/Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst
new file mode 100644
index 000000000000..3bbe7c495f82
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst
@@ -0,0 +1 @@
+Ensure that ``truncate()`` preserves the file position (as reported by ``tell()``) after writes longer than the buffer size.
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
index 2b7aaaf7c7d0..358a654a98ca 100644
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -1292,7 +1292,6 @@ _io__Buffered_seek_impl(buffered *self, PyObject *targetobj, int whence)
         if (res == NULL)
             goto end;
         Py_CLEAR(res);
-        _bufferedwriter_reset_buf(self);
     }
 
     /* TODO: align on block boundary and read buffer if needed? */
@@ -1852,8 +1851,6 @@ _bufferedwriter_raw_write(buffered *self, char *start, Py_ssize_t len)
     return n;
 }
 
-/* `restore_pos` is 1 if we need to restore the raw stream position at
-   the end, 0 otherwise. */
 static PyObject *
 _bufferedwriter_flush_unlocked(buffered *self)
 {
@@ -1894,9 +1891,18 @@ _bufferedwriter_flush_unlocked(buffered *self)
             goto error;
     }
 
-    _bufferedwriter_reset_buf(self);
 
 end:
+    /* This ensures that after return from this function,
+       VALID_WRITE_BUFFER(self) returns false.
+
+       This is a required condition because when a tell() is called
+       after flushing and if VALID_READ_BUFFER(self) is false, we need
+       VALID_WRITE_BUFFER(self) to be false to have
+       RAW_OFFSET(self) == 0.
+
+       Issue: https://bugs.python.org/issue32228 */
+    _bufferedwriter_reset_buf(self);
     Py_RETURN_NONE;
 
 error:


From webhook-mailer at python.org  Sun Jan 28 12:40:27 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 28 Jan 2018 17:40:27 -0000
Subject: [Python-checkins] bpo-32690: Preserve order of locals() (#5379)
Message-ID: <mailman.16.1517161229.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/a4d00012565d716db6e6abe1b8f33eaaa4de416e
commit: a4d00012565d716db6e6abe1b8f33eaaa4de416e
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T09:40:24-08:00
summary:

bpo-32690: Preserve order of locals() (#5379)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst
M Objects/frameobject.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst
new file mode 100644
index 000000000000..1663b96cb927
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst	
@@ -0,0 +1,2 @@
+The locals() dictionary now displays in the lexical order that variables
+were defined.  Previously, the order was reversed.
diff --git a/Objects/frameobject.c b/Objects/frameobject.c
index 1ac3d7525752..3083e5b6445c 100644
--- a/Objects/frameobject.c
+++ b/Objects/frameobject.c
@@ -791,7 +791,7 @@ map_to_dict(PyObject *map, Py_ssize_t nmap, PyObject *dict, PyObject **values,
     assert(PyTuple_Check(map));
     assert(PyDict_Check(dict));
     assert(PyTuple_Size(map) >= nmap);
-    for (j = nmap; --j >= 0; ) {
+    for (j=0; j < nmap; j++) {
         PyObject *key = PyTuple_GET_ITEM(map, j);
         PyObject *value = values[j];
         assert(PyUnicode_Check(key));
@@ -844,7 +844,7 @@ dict_to_map(PyObject *map, Py_ssize_t nmap, PyObject *dict, PyObject **values,
     assert(PyTuple_Check(map));
     assert(PyDict_Check(dict));
     assert(PyTuple_Size(map) >= nmap);
-    for (j = nmap; --j >= 0; ) {
+    for (j=0; j < nmap; j++) {
         PyObject *key = PyTuple_GET_ITEM(map, j);
         PyObject *value = PyObject_GetItem(dict, key);
         assert(PyUnicode_Check(key));


From webhook-mailer at python.org  Sun Jan 28 12:42:34 2018
From: webhook-mailer at python.org (Antoine Pitrou)
Date: Sun, 28 Jan 2018 17:42:34 -0000
Subject: [Python-checkins] [3.6] bpo-32228: Reset raw_pos after unwinding
 the raw stream (GH-4858) (#5389)
Message-ID: <mailman.17.1517161356.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/1d896ed2cddada4455f40782e4120249defbfa70
commit: 1d896ed2cddada4455f40782e4120249defbfa70
branch: 3.6
author: Antoine Pitrou <pitrou at free.fr>
committer: GitHub <noreply at github.com>
date: 2018-01-28T18:42:31+01:00
summary:

[3.6] bpo-32228: Reset raw_pos after unwinding the raw stream (GH-4858) (#5389)

Ensure that ``truncate()`` preserves the file position (as reported by ``tell()``) after writes longer than the buffer size..
(cherry picked from commit 059f58ce938d9c3f0286412a4efb1b9131339421)

files:
A Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst
M Lib/test/test_io.py
M Modules/_io/bufferedio.c

diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
index 42dbbd35dc95..6c30a44e62a6 100644
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -1703,6 +1703,23 @@ def test_truncate(self):
         with self.open(support.TESTFN, "rb", buffering=0) as f:
             self.assertEqual(f.read(), b"abc")
 
+    def test_truncate_after_write(self):
+        # Ensure that truncate preserves the file position after
+        # writes longer than the buffer size.
+        # Issue: https://bugs.python.org/issue32228
+        with self.open(support.TESTFN, "wb") as f:
+            # Fill with some buffer
+            f.write(b'\x00' * 10000)
+        buffer_sizes = [8192, 4096, 200]
+        for buffer_size in buffer_sizes:
+            with self.open(support.TESTFN, "r+b", buffering=buffer_size) as f:
+                f.write(b'\x00' * (buffer_size + 1))
+                # After write write_pos and write_end are set to 0
+                f.read(1)
+                # read operation makes sure that pos != raw_pos
+                f.truncate()
+                self.assertEqual(f.tell(), buffer_size + 2)
+
     @unittest.skipUnless(threading, 'Threading required for this test.')
     @support.requires_resource('cpu')
     def test_threads(self):
diff --git a/Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst b/Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst
new file mode 100644
index 000000000000..3bbe7c495f82
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-22-16-47-41.bpo-32228.waPx3q.rst
@@ -0,0 +1 @@
+Ensure that ``truncate()`` preserves the file position (as reported by ``tell()``) after writes longer than the buffer size.
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
index efc7d05b7d01..32355813e88d 100644
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -1311,7 +1311,6 @@ _io__Buffered_seek_impl(buffered *self, PyObject *targetobj, int whence)
         if (res == NULL)
             goto end;
         Py_CLEAR(res);
-        _bufferedwriter_reset_buf(self);
     }
 
     /* TODO: align on block boundary and read buffer if needed? */
@@ -1878,8 +1877,6 @@ _bufferedwriter_raw_write(buffered *self, char *start, Py_ssize_t len)
     return n;
 }
 
-/* `restore_pos` is 1 if we need to restore the raw stream position at
-   the end, 0 otherwise. */
 static PyObject *
 _bufferedwriter_flush_unlocked(buffered *self)
 {
@@ -1920,9 +1917,18 @@ _bufferedwriter_flush_unlocked(buffered *self)
             goto error;
     }
 
-    _bufferedwriter_reset_buf(self);
 
 end:
+    /* This ensures that after return from this function,
+       VALID_WRITE_BUFFER(self) returns false.
+
+       This is a required condition because when a tell() is called
+       after flushing and if VALID_READ_BUFFER(self) is false, we need
+       VALID_WRITE_BUFFER(self) to be false to have
+       RAW_OFFSET(self) == 0.
+
+       Issue: https://bugs.python.org/issue32228 */
+    _bufferedwriter_reset_buf(self);
     Py_RETURN_NONE;
 
 error:


From webhook-mailer at python.org  Sun Jan 28 13:13:20 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 28 Jan 2018 18:13:20 -0000
Subject: [Python-checkins] bpo-32690: Preserve order of locals() (GH-5379)
 (#5390)
Message-ID: <mailman.18.1517163202.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/9105879bfd7133ecbac67f3e9c0bacf6e477de5a
commit: 9105879bfd7133ecbac67f3e9c0bacf6e477de5a
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-28T10:13:17-08:00
summary:

bpo-32690: Preserve order of locals() (GH-5379) (#5390)

(cherry picked from commit a4d00012565d716db6e6abe1b8f33eaaa4de416e)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst
M Objects/frameobject.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst
new file mode 100644
index 000000000000..1663b96cb927
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst	
@@ -0,0 +1,2 @@
+The locals() dictionary now displays in the lexical order that variables
+were defined.  Previously, the order was reversed.
diff --git a/Objects/frameobject.c b/Objects/frameobject.c
index 62f9f34c8ebf..ca1fbfa1a28f 100644
--- a/Objects/frameobject.c
+++ b/Objects/frameobject.c
@@ -776,7 +776,7 @@ map_to_dict(PyObject *map, Py_ssize_t nmap, PyObject *dict, PyObject **values,
     assert(PyTuple_Check(map));
     assert(PyDict_Check(dict));
     assert(PyTuple_Size(map) >= nmap);
-    for (j = nmap; --j >= 0; ) {
+    for (j=0; j < nmap; j++) {
         PyObject *key = PyTuple_GET_ITEM(map, j);
         PyObject *value = values[j];
         assert(PyUnicode_Check(key));
@@ -829,7 +829,7 @@ dict_to_map(PyObject *map, Py_ssize_t nmap, PyObject *dict, PyObject **values,
     assert(PyTuple_Check(map));
     assert(PyDict_Check(dict));
     assert(PyTuple_Size(map) >= nmap);
-    for (j = nmap; --j >= 0; ) {
+    for (j=0; j < nmap; j++) {
         PyObject *key = PyTuple_GET_ITEM(map, j);
         PyObject *value = PyObject_GetItem(dict, key);
         assert(PyUnicode_Check(key));


From webhook-mailer at python.org  Sun Jan 28 14:09:42 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sun, 28 Jan 2018 19:09:42 -0000
Subject: [Python-checkins] bpo-32327: Revert loop.run_in_executor behaviour:
 return a Future. (#5392)
Message-ID: <mailman.19.1517166584.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/bec2372b7e1da5dfdbadaf242aa8e994b164cace
commit: bec2372b7e1da5dfdbadaf242aa8e994b164cace
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-28T14:09:40-05:00
summary:

bpo-32327: Revert loop.run_in_executor behaviour: return a Future. (#5392)

I've run some tests on 3.7 asyncio and it appears that too many
things assume that run_in_executor returns a Future.

files:
M Doc/library/asyncio-eventloop.rst
M Lib/asyncio/base_events.py
M Lib/test/test_asyncio/test_tasks.py
M Misc/NEWS.d/3.7.0a4.rst

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 78ae18391f73..5f915c5c4392 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -782,6 +782,12 @@ Resolve host name
    This method is a :ref:`coroutine <coroutine>`, similar to
    :meth:`socket.getnameinfo` function but non-blocking.
 
+.. versionchanged:: 3.7
+   Both *getaddrinfo* and *getnameinfo* methods were always documented
+   to return a coroutine, but prior to Python 3.7 they were, in fact,
+   returning :class:`asyncio.Future` objects.  Starting with Python 3.7
+   both methods are coroutines.
+
 
 Connect pipes
 -------------
@@ -852,7 +858,7 @@ Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or
 pool of processes). By default, an event loop uses a thread pool executor
 (:class:`~concurrent.futures.ThreadPoolExecutor`).
 
-.. coroutinemethod:: AbstractEventLoop.run_in_executor(executor, func, \*args)
+.. method:: AbstractEventLoop.run_in_executor(executor, func, \*args)
 
    Arrange for a *func* to be called in the specified executor.
 
@@ -862,6 +868,8 @@ pool of processes). By default, an event loop uses a thread pool executor
    :ref:`Use functools.partial to pass keywords to the *func*
    <asyncio-pass-keywords>`.
 
+   This method returns a :class:`asyncio.Future` object.
+
    .. versionchanged:: 3.5.3
       :meth:`BaseEventLoop.run_in_executor` no longer configures the
       ``max_workers`` of the thread pool executor it creates, instead
@@ -869,11 +877,6 @@ pool of processes). By default, an event loop uses a thread pool executor
       (:class:`~concurrent.futures.ThreadPoolExecutor`) to set the
       default.
 
-   .. versionchanged:: 3.7
-      Even though the method was always documented as a coroutine
-      method, before Python 3.7 it returned a :class:`Future`.
-      Since Python 3.7, this is an ``async def`` method.
-
 .. method:: AbstractEventLoop.set_default_executor(executor)
 
    Set the default executor used by :meth:`run_in_executor`.
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index 7442bf28b035..09eb440b0ef7 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -721,7 +721,7 @@ def call_soon_threadsafe(self, callback, *args, context=None):
         self._write_to_self()
         return handle
 
-    async def run_in_executor(self, executor, func, *args):
+    def run_in_executor(self, executor, func, *args):
         self._check_closed()
         if self._debug:
             self._check_callback(func, 'run_in_executor')
@@ -730,7 +730,7 @@ def call_soon_threadsafe(self, callback, *args, context=None):
             if executor is None:
                 executor = concurrent.futures.ThreadPoolExecutor()
                 self._default_executor = executor
-        return await futures.wrap_future(
+        return futures.wrap_future(
             executor.submit(func, *args), loop=self)
 
     def set_default_executor(self, executor):
diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index 822338f932e7..19cf1e667fc4 100644
--- a/Lib/test/test_asyncio/test_tasks.py
+++ b/Lib/test/test_asyncio/test_tasks.py
@@ -2999,9 +2999,8 @@ def test_run_coroutine_threadsafe_task_factory_exception(self):
         def task_factory(loop, coro):
             raise NameError
 
-        run = self.loop.create_task(
-            self.loop.run_in_executor(
-                None, lambda: self.target(advance_coro=True)))
+        run = self.loop.run_in_executor(
+            None, lambda: self.target(advance_coro=True))
 
         # Set exception handler
         callback = test_utils.MockCallback()
diff --git a/Misc/NEWS.d/3.7.0a4.rst b/Misc/NEWS.d/3.7.0a4.rst
index a6322ed405ba..5c3da97fa9d9 100644
--- a/Misc/NEWS.d/3.7.0a4.rst
+++ b/Misc/NEWS.d/3.7.0a4.rst
@@ -426,7 +426,7 @@ Implement asyncio.create_task(coro) shortcut
 
 Convert asyncio functions that were documented as coroutines to coroutines.
 Affected functions: loop.sock_sendall, loop.sock_recv, loop.sock_accept,
-loop.run_in_executor, loop.getaddrinfo, loop.getnameinfo.
+loop.getaddrinfo, loop.getnameinfo.
 
 ..
 


From webhook-mailer at python.org  Sun Jan 28 14:13:12 2018
From: webhook-mailer at python.org (larryhastings)
Date: Sun, 28 Jan 2018 19:13:12 -0000
Subject: [Python-checkins] bpo-32697: Definition order of kwonly params is
 now guaranteed preserved. (#5391)
Message-ID: <mailman.20.1517166794.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/f36ba12809d5db1b76464d8f1f04dad8d685ec78
commit: f36ba12809d5db1b76464d8f1f04dad8d685ec78
branch: master
author: larryhastings <larry at hastings.org>
committer: GitHub <noreply at github.com>
date: 2018-01-28T11:13:09-08:00
summary:

bpo-32697: Definition order of kwonly params is now guaranteed preserved. (#5391)

Definition order of kwonly params is now guaranteed preserved.

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-52-12.bpo-32697.RHlu6k.rst
M Doc/library/inspect.rst
M Lib/test/test_inspect.py

diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
index 147e802cac4b..bc4316fabaee 100644
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -602,7 +602,13 @@ function.
    .. attribute:: Signature.parameters
 
       An ordered mapping of parameters' names to the corresponding
-      :class:`Parameter` objects.
+      :class:`Parameter` objects.  Parameters appear in strict definition
+      order, including keyword-only parameters.
+
+      .. versionchanged:: 3.7
+         Python only explicitly guaranteed that it preserved the declaration
+         order of keyword-only parameters as of version 3.7, although in practice
+         this order had always been preserved in Python 3.
 
    .. attribute:: Signature.return_annotation
 
@@ -895,7 +901,7 @@ Classes and functions
    *defaults* is an *n*-tuple of default argument values corresponding to the
    last *n* positional parameters, or ``None`` if there are no such defaults
    defined.
-   *kwonlyargs* is a list of keyword-only parameter names.
+   *kwonlyargs* is a list of keyword-only parameter names in declaration order.
    *kwonlydefaults* is a dictionary mapping parameter names from *kwonlyargs*
    to the default values used if no argument is supplied.
    *annotations* is a dictionary mapping parameter names to annotations.
@@ -921,6 +927,11 @@ Classes and functions
       single-source Python 2/3 code migrating away from the legacy
       :func:`getargspec` API.
 
+   .. versionchanged:: 3.7
+      Python only explicitly guaranteed that it preserved the declaration
+      order of keyword-only parameters as of version 3.7, although in practice
+      this order had always been preserved in Python 3.
+
 
 .. function:: getargvalues(frame)
 
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
index cb51f8aff290..1a856f6387e8 100644
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -57,6 +57,31 @@ def revise(filename, *args):
 
 git = mod.StupidGit()
 
+
+def signatures_with_lexicographic_keyword_only_parameters():
+    """
+    Yields a whole bunch of functions with only keyword-only parameters,
+    where those parameters are always in lexicographically sorted order.
+    """
+    parameters = ['a', 'bar', 'c', 'delta', 'ephraim', 'magical', 'yoyo', 'z']
+    for i in range(1, 2**len(parameters)):
+        p = []
+        bit = 1
+        for j in range(len(parameters)):
+            if i & (bit << j):
+                p.append(parameters[j])
+        fn_text = "def foo(*, " + ", ".join(p) + "): pass"
+        symbols = {}
+        exec(fn_text, symbols, symbols)
+        yield symbols['foo']
+
+
+def unsorted_keyword_only_parameters_fn(*, throw, out, the, baby, with_,
+                                        the_, bathwater):
+    pass
+
+unsorted_keyword_only_parameters = 'throw out the baby with_ the_ bathwater'.split()
+
 class IsTestBase(unittest.TestCase):
     predicates = set([inspect.isbuiltin, inspect.isclass, inspect.iscode,
                       inspect.isframe, inspect.isfunction, inspect.ismethod,
@@ -829,6 +854,17 @@ def test_getfullagrspec_builtin_func_no_signature(self):
         with self.assertRaises(TypeError):
             inspect.getfullargspec(builtin)
 
+    def test_getfullargspec_definition_order_preserved_on_kwonly(self):
+        for fn in signatures_with_lexicographic_keyword_only_parameters():
+            signature = inspect.getfullargspec(fn)
+            l = list(signature.kwonlyargs)
+            sorted_l = sorted(l)
+            self.assertTrue(l)
+            self.assertEqual(l, sorted_l)
+        signature = inspect.getfullargspec(unsorted_keyword_only_parameters_fn)
+        l = list(signature.kwonlyargs)
+        self.assertEqual(l, unsorted_keyword_only_parameters)
+
     def test_getargspec_method(self):
         class A(object):
             def m(self):
@@ -2969,6 +3005,17 @@ class MySignature(inspect.Signature): pass
         sig = MySignature.from_callable(_pickle.Pickler)
         self.assertTrue(isinstance(sig, MySignature))
 
+    def test_signature_definition_order_preserved_on_kwonly(self):
+        for fn in signatures_with_lexicographic_keyword_only_parameters():
+            signature = inspect.signature(fn)
+            l = list(signature.parameters)
+            sorted_l = sorted(l)
+            self.assertTrue(l)
+            self.assertEqual(l, sorted_l)
+        signature = inspect.signature(unsorted_keyword_only_parameters_fn)
+        l = list(signature.parameters)
+        self.assertEqual(l, unsorted_keyword_only_parameters)
+
 
 class TestParameterObject(unittest.TestCase):
     def test_signature_parameter_kinds(self):
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-52-12.bpo-32697.RHlu6k.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-52-12.bpo-32697.RHlu6k.rst
new file mode 100644
index 000000000000..97bc310683c3
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-52-12.bpo-32697.RHlu6k.rst	
@@ -0,0 +1,3 @@
+Python now explicitly preserves the definition order of keyword-only
+parameters.  It's always preserved their order, but this behavior was never
+guaranteed before; this behavior is now guaranteed and tested.


From webhook-mailer at python.org  Sun Jan 28 16:08:35 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sun, 28 Jan 2018 21:08:35 -0000
Subject: [Python-checkins] Revert "bpo-32221: makeipaddr(): remove interface
 part + speedup (GH-4724)" (#5394)
Message-ID: <mailman.21.1517173717.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/0ceb717689b04c0540d78c1ba93c0572c66c0994
commit: 0ceb717689b04c0540d78c1ba93c0572c66c0994
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-28T16:08:32-05:00
summary:

Revert "bpo-32221: makeipaddr(): remove interface part + speedup (GH-4724)" (#5394)

This reverts commit 47c0b1f7d4115e6f15e6776c1f91d28e7d96fe0c.

files:
D Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst
M Doc/library/socket.rst
M Lib/test/test_socket.py
M Modules/socketmodule.c

diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index 777710f16f26..7edd4ba36043 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -77,11 +77,6 @@ created.  Socket addresses are represented as follows:
   backward compatibility.  Note, however, omission of *scopeid* can cause problems
   in manipulating scoped IPv6 addresses.
 
-  .. versionchanged:: 3.7
-     For multicast addresses (with *scopeid* meaningful) *address* may not contain
-     ``%scope`` (or ``zone id``) part. This information is superfluous and may
-     be safely omitted (recommended).
-
 - :const:`AF_NETLINK` sockets are represented as pairs ``(pid, groups)``.
 
 - Linux-only support for TIPC is available using the :const:`AF_TIPC`
@@ -635,10 +630,6 @@ The :mod:`socket` module also offers various network-related services:
    .. versionchanged:: 3.2
       parameters can now be passed using keyword arguments.
 
-   .. versionchanged:: 3.7
-      for IPv6 multicast addresses, string representing an address will not
-      contain ``%scope`` part.
-
 .. function:: getfqdn([name])
 
    Return a fully qualified domain name for *name*. If *name* is omitted or empty,
@@ -697,8 +688,6 @@ The :mod:`socket` module also offers various network-related services:
    or numeric address representation in *host*.  Similarly, *port* can contain a
    string port name or a numeric port number.
 
-   For IPv6 addresses, ``%scope`` is appended to the host part if *sockaddr*
-   contains meaningful *scopeid*. Usually this happens for multicast addresses.
 
 .. function:: getprotobyname(protocolname)
 
@@ -1189,10 +1178,6 @@ to sockets.
       an exception, the method now retries the system call instead of raising
       an :exc:`InterruptedError` exception (see :pep:`475` for the rationale).
 
-   .. versionchanged:: 3.7
-      For multicast IPv6 address, first item of *address* does not contain
-      ``%scope`` part anymore. In order to get full IPv6 address use
-      :func:`getnameinfo`.
 
 .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]])
 
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index a971f921546f..275384c28fb3 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1586,50 +1586,6 @@ def test_flowinfo(self):
         with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as s:
             self.assertRaises(OverflowError, s.bind, (support.HOSTv6, 0, -10))
 
-    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
-    def test_getaddrinfo_ipv6_basic(self):
-        ((*_, sockaddr),) = socket.getaddrinfo(
-            'ff02::1de:c0:face:8D',  # Note capital letter `D`.
-            1234, socket.AF_INET6,
-            socket.SOCK_DGRAM,
-            socket.IPPROTO_UDP
-        )
-        self.assertEqual(sockaddr, ('ff02::1de:c0:face:8d', 1234, 0, 0))
-
-    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
-    @unittest.skipUnless(
-        hasattr(socket, 'if_nameindex'),
-        'IPv6 scope id by interface name is not supported'
-    )
-    def test_getaddrinfo_ipv6_scopeid(self):
-        test_interface = 'lo'
-        ifindex = socket.if_nametoindex(test_interface)
-        ((*_, sockaddr),) = socket.getaddrinfo(
-            'ff02::1de:c0:face:8D%' + test_interface,
-            1234, socket.AF_INET6,
-            socket.SOCK_DGRAM,
-            socket.IPPROTO_UDP
-        )
-        # Note missing interface name part in IPv6 address
-        self.assertEqual(sockaddr, ('ff02::1de:c0:face:8d', 1234, 0, ifindex))
-
-    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
-    def test_getaddrinfo_ipv6_scopeid_numeric(self):
-        ((*_, sockaddr),) = socket.getaddrinfo(
-            'ff02::1de:c0:face:8D%42',
-            1234, socket.AF_INET6,
-            socket.SOCK_DGRAM,
-            socket.IPPROTO_UDP
-        )
-        # Note missing interface name part in IPv6 address
-        self.assertEqual(sockaddr, ('ff02::1de:c0:face:8d', 1234, 0, 42))
-
-    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.')
-    def test_getnameinfo_ipv6_scopeid(self):
-        sockaddr = ('ff02::1de:c0:face:8D', 1234, 0, 100500)  # Note capital letter `D`.
-        nameinfo = socket.getnameinfo(sockaddr, socket.NI_NUMERICHOST | socket.NI_NUMERICSERV)
-        self.assertEqual(nameinfo, ('ff02::1de:c0:face:8d%100500', '1234'))
-
     def test_str_for_enums(self):
         # Make sure that the AF_* and SOCK_* constants have enum-like string
         # reprs.
diff --git a/Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst b/Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst
deleted file mode 100644
index a88dcf48e02b..000000000000
--- a/Misc/NEWS.d/next/Library/2017-12-06-10-10-10.bpo-32221.ideco_.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Various functions returning tuple containig IPv6 addresses now omit ``%scope``
-part since the same information is already encoded in *scopeid* tuple item.
-Especially this speeds up :func:`socket.recvfrom` when it receives multicast
-packet since useless resolving of network interface name is omitted.
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index 30afe81d6352..5fe2431bee84 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -1094,33 +1094,25 @@ setipaddr(const char *name, struct sockaddr *addr_ret, size_t addr_ret_size, int
 }
 
 
-/* Convert IPv4 sockaddr to a Python str. */
+/* Create a string object representing an IP address.
+   This is always a string of the form 'dd.dd.dd.dd' (with variable
+   size numbers). */
 
 static PyObject *
-make_ipv4_addr(const struct sockaddr_in *addr)
+makeipaddr(struct sockaddr *addr, int addrlen)
 {
-    char buf[INET_ADDRSTRLEN];
-    if (inet_ntop(AF_INET, &addr->sin_addr, buf, sizeof(buf)) == NULL) {
-        PyErr_SetFromErrno(PyExc_OSError);
-        return NULL;
-    }
-    return PyUnicode_FromString(buf);
-}
-
-#ifdef ENABLE_IPV6
-/* Convert IPv6 sockaddr to a Python str. */
+    char buf[NI_MAXHOST];
+    int error;
 
-static PyObject *
-make_ipv6_addr(const struct sockaddr_in6 *addr)
-{
-    char buf[INET6_ADDRSTRLEN];
-    if (inet_ntop(AF_INET6, &addr->sin6_addr, buf, sizeof(buf)) == NULL) {
-        PyErr_SetFromErrno(PyExc_OSError);
+    error = getnameinfo(addr, addrlen, buf, sizeof(buf), NULL, 0,
+        NI_NUMERICHOST);
+    if (error) {
+        set_gaierror(error);
         return NULL;
     }
     return PyUnicode_FromString(buf);
 }
-#endif
+
 
 #ifdef USE_BLUETOOTH
 /* Convert a string representation of a Bluetooth address into a numeric
@@ -1185,10 +1177,11 @@ makesockaddr(SOCKET_T sockfd, struct sockaddr *addr, size_t addrlen, int proto)
 
     case AF_INET:
     {
-        const struct sockaddr_in *a = (const struct sockaddr_in *)addr;
-        PyObject *addrobj = make_ipv4_addr(a);
+        struct sockaddr_in *a;
+        PyObject *addrobj = makeipaddr(addr, sizeof(*a));
         PyObject *ret = NULL;
         if (addrobj) {
+            a = (struct sockaddr_in *)addr;
             ret = Py_BuildValue("Oi", addrobj, ntohs(a->sin_port));
             Py_DECREF(addrobj);
         }
@@ -1232,10 +1225,11 @@ makesockaddr(SOCKET_T sockfd, struct sockaddr *addr, size_t addrlen, int proto)
 #ifdef ENABLE_IPV6
     case AF_INET6:
     {
-        const struct sockaddr_in6 *a = (const struct sockaddr_in6 *)addr;
-        PyObject *addrobj = make_ipv6_addr(a);
+        struct sockaddr_in6 *a;
+        PyObject *addrobj = makeipaddr(addr, sizeof(*a));
         PyObject *ret = NULL;
         if (addrobj) {
+            a = (struct sockaddr_in6 *)addr;
             ret = Py_BuildValue("OiII",
                                 addrobj,
                                 ntohs(a->sin6_port),
@@ -5042,14 +5036,14 @@ static PyObject *
 socket_gethostbyname(PyObject *self, PyObject *args)
 {
     char *name;
-    struct sockaddr_in addrbuf;
+    sock_addr_t addrbuf;
     PyObject *ret = NULL;
 
     if (!PyArg_ParseTuple(args, "et:gethostbyname", "idna", &name))
         return NULL;
-    if (setipaddr(name, (struct sockaddr *)&addrbuf,  sizeof(addrbuf), AF_INET) < 0)
+    if (setipaddr(name, SAS2SA(&addrbuf),  sizeof(addrbuf), AF_INET) < 0)
         goto finally;
-    ret = make_ipv4_addr(&addrbuf);
+    ret = makeipaddr(SAS2SA(&addrbuf), sizeof(struct sockaddr_in));
 finally:
     PyMem_Free(name);
     return ret;
@@ -5151,7 +5145,7 @@ gethost_common(struct hostent *h, struct sockaddr *addr, size_t alen, int af)
             sin.sin_len = sizeof(sin);
 #endif
             memcpy(&sin.sin_addr, *pch, sizeof(sin.sin_addr));
-            tmp = make_ipv4_addr(&sin);
+            tmp = makeipaddr((struct sockaddr *)&sin, sizeof(sin));
 
             if (pch == h->h_addr_list && alen >= sizeof(sin))
                 memcpy((char *) addr, &sin, sizeof(sin));
@@ -5168,7 +5162,8 @@ gethost_common(struct hostent *h, struct sockaddr *addr, size_t alen, int af)
             sin6.sin6_len = sizeof(sin6);
 #endif
             memcpy(&sin6.sin6_addr, *pch, sizeof(sin6.sin6_addr));
-            tmp = make_ipv6_addr(&sin6);
+            tmp = makeipaddr((struct sockaddr *)&sin6,
+                sizeof(sin6));
 
             if (pch == h->h_addr_list && alen >= sizeof(sin6))
                 memcpy((char *) addr, &sin6, sizeof(sin6));
@@ -5939,11 +5934,14 @@ socket_inet_ntop(PyObject *self, PyObject *args)
     Py_buffer packed_ip;
     const char* retval;
 #ifdef ENABLE_IPV6
-    char ip[Py_MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN)];
+    char ip[Py_MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1];
 #else
-    char ip[INET_ADDRSTRLEN];
+    char ip[INET_ADDRSTRLEN + 1];
 #endif
 
+    /* Guarantee NUL-termination for PyUnicode_FromString() below */
+    memset((void *) &ip[0], '\0', sizeof(ip));
+
     if (!PyArg_ParseTuple(args, "iy*:inet_ntop", &af, &packed_ip)) {
         return NULL;
     }
@@ -5971,7 +5969,6 @@ socket_inet_ntop(PyObject *self, PyObject *args)
         return NULL;
     }
 
-    /* inet_ntop guarantee NUL-termination of resulting string. */
     retval = inet_ntop(af, packed_ip.buf, ip, sizeof(ip));
     PyBuffer_Release(&packed_ip);
     if (!retval) {


From webhook-mailer at python.org  Sun Jan 28 16:30:29 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sun, 28 Jan 2018 21:30:29 -0000
Subject: [Python-checkins] bpo-32251: Implement asyncio.BufferedProtocol.
 (#4755)
Message-ID: <mailman.22.1517175033.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/631fd38dbf04dbf0127881f3977982e401a849e4
commit: 631fd38dbf04dbf0127881f3977982e401a849e4
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-28T16:30:26-05:00
summary:

bpo-32251: Implement asyncio.BufferedProtocol. (#4755)

files:
A Lib/test/test_asyncio/test_buffered_proto.py
A Misc/NEWS.d/next/Library/2018-01-28-01-21-47.bpo-32251.fOA5qB.rst
M Doc/library/asyncio-protocol.rst
M Lib/asyncio/proactor_events.py
M Lib/asyncio/protocols.py
M Lib/asyncio/selector_events.py
M Lib/test/test_asyncio/test_proactor_events.py
M Lib/test/test_asyncio/test_selector_events.py

diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst
index 3aa1f2f2e996..a08c4d379b68 100644
--- a/Doc/library/asyncio-protocol.rst
+++ b/Doc/library/asyncio-protocol.rst
@@ -333,6 +333,16 @@ Protocol classes
    The base class for implementing streaming protocols (for use with
    e.g. TCP and SSL transports).
 
+.. class:: BufferedProtocol
+
+   A base class for implementing streaming protocols with manual
+   control of the receive buffer.
+
+   .. versionadded:: 3.7
+      **Important:** this has been been added to asyncio in Python 3.7
+      *on a provisional basis*!  Treat it as an experimental API that
+      might be changed or removed in Python 3.8.
+
 .. class:: DatagramProtocol
 
    The base class for implementing datagram protocols (for use with
@@ -428,10 +438,67 @@ and, if called, :meth:`data_received` won't be called after it.
 
 State machine:
 
-    start -> :meth:`~BaseProtocol.connection_made`
-    [-> :meth:`~Protocol.data_received` \*]
-    [-> :meth:`~Protocol.eof_received` ?]
-    -> :meth:`~BaseProtocol.connection_lost` -> end
+.. code-block:: none
+
+    start -> connection_made
+        [-> data_received]*
+        [-> eof_received]?
+    -> connection_lost -> end
+
+
+Streaming protocols with manual receive buffer control
+------------------------------------------------------
+
+.. versionadded:: 3.7
+   **Important:** :class:`BufferedProtocol` has been been added to
+   asyncio in Python 3.7 *on a provisional basis*!  Treat it as an
+   experimental API that might be changed or removed in Python 3.8.
+
+
+Event methods, such as :meth:`AbstractEventLoop.create_server` and
+:meth:`AbstractEventLoop.create_connection`, accept factories that
+return protocols that implement this interface.
+
+The idea of BufferedProtocol is that it allows to manually allocate
+and control the receive buffer.  Event loops can then use the buffer
+provided by the protocol to avoid unnecessary data copies.  This
+can result in noticeable performance improvement for protocols that
+receive big amounts of data.  Sophisticated protocols can allocate
+the buffer only once at creation time.
+
+The following callbacks are called on :class:`BufferedProtocol`
+instances:
+
+.. method:: BufferedProtocol.get_buffer()
+
+   Called to allocate a new receive buffer.  Must return an object
+   that implements the :ref:`buffer protocol <bufferobjects>`.
+
+.. method:: BufferedProtocol.buffer_updated(nbytes)
+
+   Called when the buffer was updated with the received data.
+
+   *nbytes* is the total number of bytes that were written to the buffer.
+
+.. method:: BufferedProtocol.eof_received()
+
+   See the documentation of the :meth:`Protocol.eof_received` method.
+
+
+:meth:`get_buffer` can be called an arbitrary number of times during
+a connection.  However, :meth:`eof_received` is called at most once
+and, if called, :meth:`data_received` won't be called after it.
+
+State machine:
+
+.. code-block:: none
+
+    start -> connection_made
+        [-> get_buffer
+            [-> buffer_updated]?
+        ]*
+        [-> eof_received]?
+    -> connection_lost -> end
 
 
 Datagram protocols
diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
index 6d27e5323875..10ca6f8967fb 100644
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -12,6 +12,7 @@
 from . import base_events
 from . import constants
 from . import futures
+from . import protocols
 from . import sslproto
 from . import transports
 from .log import logger
@@ -91,17 +92,19 @@ def __del__(self):
             self.close()
 
     def _fatal_error(self, exc, message='Fatal error on pipe transport'):
-        if isinstance(exc, base_events._FATAL_ERROR_IGNORE):
-            if self._loop.get_debug():
-                logger.debug("%r: %s", self, message, exc_info=True)
-        else:
-            self._loop.call_exception_handler({
-                'message': message,
-                'exception': exc,
-                'transport': self,
-                'protocol': self._protocol,
-            })
-        self._force_close(exc)
+        try:
+            if isinstance(exc, base_events._FATAL_ERROR_IGNORE):
+                if self._loop.get_debug():
+                    logger.debug("%r: %s", self, message, exc_info=True)
+            else:
+                self._loop.call_exception_handler({
+                    'message': message,
+                    'exception': exc,
+                    'transport': self,
+                    'protocol': self._protocol,
+                })
+        finally:
+            self._force_close(exc)
 
     def _force_close(self, exc):
         if self._closing:
@@ -150,6 +153,12 @@ def __init__(self, loop, sock, protocol, waiter=None,
                  extra=None, server=None):
         super().__init__(loop, sock, protocol, waiter, extra, server)
         self._paused = False
+
+        if protocols._is_buffered_protocol(protocol):
+            self._loop_reading = self._loop_reading__get_buffer
+        else:
+            self._loop_reading = self._loop_reading__data_received
+
         self._loop.call_soon(self._loop_reading)
 
     def is_reading(self):
@@ -159,6 +168,11 @@ def pause_reading(self):
         if self._closing or self._paused:
             return
         self._paused = True
+
+        if self._read_fut is not None and not self._read_fut.done():
+            self._read_fut.cancel()
+            self._read_fut = None
+
         if self._loop.get_debug():
             logger.debug("%r pauses reading", self)
 
@@ -170,11 +184,25 @@ def resume_reading(self):
         if self._loop.get_debug():
             logger.debug("%r resumes reading", self)
 
-    def _loop_reading(self, fut=None):
+    def _loop_reading__on_eof(self):
+        if self._loop.get_debug():
+            logger.debug("%r received EOF", self)
+
+        try:
+            keep_open = self._protocol.eof_received()
+        except Exception as exc:
+            self._fatal_error(
+                exc, 'Fatal error: protocol.eof_received() call failed.')
+            return
+
+        if not keep_open:
+            self.close()
+
+    def _loop_reading__data_received(self, fut=None):
         if self._paused:
             return
-        data = None
 
+        data = None
         try:
             if fut is not None:
                 assert self._read_fut is fut or (self._read_fut is None and
@@ -197,7 +225,7 @@ def _loop_reading(self, fut=None):
                 return
 
             # reschedule a new read
-            self._read_fut = self._loop._proactor.recv(self._sock, 4096)
+            self._read_fut = self._loop._proactor.recv(self._sock, 32768)
         except ConnectionAbortedError as exc:
             if not self._closing:
                 self._fatal_error(exc, 'Fatal read error on pipe transport')
@@ -216,12 +244,81 @@ def _loop_reading(self, fut=None):
         finally:
             if data:
                 self._protocol.data_received(data)
-            elif data is not None:
-                if self._loop.get_debug():
-                    logger.debug("%r received EOF", self)
-                keep_open = self._protocol.eof_received()
-                if not keep_open:
-                    self.close()
+            elif data == b'':
+                self._loop_reading__on_eof()
+
+    def _loop_reading__get_buffer(self, fut=None):
+        if self._paused:
+            return
+
+        nbytes = None
+        if fut is not None:
+            assert self._read_fut is fut or (self._read_fut is None and
+                                             self._closing)
+            self._read_fut = None
+            try:
+                if fut.done():
+                    nbytes = fut.result()
+                else:
+                    # the future will be replaced by next proactor.recv call
+                    fut.cancel()
+            except ConnectionAbortedError as exc:
+                if not self._closing:
+                    self._fatal_error(
+                        exc, 'Fatal read error on pipe transport')
+                elif self._loop.get_debug():
+                    logger.debug("Read error on pipe transport while closing",
+                                 exc_info=True)
+            except ConnectionResetError as exc:
+                self._force_close(exc)
+            except OSError as exc:
+                self._fatal_error(exc, 'Fatal read error on pipe transport')
+            except futures.CancelledError:
+                if not self._closing:
+                    raise
+
+            if nbytes is not None:
+                if nbytes == 0:
+                    # we got end-of-file so no need to reschedule a new read
+                    self._loop_reading__on_eof()
+                else:
+                    try:
+                        self._protocol.buffer_updated(nbytes)
+                    except Exception as exc:
+                        self._fatal_error(
+                            exc,
+                            'Fatal error: '
+                            'protocol.buffer_updated() call failed.')
+                        return
+
+        if self._closing or nbytes == 0:
+            # since close() has been called we ignore any read data
+            return
+
+        try:
+            buf = self._protocol.get_buffer()
+        except Exception as exc:
+            self._fatal_error(
+                exc, 'Fatal error: protocol.get_buffer() call failed.')
+            return
+
+        try:
+            # schedule a new read
+            self._read_fut = self._loop._proactor.recv_into(self._sock, buf)
+            self._read_fut.add_done_callback(self._loop_reading)
+        except ConnectionAbortedError as exc:
+            if not self._closing:
+                self._fatal_error(exc, 'Fatal read error on pipe transport')
+            elif self._loop.get_debug():
+                logger.debug("Read error on pipe transport while closing",
+                             exc_info=True)
+        except ConnectionResetError as exc:
+            self._force_close(exc)
+        except OSError as exc:
+            self._fatal_error(exc, 'Fatal read error on pipe transport')
+        except futures.CancelledError:
+            if not self._closing:
+                raise
 
 
 class _ProactorBaseWritePipeTransport(_ProactorBasePipeTransport,
diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py
index 57987ae4463f..9e786dcfbbe5 100644
--- a/Lib/asyncio/protocols.py
+++ b/Lib/asyncio/protocols.py
@@ -2,7 +2,7 @@
 
 __all__ = (
     'BaseProtocol', 'Protocol', 'DatagramProtocol',
-    'SubprocessProtocol',
+    'SubprocessProtocol', 'BufferedProtocol',
 )
 
 
@@ -102,6 +102,57 @@ def eof_received(self):
         """
 
 
+class BufferedProtocol(BaseProtocol):
+    """Interface for stream protocol with manual buffer control.
+
+    Important: this has been been added to asyncio in Python 3.7
+    *on a provisional basis*!  Treat it as an experimental API that
+    might be changed or removed in Python 3.8.
+
+    Event methods, such as `create_server` and `create_connection`,
+    accept factories that return protocols that implement this interface.
+
+    The idea of BufferedProtocol is that it allows to manually allocate
+    and control the receive buffer.  Event loops can then use the buffer
+    provided by the protocol to avoid unnecessary data copies.  This
+    can result in noticeable performance improvement for protocols that
+    receive big amounts of data.  Sophisticated protocols can allocate
+    the buffer only once at creation time.
+
+    State machine of calls:
+
+      start -> CM [-> GB [-> BU?]]* [-> ER?] -> CL -> end
+
+    * CM: connection_made()
+    * GB: get_buffer()
+    * BU: buffer_updated()
+    * ER: eof_received()
+    * CL: connection_lost()
+    """
+
+    def get_buffer(self):
+        """Called to allocate a new receive buffer.
+
+        Must return an object that implements the
+        :ref:`buffer protocol <bufferobjects>`.
+        """
+
+    def buffer_updated(self, nbytes):
+        """Called when the buffer was updated with the received data.
+
+        *nbytes* is the total number of bytes that were written to
+        the buffer.
+        """
+
+    def eof_received(self):
+        """Called when the other end calls write_eof() or equivalent.
+
+        If this returns a false value (including None), the transport
+        will close itself.  If it returns a true value, closing the
+        transport is up to the protocol.
+        """
+
+
 class DatagramProtocol(BaseProtocol):
     """Interface for datagram protocol."""
 
@@ -134,3 +185,7 @@ def pipe_connection_lost(self, fd, exc):
 
     def process_exited(self):
         """Called when subprocess has exited."""
+
+
+def _is_buffered_protocol(proto):
+    return hasattr(proto, 'get_buffer') and not hasattr(proto, 'data_received')
diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
index 5956f2d993e5..354bf9d1c2de 100644
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -22,8 +22,9 @@
 from . import constants
 from . import events
 from . import futures
-from . import transports
+from . import protocols
 from . import sslproto
+from . import transports
 from .log import logger
 
 
@@ -713,6 +714,12 @@ class _SelectorSocketTransport(_SelectorTransport):
 
     def __init__(self, loop, sock, protocol, waiter=None,
                  extra=None, server=None):
+
+        if protocols._is_buffered_protocol(protocol):
+            self._read_ready = self._read_ready__get_buffer
+        else:
+            self._read_ready = self._read_ready__data_received
+
         super().__init__(loop, sock, protocol, extra, server)
         self._eof = False
         self._paused = False
@@ -751,29 +758,74 @@ def resume_reading(self):
         if self._loop.get_debug():
             logger.debug("%r resumes reading", self)
 
-    def _read_ready(self):
+    def _read_ready__get_buffer(self):
+        if self._conn_lost:
+            return
+
+        try:
+            buf = self._protocol.get_buffer()
+        except Exception as exc:
+            self._fatal_error(
+                exc, 'Fatal error: protocol.get_buffer() call failed.')
+            return
+
+        try:
+            nbytes = self._sock.recv_into(buf)
+        except (BlockingIOError, InterruptedError):
+            return
+        except Exception as exc:
+            self._fatal_error(exc, 'Fatal read error on socket transport')
+            return
+
+        if not nbytes:
+            self._read_ready__on_eof()
+            return
+
+        try:
+            self._protocol.buffer_updated(nbytes)
+        except Exception as exc:
+            self._fatal_error(
+                exc, 'Fatal error: protocol.buffer_updated() call failed.')
+
+    def _read_ready__data_received(self):
         if self._conn_lost:
             return
         try:
             data = self._sock.recv(self.max_size)
         except (BlockingIOError, InterruptedError):
-            pass
+            return
         except Exception as exc:
             self._fatal_error(exc, 'Fatal read error on socket transport')
+            return
+
+        if not data:
+            self._read_ready__on_eof()
+            return
+
+        try:
+            self._protocol.data_received(data)
+        except Exception as exc:
+            self._fatal_error(
+                exc, 'Fatal error: protocol.data_received() call failed.')
+
+    def _read_ready__on_eof(self):
+        if self._loop.get_debug():
+            logger.debug("%r received EOF", self)
+
+        try:
+            keep_open = self._protocol.eof_received()
+        except Exception as exc:
+            self._fatal_error(
+                exc, 'Fatal error: protocol.eof_received() call failed.')
+            return
+
+        if keep_open:
+            # We're keeping the connection open so the
+            # protocol can write more, but we still can't
+            # receive more, so remove the reader callback.
+            self._loop._remove_reader(self._sock_fd)
         else:
-            if data:
-                self._protocol.data_received(data)
-            else:
-                if self._loop.get_debug():
-                    logger.debug("%r received EOF", self)
-                keep_open = self._protocol.eof_received()
-                if keep_open:
-                    # We're keeping the connection open so the
-                    # protocol can write more, but we still can't
-                    # receive more, so remove the reader callback.
-                    self._loop._remove_reader(self._sock_fd)
-                else:
-                    self.close()
+            self.close()
 
     def write(self, data):
         if not isinstance(data, (bytes, bytearray, memoryview)):
diff --git a/Lib/test/test_asyncio/test_buffered_proto.py b/Lib/test/test_asyncio/test_buffered_proto.py
new file mode 100644
index 000000000000..22f9269e814f
--- /dev/null
+++ b/Lib/test/test_asyncio/test_buffered_proto.py
@@ -0,0 +1,85 @@
+import asyncio
+import unittest
+
+from test.test_asyncio import functional as func_tests
+
+
+class ReceiveStuffProto(asyncio.BufferedProtocol):
+    def __init__(self, cb, con_lost_fut):
+        self.cb = cb
+        self.con_lost_fut = con_lost_fut
+
+    def get_buffer(self):
+        self.buffer = bytearray(100)
+        return self.buffer
+
+    def buffer_updated(self, nbytes):
+        self.cb(self.buffer[:nbytes])
+
+    def connection_lost(self, exc):
+        if exc is None:
+            self.con_lost_fut.set_result(None)
+        else:
+            self.con_lost_fut.set_exception(exc)
+
+
+class BaseTestBufferedProtocol(func_tests.FunctionalTestCaseMixin):
+
+    def new_loop(self):
+        raise NotImplementedError
+
+    def test_buffered_proto_create_connection(self):
+
+        NOISE = b'12345678+' * 1024
+
+        async def client(addr):
+            data = b''
+
+            def on_buf(buf):
+                nonlocal data
+                data += buf
+                if data == NOISE:
+                    tr.write(b'1')
+
+            conn_lost_fut = self.loop.create_future()
+
+            tr, pr = await self.loop.create_connection(
+                lambda: ReceiveStuffProto(on_buf, conn_lost_fut), *addr)
+
+            await conn_lost_fut
+
+        async def on_server_client(reader, writer):
+            writer.write(NOISE)
+            await reader.readexactly(1)
+            writer.close()
+            await writer.wait_closed()
+
+        srv = self.loop.run_until_complete(
+            asyncio.start_server(
+                on_server_client, '127.0.0.1', 0))
+
+        addr = srv.sockets[0].getsockname()
+        self.loop.run_until_complete(
+            asyncio.wait_for(client(addr), 5, loop=self.loop))
+
+        srv.close()
+        self.loop.run_until_complete(srv.wait_closed())
+
+
+class BufferedProtocolSelectorTests(BaseTestBufferedProtocol,
+                                    unittest.TestCase):
+
+    def new_loop(self):
+        return asyncio.SelectorEventLoop()
+
+
+ at unittest.skipUnless(hasattr(asyncio, 'ProactorEventLoop'), 'Windows only')
+class BufferedProtocolProactorTests(BaseTestBufferedProtocol,
+                                    unittest.TestCase):
+
+    def new_loop(self):
+        return asyncio.ProactorEventLoop()
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py
index 82bff28d7d05..f627dfce0e17 100644
--- a/Lib/test/test_asyncio/test_proactor_events.py
+++ b/Lib/test/test_asyncio/test_proactor_events.py
@@ -44,12 +44,12 @@ def test_ctor(self):
         test_utils.run_briefly(self.loop)
         self.assertIsNone(fut.result())
         self.protocol.connection_made(tr)
-        self.proactor.recv.assert_called_with(self.sock, 4096)
+        self.proactor.recv.assert_called_with(self.sock, 32768)
 
     def test_loop_reading(self):
         tr = self.socket_transport()
         tr._loop_reading()
-        self.loop._proactor.recv.assert_called_with(self.sock, 4096)
+        self.loop._proactor.recv.assert_called_with(self.sock, 32768)
         self.assertFalse(self.protocol.data_received.called)
         self.assertFalse(self.protocol.eof_received.called)
 
@@ -60,7 +60,7 @@ def test_loop_reading_data(self):
         tr = self.socket_transport()
         tr._read_fut = res
         tr._loop_reading(res)
-        self.loop._proactor.recv.assert_called_with(self.sock, 4096)
+        self.loop._proactor.recv.assert_called_with(self.sock, 32768)
         self.protocol.data_received.assert_called_with(b'data')
 
     def test_loop_reading_no_data(self):
@@ -444,6 +444,197 @@ def test_dont_pause_writing(self):
         self.assertFalse(self.protocol.pause_writing.called)
 
 
+class ProactorSocketTransportBufferedProtoTests(test_utils.TestCase):
+
+    def setUp(self):
+        super().setUp()
+        self.loop = self.new_test_loop()
+        self.addCleanup(self.loop.close)
+        self.proactor = mock.Mock()
+        self.loop._proactor = self.proactor
+
+        self.protocol = test_utils.make_test_protocol(asyncio.BufferedProtocol)
+        self.buf = mock.Mock()
+        self.protocol.get_buffer.side_effect = lambda: self.buf
+
+        self.sock = mock.Mock(socket.socket)
+
+    def socket_transport(self, waiter=None):
+        transport = _ProactorSocketTransport(self.loop, self.sock,
+                                             self.protocol, waiter=waiter)
+        self.addCleanup(close_transport, transport)
+        return transport
+
+    def test_ctor(self):
+        fut = asyncio.Future(loop=self.loop)
+        tr = self.socket_transport(waiter=fut)
+        test_utils.run_briefly(self.loop)
+        self.assertIsNone(fut.result())
+        self.protocol.connection_made(tr)
+        self.proactor.recv_into.assert_called_with(self.sock, self.buf)
+
+    def test_loop_reading(self):
+        tr = self.socket_transport()
+        tr._loop_reading()
+        self.loop._proactor.recv_into.assert_called_with(self.sock, self.buf)
+        self.assertTrue(self.protocol.get_buffer.called)
+        self.assertFalse(self.protocol.buffer_updated.called)
+        self.assertFalse(self.protocol.eof_received.called)
+
+    def test_get_buffer_error(self):
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+        self.protocol.get_buffer.side_effect = LookupError()
+
+        transport._loop_reading()
+
+        self.assertTrue(transport._fatal_error.called)
+        self.assertTrue(self.protocol.get_buffer.called)
+        self.assertFalse(self.protocol.buffer_updated.called)
+
+    def test_buffer_updated_error(self):
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+        self.protocol.buffer_updated.side_effect = LookupError()
+
+        res = asyncio.Future(loop=self.loop)
+        res.set_result(10)
+        transport._read_fut = res
+        transport._loop_reading(res)
+
+        self.assertTrue(transport._fatal_error.called)
+        self.assertFalse(self.protocol.get_buffer.called)
+        self.assertTrue(self.protocol.buffer_updated.called)
+
+    def test_loop_eof_received_error(self):
+        res = asyncio.Future(loop=self.loop)
+        res.set_result(0)
+
+        self.protocol.eof_received.side_effect = LookupError()
+
+        tr = self.socket_transport()
+        tr._fatal_error = mock.Mock()
+
+        tr.close = mock.Mock()
+        tr._read_fut = res
+        tr._loop_reading(res)
+        self.assertFalse(self.loop._proactor.recv_into.called)
+        self.assertTrue(self.protocol.eof_received.called)
+        self.assertTrue(tr._fatal_error.called)
+
+    def test_loop_reading_data(self):
+        res = asyncio.Future(loop=self.loop)
+        res.set_result(4)
+
+        tr = self.socket_transport()
+        tr._read_fut = res
+        tr._loop_reading(res)
+        self.loop._proactor.recv_into.assert_called_with(self.sock, self.buf)
+        self.protocol.buffer_updated.assert_called_with(4)
+
+    def test_loop_reading_no_data(self):
+        res = asyncio.Future(loop=self.loop)
+        res.set_result(0)
+
+        tr = self.socket_transport()
+        self.assertRaises(AssertionError, tr._loop_reading, res)
+
+        tr.close = mock.Mock()
+        tr._read_fut = res
+        tr._loop_reading(res)
+        self.assertFalse(self.loop._proactor.recv_into.called)
+        self.assertTrue(self.protocol.eof_received.called)
+        self.assertTrue(tr.close.called)
+
+    def test_loop_reading_aborted(self):
+        err = self.loop._proactor.recv_into.side_effect = \
+            ConnectionAbortedError()
+
+        tr = self.socket_transport()
+        tr._fatal_error = mock.Mock()
+        tr._loop_reading()
+        tr._fatal_error.assert_called_with(
+            err, 'Fatal read error on pipe transport')
+
+    def test_loop_reading_aborted_closing(self):
+        self.loop._proactor.recv.side_effect = ConnectionAbortedError()
+
+        tr = self.socket_transport()
+        tr._closing = True
+        tr._fatal_error = mock.Mock()
+        tr._loop_reading()
+        self.assertFalse(tr._fatal_error.called)
+
+    def test_loop_reading_aborted_is_fatal(self):
+        self.loop._proactor.recv_into.side_effect = ConnectionAbortedError()
+        tr = self.socket_transport()
+        tr._closing = False
+        tr._fatal_error = mock.Mock()
+        tr._loop_reading()
+        self.assertTrue(tr._fatal_error.called)
+
+    def test_loop_reading_conn_reset_lost(self):
+        err = self.loop._proactor.recv_into.side_effect = ConnectionResetError()
+
+        tr = self.socket_transport()
+        tr._closing = False
+        tr._fatal_error = mock.Mock()
+        tr._force_close = mock.Mock()
+        tr._loop_reading()
+        self.assertFalse(tr._fatal_error.called)
+        tr._force_close.assert_called_with(err)
+
+    def test_loop_reading_exception(self):
+        err = self.loop._proactor.recv_into.side_effect = OSError()
+
+        tr = self.socket_transport()
+        tr._fatal_error = mock.Mock()
+        tr._loop_reading()
+        tr._fatal_error.assert_called_with(
+            err, 'Fatal read error on pipe transport')
+
+    def test_pause_resume_reading(self):
+        tr = self.socket_transport()
+        futures = []
+        for msg in [10, 20, 30, 40, 0]:
+            f = asyncio.Future(loop=self.loop)
+            f.set_result(msg)
+            futures.append(f)
+
+        self.loop._proactor.recv_into.side_effect = futures
+        self.loop._run_once()
+        self.assertFalse(tr._paused)
+        self.assertTrue(tr.is_reading())
+        self.loop._run_once()
+        self.protocol.buffer_updated.assert_called_with(10)
+        self.loop._run_once()
+        self.protocol.buffer_updated.assert_called_with(20)
+
+        tr.pause_reading()
+        tr.pause_reading()
+        self.assertTrue(tr._paused)
+        self.assertFalse(tr.is_reading())
+        for i in range(10):
+            self.loop._run_once()
+        self.protocol.buffer_updated.assert_called_with(20)
+
+        tr.resume_reading()
+        tr.resume_reading()
+        self.assertFalse(tr._paused)
+        self.assertTrue(tr.is_reading())
+        self.loop._run_once()
+        self.protocol.buffer_updated.assert_called_with(30)
+        self.loop._run_once()
+        self.protocol.buffer_updated.assert_called_with(40)
+        tr.close()
+
+        self.assertFalse(tr.is_reading())
+
+
 class BaseProactorEventLoopTests(test_utils.TestCase):
 
     def setUp(self):
diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py
index 89c3d5e16c1d..219ab0eb5b8b 100644
--- a/Lib/test/test_asyncio/test_selector_events.py
+++ b/Lib/test/test_asyncio/test_selector_events.py
@@ -926,6 +926,34 @@ def test_pause_resume_reading(self):
         self.assertFalse(tr.is_reading())
         self.loop.assert_no_reader(7)
 
+    def test_read_eof_received_error(self):
+        transport = self.socket_transport()
+        transport.close = mock.Mock()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+
+        self.protocol.eof_received.side_effect = LookupError()
+
+        self.sock.recv.return_value = b''
+        transport._read_ready()
+
+        self.protocol.eof_received.assert_called_with()
+        self.assertTrue(transport._fatal_error.called)
+
+    def test_data_received_error(self):
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+        self.protocol.data_received.side_effect = LookupError()
+
+        self.sock.recv.return_value = b'data'
+        transport._read_ready()
+
+        self.assertTrue(transport._fatal_error.called)
+        self.assertTrue(self.protocol.data_received.called)
+
     def test_read_ready(self):
         transport = self.socket_transport()
 
@@ -1229,6 +1257,149 @@ def test_transport_close_remove_writer(self, m_log):
         remove_writer.assert_called_with(self.sock_fd)
 
 
+class SelectorSocketTransportBufferedProtocolTests(test_utils.TestCase):
+
+    def setUp(self):
+        super().setUp()
+        self.loop = self.new_test_loop()
+
+        self.protocol = test_utils.make_test_protocol(asyncio.BufferedProtocol)
+        self.buf = mock.Mock()
+        self.protocol.get_buffer.side_effect = lambda: self.buf
+
+        self.sock = mock.Mock(socket.socket)
+        self.sock_fd = self.sock.fileno.return_value = 7
+
+    def socket_transport(self, waiter=None):
+        transport = _SelectorSocketTransport(self.loop, self.sock,
+                                             self.protocol, waiter=waiter)
+        self.addCleanup(close_transport, transport)
+        return transport
+
+    def test_ctor(self):
+        waiter = asyncio.Future(loop=self.loop)
+        tr = self.socket_transport(waiter=waiter)
+        self.loop.run_until_complete(waiter)
+
+        self.loop.assert_reader(7, tr._read_ready)
+        test_utils.run_briefly(self.loop)
+        self.protocol.connection_made.assert_called_with(tr)
+
+    def test_get_buffer_error(self):
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+        self.protocol.get_buffer.side_effect = LookupError()
+
+        transport._read_ready()
+
+        self.assertTrue(transport._fatal_error.called)
+        self.assertTrue(self.protocol.get_buffer.called)
+        self.assertFalse(self.protocol.buffer_updated.called)
+
+    def test_buffer_updated_error(self):
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+        self.protocol.buffer_updated.side_effect = LookupError()
+
+        self.sock.recv_into.return_value = 10
+        transport._read_ready()
+
+        self.assertTrue(transport._fatal_error.called)
+        self.assertTrue(self.protocol.get_buffer.called)
+        self.assertTrue(self.protocol.buffer_updated.called)
+
+    def test_read_eof_received_error(self):
+        transport = self.socket_transport()
+        transport.close = mock.Mock()
+        transport._fatal_error = mock.Mock()
+
+        self.loop.call_exception_handler = mock.Mock()
+
+        self.protocol.eof_received.side_effect = LookupError()
+
+        self.sock.recv_into.return_value = 0
+        transport._read_ready()
+
+        self.protocol.eof_received.assert_called_with()
+        self.assertTrue(transport._fatal_error.called)
+
+    def test_read_ready(self):
+        transport = self.socket_transport()
+
+        self.sock.recv_into.return_value = 10
+        transport._read_ready()
+
+        self.protocol.get_buffer.assert_called_with()
+        self.protocol.buffer_updated.assert_called_with(10)
+
+    def test_read_ready_eof(self):
+        transport = self.socket_transport()
+        transport.close = mock.Mock()
+
+        self.sock.recv_into.return_value = 0
+        transport._read_ready()
+
+        self.protocol.eof_received.assert_called_with()
+        transport.close.assert_called_with()
+
+    def test_read_ready_eof_keep_open(self):
+        transport = self.socket_transport()
+        transport.close = mock.Mock()
+
+        self.sock.recv_into.return_value = 0
+        self.protocol.eof_received.return_value = True
+        transport._read_ready()
+
+        self.protocol.eof_received.assert_called_with()
+        self.assertFalse(transport.close.called)
+
+    @mock.patch('logging.exception')
+    def test_read_ready_tryagain(self, m_exc):
+        self.sock.recv_into.side_effect = BlockingIOError
+
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+        transport._read_ready()
+
+        self.assertFalse(transport._fatal_error.called)
+
+    @mock.patch('logging.exception')
+    def test_read_ready_tryagain_interrupted(self, m_exc):
+        self.sock.recv_into.side_effect = InterruptedError
+
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+        transport._read_ready()
+
+        self.assertFalse(transport._fatal_error.called)
+
+    @mock.patch('logging.exception')
+    def test_read_ready_conn_reset(self, m_exc):
+        err = self.sock.recv_into.side_effect = ConnectionResetError()
+
+        transport = self.socket_transport()
+        transport._force_close = mock.Mock()
+        with test_utils.disable_logger():
+            transport._read_ready()
+        transport._force_close.assert_called_with(err)
+
+    @mock.patch('logging.exception')
+    def test_read_ready_err(self, m_exc):
+        err = self.sock.recv_into.side_effect = OSError()
+
+        transport = self.socket_transport()
+        transport._fatal_error = mock.Mock()
+        transport._read_ready()
+
+        transport._fatal_error.assert_called_with(
+                                   err,
+                                   'Fatal read error on socket transport')
+
+
 class SelectorDatagramTransportTests(test_utils.TestCase):
 
     def setUp(self):
diff --git a/Misc/NEWS.d/next/Library/2018-01-28-01-21-47.bpo-32251.fOA5qB.rst b/Misc/NEWS.d/next/Library/2018-01-28-01-21-47.bpo-32251.fOA5qB.rst
new file mode 100644
index 000000000000..869be1ccce05
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-28-01-21-47.bpo-32251.fOA5qB.rst
@@ -0,0 +1 @@
+Implement asyncio.BufferedProtocol (provisional API).


From webhook-mailer at python.org  Sun Jan 28 17:27:41 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sun, 28 Jan 2018 22:27:41 -0000
Subject: [Python-checkins] bpo-32373: Add socket.getblocking() method.
 (#4926)
Message-ID: <mailman.23.1517178463.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/f11b460d8717fc3a5810684713b8b818f68789e8
commit: f11b460d8717fc3a5810684713b8b818f68789e8
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-28T17:27:38-05:00
summary:

bpo-32373: Add socket.getblocking() method. (#4926)

files:
A Misc/NEWS.d/next/Library/2017-12-19-09-23-46.bpo-32373.8qAkoW.rst
M Doc/library/socket.rst
M Lib/test/test_socket.py
M Modules/socketmodule.c

diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index 7edd4ba36043..faa260e5d6c5 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1087,6 +1087,16 @@ to sockets.
    to decode C structures encoded as byte strings).
 
 
+.. method:: socket.getblocking()
+
+   Return ``True`` if socket is in blocking mode, ``False`` if in
+   non-blocking.
+
+   This is equivalent to checking ``socket.gettimeout() == 0``.
+
+   .. versionadded:: 3.7
+
+
 .. method:: socket.gettimeout()
 
    Return the timeout in seconds (float) associated with socket operations,
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 275384c28fb3..365da36c4210 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -98,6 +98,12 @@ def _have_socket_vsock():
     ret = get_cid() is not None
     return ret
 
+
+def _is_fd_in_blocking_mode(sock):
+    return not bool(
+        fcntl.fcntl(sock, fcntl.F_GETFL, os.O_NONBLOCK) & os.O_NONBLOCK)
+
+
 HAVE_SOCKET_CAN = _have_socket_can()
 
 HAVE_SOCKET_CAN_ISOTP = _have_socket_can_isotp()
@@ -4079,8 +4085,44 @@ def testSetBlocking(self):
         # Testing whether set blocking works
         self.serv.setblocking(True)
         self.assertIsNone(self.serv.gettimeout())
+        self.assertTrue(self.serv.getblocking())
+        if fcntl:
+            self.assertTrue(_is_fd_in_blocking_mode(self.serv))
+
         self.serv.setblocking(False)
         self.assertEqual(self.serv.gettimeout(), 0.0)
+        self.assertFalse(self.serv.getblocking())
+        if fcntl:
+            self.assertFalse(_is_fd_in_blocking_mode(self.serv))
+
+        self.serv.settimeout(None)
+        self.assertTrue(self.serv.getblocking())
+        if fcntl:
+            self.assertTrue(_is_fd_in_blocking_mode(self.serv))
+
+        self.serv.settimeout(0)
+        self.assertFalse(self.serv.getblocking())
+        self.assertEqual(self.serv.gettimeout(), 0)
+        if fcntl:
+            self.assertFalse(_is_fd_in_blocking_mode(self.serv))
+
+        self.serv.settimeout(10)
+        self.assertTrue(self.serv.getblocking())
+        self.assertEqual(self.serv.gettimeout(), 10)
+        if fcntl:
+            # When a Python socket has a non-zero timeout, it's
+            # switched internally to a non-blocking mode.
+            # Later, sock.sendall(), sock.recv(), and other socket
+            # operations use a `select()` call and handle EWOULDBLOCK/EGAIN
+            # on all socket operations.  That's how timeouts are
+            # enforced.
+            self.assertFalse(_is_fd_in_blocking_mode(self.serv))
+
+        self.serv.settimeout(0)
+        self.assertFalse(self.serv.getblocking())
+        if fcntl:
+            self.assertFalse(_is_fd_in_blocking_mode(self.serv))
+
         start = time.time()
         try:
             self.serv.accept()
@@ -4113,6 +4155,8 @@ def testInitNonBlocking(self):
         self.serv.close()
         self.serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM |
                                                   socket.SOCK_NONBLOCK)
+        self.assertFalse(self.serv.getblocking())
+        self.assertEqual(self.serv.gettimeout(), 0)
         self.port = support.bind_port(self.serv)
         self.serv.listen()
         # actual testing
@@ -5190,11 +5234,24 @@ def checkNonblock(self, s, nonblock=True, timeout=0.0):
             self.assertEqual(s.gettimeout(), timeout)
             self.assertTrue(
                 fcntl.fcntl(s, fcntl.F_GETFL, os.O_NONBLOCK) & os.O_NONBLOCK)
+            if timeout == 0:
+                # timeout == 0: means that getblocking() must be False.
+                self.assertFalse(s.getblocking())
+            else:
+                # If timeout > 0, the socket will be in a "blocking" mode
+                # from the standpoint of the Python API.  For Python socket
+                # object, "blocking" means that operations like 'sock.recv()'
+                # will block.  Internally, file descriptors for
+                # "blocking" Python sockets *with timeouts* are in a
+                # *non-blocking* mode, and 'sock.recv()' uses 'select()'
+                # and handles EWOULDBLOCK/EAGAIN to enforce the timeout.
+                self.assertTrue(s.getblocking())
         else:
             self.assertEqual(s.type, socket.SOCK_STREAM)
             self.assertEqual(s.gettimeout(), None)
             self.assertFalse(
                 fcntl.fcntl(s, fcntl.F_GETFL, os.O_NONBLOCK) & os.O_NONBLOCK)
+            self.assertTrue(s.getblocking())
 
     @support.requires_linux_version(2, 6, 28)
     def test_SOCK_NONBLOCK(self):
@@ -5204,15 +5261,15 @@ def test_SOCK_NONBLOCK(self):
                            socket.SOCK_STREAM | socket.SOCK_NONBLOCK) as s:
             self.checkNonblock(s)
             s.setblocking(1)
-            self.checkNonblock(s, False)
+            self.checkNonblock(s, nonblock=False)
             s.setblocking(0)
             self.checkNonblock(s)
             s.settimeout(None)
-            self.checkNonblock(s, False)
+            self.checkNonblock(s, nonblock=False)
             s.settimeout(2.0)
             self.checkNonblock(s, timeout=2.0)
             s.setblocking(1)
-            self.checkNonblock(s, False)
+            self.checkNonblock(s, nonblock=False)
         # defaulttimeout
         t = socket.getdefaulttimeout()
         socket.setdefaulttimeout(0.0)
diff --git a/Misc/NEWS.d/next/Library/2017-12-19-09-23-46.bpo-32373.8qAkoW.rst b/Misc/NEWS.d/next/Library/2017-12-19-09-23-46.bpo-32373.8qAkoW.rst
new file mode 100644
index 000000000000..9772dda35a30
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-19-09-23-46.bpo-32373.8qAkoW.rst
@@ -0,0 +1 @@
+Add socket.getblocking() method.
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index 5fe2431bee84..8744ea0be87c 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -136,6 +136,7 @@ sendall(data[, flags]) -- send all data\n\
 send(data[, flags]) -- send data, may not send all of it\n\
 sendto(data[, flags], addr) -- send data to a given address\n\
 setblocking(0 | 1) -- set or clear the blocking I/O flag\n\
+getblocking() -- return True if socket is blocking, False if non-blocking\n\
 setsockopt(level, optname, value[, optlen]) -- set socket options\n\
 settimeout(None | float) -- set or clear the timeout\n\
 shutdown(how) -- shut down traffic in one or both directions\n\
@@ -2525,6 +2526,27 @@ Set the socket to blocking (flag is true) or non-blocking (false).\n\
 setblocking(True) is equivalent to settimeout(None);\n\
 setblocking(False) is equivalent to settimeout(0.0).");
 
+/* s.getblocking() method.
+   Returns True if socket is in blocking mode,
+   False if it is in non-blocking mode.
+*/
+static PyObject *
+sock_getblocking(PySocketSockObject *s)
+{
+    if (s->sock_timeout) {
+        Py_RETURN_TRUE;
+    }
+    else {
+        Py_RETURN_FALSE;
+    }
+}
+
+PyDoc_STRVAR(getblocking_doc,
+"getblocking()\n\
+\n\
+Returns True if socket is in blocking mode, or False if it\n\
+is in non-blocking mode.");
+
 static int
 socket_parse_timeout(_PyTime_t *timeout, PyObject *timeout_obj)
 {
@@ -2581,7 +2603,30 @@ sock_settimeout(PySocketSockObject *s, PyObject *arg)
         return NULL;
 
     s->sock_timeout = timeout;
-    if (internal_setblocking(s, timeout < 0) == -1) {
+
+    int block = timeout < 0;
+    /* Blocking mode for a Python socket object means that operations
+       like :meth:`recv` or :meth:`sendall` will block the execution of
+       the current thread until they are complete or aborted with a
+       `socket.timeout` or `socket.error` errors.  When timeout is `None`,
+       the underlying FD is in a blocking mode.  When timeout is a positive
+       number, the FD is in a non-blocking mode, and socket ops are
+       implemented with a `select()` call.
+
+       When timeout is 0.0, the FD is in a non-blocking mode.
+
+       This table summarizes all states in which the socket object and
+       its underlying FD can be:
+
+       ==================== ===================== ==============
+        `gettimeout()`       `getblocking()`       FD
+       ==================== ===================== ==============
+        ``None``             ``True``              blocking
+        ``0.0``              ``False``             non-blocking
+        ``> 0``              ``True``              non-blocking
+    */
+
+    if (internal_setblocking(s, block) == -1) {
         return NULL;
     }
     Py_RETURN_NONE;
@@ -4601,6 +4646,8 @@ static PyMethodDef sock_methods[] = {
                       sendto_doc},
     {"setblocking",       (PyCFunction)sock_setblocking, METH_O,
                       setblocking_doc},
+    {"getblocking",   (PyCFunction)sock_getblocking, METH_NOARGS,
+                      getblocking_doc},
     {"settimeout",    (PyCFunction)sock_settimeout, METH_O,
                       settimeout_doc},
     {"gettimeout",    (PyCFunction)sock_gettimeout, METH_NOARGS,


From webhook-mailer at python.org  Sun Jan 28 18:34:11 2018
From: webhook-mailer at python.org (Steve Dower)
Date: Sun, 28 Jan 2018 23:34:11 -0000
Subject: [Python-checkins] bpo-32699: Improves doc for .pth files in
 presense of a ._pth file (#5399)
Message-ID: <mailman.24.1517182454.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/15ea3a6a320623328530e5d8da5b83c75766180b
commit: 15ea3a6a320623328530e5d8da5b83c75766180b
branch: master
author: Steve Dower <steve.dower at microsoft.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T10:34:09+11:00
summary:

bpo-32699: Improves doc for .pth files in presense of a ._pth file (#5399)

files:
M Doc/using/windows.rst

diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index 1c7b8142af29..47d423f42d6e 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -721,7 +721,7 @@ installation directory.  So, if you had installed Python to
 :file:`C:\\Python\\Lib\\site-packages\\`.
 
 To completely override :data:`sys.path`, create a ``._pth`` file with the same
-name as the DLL (``python36._pth``) or the executable (``python._pth``) and
+name as the DLL (``python37._pth``) or the executable (``python._pth``) and
 specify one line for each path to add to :data:`sys.path`. The file based on the
 DLL name overrides the one based on the executable, which allows paths to be
 restricted for any program loading the runtime if desired.
@@ -734,7 +734,7 @@ Import statements other than to ``site`` are not permitted, and arbitrary code
 cannot be specified.
 
 Note that ``.pth`` files (without leading underscore) will be processed normally
-by the :mod:`site` module.
+by the :mod:`site` module when ``import site`` has been specified.
 
 When no ``._pth`` file is found, this is how :data:`sys.path` is populated on
 Windows:
@@ -796,7 +796,7 @@ following advice will prevent conflicts with other installations:
   environment variables, and also ignore :mod:`site` unless ``import site`` is
   listed.
 
-* If you are loading :file:`python3.dll` or :file:`python36.dll` in your own
+* If you are loading :file:`python3.dll` or :file:`python37.dll` in your own
   executable, explicitly call :c:func:`Py_SetPath` or (at least)
   :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`.
 
@@ -930,7 +930,7 @@ directly accessed by end-users.
 When extracted, the embedded distribution is (almost) fully isolated from the
 user's system, including environment variables, system registry settings, and
 installed packages. The standard library is included as pre-compiled and
-optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python36.dll``,
+optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python37.dll``,
 ``python.exe`` and ``pythonw.exe`` are all provided. Tcl/tk (including all
 dependants, such as Idle), pip and the Python documentation are not included.
 


From webhook-mailer at python.org  Sun Jan 28 19:18:58 2018
From: webhook-mailer at python.org (Eric V. Smith)
Date: Mon, 29 Jan 2018 00:18:58 -0000
Subject: [Python-checkins] Fix trivial typo in test_dataclasses.py. (GH-5398)
Message-ID: <mailman.25.1517185139.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/7c99e931a90d468e8019a0409f16213b3aba7067
commit: 7c99e931a90d468e8019a0409f16213b3aba7067
branch: master
author: Eric V. Smith <ericvsmith at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T19:18:55-05:00
summary:

Fix trivial typo in test_dataclasses.py. (GH-5398)

files:
M Lib/test/test_dataclasses.py

diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index 53281f9dd9de..736bc490867b 100755
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -668,7 +668,7 @@ class Date:
         with self.assertRaisesRegex(TypeError, 'unpack'):
             x, y, z = Point3D(4, 5, 6)
 
-        # Maka sure another class with the same field names isn't
+        # Make sure another class with the same field names isn't
         #  equal.
         @dataclass
         class Point3Dv1:


From webhook-mailer at python.org  Sun Jan 28 19:25:08 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Mon, 29 Jan 2018 00:25:08 -0000
Subject: [Python-checkins] bpo-32650: Add native coroutine support to bdb
 when stepping over line (GH-5400)
Message-ID: <mailman.26.1517185511.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/46877024423e98d1b872bf308dacacd583327207
commit: 46877024423e98d1b872bf308dacacd583327207
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-29T02:25:05+02:00
summary:

bpo-32650: Add native coroutine support to bdb when stepping over line (GH-5400)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst
M Lib/bdb.py
M Lib/test/test_pdb.py

diff --git a/Lib/bdb.py b/Lib/bdb.py
index eb2fa4fb3e97..a10362a95e3a 100644
--- a/Lib/bdb.py
+++ b/Lib/bdb.py
@@ -3,7 +3,7 @@
 import fnmatch
 import sys
 import os
-from inspect import CO_GENERATOR
+from inspect import CO_GENERATOR, CO_COROUTINE
 
 __all__ = ["BdbQuit", "Bdb", "Breakpoint"]
 
@@ -127,7 +127,7 @@ def dispatch_call(self, frame, arg):
             # No need to trace this function
             return # None
         # Ignore call events in generator except when stepping.
-        if self.stopframe and frame.f_code.co_flags & CO_GENERATOR:
+        if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
             return self.trace_dispatch
         self.user_call(frame, arg)
         if self.quitting: raise BdbQuit
@@ -142,7 +142,7 @@ def dispatch_return(self, frame, arg):
         """
         if self.stop_here(frame) or frame == self.returnframe:
             # Ignore return events in generator except when stepping.
-            if self.stopframe and frame.f_code.co_flags & CO_GENERATOR:
+            if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
                 return self.trace_dispatch
             try:
                 self.frame_returning = frame
@@ -166,7 +166,7 @@ def dispatch_exception(self, frame, arg):
             # When stepping with next/until/return in a generator frame, skip
             # the internal StopIteration exception (with no traceback)
             # triggered by a subiterator run with the 'yield from' statement.
-            if not (frame.f_code.co_flags & CO_GENERATOR
+            if not (frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
                     and arg[0] is StopIteration and arg[2] is None):
                 self.user_exception(frame, arg)
                 if self.quitting: raise BdbQuit
@@ -175,7 +175,7 @@ def dispatch_exception(self, frame, arg):
         # next/until command at the last statement in the generator before the
         # exception.
         elif (self.stopframe and frame is not self.stopframe
-                and self.stopframe.f_code.co_flags & CO_GENERATOR
+                and self.stopframe.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
                 and arg[0] in (StopIteration, GeneratorExit)):
             self.user_exception(frame, arg)
             if self.quitting: raise BdbQuit
@@ -309,7 +309,7 @@ def set_next(self, frame):
 
     def set_return(self, frame):
         """Stop when returning from the given frame."""
-        if frame.f_code.co_flags & CO_GENERATOR:
+        if frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
             self._set_stopinfo(frame, None, -1)
         else:
             self._set_stopinfo(frame.f_back, frame)
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 85b47d533491..9ee1d9dacd37 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -727,6 +727,61 @@ def test_pdb_next_command_for_generator():
     finished
     """
 
+def test_pdb_next_command_for_coroutine():
+    """Testing skip unwindng stack on yield for coroutines for "next" command
+
+    >>> import asyncio
+
+    >>> async def test_coro():
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.get_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'step',
+    ...                    'next',
+    ...                    'next',
+    ...                    'next',
+    ...                    'step',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) step
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(2)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(3)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(4)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    Internal StopIteration
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Return--
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[2]>(3)test_main()->None
+    -> await test_coro()
+    (Pdb) continue
+    finished
+    """
+
 def test_pdb_return_command_for_generator():
     """Testing no unwindng stack on yield for generators
        for "return" command
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst
new file mode 100644
index 000000000000..f0b9e3ff459d
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst	
@@ -0,0 +1,2 @@
+Pdb and other debuggers dependent on bdb.py will correctly step over (next
+command) native coroutines. Patch by Pablo Galindo.


From webhook-mailer at python.org  Sun Jan 28 20:31:03 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 29 Jan 2018 01:31:03 -0000
Subject: [Python-checkins] bpo-32650 Add support for async generators and
 more test for coroutines in pdb (#5403)
Message-ID: <mailman.27.1517189464.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/c7ab581db216aeeb1c2aa7af2f2198d2b7516383
commit: c7ab581db216aeeb1c2aa7af2f2198d2b7516383
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2018-01-28T20:31:00-05:00
summary:

bpo-32650 Add support for async generators and more test for coroutines in pdb  (#5403)

files:
M Lib/bdb.py
M Lib/test/test_pdb.py

diff --git a/Lib/bdb.py b/Lib/bdb.py
index a10362a95e3a..c6a10359ac6f 100644
--- a/Lib/bdb.py
+++ b/Lib/bdb.py
@@ -3,10 +3,12 @@
 import fnmatch
 import sys
 import os
-from inspect import CO_GENERATOR, CO_COROUTINE
+from inspect import CO_GENERATOR, CO_COROUTINE, CO_ASYNC_GENERATOR
 
 __all__ = ["BdbQuit", "Bdb", "Breakpoint"]
 
+GENERATOR_AND_COROUTINE_FLAGS = CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR
+
 
 class BdbQuit(Exception):
     """Exception to give up completely."""
@@ -127,7 +129,7 @@ def dispatch_call(self, frame, arg):
             # No need to trace this function
             return # None
         # Ignore call events in generator except when stepping.
-        if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
+        if self.stopframe and frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
             return self.trace_dispatch
         self.user_call(frame, arg)
         if self.quitting: raise BdbQuit
@@ -142,7 +144,7 @@ def dispatch_return(self, frame, arg):
         """
         if self.stop_here(frame) or frame == self.returnframe:
             # Ignore return events in generator except when stepping.
-            if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
+            if self.stopframe and frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
                 return self.trace_dispatch
             try:
                 self.frame_returning = frame
@@ -166,7 +168,7 @@ def dispatch_exception(self, frame, arg):
             # When stepping with next/until/return in a generator frame, skip
             # the internal StopIteration exception (with no traceback)
             # triggered by a subiterator run with the 'yield from' statement.
-            if not (frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
+            if not (frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS
                     and arg[0] is StopIteration and arg[2] is None):
                 self.user_exception(frame, arg)
                 if self.quitting: raise BdbQuit
@@ -175,7 +177,7 @@ def dispatch_exception(self, frame, arg):
         # next/until command at the last statement in the generator before the
         # exception.
         elif (self.stopframe and frame is not self.stopframe
-                and self.stopframe.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
+                and self.stopframe.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS
                 and arg[0] in (StopIteration, GeneratorExit)):
             self.user_exception(frame, arg)
             if self.quitting: raise BdbQuit
@@ -309,7 +311,7 @@ def set_next(self, frame):
 
     def set_return(self, frame):
         """Stop when returning from the given frame."""
-        if frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
+        if frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
             self._set_stopinfo(frame, None, -1)
         else:
             self._set_stopinfo(frame.f_back, frame)
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 9ee1d9dacd37..47a669f2cea2 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -742,7 +742,7 @@ def test_pdb_next_command_for_coroutine():
     ...     await test_coro()
 
     >>> def test_function():
-    ...     loop = asyncio.get_event_loop()
+    ...     loop = asyncio.new_event_loop()
     ...     loop.run_until_complete(test_main())
     ...     loop.close()
     ...     print("finished")
@@ -837,6 +837,47 @@ def test_pdb_return_command_for_generator():
     finished
     """
 
+def test_pdb_return_command_for_coroutine():
+    """Testing no unwindng stack on yield for coroutines for "return" command
+
+    >>> import asyncio
+
+    >>> async def test_coro():
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.new_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'step',
+    ...                    'next',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[1]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) step
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[1]>(2)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[1]>(3)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) continue
+    finished
+    """
+
 def test_pdb_until_command_for_generator():
     """Testing no unwindng stack on yield for generators
        for "until" command if target breakpoing is not reached


From webhook-mailer at python.org  Sun Jan 28 20:48:33 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Mon, 29 Jan 2018 01:48:33 -0000
Subject: [Python-checkins] Document the error return of PyLong_As* APIs.
 (#5396)
Message-ID: <mailman.28.1517190515.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/f5b04a360e44aa9733f7a92dd66d2292d6c52955
commit: f5b04a360e44aa9733f7a92dd66d2292d6c52955
branch: master
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2018-01-28T17:48:31-08:00
summary:

Document the error return of PyLong_As* APIs. (#5396)

Document the error return of PyLong_As* APIs.

A frequent Python C API usage error is neglecting to check the return
value and/or PyErr_Occurred().

files:
M Doc/c-api/long.rst

diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst
index f50680b3d295..4f16b578eb59 100644
--- a/Doc/c-api/long.rst
+++ b/Doc/c-api/long.rst
@@ -10,6 +10,9 @@ Integer Objects
 
 All integers are implemented as "long" integer objects of arbitrary size.
 
+On error, most ``PyLong_As*`` APIs return ``(return type)-1`` which cannot be
+distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 .. c:type:: PyLongObject
 
    This subtype of :c:type:`PyObject` represents a Python integer object.
@@ -134,6 +137,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *obj* is out of range for a
    :c:type:`long`.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: long PyLong_AsLongAndOverflow(PyObject *obj, int *overflow)
 
@@ -146,6 +151,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    return ``-1``; otherwise, set *\*overflow* to ``0``.  If any other exception
    occurs set *\*overflow* to ``0`` and return ``-1`` as usual.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: long long PyLong_AsLongLong(PyObject *obj)
 
@@ -159,6 +166,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *obj* is out of range for a
    :c:type:`long`.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: long long PyLong_AsLongLongAndOverflow(PyObject *obj, int *overflow)
 
@@ -171,6 +180,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    and return ``-1``; otherwise, set *\*overflow* to ``0``.  If any other
    exception occurs set *\*overflow* to ``0`` and return ``-1`` as usual.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
    .. versionadded:: 3.2
 
 
@@ -186,6 +197,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`Py_ssize_t`.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: unsigned long PyLong_AsUnsignedLong(PyObject *pylong)
 
@@ -199,15 +212,25 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`unsigned long`.
 
+   Returns ``(unsigned long)-1`` on error.
+   Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: size_t PyLong_AsSize_t(PyObject *pylong)
 
+   .. index::
+      single: SIZE_MAX
+      single: OverflowError (built-in exception)
+
    Return a C :c:type:`size_t` representation of *pylong*.  *pylong* must be
    an instance of :c:type:`PyLongObject`.
 
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`size_t`.
 
+   Returns ``(size_t)-1`` on error.
+   Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: unsigned long long PyLong_AsUnsignedLongLong(PyObject *pylong)
 
@@ -220,6 +243,9 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for an
    :c:type:`unsigned long long`.
 
+   Returns ``(unsigned long long)-1`` on error.
+   Use :c:func:`PyErr_Occurred` to disambiguate.
+
    .. versionchanged:: 3.1
       A negative *pylong* now raises :exc:`OverflowError`, not :exc:`TypeError`.
 
@@ -233,6 +259,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    If the value of *obj* is out of range for an :c:type:`unsigned long`,
    return the reduction of that value modulo ``ULONG_MAX + 1``.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: unsigned long long PyLong_AsUnsignedLongLongMask(PyObject *obj)
 
@@ -243,6 +271,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    If the value of *obj* is out of range for an :c:type:`unsigned long long`,
    return the reduction of that value modulo ``PY_ULLONG_MAX + 1``.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: double PyLong_AsDouble(PyObject *pylong)
 
@@ -252,6 +282,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`double`.
 
+   Returns -1.0 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: void* PyLong_AsVoidPtr(PyObject *pylong)
 
@@ -259,3 +291,5 @@ All integers are implemented as "long" integer objects of arbitrary size.
    If *pylong* cannot be converted, an :exc:`OverflowError` will be raised.  This
    is only assured to produce a usable :c:type:`void` pointer for values created
    with :c:func:`PyLong_FromVoidPtr`.
+
+   Returns NULL on error.  Use :c:func:`PyErr_Occurred` to disambiguate.


From webhook-mailer at python.org  Sun Jan 28 20:56:13 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Mon, 29 Jan 2018 01:56:13 -0000
Subject: [Python-checkins] bpo-20104: Expose `posix_spawn` in the os module
 (GH-5109)
Message-ID: <mailman.29.1517190978.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/6c6ddf97c402709713d668d0ed53836a7749ba99
commit: 6c6ddf97c402709713d668d0ed53836a7749ba99
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-28T17:56:10-08:00
summary:

bpo-20104: Expose `posix_spawn` in the os module (GH-5109)

Add os.posix_spawn to wrap the low level POSIX API of the same name.

Contributed by Pablo Galindo.

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-06-01-14-53.bpo-20104.9DkKb8.rst
M Lib/test/test_posix.py
M Modules/clinic/posixmodule.c.h
M Modules/posixmodule.c
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
index a7f3d34f66db..8ada0e3ab3c7 100644
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -176,6 +176,23 @@ def test_fexecve(self):
         finally:
             os.close(fp)
 
+
+    @unittest.skipUnless(hasattr(os, 'posix_spawn'), "test needs os.posix_spawn")
+    def test_posix_spawn(self):
+        pid = posix.posix_spawn(sys.executable, [sys.executable, "-c", "pass"], os.environ,[])
+        self.assertEqual(os.waitpid(pid,0),(pid,0))
+
+
+    @unittest.skipUnless(hasattr(os, 'posix_spawn'), "test needs os.posix_spawn")
+    def test_posix_spawn_file_actions(self):
+        file_actions = []
+        file_actions.append((0,3,os.path.realpath(__file__),0,0))
+        file_actions.append((os.POSIX_SPAWN_CLOSE,2))
+        file_actions.append((os.POSIX_SPAWN_DUP2,1,4))
+        pid = posix.posix_spawn(sys.executable, [sys.executable, "-c", "pass"], os.environ, file_actions)
+        self.assertEqual(os.waitpid(pid,0),(pid,0))
+
+
     @unittest.skipUnless(hasattr(posix, 'waitid'), "test needs posix.waitid()")
     @unittest.skipUnless(hasattr(os, 'fork'), "test needs os.fork()")
     def test_waitid(self):
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-06-01-14-53.bpo-20104.9DkKb8.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-06-01-14-53.bpo-20104.9DkKb8.rst
new file mode 100644
index 000000000000..cb69f320dfb8
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-06-01-14-53.bpo-20104.9DkKb8.rst	
@@ -0,0 +1 @@
+Expose posix_spawn as a low level API in the os module.
diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h
index f432437e99ef..d6af15ffc2ca 100644
--- a/Modules/clinic/posixmodule.c.h
+++ b/Modules/clinic/posixmodule.c.h
@@ -1721,6 +1721,54 @@ os_execve(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *k
 
 #endif /* defined(HAVE_EXECV) */
 
+#if defined(HAVE_POSIX_SPAWN)
+
+PyDoc_STRVAR(os_posix_spawn__doc__,
+"posix_spawn($module, path, argv, env, file_actions=None, /)\n"
+"--\n"
+"\n"
+"Execute the program specified by path in a new process.\n"
+"\n"
+"  path\n"
+"    Path of executable file.\n"
+"  argv\n"
+"    Tuple or list of strings.\n"
+"  env\n"
+"    Dictionary of strings mapping to strings.\n"
+"  file_actions\n"
+"    FileActions object.");
+
+#define OS_POSIX_SPAWN_METHODDEF    \
+    {"posix_spawn", (PyCFunction)os_posix_spawn, METH_FASTCALL, os_posix_spawn__doc__},
+
+static PyObject *
+os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv,
+                    PyObject *env, PyObject *file_actions);
+
+static PyObject *
+os_posix_spawn(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
+{
+    PyObject *return_value = NULL;
+    path_t path = PATH_T_INITIALIZE("posix_spawn", "path", 0, 0);
+    PyObject *argv;
+    PyObject *env;
+    PyObject *file_actions = Py_None;
+
+    if (!_PyArg_ParseStack(args, nargs, "O&OO|O:posix_spawn",
+        path_converter, &path, &argv, &env, &file_actions)) {
+        goto exit;
+    }
+    return_value = os_posix_spawn_impl(module, &path, argv, env, file_actions);
+
+exit:
+    /* Cleanup for path */
+    path_cleanup(&path);
+
+    return return_value;
+}
+
+#endif /* defined(HAVE_POSIX_SPAWN) */
+
 #if (defined(HAVE_SPAWNV) || defined(HAVE_WSPAWNV))
 
 PyDoc_STRVAR(os_spawnv__doc__,
@@ -6137,6 +6185,10 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject
     #define OS_EXECVE_METHODDEF
 #endif /* !defined(OS_EXECVE_METHODDEF) */
 
+#ifndef OS_POSIX_SPAWN_METHODDEF
+    #define OS_POSIX_SPAWN_METHODDEF
+#endif /* !defined(OS_POSIX_SPAWN_METHODDEF) */
+
 #ifndef OS_SPAWNV_METHODDEF
     #define OS_SPAWNV_METHODDEF
 #endif /* !defined(OS_SPAWNV_METHODDEF) */
@@ -6528,4 +6580,4 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject
 #ifndef OS_GETRANDOM_METHODDEF
     #define OS_GETRANDOM_METHODDEF
 #endif /* !defined(OS_GETRANDOM_METHODDEF) */
-/*[clinic end generated code: output=06ace805893aa10c input=a9049054013a1b77]*/
+/*[clinic end generated code: output=8e5d4a01257b6292 input=a9049054013a1b77]*/
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index ceea85559c85..4c0392e22c3b 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -176,6 +176,7 @@ corresponding Unix manual entries for more information on calls.");
 #else
 /* Unix functions that the configure script doesn't check for */
 #define HAVE_EXECV      1
+#define HAVE_POSIX_SPAWN 1
 #define HAVE_FORK       1
 #if defined(__USLC__) && defined(__SCO_VERSION__)       /* SCO UDK Compiler */
 #define HAVE_FORK1      1
@@ -246,6 +247,10 @@ extern int lstat(const char *, struct stat *);
 
 #endif /* !_MSC_VER */
 
+#ifdef HAVE_POSIX_SPAWN
+#include <spawn.h>
+#endif
+
 #ifdef HAVE_UTIME_H
 #include <utime.h>
 #endif /* HAVE_UTIME_H */
@@ -5097,6 +5102,194 @@ os_execve_impl(PyObject *module, path_t *path, PyObject *argv, PyObject *env)
 
 #endif /* HAVE_EXECV */
 
+#ifdef HAVE_POSIX_SPAWN
+
+enum posix_spawn_file_actions_identifier {
+    POSIX_SPAWN_OPEN,
+    POSIX_SPAWN_CLOSE,
+    POSIX_SPAWN_DUP2
+};
+
+/*[clinic input]
+
+os.posix_spawn
+    path: path_t
+        Path of executable file.
+    argv: object
+        Tuple or list of strings.
+    env: object
+        Dictionary of strings mapping to strings.
+    file_actions: object = None
+        FileActions object.
+    /
+
+Execute the program specified by path in a new process.
+[clinic start generated code]*/
+
+static PyObject *
+os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv,
+                    PyObject *env, PyObject *file_actions)
+/*[clinic end generated code: output=d023521f541c709c input=0ec9f1cfdc890be5]*/
+{
+    EXECV_CHAR **argvlist = NULL;
+    EXECV_CHAR **envlist;
+    Py_ssize_t argc, envc;
+
+    /* posix_spawn has three arguments: (path, argv, env), where
+   argv is a list or tuple of strings and env is a dictionary
+       like posix.environ. */
+
+    if (!PySequence_Check(argv)){
+        PyErr_SetString(PyExc_TypeError,
+                        "posix_spawn: argv must be a tuple or list");
+        goto fail;
+    }
+    argc = PySequence_Size(argv);
+    if (argc < 1) {
+        PyErr_SetString(PyExc_ValueError, "posix_spawn: argv must not be empty");
+        return NULL;
+    }
+
+    if (!PyMapping_Check(env)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "posix_spawn: environment must be a mapping object");
+        goto fail;
+    }
+
+    argvlist = parse_arglist(argv, &argc);
+    if (argvlist == NULL) {
+        goto fail;
+    }
+    if (!argvlist[0][0]) {
+        PyErr_SetString(PyExc_ValueError,
+            "posix_spawn: argv first element cannot be empty");
+        goto fail;
+    }
+
+    envlist = parse_envlist(env, &envc);
+    if (envlist == NULL)
+        goto fail;
+
+    pid_t pid;
+    posix_spawn_file_actions_t *file_actionsp = NULL;
+    if (file_actions  != NULL && file_actions != Py_None){
+        posix_spawn_file_actions_t _file_actions;
+        if(posix_spawn_file_actions_init(&_file_actions) != 0){
+            PyErr_SetString(PyExc_TypeError,
+                            "Error initializing file actions");
+            goto fail;
+        }
+
+
+        file_actionsp = &_file_actions;
+
+
+        PyObject* seq = PySequence_Fast(file_actions, "file_actions must be a sequence");
+        if(seq == NULL){
+            goto fail;
+        }
+        PyObject* file_actions_obj;
+        PyObject* mode_obj;
+
+        for (int i = 0; i < PySequence_Fast_GET_SIZE(seq); ++i) {
+            file_actions_obj = PySequence_Fast_GET_ITEM(seq, i);
+
+            if(!PySequence_Check(file_actions_obj) | !PySequence_Size(file_actions_obj)){
+                PyErr_SetString(PyExc_TypeError,"Each file_action element must be a non empty sequence");
+                goto fail;
+            }
+
+
+            mode_obj = PySequence_Fast_GET_ITEM(file_actions_obj, 0);
+            int mode = PyLong_AsLong(mode_obj);
+
+            /* Populate the file_actions object */
+
+            switch(mode) {
+
+                case POSIX_SPAWN_OPEN:
+                    if(PySequence_Size(file_actions_obj) != 5){
+                        PyErr_SetString(PyExc_TypeError,"A open file_action object must have 5 elements");
+                        goto fail;
+                    }
+
+                    long open_fd = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 1));
+                    if(PyErr_Occurred()) {
+                        goto fail;
+                    }
+                    const char* open_path = PyUnicode_AsUTF8(PySequence_GetItem(file_actions_obj, 2));
+                    if(open_path == NULL){
+                        goto fail;
+                    }
+                    long open_oflag = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 3));
+                    if(PyErr_Occurred()) {
+                        goto fail;
+                    }
+                    long open_mode = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 4));
+                    if(PyErr_Occurred()) {
+                        goto fail;
+                    }
+                    posix_spawn_file_actions_addopen(file_actionsp, open_fd, open_path, open_oflag, open_mode);
+                    break;
+
+                case POSIX_SPAWN_CLOSE:
+                    if(PySequence_Size(file_actions_obj) != 2){
+                        PyErr_SetString(PyExc_TypeError,"A close file_action object must have 2 elements");
+                        goto fail;
+                    }
+
+                    long close_fd = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 1));
+                    if(PyErr_Occurred()) {
+                        goto fail;
+                    }
+                    posix_spawn_file_actions_addclose(file_actionsp, close_fd);
+                    break;
+
+                case POSIX_SPAWN_DUP2:
+                    if(PySequence_Size(file_actions_obj) != 3){
+                        PyErr_SetString(PyExc_TypeError,"A dup2 file_action object must have 3 elements");
+                        goto fail;
+                    }
+
+                    long fd1 = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 1));
+                    if(PyErr_Occurred()) {
+                        goto fail;
+                    }
+                    long fd2 = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 2));
+                    if(PyErr_Occurred()) {
+                        goto fail;
+                    }
+                    posix_spawn_file_actions_adddup2(file_actionsp, fd1, fd2);
+                    break;
+
+                default:
+                    PyErr_SetString(PyExc_TypeError,"Unknown file_actions identifier");
+                    goto fail;
+            }
+        }
+        Py_DECREF(seq);
+}
+
+    _Py_BEGIN_SUPPRESS_IPH
+        posix_spawn(&pid, path->narrow, file_actionsp, NULL, argvlist, envlist);
+        return PyLong_FromPid(pid);
+    _Py_END_SUPPRESS_IPH
+
+    path_error(path);
+
+    free_string_array(envlist, envc);
+
+fail:
+
+    if (argvlist) {
+        free_string_array(argvlist, argc);
+    }
+    return NULL;
+
+
+}
+#endif /* HAVE_POSIX_SPAWN */
+
 
 #if defined(HAVE_SPAWNV) || defined(HAVE_WSPAWNV)
 /*[clinic input]
@@ -5189,7 +5382,6 @@ os_spawnv_impl(PyObject *module, int mode, path_t *path, PyObject *argv)
         return Py_BuildValue(_Py_PARSE_INTPTR, spawnval);
 }
 
-
 /*[clinic input]
 os.spawnve
 
@@ -12610,6 +12802,7 @@ static PyMethodDef posix_methods[] = {
     OS_NICE_METHODDEF
     OS_GETPRIORITY_METHODDEF
     OS_SETPRIORITY_METHODDEF
+    OS_POSIX_SPAWN_METHODDEF
 #ifdef HAVE_READLINK
     {"readlink",        (PyCFunction)posix_readlink,
                         METH_VARARGS | METH_KEYWORDS,
@@ -13164,6 +13357,13 @@ all_ins(PyObject *m)
     if (PyModule_AddIntConstant(m, "RWF_NOWAIT", RWF_NOWAIT)) return -1;
 #endif
 
+/* constants for posix_spawn */
+#ifdef HAVE_POSIX_SPAWN
+    if (PyModule_AddIntConstant(m, "POSIX_SPAWN_OPEN", POSIX_SPAWN_OPEN)) return -1;
+    if (PyModule_AddIntConstant(m, "POSIX_SPAWN_CLOSE", POSIX_SPAWN_CLOSE)) return -1;
+    if (PyModule_AddIntConstant(m, "POSIX_SPAWN_DUP2", POSIX_SPAWN_DUP2)) return -1;
+#endif
+
 #ifdef HAVE_SPAWNV
     if (PyModule_AddIntConstant(m, "P_WAIT", _P_WAIT)) return -1;
     if (PyModule_AddIntConstant(m, "P_NOWAIT", _P_NOWAIT)) return -1;
diff --git a/configure b/configure
index 6ce0728af119..f94d16bf0b0c 100755
--- a/configure
+++ b/configure
@@ -11197,7 +11197,7 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \
  initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
  memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
- posix_fallocate posix_fadvise pread preadv preadv2 \
+ posix_fallocate posix_fadvise posix_spawn pread preadv preadv2 \
  pthread_init pthread_kill putenv pwrite pwritev pwritev2 readlink readlinkat readv realpath renameat \
  sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \
  setgid sethostname \
diff --git a/configure.ac b/configure.ac
index b901f807657c..65248630b6e1 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3431,7 +3431,7 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \
  initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
  memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
- posix_fallocate posix_fadvise pread preadv preadv2 \
+ posix_fallocate posix_fadvise posix_spawn pread preadv preadv2 \
  pthread_init pthread_kill putenv pwrite pwritev pwritev2 readlink readlinkat readv realpath renameat \
  sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \
  setgid sethostname \
diff --git a/pyconfig.h.in b/pyconfig.h.in
index 27fe136a9764..a18e3ca854e6 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -707,6 +707,9 @@
 /* Define to 1 if you have the `posix_fallocate' function. */
 #undef HAVE_POSIX_FALLOCATE
 
+/* Define to 1 if you have the `posix_spawn' function. */
+#undef HAVE_POSIX_SPAWN
+
 /* Define to 1 if you have the `pread' function. */
 #undef HAVE_PREAD
 


From webhook-mailer at python.org  Sun Jan 28 22:03:26 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 29 Jan 2018 03:03:26 -0000
Subject: [Python-checkins] bpo-28685: Optimize sorted() list.sort() with
 type-specialized comparisons (#582)
Message-ID: <mailman.30.1517195007.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/1e34da49ef22004ca25c517b3f07c6d25f083ece
commit: 1e34da49ef22004ca25c517b3f07c6d25f083ece
branch: master
author: embg <elliot.gorokhovsky at gmail.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-28T19:03:23-08:00
summary:

bpo-28685: Optimize sorted() list.sort() with type-specialized comparisons (#582)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-15-09-33.bpo-28685.cHThLM.rst
M Lib/test/test_sort.py
M Misc/ACKS
M Objects/listobject.c
M Objects/listsort.txt

diff --git a/Lib/test/test_sort.py b/Lib/test/test_sort.py
index 98ccab5c3930..f2f53cb1a72f 100644
--- a/Lib/test/test_sort.py
+++ b/Lib/test/test_sort.py
@@ -260,6 +260,120 @@ def my_cmp_reversed(x, y):
         self.assertEqual(data, copy2)
 
 #==============================================================================
+def check_against_PyObject_RichCompareBool(self, L):
+    ## The idea here is to exploit the fact that unsafe_tuple_compare uses
+    ## PyObject_RichCompareBool for the second elements of tuples. So we have,
+    ## for (most) L, sorted(L) == [y[1] for y in sorted([(0,x) for x in L])]
+    ## This will work as long as __eq__ => not __lt__ for all the objects in L,
+    ## which holds for all the types used below.
+    ##
+    ## Testing this way ensures that the optimized implementation remains consistent
+    ## with the naive implementation, even if changes are made to any of the
+    ## richcompares.
+    ##
+    ## This function tests sorting for three lists (it randomly shuffles each one):
+    ##                        1. L
+    ##                        2. [(x,) for x in L]
+    ##                        3. [((x,),) for x in L]
+
+    random.seed(0)
+    random.shuffle(L)
+    L_1 = L[:]
+    L_2 = [(x,) for x in L]
+    L_3 = [((x,),) for x in L]
+    for L in [L_1, L_2, L_3]:
+        optimized = sorted(L)
+        reference = [y[1] for y in sorted([(0,x) for x in L])]
+        for (opt, ref) in zip(optimized, reference):
+            self.assertIs(opt, ref)
+            #note: not assertEqual! We want to ensure *identical* behavior.
+
+class TestOptimizedCompares(unittest.TestCase):
+    def test_safe_object_compare(self):
+        heterogeneous_lists = [[0, 'foo'],
+                               [0.0, 'foo'],
+                               [('foo',), 'foo']]
+        for L in heterogeneous_lists:
+            self.assertRaises(TypeError, L.sort)
+            self.assertRaises(TypeError, [(x,) for x in L].sort)
+            self.assertRaises(TypeError, [((x,),) for x in L].sort)
+
+        float_int_lists = [[1,1.1],
+                           [1<<70,1.1],
+                           [1.1,1],
+                           [1.1,1<<70]]
+        for L in float_int_lists:
+            check_against_PyObject_RichCompareBool(self, L)
+
+    def test_unsafe_object_compare(self):
+
+        # This test is by ppperry. It ensures that unsafe_object_compare is
+        # verifying ms->key_richcompare == tp->richcompare before comparing.
+
+        class WackyComparator(int):
+            def __lt__(self, other):
+                elem.__class__ = WackyList2
+                return int.__lt__(self, other)
+
+        class WackyList1(list):
+            pass
+
+        class WackyList2(list):
+            def __lt__(self, other):
+                raise ValueError
+
+        L = [WackyList1([WackyComparator(i), i]) for i in range(10)]
+        elem = L[-1]
+        with self.assertRaises(ValueError):
+            L.sort()
+
+        L = [WackyList1([WackyComparator(i), i]) for i in range(10)]
+        elem = L[-1]
+        with self.assertRaises(ValueError):
+            [(x,) for x in L].sort()
+
+        # The following test is also by ppperry. It ensures that
+        # unsafe_object_compare handles Py_NotImplemented appropriately.
+        class PointlessComparator:
+            def __lt__(self, other):
+                return NotImplemented
+        L = [PointlessComparator(), PointlessComparator()]
+        self.assertRaises(TypeError, L.sort)
+        self.assertRaises(TypeError, [(x,) for x in L].sort)
+
+        # The following tests go through various types that would trigger
+        # ms->key_compare = unsafe_object_compare
+        lists = [list(range(100)) + [(1<<70)],
+                 [str(x) for x in range(100)] + ['\uffff'],
+                 [bytes(x) for x in range(100)],
+                 [cmp_to_key(lambda x,y: x<y)(x) for x in range(100)]]
+        for L in lists:
+            check_against_PyObject_RichCompareBool(self, L)
+
+    def test_unsafe_latin_compare(self):
+        check_against_PyObject_RichCompareBool(self, [str(x) for
+                                                      x in range(100)])
+
+    def test_unsafe_long_compare(self):
+        check_against_PyObject_RichCompareBool(self, [x for
+                                                      x in range(100)])
+
+    def test_unsafe_float_compare(self):
+        check_against_PyObject_RichCompareBool(self, [float(x) for
+                                                      x in range(100)])
+
+    def test_unsafe_tuple_compare(self):
+        # This test was suggested by Tim Peters. It verifies that the tuple
+        # comparison respects the current tuple compare semantics, which do not
+        # guarantee that x < x <=> (x,) < (x,)
+        #
+        # Note that we don't have to put anything in tuples here, because
+        # the check function does a tuple test automatically.
+
+        check_against_PyObject_RichCompareBool(self, [float('nan')]*100)
+        check_against_PyObject_RichCompareBool(self, [float('nan') for
+                                                      _ in range(100)])
+#==============================================================================
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Misc/ACKS b/Misc/ACKS
index 20210e8089b4..bfcdd33b8ed7 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -554,6 +554,7 @@ Tiago Gon?alves
 Chris Gonnerman
 Shelley Gooch
 David Goodger
+Elliot Gorokhovsky
 Hans de Graaff
 Tim Graham
 Kim Gr?sman
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-15-09-33.bpo-28685.cHThLM.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-15-09-33.bpo-28685.cHThLM.rst
new file mode 100644
index 000000000000..ccc3c0857bc0
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-15-09-33.bpo-28685.cHThLM.rst	
@@ -0,0 +1,2 @@
+Optimize list.sort() and sorted() by using type specialized comparisons when
+possible.
diff --git a/Objects/listobject.c b/Objects/listobject.c
index 8794e37364a8..9e321371aae9 100644
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -1081,11 +1081,12 @@ sortslice_advance(sortslice *slice, Py_ssize_t n)
         slice->values += n;
 }
 
-/* Comparison function: PyObject_RichCompareBool with Py_LT.
+/* Comparison function: ms->key_compare, which is set at run-time in
+ * listsort_impl to optimize for various special cases.
  * Returns -1 on error, 1 if x < y, 0 if x >= y.
  */
 
-#define ISLT(X, Y) (PyObject_RichCompareBool(X, Y, Py_LT))
+#define ISLT(X, Y) (*(ms->key_compare))(X, Y, ms)
 
 /* Compare X to Y via "<".  Goto "fail" if the comparison raises an
    error.  Else "k" is set to true iff X<Y, and an "if (k)" block is
@@ -1094,6 +1095,75 @@ sortslice_advance(sortslice *slice, Py_ssize_t n)
 #define IFLT(X, Y) if ((k = ISLT(X, Y)) < 0) goto fail;  \
            if (k)
 
+/* The maximum number of entries in a MergeState's pending-runs stack.
+ * This is enough to sort arrays of size up to about
+ *     32 * phi ** MAX_MERGE_PENDING
+ * where phi ~= 1.618.  85 is ridiculouslylarge enough, good for an array
+ * with 2**64 elements.
+ */
+#define MAX_MERGE_PENDING 85
+
+/* When we get into galloping mode, we stay there until both runs win less
+ * often than MIN_GALLOP consecutive times.  See listsort.txt for more info.
+ */
+#define MIN_GALLOP 7
+
+/* Avoid malloc for small temp arrays. */
+#define MERGESTATE_TEMP_SIZE 256
+
+/* One MergeState exists on the stack per invocation of mergesort.  It's just
+ * a convenient way to pass state around among the helper functions.
+ */
+struct s_slice {
+    sortslice base;
+    Py_ssize_t len;
+};
+
+typedef struct s_MergeState MergeState;
+struct s_MergeState {
+    /* This controls when we get *into* galloping mode.  It's initialized
+     * to MIN_GALLOP.  merge_lo and merge_hi tend to nudge it higher for
+     * random data, and lower for highly structured data.
+     */
+    Py_ssize_t min_gallop;
+
+    /* 'a' is temp storage to help with merges.  It contains room for
+     * alloced entries.
+     */
+    sortslice a;        /* may point to temparray below */
+    Py_ssize_t alloced;
+
+    /* A stack of n pending runs yet to be merged.  Run #i starts at
+     * address base[i] and extends for len[i] elements.  It's always
+     * true (so long as the indices are in bounds) that
+     *
+     *     pending[i].base + pending[i].len == pending[i+1].base
+     *
+     * so we could cut the storage for this, but it's a minor amount,
+     * and keeping all the info explicit simplifies the code.
+     */
+    int n;
+    struct s_slice pending[MAX_MERGE_PENDING];
+
+    /* 'a' points to this when possible, rather than muck with malloc. */
+    PyObject *temparray[MERGESTATE_TEMP_SIZE];
+
+    /* This is the function we will use to compare two keys,
+     * even when none of our special cases apply and we have to use
+     * safe_object_compare. */
+    int (*key_compare)(PyObject *, PyObject *, MergeState *);
+
+    /* This function is used by unsafe_object_compare to optimize comparisons
+     * when we know our list is type-homogeneous but we can't assume anything else.
+     * In the pre-sort check it is set equal to key->ob_type->tp_richcompare */
+    PyObject *(*key_richcompare)(PyObject *, PyObject *, int);
+
+    /* This function is used by unsafe_tuple_compare to compare the first elements
+     * of tuples. It may be set to safe_object_compare, but the idea is that hopefully
+     * we can assume more, and use one of the special-case compares. */
+    int (*tuple_elem_compare)(PyObject *, PyObject *, MergeState *);
+};
+
 /* binarysort is the best method for sorting small arrays: it does
    few compares, but can do data movement quadratic in the number of
    elements.
@@ -1106,7 +1176,7 @@ sortslice_advance(sortslice *slice, Py_ssize_t n)
    the input (nothing is lost or duplicated).
 */
 static int
-binarysort(sortslice lo, PyObject **hi, PyObject **start)
+binarysort(MergeState *ms, sortslice lo, PyObject **hi, PyObject **start)
 {
     Py_ssize_t k;
     PyObject **l, **p, **r;
@@ -1180,7 +1250,7 @@ elements to get out of order).
 Returns -1 in case of error.
 */
 static Py_ssize_t
-count_run(PyObject **lo, PyObject **hi, int *descending)
+count_run(MergeState *ms, PyObject **lo, PyObject **hi, int *descending)
 {
     Py_ssize_t k;
     Py_ssize_t n;
@@ -1235,7 +1305,7 @@ key, and the last n-k should follow key.
 Returns -1 on error.  See listsort.txt for info on the method.
 */
 static Py_ssize_t
-gallop_left(PyObject *key, PyObject **a, Py_ssize_t n, Py_ssize_t hint)
+gallop_left(MergeState *ms, PyObject *key, PyObject **a, Py_ssize_t n, Py_ssize_t hint)
 {
     Py_ssize_t ofs;
     Py_ssize_t lastofs;
@@ -1326,7 +1396,7 @@ we're sticking to "<" comparisons that it's much harder to follow if
 written as one routine with yet another "left or right?" flag.
 */
 static Py_ssize_t
-gallop_right(PyObject *key, PyObject **a, Py_ssize_t n, Py_ssize_t hint)
+gallop_right(MergeState *ms, PyObject *key, PyObject **a, Py_ssize_t n, Py_ssize_t hint)
 {
     Py_ssize_t ofs;
     Py_ssize_t lastofs;
@@ -1402,59 +1472,6 @@ gallop_right(PyObject *key, PyObject **a, Py_ssize_t n, Py_ssize_t hint)
     return -1;
 }
 
-/* The maximum number of entries in a MergeState's pending-runs stack.
- * This is enough to sort arrays of size up to about
- *     32 * phi ** MAX_MERGE_PENDING
- * where phi ~= 1.618.  85 is ridiculouslylarge enough, good for an array
- * with 2**64 elements.
- */
-#define MAX_MERGE_PENDING 85
-
-/* When we get into galloping mode, we stay there until both runs win less
- * often than MIN_GALLOP consecutive times.  See listsort.txt for more info.
- */
-#define MIN_GALLOP 7
-
-/* Avoid malloc for small temp arrays. */
-#define MERGESTATE_TEMP_SIZE 256
-
-/* One MergeState exists on the stack per invocation of mergesort.  It's just
- * a convenient way to pass state around among the helper functions.
- */
-struct s_slice {
-    sortslice base;
-    Py_ssize_t len;
-};
-
-typedef struct s_MergeState {
-    /* This controls when we get *into* galloping mode.  It's initialized
-     * to MIN_GALLOP.  merge_lo and merge_hi tend to nudge it higher for
-     * random data, and lower for highly structured data.
-     */
-    Py_ssize_t min_gallop;
-
-    /* 'a' is temp storage to help with merges.  It contains room for
-     * alloced entries.
-     */
-    sortslice a;        /* may point to temparray below */
-    Py_ssize_t alloced;
-
-    /* A stack of n pending runs yet to be merged.  Run #i starts at
-     * address base[i] and extends for len[i] elements.  It's always
-     * true (so long as the indices are in bounds) that
-     *
-     *     pending[i].base + pending[i].len == pending[i+1].base
-     *
-     * so we could cut the storage for this, but it's a minor amount,
-     * and keeping all the info explicit simplifies the code.
-     */
-    int n;
-    struct s_slice pending[MAX_MERGE_PENDING];
-
-    /* 'a' points to this when possible, rather than muck with malloc. */
-    PyObject *temparray[MERGESTATE_TEMP_SIZE];
-} MergeState;
-
 /* Conceptually a MergeState's constructor. */
 static void
 merge_init(MergeState *ms, Py_ssize_t list_size, int has_keyfunc)
@@ -1514,11 +1531,11 @@ merge_getmem(MergeState *ms, Py_ssize_t need)
      * we don't care what's in the block.
      */
     merge_freemem(ms);
-    if ((size_t)need > PY_SSIZE_T_MAX / sizeof(PyObject*) / multiplier) {
+    if ((size_t)need > PY_SSIZE_T_MAX / sizeof(PyObject *) / multiplier) {
         PyErr_NoMemory();
         return -1;
     }
-    ms->a.keys = (PyObject**)PyMem_Malloc(multiplier * need
+    ms->a.keys = (PyObject **)PyMem_Malloc(multiplier * need
                                           * sizeof(PyObject *));
     if (ms->a.keys != NULL) {
         ms->alloced = need;
@@ -1607,7 +1624,7 @@ merge_lo(MergeState *ms, sortslice ssa, Py_ssize_t na,
             assert(na > 1 && nb > 0);
             min_gallop -= min_gallop > 1;
             ms->min_gallop = min_gallop;
-            k = gallop_right(ssb.keys[0], ssa.keys, na, 0);
+            k = gallop_right(ms, ssb.keys[0], ssa.keys, na, 0);
             acount = k;
             if (k) {
                 if (k < 0)
@@ -1630,7 +1647,7 @@ merge_lo(MergeState *ms, sortslice ssa, Py_ssize_t na,
             if (nb == 0)
                 goto Succeed;
 
-            k = gallop_left(ssa.keys[0], ssb.keys, nb, 0);
+            k = gallop_left(ms, ssa.keys[0], ssb.keys, nb, 0);
             bcount = k;
             if (k) {
                 if (k < 0)
@@ -1745,7 +1762,7 @@ merge_hi(MergeState *ms, sortslice ssa, Py_ssize_t na,
             assert(na > 0 && nb > 1);
             min_gallop -= min_gallop > 1;
             ms->min_gallop = min_gallop;
-            k = gallop_right(ssb.keys[0], basea.keys, na, na-1);
+            k = gallop_right(ms, ssb.keys[0], basea.keys, na, na-1);
             if (k < 0)
                 goto Fail;
             k = na - k;
@@ -1763,7 +1780,7 @@ merge_hi(MergeState *ms, sortslice ssa, Py_ssize_t na,
             if (nb == 1)
                 goto CopyA;
 
-            k = gallop_left(ssa.keys[0], baseb.keys, nb, nb-1);
+            k = gallop_left(ms, ssa.keys[0], baseb.keys, nb, nb-1);
             if (k < 0)
                 goto Fail;
             k = nb - k;
@@ -1840,7 +1857,7 @@ merge_at(MergeState *ms, Py_ssize_t i)
     /* Where does b start in a?  Elements in a before that can be
      * ignored (already in place).
      */
-    k = gallop_right(*ssb.keys, ssa.keys, na, 0);
+    k = gallop_right(ms, *ssb.keys, ssa.keys, na, 0);
     if (k < 0)
         return -1;
     sortslice_advance(&ssa, k);
@@ -1851,7 +1868,7 @@ merge_at(MergeState *ms, Py_ssize_t i)
     /* Where does a end in b?  Elements in b after that can be
      * ignored (already in place).
      */
-    nb = gallop_left(ssa.keys[na-1], ssb.keys, nb, nb-1);
+    nb = gallop_left(ms, ssa.keys[na-1], ssb.keys, nb, nb-1);
     if (nb <= 0)
         return nb;
 
@@ -1890,8 +1907,8 @@ merge_collapse(MergeState *ms)
                 return -1;
         }
         else if (p[n].len <= p[n+1].len) {
-                 if (merge_at(ms, n) < 0)
-                        return -1;
+            if (merge_at(ms, n) < 0)
+                return -1;
         }
         else
             break;
@@ -1951,6 +1968,170 @@ reverse_sortslice(sortslice *s, Py_ssize_t n)
         reverse_slice(s->values, &s->values[n]);
 }
 
+/* Here we define custom comparison functions to optimize for the cases one commonly
+ * encounters in practice: homogeneous lists, often of one of the basic types. */
+
+/* This struct holds the comparison function and helper functions
+ * selected in the pre-sort check. */
+
+/* These are the special case compare functions.
+ * ms->key_compare will always point to one of these: */
+
+/* Heterogeneous compare: default, always safe to fall back on. */
+static int
+safe_object_compare(PyObject *v, PyObject *w, MergeState *ms)
+{
+    /* No assumptions necessary! */
+    return PyObject_RichCompareBool(v, w, Py_LT);
+}
+
+/* Homogeneous compare: safe for any two compareable objects of the same type.
+ * (ms->key_richcompare is set to ob_type->tp_richcompare in the
+ *  pre-sort check.)
+ */
+static int
+unsafe_object_compare(PyObject *v, PyObject *w, MergeState *ms)
+{
+    PyObject *res_obj; int res;
+
+    /* No assumptions, because we check first: */
+    if (v->ob_type->tp_richcompare != ms->key_richcompare)
+        return PyObject_RichCompareBool(v, w, Py_LT);
+
+    assert(ms->key_richcompare != NULL);
+    res_obj = (*(ms->key_richcompare))(v, w, Py_LT);
+
+    if (res_obj == Py_NotImplemented) {
+        Py_DECREF(res_obj);
+        return PyObject_RichCompareBool(v, w, Py_LT);
+    }
+    if (res_obj == NULL)
+        return -1;
+
+    if (PyBool_Check(res_obj)) {
+        res = (res_obj == Py_True);
+    }
+    else {
+        res = PyObject_IsTrue(res_obj);
+    }
+    Py_DECREF(res_obj);
+
+    /* Note that we can't assert
+     *     res == PyObject_RichCompareBool(v, w, Py_LT);
+     * because of evil compare functions like this:
+     *     lambda a, b:  int(random.random() * 3) - 1)
+     * (which is actually in test_sort.py) */
+    return res;
+}
+
+/* Latin string compare: safe for any two latin (one byte per char) strings. */
+static int
+unsafe_latin_compare(PyObject *v, PyObject *w, MergeState *ms)
+{
+    int len, res;
+
+    /* Modified from Objects/unicodeobject.c:unicode_compare, assuming: */
+    assert(v->ob_type == w->ob_type);
+    assert(v->ob_type == &PyUnicode_Type);
+    assert(PyUnicode_KIND(v) == PyUnicode_KIND(w));
+    assert(PyUnicode_KIND(v) == PyUnicode_1BYTE_KIND);
+
+    len = Py_MIN(PyUnicode_GET_LENGTH(v), PyUnicode_GET_LENGTH(w));
+    res = memcmp(PyUnicode_DATA(v), PyUnicode_DATA(w), len);
+
+    res = (res != 0 ?
+           res < 0 :
+           PyUnicode_GET_LENGTH(v) < PyUnicode_GET_LENGTH(w));
+
+    assert(res == PyObject_RichCompareBool(v, w, Py_LT));;
+    return res;
+}
+
+/* Bounded int compare: compare any two longs that fit in a single machine word. */
+static int
+unsafe_long_compare(PyObject *v, PyObject *w, MergeState *ms)
+{
+    PyLongObject *vl, *wl; sdigit v0, w0; int res;
+
+    /* Modified from Objects/longobject.c:long_compare, assuming: */
+    assert(v->ob_type == w->ob_type);
+    assert(v->ob_type == &PyLong_Type);
+    assert(Py_ABS(Py_SIZE(v)) <= 1);
+    assert(Py_ABS(Py_SIZE(w)) <= 1);
+
+    vl = (PyLongObject*)v;
+    wl = (PyLongObject*)w;
+
+    v0 = Py_SIZE(vl) == 0 ? 0 : (sdigit)vl->ob_digit[0];
+    w0 = Py_SIZE(wl) == 0 ? 0 : (sdigit)wl->ob_digit[0];
+
+    if (Py_SIZE(vl) < 0)
+        v0 = -v0;
+    if (Py_SIZE(wl) < 0)
+        w0 = -w0;
+
+    res = v0 < w0;
+    assert(res == PyObject_RichCompareBool(v, w, Py_LT));
+    return res;
+}
+
+/* Float compare: compare any two floats. */
+static int
+unsafe_float_compare(PyObject *v, PyObject *w, MergeState *ms)
+{
+    int res;
+
+    /* Modified from Objects/floatobject.c:float_richcompare, assuming: */
+    assert(v->ob_type == w->ob_type);
+    assert(v->ob_type == &PyFloat_Type);
+
+    res = PyFloat_AS_DOUBLE(v) < PyFloat_AS_DOUBLE(w);
+    assert(res == PyObject_RichCompareBool(v, w, Py_LT));
+    return res;
+}
+
+/* Tuple compare: compare *any* two tuples, using
+ * ms->tuple_elem_compare to compare the first elements, which is set
+ * using the same pre-sort check as we use for ms->key_compare,
+ * but run on the list [x[0] for x in L]. This allows us to optimize compares
+ * on two levels (as long as [x[0] for x in L] is type-homogeneous.) The idea is
+ * that most tuple compares don't involve x[1:]. */
+static int
+unsafe_tuple_compare(PyObject *v, PyObject *w, MergeState *ms)
+{
+    PyTupleObject *vt, *wt;
+    Py_ssize_t i, vlen, wlen;
+    int k;
+
+    /* Modified from Objects/tupleobject.c:tuplerichcompare, assuming: */
+    assert(v->ob_type == w->ob_type);
+    assert(v->ob_type == &PyTuple_Type);
+    assert(Py_SIZE(v) > 0);
+    assert(Py_SIZE(w) > 0);
+
+    vt = (PyTupleObject *)v;
+    wt = (PyTupleObject *)w;
+
+    vlen = Py_SIZE(vt);
+    wlen = Py_SIZE(wt);
+
+    for (i = 0; i < vlen && i < wlen; i++) {
+        k = PyObject_RichCompareBool(vt->ob_item[i], wt->ob_item[i], Py_EQ);
+        if (k < 0)
+            return -1;
+        if (!k)
+            break;
+    }
+
+    if (i >= vlen || i >= wlen)
+        return vlen < wlen;
+
+    if (i == 0)
+        return ms->tuple_elem_compare(vt->ob_item[i], wt->ob_item[i], ms);
+    else
+        return PyObject_RichCompareBool(vt->ob_item[i], wt->ob_item[i], Py_LT);
+}
+
 /* An adaptive, stable, natural mergesort.  See listsort.txt.
  * Returns Py_None on success, NULL on error.  Even in case of error, the
  * list will be some permutation of its input state (nothing is lost or
@@ -2031,6 +2212,91 @@ list_sort_impl(PyListObject *self, PyObject *keyfunc, int reverse)
         lo.values = saved_ob_item;
     }
 
+
+    /* The pre-sort check: here's where we decide which compare function to use.
+     * How much optimization is safe? We test for homogeneity with respect to
+     * several properties that are expensive to check at compare-time, and
+     * set ms appropriately. */
+    if (saved_ob_size > 1) {
+        /* Assume the first element is representative of the whole list. */
+        int keys_are_in_tuples = (lo.keys[0]->ob_type == &PyTuple_Type &&
+                                  Py_SIZE(lo.keys[0]) > 0);
+
+        PyTypeObject* key_type = (keys_are_in_tuples ?
+                                  PyTuple_GET_ITEM(lo.keys[0], 0)->ob_type :
+                                  lo.keys[0]->ob_type);
+
+        int keys_are_all_same_type = 1;
+        int strings_are_latin = 1;
+        int ints_are_bounded = 1;
+
+        /* Prove that assumption by checking every key. */
+        int i;
+        for (i=0; i < saved_ob_size; i++) {
+
+            if (keys_are_in_tuples &&
+                !(lo.keys[i]->ob_type == &PyTuple_Type && Py_SIZE(lo.keys[i]) != 0)) {
+                keys_are_in_tuples = 0;
+                keys_are_all_same_type = 0;
+                break;
+            }
+
+            /* Note: for lists of tuples, key is the first element of the tuple
+             * lo.keys[i], not lo.keys[i] itself! We verify type-homogeneity
+             * for lists of tuples in the if-statement directly above. */
+            PyObject *key = (keys_are_in_tuples ?
+                             PyTuple_GET_ITEM(lo.keys[i], 0) :
+                             lo.keys[i]);
+
+            if (key->ob_type != key_type) {
+                keys_are_all_same_type = 0;
+                break;
+            }
+
+            if (key_type == &PyLong_Type) {
+                if (ints_are_bounded && Py_ABS(Py_SIZE(key)) > 1)
+                    ints_are_bounded = 0;
+            }
+            else if (key_type == &PyUnicode_Type){
+                if (strings_are_latin &&
+                    PyUnicode_KIND(key) != PyUnicode_1BYTE_KIND)
+                strings_are_latin = 0;
+            }
+        }
+
+        /* Choose the best compare, given what we now know about the keys. */
+        if (keys_are_all_same_type) {
+
+            if (key_type == &PyUnicode_Type && strings_are_latin) {
+                ms.key_compare = unsafe_latin_compare;
+            }
+            else if (key_type == &PyLong_Type && ints_are_bounded) {
+                ms.key_compare = unsafe_long_compare;
+            }
+            else if (key_type == &PyFloat_Type) {
+                ms.key_compare = unsafe_float_compare;
+            }
+            else if ((ms.key_richcompare = key_type->tp_richcompare) != NULL) {
+                ms.key_compare = unsafe_object_compare;
+            }
+        }
+        else {
+            ms.key_compare = safe_object_compare;
+        }
+
+        if (keys_are_in_tuples) {
+            /* Make sure we're not dealing with tuples of tuples
+             * (remember: here, key_type refers list [key[0] for key in keys]) */
+            if (key_type == &PyTuple_Type)
+                ms.tuple_elem_compare = safe_object_compare;
+            else
+                ms.tuple_elem_compare = ms.key_compare;
+
+            ms.key_compare = unsafe_tuple_compare;
+        }
+    }
+    /* End of pre-sort check: ms is now set properly! */
+
     merge_init(&ms, saved_ob_size, keys != NULL);
 
     nremaining = saved_ob_size;
@@ -2054,7 +2320,7 @@ list_sort_impl(PyListObject *self, PyObject *keyfunc, int reverse)
         Py_ssize_t n;
 
         /* Identify next run. */
-        n = count_run(lo.keys, lo.keys + nremaining, &descending);
+        n = count_run(&ms, lo.keys, lo.keys + nremaining, &descending);
         if (n < 0)
             goto fail;
         if (descending)
@@ -2063,7 +2329,7 @@ list_sort_impl(PyListObject *self, PyObject *keyfunc, int reverse)
         if (n < minrun) {
             const Py_ssize_t force = nremaining <= minrun ?
                               nremaining : minrun;
-            if (binarysort(lo, lo.keys + force, lo.keys + n) < 0)
+            if (binarysort(&ms, lo, lo.keys + force, lo.keys + n) < 0)
                 goto fail;
             n = force;
         }
diff --git a/Objects/listsort.txt b/Objects/listsort.txt
index 17d27973f826..8c877515c72e 100644
--- a/Objects/listsort.txt
+++ b/Objects/listsort.txt
@@ -753,3 +753,11 @@ example, with the region of uncertainty B[4], B[5], B[6], there are 4
 locations:  before B[4], between B[4] and B[5], between B[5] and B[6], and
 after B[6].  In general, across 2**(k-1)-1 elements, there are 2**(k-1)
 locations.  That's why k-1 binary searches are necessary and sufficient.
+
+OPTIMIZATION OF INDIVIDUAL COMPARISONS
+As noted above, even the simplest Python comparison triggers a large pile of
+C-level pointer dereferences, conditionals, and function calls.  This can be
+partially mitigated by pre-scanning the data to determine whether the data is
+homogenous with respect to type.  If so, it is sometimes possible to
+substitute faster type-specific comparisons for the slower, generic
+PyObject_RichCompareBool.


From webhook-mailer at python.org  Sun Jan 28 22:32:15 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Mon, 29 Jan 2018 03:32:15 -0000
Subject: [Python-checkins] bpo-32688: Make why the line is commented out
 clear (GH-5405)
Message-ID: <mailman.31.1517196737.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/997478eb797b31bd724135ca17781d1cf4b89cfa
commit: 997478eb797b31bd724135ca17781d1cf4b89cfa
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T11:32:12+08:00
summary:

bpo-32688: Make why the line is commented out clear (GH-5405)

files:
M Python/ceval.c

diff --git a/Python/ceval.c b/Python/ceval.c
index 3d7bed0111b2..128ec2c00454 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -3525,7 +3525,7 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
                                 tstate, f,
                                 PyTrace_RETURN, retval)) {
                 Py_CLEAR(retval);
-                /* why = WHY_EXCEPTION; */
+                /* why = WHY_EXCEPTION; useless yet but cause compiler warnings */
             }
         }
     }


From webhook-mailer at python.org  Sun Jan 28 22:43:49 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 29 Jan 2018 03:43:49 -0000
Subject: [Python-checkins] bpo-32650: Add an asyncgen pdb test (#5406)
Message-ID: <mailman.32.1517197431.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/9ee1bf9ab5af8233ed8ec5c53d12a29dc1bd9c9d
commit: 9ee1bf9ab5af8233ed8ec5c53d12a29dc1bd9c9d
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-28T22:43:46-05:00
summary:

bpo-32650: Add an asyncgen pdb test (#5406)

files:
M Lib/test/test_pdb.py

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 47a669f2cea2..08d374ca834a 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -782,6 +782,66 @@ def test_pdb_next_command_for_coroutine():
     finished
     """
 
+def test_pdb_next_command_for_asyncgen():
+    """Testing skip unwindng stack on yield for coroutines for "next" command
+
+    >>> import asyncio
+
+    >>> async def agen():
+    ...     yield 1
+    ...     await asyncio.sleep(0)
+    ...     yield 2
+
+    >>> async def test_coro():
+    ...     async for x in agen():
+    ...         print(x)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.new_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'step',
+    ...                    'next',
+    ...                    'next',
+    ...                    'step',
+    ...                    'next',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[3]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) step
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(2)test_coro()
+    -> async for x in agen():
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(3)test_coro()
+    -> print(x)
+    (Pdb) next
+    1
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(2)test_coro()
+    -> async for x in agen():
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[1]>(2)agen()
+    -> yield 1
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[1]>(3)agen()
+    -> await asyncio.sleep(0)
+    (Pdb) continue
+    2
+    finished
+    """
+
 def test_pdb_return_command_for_generator():
     """Testing no unwindng stack on yield for generators
        for "return" command


From webhook-mailer at python.org  Sun Jan 28 23:51:11 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 29 Jan 2018 04:51:11 -0000
Subject: [Python-checkins] bpo-32251: Fix docs (#5408)
Message-ID: <mailman.33.1517201474.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/07627e9a6a5f418354ff3dc99a0f36bc5b79dcd8
commit: 07627e9a6a5f418354ff3dc99a0f36bc5b79dcd8
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-28T23:51:08-05:00
summary:

bpo-32251: Fix docs (#5408)

files:
M Doc/library/asyncio-protocol.rst
M Lib/asyncio/protocols.py

diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst
index a08c4d379b68..004cac80d90c 100644
--- a/Doc/library/asyncio-protocol.rst
+++ b/Doc/library/asyncio-protocol.rst
@@ -451,7 +451,7 @@ Streaming protocols with manual receive buffer control
 
 .. versionadded:: 3.7
    **Important:** :class:`BufferedProtocol` has been been added to
-   asyncio in Python 3.7 *on a provisional basis*!  Treat it as an
+   asyncio in Python 3.7 *on a provisional basis*!  Consider it as an
    experimental API that might be changed or removed in Python 3.8.
 
 
@@ -487,7 +487,8 @@ instances:
 
 :meth:`get_buffer` can be called an arbitrary number of times during
 a connection.  However, :meth:`eof_received` is called at most once
-and, if called, :meth:`data_received` won't be called after it.
+and, if called, :meth:`get_buffer` and :meth:`buffer_updated`
+won't be called after it.
 
 State machine:
 
diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py
index 9e786dcfbbe5..8904478f1ab2 100644
--- a/Lib/asyncio/protocols.py
+++ b/Lib/asyncio/protocols.py
@@ -106,7 +106,7 @@ class BufferedProtocol(BaseProtocol):
     """Interface for stream protocol with manual buffer control.
 
     Important: this has been been added to asyncio in Python 3.7
-    *on a provisional basis*!  Treat it as an experimental API that
+    *on a provisional basis*!  Consider it as an experimental API that
     might be changed or removed in Python 3.8.
 
     Event methods, such as `create_server` and `create_connection`,


From webhook-mailer at python.org  Mon Jan 29 00:25:08 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 29 Jan 2018 05:25:08 -0000
Subject: [Python-checkins] bpo-23749: Make start-tls tests more stable on
 win7 buildbot (GH-5409)
Message-ID: <mailman.34.1517203509.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/1e5b25b8c0c45ccfd58da2cb82fdf231c6823fef
commit: 1e5b25b8c0c45ccfd58da2cb82fdf231c6823fef
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-29T00:25:05-05:00
summary:

bpo-23749: Make start-tls tests more stable on win7 buildbot (GH-5409)

To mitigate the situation when the buildbot is under load
and is unable to send/receive data fast enough:

* reduce the size of the payload
* set a generous timeout for socket ops

files:
M Lib/test/test_asyncio/test_sslproto.py

diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py
index 475f494b6705..c4c30bedb4ec 100644
--- a/Lib/test/test_asyncio/test_sslproto.py
+++ b/Lib/test/test_asyncio/test_sslproto.py
@@ -2,6 +2,7 @@
 
 import os
 import logging
+import time
 import unittest
 from unittest import mock
 try:
@@ -171,12 +172,14 @@ def new_loop(self):
         raise NotImplementedError
 
     def test_start_tls_client_1(self):
-        HELLO_MSG = b'1' * 1024 * 1024 * 5
+        HELLO_MSG = b'1' * 1024 * 1024
 
         server_context = test_utils.simple_server_sslcontext()
         client_context = test_utils.simple_client_sslcontext()
 
         def serve(sock):
+            sock.settimeout(5)
+
             data = sock.recv_all(len(HELLO_MSG))
             self.assertEqual(len(data), len(HELLO_MSG))
 
@@ -205,6 +208,8 @@ def eof_received(self):
                 self.on_eof.set_result(True)
 
         async def client(addr):
+            await asyncio.sleep(0.5, loop=self.loop)
+
             on_data = self.loop.create_future()
             on_eof = self.loop.create_future()
 
@@ -225,12 +230,15 @@ def eof_received(self):
                 asyncio.wait_for(client(srv.addr), loop=self.loop, timeout=10))
 
     def test_start_tls_server_1(self):
-        HELLO_MSG = b'1' * 1024 * 1024 * 5
+        HELLO_MSG = b'1' * 1024 * 1024
 
         server_context = test_utils.simple_server_sslcontext()
         client_context = test_utils.simple_client_sslcontext()
 
         def client(sock, addr):
+            time.sleep(0.5)
+            sock.settimeout(5)
+
             sock.connect(addr)
             data = sock.recv_all(len(HELLO_MSG))
             self.assertEqual(len(data), len(HELLO_MSG))


From webhook-mailer at python.org  Mon Jan 29 00:57:10 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Mon, 29 Jan 2018 05:57:10 -0000
Subject: [Python-checkins] bpo-32650: Add native coroutine support to bdb
 when stepping over line (GH-5400) (#5402)
Message-ID: <mailman.35.1517205433.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/543ec005a4c83fcc0d14ac1e4c0f1a36bf84640b
commit: 543ec005a4c83fcc0d14ac1e4c0f1a36bf84640b
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-29T07:57:06+02:00
summary:

bpo-32650: Add native coroutine support to bdb when stepping over line (GH-5400) (#5402)

(cherry picked from commit 46877024423e98d1b872bf308dacacd583327207)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst
M Lib/bdb.py
M Lib/test/test_pdb.py

diff --git a/Lib/bdb.py b/Lib/bdb.py
index 67a08463fa1b..fe6af903b732 100644
--- a/Lib/bdb.py
+++ b/Lib/bdb.py
@@ -3,7 +3,7 @@
 import fnmatch
 import sys
 import os
-from inspect import CO_GENERATOR
+from inspect import CO_GENERATOR, CO_COROUTINE
 
 __all__ = ["BdbQuit", "Bdb", "Breakpoint"]
 
@@ -77,7 +77,7 @@ def dispatch_call(self, frame, arg):
             # No need to trace this function
             return # None
         # Ignore call events in generator except when stepping.
-        if self.stopframe and frame.f_code.co_flags & CO_GENERATOR:
+        if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
             return self.trace_dispatch
         self.user_call(frame, arg)
         if self.quitting: raise BdbQuit
@@ -86,7 +86,7 @@ def dispatch_call(self, frame, arg):
     def dispatch_return(self, frame, arg):
         if self.stop_here(frame) or frame == self.returnframe:
             # Ignore return events in generator except when stepping.
-            if self.stopframe and frame.f_code.co_flags & CO_GENERATOR:
+            if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
                 return self.trace_dispatch
             try:
                 self.frame_returning = frame
@@ -104,7 +104,7 @@ def dispatch_exception(self, frame, arg):
             # When stepping with next/until/return in a generator frame, skip
             # the internal StopIteration exception (with no traceback)
             # triggered by a subiterator run with the 'yield from' statement.
-            if not (frame.f_code.co_flags & CO_GENERATOR
+            if not (frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
                     and arg[0] is StopIteration and arg[2] is None):
                 self.user_exception(frame, arg)
                 if self.quitting: raise BdbQuit
@@ -113,7 +113,7 @@ def dispatch_exception(self, frame, arg):
         # next/until command at the last statement in the generator before the
         # exception.
         elif (self.stopframe and frame is not self.stopframe
-                and self.stopframe.f_code.co_flags & CO_GENERATOR
+                and self.stopframe.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
                 and arg[0] in (StopIteration, GeneratorExit)):
             self.user_exception(frame, arg)
             if self.quitting: raise BdbQuit
@@ -230,7 +230,7 @@ def set_next(self, frame):
 
     def set_return(self, frame):
         """Stop when returning from the given frame."""
-        if frame.f_code.co_flags & CO_GENERATOR:
+        if frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
             self._set_stopinfo(frame, None, -1)
         else:
             self._set_stopinfo(frame.f_back, frame)
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 0ea2af554175..a7656196371b 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -724,6 +724,61 @@ def test_pdb_next_command_for_generator():
     finished
     """
 
+def test_pdb_next_command_for_coroutine():
+    """Testing skip unwindng stack on yield for coroutines for "next" command
+
+    >>> import asyncio
+
+    >>> async def test_coro():
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.get_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'step',
+    ...                    'next',
+    ...                    'next',
+    ...                    'next',
+    ...                    'step',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) step
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(2)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(3)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[1]>(4)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    Internal StopIteration
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Return--
+    > <doctest test.test_pdb.test_pdb_next_command_for_coroutine[2]>(3)test_main()->None
+    -> await test_coro()
+    (Pdb) continue
+    finished
+    """
+
 def test_pdb_return_command_for_generator():
     """Testing no unwindng stack on yield for generators
        for "return" command
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst
new file mode 100644
index 000000000000..f0b9e3ff459d
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-23-01-39.bpo-32650.Bbi7ek.rst	
@@ -0,0 +1,2 @@
+Pdb and other debuggers dependent on bdb.py will correctly step over (next
+command) native coroutines. Patch by Pablo Galindo.


From webhook-mailer at python.org  Mon Jan 29 01:51:09 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Mon, 29 Jan 2018 06:51:09 -0000
Subject: [Python-checkins] [3.6] bpo-32650 Add support for async generators
 and more test for coroutines in pdb (GH-5403). (#5411)
Message-ID: <mailman.36.1517208673.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/3cfb84c65790f5f9377574f9be0799bdd9d0132c
commit: 3cfb84c65790f5f9377574f9be0799bdd9d0132c
branch: 3.6
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T08:51:07+02:00
summary:

[3.6] bpo-32650 Add support for async generators and more test for coroutines in pdb  (GH-5403). (#5411)

(cherry picked from commit c7ab581db216aeeb1c2aa7af2f2198d2b7516383)

files:
M Lib/bdb.py
M Lib/test/test_pdb.py

diff --git a/Lib/bdb.py b/Lib/bdb.py
index fe6af903b732..63d247b4b221 100644
--- a/Lib/bdb.py
+++ b/Lib/bdb.py
@@ -3,10 +3,13 @@
 import fnmatch
 import sys
 import os
-from inspect import CO_GENERATOR, CO_COROUTINE
+from inspect import CO_GENERATOR, CO_COROUTINE, CO_ASYNC_GENERATOR
 
 __all__ = ["BdbQuit", "Bdb", "Breakpoint"]
 
+GENERATOR_AND_COROUTINE_FLAGS = CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR
+
+
 class BdbQuit(Exception):
     """Exception to give up completely."""
 
@@ -77,7 +80,7 @@ def dispatch_call(self, frame, arg):
             # No need to trace this function
             return # None
         # Ignore call events in generator except when stepping.
-        if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
+        if self.stopframe and frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
             return self.trace_dispatch
         self.user_call(frame, arg)
         if self.quitting: raise BdbQuit
@@ -86,7 +89,7 @@ def dispatch_call(self, frame, arg):
     def dispatch_return(self, frame, arg):
         if self.stop_here(frame) or frame == self.returnframe:
             # Ignore return events in generator except when stepping.
-            if self.stopframe and frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
+            if self.stopframe and frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
                 return self.trace_dispatch
             try:
                 self.frame_returning = frame
@@ -104,7 +107,7 @@ def dispatch_exception(self, frame, arg):
             # When stepping with next/until/return in a generator frame, skip
             # the internal StopIteration exception (with no traceback)
             # triggered by a subiterator run with the 'yield from' statement.
-            if not (frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
+            if not (frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS
                     and arg[0] is StopIteration and arg[2] is None):
                 self.user_exception(frame, arg)
                 if self.quitting: raise BdbQuit
@@ -113,7 +116,7 @@ def dispatch_exception(self, frame, arg):
         # next/until command at the last statement in the generator before the
         # exception.
         elif (self.stopframe and frame is not self.stopframe
-                and self.stopframe.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE)
+                and self.stopframe.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS
                 and arg[0] in (StopIteration, GeneratorExit)):
             self.user_exception(frame, arg)
             if self.quitting: raise BdbQuit
@@ -230,7 +233,7 @@ def set_next(self, frame):
 
     def set_return(self, frame):
         """Stop when returning from the given frame."""
-        if frame.f_code.co_flags & (CO_GENERATOR | CO_COROUTINE):
+        if frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
             self._set_stopinfo(frame, None, -1)
         else:
             self._set_stopinfo(frame.f_back, frame)
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index a7656196371b..6d7e149805cb 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -739,7 +739,7 @@ def test_pdb_next_command_for_coroutine():
     ...     await test_coro()
 
     >>> def test_function():
-    ...     loop = asyncio.get_event_loop()
+    ...     loop = asyncio.new_event_loop()
     ...     loop.run_until_complete(test_main())
     ...     loop.close()
     ...     print("finished")
@@ -834,6 +834,47 @@ def test_pdb_return_command_for_generator():
     finished
     """
 
+def test_pdb_return_command_for_coroutine():
+    """Testing no unwindng stack on yield for coroutines for "return" command
+
+    >>> import asyncio
+
+    >>> async def test_coro():
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+    ...     await asyncio.sleep(0)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.new_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'step',
+    ...                    'next',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[1]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) step
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[1]>(2)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_return_command_for_coroutine[1]>(3)test_coro()
+    -> await asyncio.sleep(0)
+    (Pdb) continue
+    finished
+    """
+
 def test_pdb_until_command_for_generator():
     """Testing no unwindng stack on yield for generators
        for "until" command if target breakpoing is not reached


From webhook-mailer at python.org  Mon Jan 29 02:34:28 2018
From: webhook-mailer at python.org (Nathaniel J. Smith)
Date: Mon, 29 Jan 2018 07:34:28 -0000
Subject: [Python-checkins] bpo-32591: silence deprecation warnings in
 test_coroutine (GH-5412)
Message-ID: <mailman.37.1517211272.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/95e214713a7da4e5cbfb73f35b2119f76074fc3f
commit: 95e214713a7da4e5cbfb73f35b2119f76074fc3f
branch: master
author: Nathaniel J. Smith <njs at pobox.com>
committer: GitHub <noreply at github.com>
date: 2018-01-28T23:34:26-08:00
summary:

bpo-32591: silence deprecation warnings in test_coroutine (GH-5412)

files:
M Lib/test/test_coroutines.py

diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py
index a97535a12a89..19a3444b6ad7 100644
--- a/Lib/test/test_coroutines.py
+++ b/Lib/test/test_coroutines.py
@@ -1981,16 +1981,19 @@ def wrap(gen):
 
         with self.assertWarns(DeprecationWarning):
             sys.set_coroutine_wrapper(wrap)
-        self.assertIs(sys.get_coroutine_wrapper(), wrap)
+        with self.assertWarns(DeprecationWarning):
+            self.assertIs(sys.get_coroutine_wrapper(), wrap)
         try:
             f = foo()
             self.assertTrue(wrapped)
 
             self.assertEqual(run_async(f), ([], 'spam'))
         finally:
-            sys.set_coroutine_wrapper(None)
+            with self.assertWarns(DeprecationWarning):
+                sys.set_coroutine_wrapper(None)
 
-        self.assertIsNone(sys.get_coroutine_wrapper())
+        with self.assertWarns(DeprecationWarning):
+            self.assertIsNone(sys.get_coroutine_wrapper())
 
         wrapped = None
         with silence_coro_gc():
@@ -1998,10 +2001,13 @@ def wrap(gen):
         self.assertFalse(wrapped)
 
     def test_set_wrapper_2(self):
-        self.assertIsNone(sys.get_coroutine_wrapper())
+        with self.assertWarns(DeprecationWarning):
+            self.assertIsNone(sys.get_coroutine_wrapper())
         with self.assertRaisesRegex(TypeError, "callable expected, got int"):
-            sys.set_coroutine_wrapper(1)
-        self.assertIsNone(sys.get_coroutine_wrapper())
+            with self.assertWarns(DeprecationWarning):
+                sys.set_coroutine_wrapper(1)
+        with self.assertWarns(DeprecationWarning):
+            self.assertIsNone(sys.get_coroutine_wrapper())
 
     def test_set_wrapper_3(self):
         async def foo():
@@ -2012,7 +2018,8 @@ def wrapper(coro):
                 return await coro
             return wrap(coro)
 
-        sys.set_coroutine_wrapper(wrapper)
+        with self.assertWarns(DeprecationWarning):
+            sys.set_coroutine_wrapper(wrapper)
         try:
             with silence_coro_gc(), self.assertRaisesRegex(
                 RuntimeError,
@@ -2021,7 +2028,8 @@ def wrapper(coro):
 
                 foo()
         finally:
-            sys.set_coroutine_wrapper(None)
+            with self.assertWarns(DeprecationWarning):
+                sys.set_coroutine_wrapper(None)
 
     def test_set_wrapper_4(self):
         @types.coroutine
@@ -2034,7 +2042,8 @@ def wrap(gen):
             wrapped = gen
             return gen
 
-        sys.set_coroutine_wrapper(wrap)
+        with self.assertWarns(DeprecationWarning):
+            sys.set_coroutine_wrapper(wrap)
         try:
             foo()
             self.assertIs(
@@ -2042,7 +2051,8 @@ def wrap(gen):
                 "generator-based coroutine was wrapped via "
                 "sys.set_coroutine_wrapper")
         finally:
-            sys.set_coroutine_wrapper(None)
+            with self.assertWarns(DeprecationWarning):
+                sys.set_coroutine_wrapper(None)
 
 
 class OriginTrackingTest(unittest.TestCase):


From webhook-mailer at python.org  Mon Jan 29 03:30:51 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 29 Jan 2018 08:30:51 -0000
Subject: [Python-checkins] Fix minor markup typo (#5407)
Message-ID: <mailman.38.1517214652.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/2a3260bb030b0d4620242110c0ee5abc379afa8d
commit: 2a3260bb030b0d4620242110c0ee5abc379afa8d
branch: master
author: cocoatomo <cocoatomo77 at gmail.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-29T00:30:48-08:00
summary:

Fix minor markup typo (#5407)

files:
M Doc/library/functions.rst

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index e47225718c1b..bfb813cf3906 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1423,7 +1423,7 @@ are always available.  They are listed here in alphabetical order.
    a regular function and do something with its result.  This is needed
    in some cases where you need a reference to a function from a class
    body and you want to avoid the automatic transformation to instance
-   method.  For these cases, use this idiom:
+   method.  For these cases, use this idiom::
 
       class C:
           builtin_open = staticmethod(open)


From solipsis at pitrou.net  Mon Jan 29 04:12:05 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 29 Jan 2018 09:12:05 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-3
Message-ID: <20180129091205.1.EFEE476A5342AECD@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [0, 0, -7] memory blocks, sum=-7
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [-1, -1, 2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflog0QiFPm', '--timeout', '7200']

From webhook-mailer at python.org  Mon Jan 29 04:24:17 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 29 Jan 2018 09:24:17 -0000
Subject: [Python-checkins] Fix minor markup typo (GH-5407) (#5416)
Message-ID: <mailman.39.1517217859.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/190de183e5e2a11591e28e2c413a70079d4c92aa
commit: 190de183e5e2a11591e28e2c413a70079d4c92aa
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-29T01:24:09-08:00
summary:

Fix minor markup typo (GH-5407) (#5416)

(cherry picked from commit 2a3260bb030b0d4620242110c0ee5abc379afa8d)

files:
M Doc/library/functions.rst

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 7a7a84d1d424..3b479bd64294 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1409,7 +1409,7 @@ are always available.  They are listed here in alphabetical order.
    a regular function and do something with its result.  This is needed
    in some cases where you need a reference to a function from a class
    body and you want to avoid the automatic transformation to instance
-   method.  For these cases, use this idiom:
+   method.  For these cases, use this idiom::
 
       class C:
           builtin_open = staticmethod(open)


From webhook-mailer at python.org  Mon Jan 29 04:50:43 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 09:50:43 -0000
Subject: [Python-checkins] bpo-32705: Current Android does not have
 posix_spawn (#5413)
Message-ID: <mailman.40.1517219446.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/8997f9cd1a59f04fbb8c7b590295a9f38c548744
commit: 8997f9cd1a59f04fbb8c7b590295a9f38c548744
branch: master
author: Chih-Hsuan Yen <yan12125 at gmail.com>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-29T10:50:35+01:00
summary:

bpo-32705: Current Android does not have posix_spawn (#5413)

files:
M Modules/posixmodule.c

diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 4c0392e22c3b..8b11e981f115 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -176,7 +176,14 @@ corresponding Unix manual entries for more information on calls.");
 #else
 /* Unix functions that the configure script doesn't check for */
 #define HAVE_EXECV      1
+/* bpo-32705: Current Android does not have posix_spawn
+ * Most likely posix_spawn will be available in next Android version (Android
+ * P, API 28). Need revisit then. See
+ * https://android-review.googlesource.com/c/platform/bionic/+/504842
+ **/
+#ifndef __ANDROID__
 #define HAVE_POSIX_SPAWN 1
+#endif
 #define HAVE_FORK       1
 #if defined(__USLC__) && defined(__SCO_VERSION__)       /* SCO UDK Compiler */
 #define HAVE_FORK1      1


From webhook-mailer at python.org  Mon Jan 29 05:41:37 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Mon, 29 Jan 2018 10:41:37 -0000
Subject: [Python-checkins] bpo-32650: Add an asyncgen pdb test (GH-5406)
 (#5419)
Message-ID: <mailman.41.1517222500.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/d9c743b2d118530ca13a8e29f96f30950866bd56
commit: d9c743b2d118530ca13a8e29f96f30950866bd56
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-29T12:41:34+02:00
summary:

bpo-32650: Add an asyncgen pdb test (GH-5406) (#5419)

(cherry picked from commit 9ee1bf9ab5af8233ed8ec5c53d12a29dc1bd9c9d)

files:
M Lib/test/test_pdb.py

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 6d7e149805cb..369a0a91e305 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -779,6 +779,66 @@ def test_pdb_next_command_for_coroutine():
     finished
     """
 
+def test_pdb_next_command_for_asyncgen():
+    """Testing skip unwindng stack on yield for coroutines for "next" command
+
+    >>> import asyncio
+
+    >>> async def agen():
+    ...     yield 1
+    ...     await asyncio.sleep(0)
+    ...     yield 2
+
+    >>> async def test_coro():
+    ...     async for x in agen():
+    ...         print(x)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.new_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'step',
+    ...                    'next',
+    ...                    'next',
+    ...                    'step',
+    ...                    'next',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[3]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) step
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(2)test_coro()
+    -> async for x in agen():
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(3)test_coro()
+    -> print(x)
+    (Pdb) next
+    1
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[2]>(2)test_coro()
+    -> async for x in agen():
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[1]>(2)agen()
+    -> yield 1
+    (Pdb) next
+    > <doctest test.test_pdb.test_pdb_next_command_for_asyncgen[1]>(3)agen()
+    -> await asyncio.sleep(0)
+    (Pdb) continue
+    2
+    finished
+    """
+
 def test_pdb_return_command_for_generator():
     """Testing no unwindng stack on yield for generators
        for "return" command


From webhook-mailer at python.org  Mon Jan 29 05:57:47 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 10:57:47 -0000
Subject: [Python-checkins] bpo-20891: Py_Initialize() now creates the GIL
 (#4700)
Message-ID: <mailman.43.1517223469.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/2914bb32e2adf8dff77c0ca58b33201bc94e398c
commit: 2914bb32e2adf8dff77c0ca58b33201bc94e398c
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T11:57:45+01:00
summary:

bpo-20891: Py_Initialize() now creates the GIL (#4700)

The GIL is no longer created "on demand" to fix a race condition when
PyGILState_Ensure() is called in a non-Python thread.

files:
A Misc/NEWS.d/next/Core and Builtins/2017-12-04-18-34-11.bpo-20891.C2TsfR.rst
M Doc/c-api/init.rst
M Python/ceval.c
M Python/pylifecycle.c

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 02189a9e3591..bae49d5ba812 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -687,15 +687,14 @@ This is so common that a pair of macros exists to simplify it::
 
 The :c:macro:`Py_BEGIN_ALLOW_THREADS` macro opens a new block and declares a
 hidden local variable; the :c:macro:`Py_END_ALLOW_THREADS` macro closes the
-block.  These two macros are still available when Python is compiled without
-thread support (they simply have an empty expansion).
+block.
 
-When thread support is enabled, the block above expands to the following code::
+The block above expands to the following code::
 
    PyThreadState *_save;
 
    _save = PyEval_SaveThread();
-   ...Do some blocking I/O operation...
+   ... Do some blocking I/O operation ...
    PyEval_RestoreThread(_save);
 
 .. index::
@@ -818,36 +817,24 @@ code, or when embedding the Python interpreter:
 
    This is a no-op when called for a second time.
 
+   .. versionchanged:: 3.7
+      This function is now called by :c:func:`Py_Initialize()`, so you don't
+      have to call it yourself anymore.
+
    .. versionchanged:: 3.2
       This function cannot be called before :c:func:`Py_Initialize()` anymore.
 
    .. index:: module: _thread
 
-   .. note::
-
-      When only the main thread exists, no GIL operations are needed. This is a
-      common situation (most Python programs do not use threads), and the lock
-      operations slow the interpreter down a bit. Therefore, the lock is not
-      created initially.  This situation is equivalent to having acquired the lock:
-      when there is only a single thread, all object accesses are safe.  Therefore,
-      when this function initializes the global interpreter lock, it also acquires
-      it.  Before the Python :mod:`_thread` module creates a new thread, knowing
-      that either it has the lock or the lock hasn't been created yet, it calls
-      :c:func:`PyEval_InitThreads`.  When this call returns, it is guaranteed that
-      the lock has been created and that the calling thread has acquired it.
-
-      It is **not** safe to call this function when it is unknown which thread (if
-      any) currently has the global interpreter lock.
-
-      This function is not available when thread support is disabled at compile time.
-
 
 .. c:function:: int PyEval_ThreadsInitialized()
 
    Returns a non-zero value if :c:func:`PyEval_InitThreads` has been called.  This
    function can be called without holding the GIL, and therefore can be used to
-   avoid calls to the locking API when running single-threaded.  This function is
-   not available when thread support is disabled at compile time.
+   avoid calls to the locking API when running single-threaded.
+
+   .. versionchanged:: 3.7
+      The :term:`GIL` is now initialized by :c:func:`Py_Initialize()`.
 
 
 .. c:function:: PyThreadState* PyEval_SaveThread()
@@ -855,8 +842,7 @@ code, or when embedding the Python interpreter:
    Release the global interpreter lock (if it has been created and thread
    support is enabled) and reset the thread state to *NULL*, returning the
    previous thread state (which is not *NULL*).  If the lock has been created,
-   the current thread must have acquired it.  (This function is available even
-   when thread support is disabled at compile time.)
+   the current thread must have acquired it.
 
 
 .. c:function:: void PyEval_RestoreThread(PyThreadState *tstate)
@@ -864,8 +850,7 @@ code, or when embedding the Python interpreter:
    Acquire the global interpreter lock (if it has been created and thread
    support is enabled) and set the thread state to *tstate*, which must not be
    *NULL*.  If the lock has been created, the current thread must not have
-   acquired it, otherwise deadlock ensues.  (This function is available even
-   when thread support is disabled at compile time.)
+   acquired it, otherwise deadlock ensues.
 
 
 .. c:function:: PyThreadState* PyThreadState_Get()
@@ -957,7 +942,7 @@ example usage in the Python source distribution.
    This macro expands to ``{ PyThreadState *_save; _save = PyEval_SaveThread();``.
    Note that it contains an opening brace; it must be matched with a following
    :c:macro:`Py_END_ALLOW_THREADS` macro.  See above for further discussion of this
-   macro.  It is a no-op when thread support is disabled at compile time.
+   macro.
 
 
 .. c:macro:: Py_END_ALLOW_THREADS
@@ -965,29 +950,29 @@ example usage in the Python source distribution.
    This macro expands to ``PyEval_RestoreThread(_save); }``. Note that it contains
    a closing brace; it must be matched with an earlier
    :c:macro:`Py_BEGIN_ALLOW_THREADS` macro.  See above for further discussion of
-   this macro.  It is a no-op when thread support is disabled at compile time.
+   this macro.
 
 
 .. c:macro:: Py_BLOCK_THREADS
 
    This macro expands to ``PyEval_RestoreThread(_save);``: it is equivalent to
-   :c:macro:`Py_END_ALLOW_THREADS` without the closing brace.  It is a no-op when
-   thread support is disabled at compile time.
+   :c:macro:`Py_END_ALLOW_THREADS` without the closing brace.
 
 
 .. c:macro:: Py_UNBLOCK_THREADS
 
    This macro expands to ``_save = PyEval_SaveThread();``: it is equivalent to
    :c:macro:`Py_BEGIN_ALLOW_THREADS` without the opening brace and variable
-   declaration.  It is a no-op when thread support is disabled at compile time.
+   declaration.
 
 
 Low-level API
 -------------
 
-All of the following functions are only available when thread support is enabled
-at compile time, and must be called only when the global interpreter lock has
-been created.
+All of the following functions must be called after :c:func:`Py_Initialize`.
+
+.. versionchanged:: 3.7
+   :c:func:`Py_Initialize()` now initializes the :term:`GIL`.
 
 
 .. c:function:: PyInterpreterState* PyInterpreterState_New()
@@ -1068,8 +1053,7 @@ been created.
    If this thread already has the lock, deadlock ensues.
 
    :c:func:`PyEval_RestoreThread` is a higher-level function which is always
-   available (even when thread support isn't enabled or when threads have
-   not been initialized).
+   available (even when threads have not been initialized).
 
 
 .. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate)
@@ -1081,8 +1065,7 @@ been created.
    reported.
 
    :c:func:`PyEval_SaveThread` is a higher-level function which is always
-   available (even when thread support isn't enabled or when threads have
-   not been initialized).
+   available (even when threads have not been initialized).
 
 
 .. c:function:: void PyEval_AcquireLock()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-12-04-18-34-11.bpo-20891.C2TsfR.rst b/Misc/NEWS.d/next/Core and Builtins/2017-12-04-18-34-11.bpo-20891.C2TsfR.rst
new file mode 100644
index 000000000000..abf9c3c2e390
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-12-04-18-34-11.bpo-20891.C2TsfR.rst	
@@ -0,0 +1,3 @@
+Py_Initialize() now creates the GIL. The GIL is no longer created "on demand"
+to fix a race condition when PyGILState_Ensure() is called in a non-Python
+thread.
diff --git a/Python/ceval.c b/Python/ceval.c
index 128ec2c00454..52a42b00724e 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -254,8 +254,8 @@ PyEval_SaveThread(void)
     PyThreadState *tstate = PyThreadState_Swap(NULL);
     if (tstate == NULL)
         Py_FatalError("PyEval_SaveThread: NULL tstate");
-    if (gil_created())
-        drop_gil(tstate);
+    assert(gil_created());
+    drop_gil(tstate);
     return tstate;
 }
 
@@ -264,17 +264,18 @@ PyEval_RestoreThread(PyThreadState *tstate)
 {
     if (tstate == NULL)
         Py_FatalError("PyEval_RestoreThread: NULL tstate");
-    if (gil_created()) {
-        int err = errno;
-        take_gil(tstate);
-        /* _Py_Finalizing is protected by the GIL */
-        if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) {
-            drop_gil(tstate);
-            PyThread_exit_thread();
-            Py_UNREACHABLE();
-        }
-        errno = err;
+    assert(gil_created());
+
+    int err = errno;
+    take_gil(tstate);
+    /* _Py_Finalizing is protected by the GIL */
+    if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) {
+        drop_gil(tstate);
+        PyThread_exit_thread();
+        Py_UNREACHABLE();
     }
+    errno = err;
+
     PyThreadState_Swap(tstate);
 }
 
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index d46784a2f6b7..82ab9154256f 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -681,9 +681,13 @@ _Py_InitializeCore(const _PyCoreConfig *core_config)
        Instead we destroy the previously created GIL here, which ensures
        that we can call Py_Initialize / Py_FinalizeEx multiple times. */
     _PyEval_FiniThreads();
+
     /* Auto-thread-state API */
     _PyGILState_Init(interp, tstate);
 
+    /* Create the GIL */
+    PyEval_InitThreads();
+
     _Py_ReadyTypes();
 
     if (!_PyFrame_Init())


From webhook-mailer at python.org  Mon Jan 29 06:10:25 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 11:10:25 -0000
Subject: [Python-checkins] bpo-31233,
 bpo-31151: Document socketserver changes (#5417)
Message-ID: <mailman.44.1517224229.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/db8189bb8db609ca3993dec31fb95fdbe134469a
commit: db8189bb8db609ca3993dec31fb95fdbe134469a
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T12:10:22+01:00
summary:

bpo-31233, bpo-31151: Document socketserver changes (#5417)

socketserver.ForkingMixIn.server_close() and
socketserver.ThreadingMixIn.server_close() now waits until all child
processes and non-daemonic threads complete.

files:
M Doc/library/socketserver.rst
M Doc/whatsnew/3.7.rst

diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst
index e12c8c97844f..560002738375 100644
--- a/Doc/library/socketserver.rst
+++ b/Doc/library/socketserver.rst
@@ -115,6 +115,21 @@ server classes.
    :class:`ForkingMixIn` and the Forking classes mentioned below are
    only available on POSIX platforms that support :func:`~os.fork`.
 
+   :meth:`socketserver.ForkingMixIn.server_close` waits until all child
+   processes complete.
+
+   :meth:`socketserver.ThreadingMixIn.server_close` waits until all non-daemon
+   threads complete. Use daemonic threads by setting
+   :data:`ThreadingMixIn.daemon_threads` to ``True`` to not wait until threads
+   complete.
+
+   .. versionchanged:: 3.7
+
+      :meth:`socketserver.ForkingMixIn.server_close` and
+      :meth:`socketserver.ThreadingMixIn.server_close` now waits until all
+      child processes and non-daemonic threads complete.
+
+
 .. class:: ForkingTCPServer
            ForkingUDPServer
            ThreadingTCPServer
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 9979e6923990..b70945f4708e 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -1021,6 +1021,14 @@ Changes in Python behavior
 Changes in the Python API
 -------------------------
 
+* :meth:`socketserver.ThreadingMixIn.server_close` now waits until all
+  non-daemon threads complete. Use daemonic threads by setting
+  :data:`ThreadingMixIn.daemon_threads` to ``True`` to not wait until threads
+  complete. (Contributed by Victor Stinner in :issue:`31233`.)
+
+* :meth:`socketserver.ForkingMixIn.server_close` now waits until all
+  child processes complete. (Contributed by Victor Stinner in :issue:`31151`.)
+
 * The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE``
   locale to the ``LC_NUMERIC`` locale in some cases.
 


From webhook-mailer at python.org  Mon Jan 29 07:21:38 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 12:21:38 -0000
Subject: [Python-checkins] bpo-32706: Skip test_ftplib.test_check_hostname()
 (#5422)
Message-ID: <mailman.45.1517228499.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/51500f37453bbcbc5f1dbc3c38d079c3c1efe5de
commit: 51500f37453bbcbc5f1dbc3c38d079c3c1efe5de
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T13:21:34+01:00
summary:

bpo-32706: Skip test_ftplib.test_check_hostname() (#5422)

This test is unstable and currently prevents to make any new change
since the test always fails on Travis CI.

Skip the test to get more time to fix it.

files:
M Lib/test/test_ftplib.py

diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py
index bb5a67f27e31..32aab047df85 100644
--- a/Lib/test/test_ftplib.py
+++ b/Lib/test/test_ftplib.py
@@ -936,6 +936,7 @@ def test_ccc(self):
         self.client.ccc()
         self.assertRaises(ValueError, self.client.sock.unwrap)
 
+    @skipUnless(False, "FIXME: bpo-32706")
     def test_check_hostname(self):
         self.client.quit()
         ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)


From webhook-mailer at python.org  Mon Jan 29 07:47:09 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 12:47:09 -0000
Subject: [Python-checkins] bpo-28685: Fix compiler warning (GH-5423)
Message-ID: <mailman.46.1517230032.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/8017b804a06804793bcc0a7f734d8a846b0fe18a
commit: 8017b804a06804793bcc0a7f734d8a846b0fe18a
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T13:47:06+01:00
summary:

bpo-28685: Fix compiler warning (GH-5423)

files:
M Objects/listobject.c

diff --git a/Objects/listobject.c b/Objects/listobject.c
index 9e321371aae9..f0fe962be941 100644
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2028,7 +2028,8 @@ unsafe_object_compare(PyObject *v, PyObject *w, MergeState *ms)
 static int
 unsafe_latin_compare(PyObject *v, PyObject *w, MergeState *ms)
 {
-    int len, res;
+    Py_ssize_t len;
+    int res;
 
     /* Modified from Objects/unicodeobject.c:unicode_compare, assuming: */
     assert(v->ob_type == w->ob_type);


From webhook-mailer at python.org  Mon Jan 29 08:04:46 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 13:04:46 -0000
Subject: [Python-checkins] bpo-20891: Reenable test_embed.test_bpo20891()
 (GH-5420)
Message-ID: <mailman.47.1517231088.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/d951157268b2122109098c792562b71ccc41920b
commit: d951157268b2122109098c792562b71ccc41920b
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T14:04:44+01:00
summary:

bpo-20891: Reenable test_embed.test_bpo20891() (GH-5420)

files:
M Lib/test/test_embed.py

diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py
index c9b65485ac7e..c7f45b59acc8 100644
--- a/Lib/test/test_embed.py
+++ b/Lib/test/test_embed.py
@@ -198,9 +198,6 @@ def test_pre_initialization_api(self):
         self.assertEqual(out, '')
         self.assertEqual(err, '')
 
-    @unittest.skipIf(True,
-                     "FIXME: test fails randomly because of a race conditon, "
-                     "see bpo-20891")
     def test_bpo20891(self):
         """
         bpo-20891: Calling PyGILState_Ensure in a non-Python thread before


From webhook-mailer at python.org  Mon Jan 29 08:10:21 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Mon, 29 Jan 2018 13:10:21 -0000
Subject: [Python-checkins] bpo-31429: Define TLS cipher suite on build time
 (#3532)
Message-ID: <mailman.48.1517231425.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/892d66e422d5367673163d62ba40cd70a37d5cf7
commit: 892d66e422d5367673163d62ba40cd70a37d5cf7
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T14:10:18+01:00
summary:

bpo-31429: Define TLS cipher suite on build time (#3532)

Until now Python used a hard coded white list of default TLS cipher
suites. The old approach has multiple downsides. OpenSSL's default
selection was completely overruled. Python did neither benefit from new
cipher suites (ChaCha20, TLS 1.3 suites) nor blacklisted cipher suites.
For example we used to re-enable 3DES.

Python now defaults to OpenSSL DEFAULT cipher suite selection and black
lists all unwanted ciphers. Downstream vendors can override the default
cipher list with --with-ssl-default-suites.

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2018-01-28-22-40-05.bpo-31429.qNt8rQ.rst
M Doc/whatsnew/3.7.rst
M Lib/ssl.py
M Lib/test/test_ssl.py
M Modules/_ssl.c
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index b70945f4708e..e36e505fd8ae 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -623,6 +623,12 @@ wildcard matching disabled by default.
 (Contributed by Mandeep Singh in :issue:`23033` and Christian Heimes in
 :issue:`31399`.)
 
+The default cipher suite selection of the ssl module now uses a blacklist
+approach rather than a hard-coded whitelist. Python no longer re-enables
+ciphers that have been blocked by OpenSSL security update. Default cipher
+suite selection can be configured on compile time.
+(Contributed by Christian Heimes in :issue:`31429`.)
+
 string
 ------
 
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 5f972e1d3d8a..b6161d0f178d 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -115,6 +115,7 @@
 
 
 from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN, HAS_ALPN, HAS_TLSv1_3
+from _ssl import _DEFAULT_CIPHERS
 from _ssl import _OPENSSL_API_VERSION
 
 
@@ -174,48 +175,7 @@
 HAS_NEVER_CHECK_COMMON_NAME = hasattr(_ssl, 'HOSTFLAG_NEVER_CHECK_SUBJECT')
 
 
-# Disable weak or insecure ciphers by default
-# (OpenSSL's default setting is 'DEFAULT:!aNULL:!eNULL')
-# Enable a better set of ciphers by default
-# This list has been explicitly chosen to:
-#   * TLS 1.3 ChaCha20 and AES-GCM cipher suites
-#   * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE)
-#   * Prefer ECDHE over DHE for better performance
-#   * Prefer AEAD over CBC for better performance and security
-#   * Prefer AES-GCM over ChaCha20 because most platforms have AES-NI
-#     (ChaCha20 needs OpenSSL 1.1.0 or patched 1.0.2)
-#   * Prefer any AES-GCM and ChaCha20 over any AES-CBC for better
-#     performance and security
-#   * Then Use HIGH cipher suites as a fallback
-#   * Disable NULL authentication, NULL encryption, 3DES and MD5 MACs
-#     for security reasons
-_DEFAULT_CIPHERS = (
-    'TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:'
-    'TLS13-AES-128-GCM-SHA256:'
-    'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:'
-    'ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:RSA+AESGCM:RSA+AES:RSA+HIGH:'
-    '!aNULL:!eNULL:!MD5:!3DES'
-    )
-
-# Restricted and more secure ciphers for the server side
-# This list has been explicitly chosen to:
-#   * TLS 1.3 ChaCha20 and AES-GCM cipher suites
-#   * Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE)
-#   * Prefer ECDHE over DHE for better performance
-#   * Prefer AEAD over CBC for better performance and security
-#   * Prefer AES-GCM over ChaCha20 because most platforms have AES-NI
-#   * Prefer any AES-GCM and ChaCha20 over any AES-CBC for better
-#     performance and security
-#   * Then Use HIGH cipher suites as a fallback
-#   * Disable NULL authentication, NULL encryption, MD5 MACs, DSS, RC4, and
-#     3DES for security reasons
-_RESTRICTED_SERVER_CIPHERS = (
-    'TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:'
-    'TLS13-AES-128-GCM-SHA256:'
-    'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:'
-    'ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:RSA+AESGCM:RSA+AES:RSA+HIGH:'
-    '!aNULL:!eNULL:!MD5:!DSS:!RC4:!3DES'
-)
+_RESTRICTED_SERVER_CIPHERS = _DEFAULT_CIPHERS
 
 CertificateError = SSLCertVerificationError
 
@@ -393,8 +353,6 @@ class SSLContext(_SSLContext):
 
     def __new__(cls, protocol=PROTOCOL_TLS, *args, **kwargs):
         self = _SSLContext.__new__(cls, protocol)
-        if protocol != _SSLv2_IF_EXISTS:
-            self.set_ciphers(_DEFAULT_CIPHERS)
         return self
 
     def __init__(self, protocol=PROTOCOL_TLS):
@@ -530,8 +488,6 @@ def create_default_context(purpose=Purpose.SERVER_AUTH, *, cafile=None,
         # verify certs and host name in client mode
         context.verify_mode = CERT_REQUIRED
         context.check_hostname = True
-    elif purpose == Purpose.CLIENT_AUTH:
-        context.set_ciphers(_RESTRICTED_SERVER_CIPHERS)
 
     if cafile or capath or cadata:
         context.load_verify_locations(cafile, capath, cadata)
diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index fdf727f788ca..6a8bf0ef7be4 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -18,6 +18,7 @@
 import weakref
 import platform
 import functools
+import sysconfig
 try:
     import ctypes
 except ImportError:
@@ -30,7 +31,7 @@
 HOST = support.HOST
 IS_LIBRESSL = ssl.OPENSSL_VERSION.startswith('LibreSSL')
 IS_OPENSSL_1_1 = not IS_LIBRESSL and ssl.OPENSSL_VERSION_INFO >= (1, 1, 0)
-
+PY_SSL_DEFAULT_CIPHERS = sysconfig.get_config_var('PY_SSL_DEFAULT_CIPHERS')
 
 def data_file(*name):
     return os.path.join(os.path.dirname(__file__), *name)
@@ -936,6 +937,19 @@ def test_ciphers(self):
         with self.assertRaisesRegex(ssl.SSLError, "No cipher can be selected"):
             ctx.set_ciphers("^$:,;?*'dorothyx")
 
+    @unittest.skipUnless(PY_SSL_DEFAULT_CIPHERS == 1,
+                         "Test applies only to Python default ciphers")
+    def test_python_ciphers(self):
+        ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+        ciphers = ctx.get_ciphers()
+        for suite in ciphers:
+            name = suite['name']
+            self.assertNotIn("PSK", name)
+            self.assertNotIn("SRP", name)
+            self.assertNotIn("MD5", name)
+            self.assertNotIn("RC4", name)
+            self.assertNotIn("3DES", name)
+
     @unittest.skipIf(ssl.OPENSSL_VERSION_INFO < (1, 0, 2, 0, 0), 'OpenSSL too old')
     def test_get_ciphers(self):
         ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
diff --git a/Misc/NEWS.d/next/Library/2018-01-28-22-40-05.bpo-31429.qNt8rQ.rst b/Misc/NEWS.d/next/Library/2018-01-28-22-40-05.bpo-31429.qNt8rQ.rst
new file mode 100644
index 000000000000..1ac537b07aaf
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-28-22-40-05.bpo-31429.qNt8rQ.rst
@@ -0,0 +1,4 @@
+The default cipher suite selection of the ssl module now uses a blacklist
+approach rather than a hard-coded whitelist. Python no longer re-enables
+ciphers that have been blocked by OpenSSL security update. Default cipher
+suite selection can be configured on compile time.
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index ec8c8af54bdd..7545e91babdb 100644
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -234,6 +234,31 @@ SSL_SESSION_get_ticket_lifetime_hint(const SSL_SESSION *s)
 
 #endif /* OpenSSL < 1.1.0 or LibreSSL */
 
+/* Default cipher suites */
+#ifndef PY_SSL_DEFAULT_CIPHERS
+#define PY_SSL_DEFAULT_CIPHERS 1
+#endif
+
+#if PY_SSL_DEFAULT_CIPHERS == 0
+  #ifndef PY_SSL_DEFAULT_CIPHER_STRING
+     #error "Py_SSL_DEFAULT_CIPHERS 0 needs Py_SSL_DEFAULT_CIPHER_STRING"
+  #endif
+#elif PY_SSL_DEFAULT_CIPHERS == 1
+/* Python custom selection of sensible ciper suites
+ * DEFAULT: OpenSSL's default cipher list. Since 1.0.2 the list is in sensible order.
+ * !aNULL:!eNULL: really no NULL ciphers
+ * !MD5:!3DES:!DES:!RC4:!IDEA:!SEED: no weak or broken algorithms on old OpenSSL versions.
+ * !aDSS: no authentication with discrete logarithm DSA algorithm
+ * !SRP:!PSK: no secure remote password or pre-shared key authentication
+ */
+  #define PY_SSL_DEFAULT_CIPHER_STRING "DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK"
+#elif PY_SSL_DEFAULT_CIPHERS == 2
+/* Ignored in SSLContext constructor, only used to as _ssl.DEFAULT_CIPHER_STRING */
+  #define PY_SSL_DEFAULT_CIPHER_STRING SSL_DEFAULT_CIPHER_LIST
+#else
+  #error "Unsupported PY_SSL_DEFAULT_CIPHERS"
+#endif
+
 
 enum py_ssl_error {
     /* these mirror ssl.h */
@@ -2873,7 +2898,12 @@ _ssl__SSLContext_impl(PyTypeObject *type, int proto_version)
     /* A bare minimum cipher list without completely broken cipher suites.
      * It's far from perfect but gives users a better head start. */
     if (proto_version != PY_SSL_VERSION_SSL2) {
-        result = SSL_CTX_set_cipher_list(ctx, "HIGH:!aNULL:!eNULL:!MD5");
+#if PY_SSL_DEFAULT_CIPHERS == 2
+        /* stick to OpenSSL's default settings */
+        result = 1;
+#else
+        result = SSL_CTX_set_cipher_list(ctx, PY_SSL_DEFAULT_CIPHER_STRING);
+#endif
     } else {
         /* SSLv2 needs MD5 */
         result = SSL_CTX_set_cipher_list(ctx, "HIGH:!aNULL:!eNULL");
@@ -5430,6 +5460,9 @@ PyInit__ssl(void)
                              (PyObject *)&PySSLSession_Type) != 0)
         return NULL;
 
+    PyModule_AddStringConstant(m, "_DEFAULT_CIPHERS",
+                               PY_SSL_DEFAULT_CIPHER_STRING);
+
     PyModule_AddIntConstant(m, "SSL_ERROR_ZERO_RETURN",
                             PY_SSL_ERROR_ZERO_RETURN);
     PyModule_AddIntConstant(m, "SSL_ERROR_WANT_READ",
diff --git a/configure b/configure
index f94d16bf0b0c..caa86677ea11 100755
--- a/configure
+++ b/configure
@@ -840,6 +840,7 @@ enable_big_digits
 with_computed_gotos
 with_ensurepip
 with_openssl
+with_ssl_default_suites
 '
       ac_precious_vars='build_alias
 host_alias
@@ -1538,6 +1539,11 @@ Optional Packages:
   --with(out)-ensurepip=[=upgrade]
                           "install" or "upgrade" using bundled pip
   --with-openssl=DIR      root of the OpenSSL directory
+  --with-ssl-default-suites=[python|openssl|STRING]
+                          Override default cipher suites string, python: use
+                          Python's preferred selection (default), openssl:
+                          leave OpenSSL's defaults untouched, STRING: use a
+                          custom string, PROTOCOL_SSLv2 ignores the setting
 
 Some influential environment variables:
   MACHDEP     name for machine-dependent library files
@@ -16931,6 +16937,48 @@ $as_echo "#define HAVE_X509_VERIFY_PARAM_SET1_HOST 1" >>confdefs.h
     LIBS="$save_LIBS"
 fi
 
+# ssl module default cipher suite string
+
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-ssl-default-suites" >&5
+$as_echo_n "checking for --with-ssl-default-suites... " >&6; }
+
+# Check whether --with-ssl-default-suites was given.
+if test "${with_ssl_default_suites+set}" = set; then :
+  withval=$with_ssl_default_suites;
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $withval" >&5
+$as_echo "$withval" >&6; }
+case "$withval" in
+    python)
+        $as_echo "#define PY_SSL_DEFAULT_CIPHERS 1" >>confdefs.h
+
+        ;;
+    openssl)
+        $as_echo "#define PY_SSL_DEFAULT_CIPHERS 2" >>confdefs.h
+
+        ;;
+    *)
+        $as_echo "#define PY_SSL_DEFAULT_CIPHERS 0" >>confdefs.h
+
+        cat >>confdefs.h <<_ACEOF
+#define PY_SSL_DEFAULT_CIPHER_STRING "$withval"
+_ACEOF
+
+        ;;
+esac
+
+else
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: python" >&5
+$as_echo "python" >&6; }
+$as_echo "#define PY_SSL_DEFAULT_CIPHERS 1" >>confdefs.h
+
+
+fi
+
+
+
 # generate output files
 ac_config_files="$ac_config_files Makefile.pre Misc/python.pc Misc/python-config.sh"
 
diff --git a/configure.ac b/configure.ac
index 65248630b6e1..af10a6d3cad5 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5497,6 +5497,43 @@ if test "$have_openssl" = yes; then
     LIBS="$save_LIBS"
 fi
 
+# ssl module default cipher suite string
+AH_TEMPLATE(PY_SSL_DEFAULT_CIPHERS,
+  [Default cipher suites list for ssl module.
+   1: Python's preferred selection, 2: leave OpenSSL defaults untouched, 0: custom string])
+AH_TEMPLATE(PY_SSL_DEFAULT_CIPHER_STRING,
+  [Cipher suite string for PY_SSL_DEFAULT_CIPHERS=0]
+)
+
+AC_MSG_CHECKING(for --with-ssl-default-suites)
+AC_ARG_WITH(ssl-default-suites,
+            AS_HELP_STRING([--with-ssl-default-suites=@<:@python|openssl|STRING@:>@],
+                           [Override default cipher suites string,
+                            python: use Python's preferred selection (default),
+                            openssl: leave OpenSSL's defaults untouched,
+                            STRING: use a custom string,
+                            PROTOCOL_SSLv2 ignores the setting]),
+[
+AC_MSG_RESULT($withval)
+case "$withval" in
+    python)
+        AC_DEFINE(PY_SSL_DEFAULT_CIPHERS, 1)
+        ;;
+    openssl)
+        AC_DEFINE(PY_SSL_DEFAULT_CIPHERS, 2)
+        ;;
+    *)
+        AC_DEFINE(PY_SSL_DEFAULT_CIPHERS, 0)
+        AC_DEFINE_UNQUOTED(PY_SSL_DEFAULT_CIPHER_STRING, "$withval")
+        ;;
+esac
+],
+[
+AC_MSG_RESULT(python)
+AC_DEFINE(PY_SSL_DEFAULT_CIPHERS, 1)
+])
+
+
 # generate output files
 AC_CONFIG_FILES(Makefile.pre Misc/python.pc Misc/python-config.sh)
 AC_CONFIG_FILES([Modules/ld_so_aix], [chmod +x Modules/ld_so_aix])
diff --git a/pyconfig.h.in b/pyconfig.h.in
index a18e3ca854e6..a0efff9777d2 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -1314,6 +1314,13 @@
 /* Define to printf format modifier for Py_ssize_t */
 #undef PY_FORMAT_SIZE_T
 
+/* Default cipher suites list for ssl module. 1: Python's preferred selection,
+   2: leave OpenSSL defaults untouched, 0: custom string */
+#undef PY_SSL_DEFAULT_CIPHERS
+
+/* Cipher suite string for PY_SSL_DEFAULT_CIPHERS=0 */
+#undef PY_SSL_DEFAULT_CIPHER_STRING
+
 /* Define to emit a locale compatibility warning in the C locale */
 #undef PY_WARN_ON_C_LOCALE
 


From webhook-mailer at python.org  Mon Jan 29 08:25:16 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Mon, 29 Jan 2018 13:25:16 -0000
Subject: [Python-checkins] bpo-28414: ssl module idna test (#5395)
Message-ID: <mailman.49.1517232322.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/66e5742becce38e69a8f09e5f7051445fc57e92e
commit: 66e5742becce38e69a8f09e5f7051445fc57e92e
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T14:25:13+01:00
summary:

bpo-28414: ssl module idna test (#5395)

Add test cases for IDNA 2003 and 2008 host names. IDNA 2003
internationalized host names are working since bpo-31399 has landed. IDNA
2008 deviations are still broken and will be fixed in another patch.

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Tests/2018-01-28-21-19-13.bpo-28414.a6Onzt.rst
M Lib/test/test_ssl.py

diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index 6a8bf0ef7be4..a253f51d2a44 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -105,6 +105,7 @@ def data_file(*name):
 SIGNING_CA = data_file("capath", "ceff1710.0")
 # cert with all kinds of subject alt names
 ALLSANFILE = data_file("allsans.pem")
+IDNSANSFILE = data_file("idnsans.pem")
 
 REMOTE_HOST = "self-signed.pythontest.net"
 
@@ -1612,7 +1613,6 @@ def test_error_types(self):
 
 
 class SimpleBackgroundTests(unittest.TestCase):
-
     """Tests that connect to a simple server running in the background"""
 
     def setUp(self):
@@ -2630,6 +2630,70 @@ def test_dual_rsa_ecc(self):
                 cipher = s.cipher()[0].split('-')
                 self.assertTrue(cipher[:2], ('ECDHE', 'ECDSA'))
 
+    def test_check_hostname_idn(self):
+        if support.verbose:
+            sys.stdout.write("\n")
+
+        server_context = ssl.SSLContext(ssl.PROTOCOL_TLS)
+        server_context.load_cert_chain(IDNSANSFILE)
+
+        context = ssl.SSLContext(ssl.PROTOCOL_TLS)
+        context.verify_mode = ssl.CERT_REQUIRED
+        context.check_hostname = True
+        context.load_verify_locations(SIGNING_CA)
+
+        # correct hostname should verify, when specified in several
+        # different ways
+        idn_hostnames = [
+            ('k?nig.idn.pythontest.net',
+             'k?nig.idn.pythontest.net',),
+            ('xn--knig-5qa.idn.pythontest.net',
+             'xn--knig-5qa.idn.pythontest.net'),
+            (b'xn--knig-5qa.idn.pythontest.net',
+             b'xn--knig-5qa.idn.pythontest.net'),
+
+            ('k?nigsg??chen.idna2003.pythontest.net',
+             'k?nigsg??chen.idna2003.pythontest.net'),
+            ('xn--knigsgsschen-lcb0w.idna2003.pythontest.net',
+             'xn--knigsgsschen-lcb0w.idna2003.pythontest.net'),
+            (b'xn--knigsgsschen-lcb0w.idna2003.pythontest.net',
+             b'xn--knigsgsschen-lcb0w.idna2003.pythontest.net'),
+        ]
+        for server_hostname, expected_hostname in idn_hostnames:
+            server = ThreadedEchoServer(context=server_context, chatty=True)
+            with server:
+                with context.wrap_socket(socket.socket(),
+                                         server_hostname=server_hostname) as s:
+                    self.assertEqual(s.server_hostname, expected_hostname)
+                    s.connect((HOST, server.port))
+                    cert = s.getpeercert()
+                    self.assertEqual(s.server_hostname, expected_hostname)
+                    self.assertTrue(cert, "Can't get peer certificate.")
+
+                with ssl.SSLSocket(socket.socket(),
+                                   server_hostname=server_hostname) as s:
+                    s.connect((HOST, server.port))
+                    s.getpeercert()
+                    self.assertEqual(s.server_hostname, expected_hostname)
+
+        # bug https://bugs.python.org/issue28414
+        # IDNA 2008 deviations are broken
+        idna2008 = 'xn--knigsgchen-b4a3dun.idna2008.pythontest.net'
+        server = ThreadedEchoServer(context=server_context, chatty=True)
+        with server:
+            with self.assertRaises(UnicodeError):
+                with context.wrap_socket(socket.socket(),
+                                         server_hostname=idna2008) as s:
+                    s.connect((HOST, server.port))
+
+        # incorrect hostname should raise an exception
+        server = ThreadedEchoServer(context=server_context, chatty=True)
+        with server:
+            with context.wrap_socket(socket.socket(),
+                                     server_hostname="python.example.org") as s:
+                with self.assertRaises(ssl.CertificateError):
+                    s.connect((HOST, server.port))
+
     def test_wrong_cert(self):
         """Connecting when the server rejects the client's certificate
 
diff --git a/Misc/NEWS.d/next/Tests/2018-01-28-21-19-13.bpo-28414.a6Onzt.rst b/Misc/NEWS.d/next/Tests/2018-01-28-21-19-13.bpo-28414.a6Onzt.rst
new file mode 100644
index 000000000000..c64bc55bb850
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-01-28-21-19-13.bpo-28414.a6Onzt.rst
@@ -0,0 +1,3 @@
+Add test cases for IDNA 2003 and 2008 host names. IDNA 2003
+internationalized host names are working since bpo-31399 has landed. IDNA
+2008 are still broken.


From webhook-mailer at python.org  Mon Jan 29 09:17:48 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Mon, 29 Jan 2018 14:17:48 -0000
Subject: [Python-checkins] Add a test for pdb until command in coroutine
 (#5427)
Message-ID: <mailman.50.1517235469.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/4f4ef0acbad81f4b05f370e8ff14ddf949773291
commit: 4f4ef0acbad81f4b05f370e8ff14ddf949773291
branch: master
author: Andrew Svetlov <andrew.svetlov at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T16:17:45+02:00
summary:

Add a test for pdb until command in coroutine (#5427)

files:
M Lib/test/test_pdb.py

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 08d374ca834a..70d8d1d32613 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -982,6 +982,52 @@ def test_pdb_until_command_for_generator():
     finished
     """
 
+def test_pdb_until_command_for_coroutine():
+    """Testing no unwindng stack for coroutines
+       for "until" command if target breakpoing is not reached
+
+    >>> import asyncio
+
+    >>> async def test_coro():
+    ...     print(0)
+    ...     await asyncio.sleep(0)
+    ...     print(1)
+    ...     await asyncio.sleep(0)
+    ...     print(2)
+    ...     await asyncio.sleep(0)
+    ...     print(3)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.new_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'until 8',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_until_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_until_command_for_coroutine[1]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) until 8
+    0
+    1
+    2
+    > <doctest test.test_pdb.test_pdb_until_command_for_coroutine[1]>(8)test_coro()
+    -> print(3)
+    (Pdb) continue
+    3
+    finished
+    """
+
 def test_pdb_next_command_in_generator_for_loop():
     """The next command on returning from a generator controlled by a for loop.
 


From webhook-mailer at python.org  Mon Jan 29 09:31:35 2018
From: webhook-mailer at python.org (=?utf-8?q?=C3=89ric?= Araujo)
Date: Mon, 29 Jan 2018 14:31:35 -0000
Subject: [Python-checkins] bpo-32304: Fix distutils upload for tar files
 ending with b'\r' (GH-5264) (GH-5331)
Message-ID: <mailman.51.1517236297.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/f5a793522d539afc84ac7888c9ad189097c43a75
commit: f5a793522d539afc84ac7888c9ad189097c43a75
branch: 2.7
author: Bo Bayles <bbayles at gmail.com>
committer: ?ric Araujo <merwok at netwok.org>
date: 2018-01-29T09:31:32-05:00
summary:

bpo-32304: Fix distutils upload for tar files ending with b'\r' (GH-5264) (GH-5331)

Patch by Bo Bayles.

files:
A Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
M Doc/whatsnew/2.7.rst
M Lib/distutils/command/upload.py
M Lib/distutils/tests/test_upload.py

diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst
index 1857aa4e8f32..53333ebf7c1a 100644
--- a/Doc/whatsnew/2.7.rst
+++ b/Doc/whatsnew/2.7.rst
@@ -1218,6 +1218,11 @@ changes, or look through the Subversion logs for all the details.
   created some new files that should be included.
   (Fixed by Tarek Ziad?; :issue:`8688`.)
 
+  The ``upload`` command now longer tries to change CR end-of-line characters
+  to CRLF.  This fixes a corruption issue with sdists that ended with a byte
+  equivalent to CR.
+  (Contributed by Bo Bayles in :issue:`32304`.)
+
 * The :mod:`doctest` module's :const:`IGNORE_EXCEPTION_DETAIL` flag
   will now ignore the name of the module containing the exception
   being tested.  (Patch by Lennart Regebro; :issue:`7490`.)
diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
index b773f47f7684..ea4b8a551a60 100644
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -155,8 +155,6 @@ def upload_file(self, command, pyversion, filename):
                 body.write(fn)
                 body.write("\r\n\r\n")
                 body.write(value)
-                if value and value[-1] == '\r':
-                    body.write('\n')  # write an extra newline (lurve Macs)
         body.write(end_boundary)
         body = body.getvalue()
 
diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py
index 3d4f30504e8d..d22575458753 100644
--- a/Lib/distutils/tests/test_upload.py
+++ b/Lib/distutils/tests/test_upload.py
@@ -128,6 +128,32 @@ def test_upload(self):
         auth = self.last_open.req.headers['Authorization']
         self.assertNotIn('\n', auth)
 
+    # bpo-32304: archives whose last byte was b'\r' were corrupted due to
+    # normalization intended for Mac OS 9.
+    def test_upload_correct_cr(self):
+        # content that ends with \r should not be modified.
+        tmp = self.mkdtemp()
+        path = os.path.join(tmp, 'xxx')
+        self.write_file(path, content='yy\r')
+        command, pyversion, filename = 'xxx', '2.6', path
+        dist_files = [(command, pyversion, filename)]
+        self.write_file(self.rc, PYPIRC_LONG_PASSWORD)
+
+        # other fields that ended with \r used to be modified, now are
+        # preserved.
+        pkg_dir, dist = self.create_dist(
+            dist_files=dist_files,
+            description='long description\r'
+        )
+        cmd = upload(dist)
+        cmd.ensure_finalized()
+        cmd.run()
+
+        headers = dict(self.last_open.req.headers)
+        self.assertEqual(headers['Content-length'], '2170')
+        self.assertIn(b'long description\r', self.last_open.req.data)
+        self.assertNotIn(b'long description\r\n', self.last_open.req.data)
+
     def test_upload_fails(self):
         self.next_msg = "Not Found"
         self.next_code = 404
diff --git a/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst b/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
new file mode 100644
index 000000000000..c199a644ad40
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-21-16-33-53.bpo-32304.TItrNv.rst
@@ -0,0 +1,2 @@
+distutils' upload command no longer corrupts tar files ending with a CR byte,
+and no longer tries to convert CR to CRLF in any of the upload text fields.


From webhook-mailer at python.org  Mon Jan 29 09:56:54 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Mon, 29 Jan 2018 14:56:54 -0000
Subject: [Python-checkins] Add a test for pdb until command in coroutine
 (GH-5427) (#5428)
Message-ID: <mailman.52.1517237815.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/757aad674808d8d4b717c3bf9994a8f90007798d
commit: 757aad674808d8d4b717c3bf9994a8f90007798d
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-29T16:56:46+02:00
summary:

Add a test for pdb until command in coroutine (GH-5427) (#5428)

(cherry picked from commit 4f4ef0acbad81f4b05f370e8ff14ddf949773291)

files:
M Lib/test/test_pdb.py

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 369a0a91e305..f2282c35c815 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -979,6 +979,52 @@ def test_pdb_until_command_for_generator():
     finished
     """
 
+def test_pdb_until_command_for_coroutine():
+    """Testing no unwindng stack for coroutines
+       for "until" command if target breakpoing is not reached
+
+    >>> import asyncio
+
+    >>> async def test_coro():
+    ...     print(0)
+    ...     await asyncio.sleep(0)
+    ...     print(1)
+    ...     await asyncio.sleep(0)
+    ...     print(2)
+    ...     await asyncio.sleep(0)
+    ...     print(3)
+
+    >>> async def test_main():
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     await test_coro()
+
+    >>> def test_function():
+    ...     loop = asyncio.new_event_loop()
+    ...     loop.run_until_complete(test_main())
+    ...     loop.close()
+    ...     print("finished")
+
+    >>> with PdbTestInput(['step',
+    ...                    'until 8',
+    ...                    'continue']):
+    ...     test_function()
+    > <doctest test.test_pdb.test_pdb_until_command_for_coroutine[2]>(3)test_main()
+    -> await test_coro()
+    (Pdb) step
+    --Call--
+    > <doctest test.test_pdb.test_pdb_until_command_for_coroutine[1]>(1)test_coro()
+    -> async def test_coro():
+    (Pdb) until 8
+    0
+    1
+    2
+    > <doctest test.test_pdb.test_pdb_until_command_for_coroutine[1]>(8)test_coro()
+    -> print(3)
+    (Pdb) continue
+    3
+    finished
+    """
+
 def test_pdb_next_command_in_generator_for_loop():
     """The next command on returning from a generator controlled by a for loop.
 


From webhook-mailer at python.org  Mon Jan 29 10:35:54 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 15:35:54 -0000
Subject: [Python-checkins] bpo-20891: Remove test_capi.test_bpo20891()
 (#5425)
Message-ID: <mailman.53.1517240155.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/0cecc22842dcc4090eb9cb99e7dababea7034a87
commit: 0cecc22842dcc4090eb9cb99e7dababea7034a87
branch: 3.6
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T16:35:50+01:00
summary:

bpo-20891: Remove test_capi.test_bpo20891() (#5425)

My first fix is not enough to make test_bpo20891() reliable. A second
fix is needed and it was decided to not backport it, so remove the
test instead.

For Python 3.6, the workaround is to call PyEval_InitThreads() before
spawning the first C thread.

Python 3.7 will have both fixes.

files:
M Lib/test/test_capi.py
M Programs/_testembed.c

diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py
index 848a5289bab7..b36a2ce585ce 100644
--- a/Lib/test/test_capi.py
+++ b/Lib/test/test_capi.py
@@ -494,19 +494,6 @@ def test_pre_initialization_api(self):
         self.assertEqual(out, '')
         self.assertEqual(err, '')
 
-    @unittest.skipIf(True,
-                     "FIXME: test fails randomly because of a race conditon, "
-                     "see bpo-20891")
-    def test_bpo20891(self):
-        """
-        bpo-20891: Calling PyGILState_Ensure in a non-Python thread before
-        calling PyEval_InitThreads() must not crash. PyGILState_Ensure() must
-        call PyEval_InitThreads() for us in this case.
-        """
-        out, err = self.run_embedded_interpreter("bpo20891")
-        self.assertEqual(out, '')
-        self.assertEqual(err, '')
-
 
 class SkipitemTest(unittest.TestCase):
 
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index b0f9087115b9..813cf307bda2 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -149,53 +149,6 @@ static int test_pre_initialization_api(void)
     return 0;
 }
 
-static void bpo20891_thread(void *lockp)
-{
-    PyThread_type_lock lock = *((PyThread_type_lock*)lockp);
-
-    PyGILState_STATE state = PyGILState_Ensure();
-    if (!PyGILState_Check()) {
-        fprintf(stderr, "PyGILState_Check failed!");
-        abort();
-    }
-
-    PyGILState_Release(state);
-
-    PyThread_release_lock(lock);
-
-    PyThread_exit_thread();
-}
-
-static int test_bpo20891(void)
-{
-    /* bpo-20891: Calling PyGILState_Ensure in a non-Python thread before
-       calling PyEval_InitThreads() must not crash. PyGILState_Ensure() must
-       call PyEval_InitThreads() for us in this case. */
-    PyThread_type_lock lock = PyThread_allocate_lock();
-    if (!lock) {
-        fprintf(stderr, "PyThread_allocate_lock failed!");
-        return 1;
-    }
-
-    _testembed_Py_Initialize();
-
-    long thrd = PyThread_start_new_thread(bpo20891_thread, &lock);
-    if (thrd == -1) {
-        fprintf(stderr, "PyThread_start_new_thread failed!");
-        return 1;
-    }
-    PyThread_acquire_lock(lock, WAIT_LOCK);
-
-    Py_BEGIN_ALLOW_THREADS
-    /* wait until the thread exit */
-    PyThread_acquire_lock(lock, WAIT_LOCK);
-    Py_END_ALLOW_THREADS
-
-    PyThread_free_lock(lock);
-
-    return 0;
-}
-
 
 /* *********************************************************
  * List of test cases and the function that implements it.
@@ -219,7 +172,6 @@ static struct TestCase TestCases[] = {
     { "forced_io_encoding", test_forced_io_encoding },
     { "repeated_init_and_subinterpreters", test_repeated_init_and_subinterpreters },
     { "pre_initialization_api", test_pre_initialization_api },
-    { "bpo20891", test_bpo20891 },
     { NULL, NULL }
 };
 


From webhook-mailer at python.org  Mon Jan 29 10:54:32 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 29 Jan 2018 15:54:32 -0000
Subject: [Python-checkins] bpo-30595: Increase
 test_queue_feeder_donot_stop_onexc() timeout (GH-2148) (GH-5429)
Message-ID: <mailman.54.1517241274.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/b60f43a0e6052b29517931eea591b0b29903f382
commit: b60f43a0e6052b29517931eea591b0b29903f382
branch: 2.7
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T16:54:29+01:00
summary:

bpo-30595: Increase test_queue_feeder_donot_stop_onexc() timeout (GH-2148) (GH-5429)

_test_multiprocessing.test_queue_feeder_donot_stop_onexc() now uses a
timeout of 1 second on Queue.get(), instead of 0.1 second, for slow
buildbots.

(cherry picked from commit 8f6eeaf21cdf4aea25fdefeec814a1ce07453fe9)

files:
M Lib/test/test_multiprocessing.py

diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
index a77bfee6718e..1071d7fac653 100644
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -671,7 +671,8 @@ def __reduce__(self):
             q = self.Queue()
             q.put(NotSerializable())
             q.put(True)
-            self.assertTrue(q.get(timeout=0.1))
+            # bpo-30595: use a timeout of 1 second for slow buildbots
+            self.assertTrue(q.get(timeout=1.0))
 
 
 #


From webhook-mailer at python.org  Mon Jan 29 11:27:53 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 29 Jan 2018 16:27:53 -0000
Subject: [Python-checkins] bpo-25988: Deprecate exposing collections.abc in
 collections GH-5414
Message-ID: <mailman.55.1517243275.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/e6d342156d2ab20fb88c0a5ec615fa8f602c0769
commit: e6d342156d2ab20fb88c0a5ec615fa8f602c0769
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T08:27:49-08:00
summary:

bpo-25988: Deprecate exposing collections.abc in collections GH-5414

files:
A Misc/NEWS.d/next/Library/2018-01-28-23-48-45.bpo-25988.I9uBct.rst
M Doc/library/collections.rst
M Doc/whatsnew/3.7.rst
M Lib/collections/__init__.py

diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst
index 18aaba65b252..772ff60fe983 100644
--- a/Doc/library/collections.rst
+++ b/Doc/library/collections.rst
@@ -35,8 +35,8 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`,
 
 .. versionchanged:: 3.3
     Moved :ref:`collections-abstract-base-classes` to the :mod:`collections.abc` module.
-    For backwards compatibility, they continue to be visible in this module
-    as well.
+    For backwards compatibility, they continue to be visible in this module through
+    Python 3.7.  Subsequently, they will be removed entirely.
 
 
 :class:`ChainMap` objects
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index e36e505fd8ae..71070df76470 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -880,6 +880,11 @@ Other CPython Implementation Changes
 Deprecated
 ==========
 
+* In Python 3.8, the abstract base classes in :mod:`collections.abc` will no
+  longer be exposed in the regular :mod:`collections` module.  This will help
+  create a clearer distinction between the concrete classes and the abstract
+  base classes.
+
 * Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated
   in comprehensions and generator expressions (aside from the iterable expression
   in the leftmost :keyword:`for` clause). This ensures that comprehensions
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 21d91fd61d60..8aeee359cd62 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -18,10 +18,14 @@
             'UserString', 'Counter', 'OrderedDict', 'ChainMap']
 
 # For backwards compatibility, continue to make the collections ABCs
-# available through the collections module.
-from _collections_abc import *
+# through Python 3.6 available through the collections module.
+# Note, no new collections ABCs were added in Python 3.7
 import _collections_abc
-__all__ += _collections_abc.__all__
+from _collections_abc import (AsyncGenerator, AsyncIterable, AsyncIterator,
+    Awaitable, ByteString, Callable, Collection, Container, Coroutine,
+    Generator, Hashable, ItemsView, Iterable, Iterator, KeysView, Mapping,
+    MappingView, MutableMapping, MutableSequence, MutableSet, Reversible,
+    Sequence, Set, Sized, ValuesView)
 
 from operator import itemgetter as _itemgetter, eq as _eq
 from keyword import iskeyword as _iskeyword
diff --git a/Misc/NEWS.d/next/Library/2018-01-28-23-48-45.bpo-25988.I9uBct.rst b/Misc/NEWS.d/next/Library/2018-01-28-23-48-45.bpo-25988.I9uBct.rst
new file mode 100644
index 000000000000..df350779ea4a
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-28-23-48-45.bpo-25988.I9uBct.rst
@@ -0,0 +1,2 @@
+Deprecate exposing the contents of collections.abc in the regular
+collections module.


From webhook-mailer at python.org  Mon Jan 29 12:24:14 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Mon, 29 Jan 2018 17:24:14 -0000
Subject: [Python-checkins] Document the error return of PyLong_As* APIs.
 (GH-5396) (#5404)
Message-ID: <mailman.56.1517246655.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/586986182343eb7475046a5429a1a0e1f368c7ea
commit: 586986182343eb7475046a5429a1a0e1f368c7ea
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T09:24:11-08:00
summary:

Document the error return of PyLong_As* APIs. (GH-5396) (#5404)

Document the error return of PyLong_As* APIs.

A frequent Python C API usage error is neglecting to check the return
value and/or PyErr_Occurred().
(cherry picked from commit f5b04a360e44aa9733f7a92dd66d2292d6c52955)

files:
M Doc/c-api/long.rst

diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst
index f50680b3d295..4f16b578eb59 100644
--- a/Doc/c-api/long.rst
+++ b/Doc/c-api/long.rst
@@ -10,6 +10,9 @@ Integer Objects
 
 All integers are implemented as "long" integer objects of arbitrary size.
 
+On error, most ``PyLong_As*`` APIs return ``(return type)-1`` which cannot be
+distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 .. c:type:: PyLongObject
 
    This subtype of :c:type:`PyObject` represents a Python integer object.
@@ -134,6 +137,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *obj* is out of range for a
    :c:type:`long`.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: long PyLong_AsLongAndOverflow(PyObject *obj, int *overflow)
 
@@ -146,6 +151,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    return ``-1``; otherwise, set *\*overflow* to ``0``.  If any other exception
    occurs set *\*overflow* to ``0`` and return ``-1`` as usual.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: long long PyLong_AsLongLong(PyObject *obj)
 
@@ -159,6 +166,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *obj* is out of range for a
    :c:type:`long`.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: long long PyLong_AsLongLongAndOverflow(PyObject *obj, int *overflow)
 
@@ -171,6 +180,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    and return ``-1``; otherwise, set *\*overflow* to ``0``.  If any other
    exception occurs set *\*overflow* to ``0`` and return ``-1`` as usual.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
    .. versionadded:: 3.2
 
 
@@ -186,6 +197,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`Py_ssize_t`.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: unsigned long PyLong_AsUnsignedLong(PyObject *pylong)
 
@@ -199,15 +212,25 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`unsigned long`.
 
+   Returns ``(unsigned long)-1`` on error.
+   Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: size_t PyLong_AsSize_t(PyObject *pylong)
 
+   .. index::
+      single: SIZE_MAX
+      single: OverflowError (built-in exception)
+
    Return a C :c:type:`size_t` representation of *pylong*.  *pylong* must be
    an instance of :c:type:`PyLongObject`.
 
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`size_t`.
 
+   Returns ``(size_t)-1`` on error.
+   Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: unsigned long long PyLong_AsUnsignedLongLong(PyObject *pylong)
 
@@ -220,6 +243,9 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for an
    :c:type:`unsigned long long`.
 
+   Returns ``(unsigned long long)-1`` on error.
+   Use :c:func:`PyErr_Occurred` to disambiguate.
+
    .. versionchanged:: 3.1
       A negative *pylong* now raises :exc:`OverflowError`, not :exc:`TypeError`.
 
@@ -233,6 +259,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    If the value of *obj* is out of range for an :c:type:`unsigned long`,
    return the reduction of that value modulo ``ULONG_MAX + 1``.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: unsigned long long PyLong_AsUnsignedLongLongMask(PyObject *obj)
 
@@ -243,6 +271,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    If the value of *obj* is out of range for an :c:type:`unsigned long long`,
    return the reduction of that value modulo ``PY_ULLONG_MAX + 1``.
 
+   Returns -1 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: double PyLong_AsDouble(PyObject *pylong)
 
@@ -252,6 +282,8 @@ All integers are implemented as "long" integer objects of arbitrary size.
    Raise :exc:`OverflowError` if the value of *pylong* is out of range for a
    :c:type:`double`.
 
+   Returns -1.0 on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+
 
 .. c:function:: void* PyLong_AsVoidPtr(PyObject *pylong)
 
@@ -259,3 +291,5 @@ All integers are implemented as "long" integer objects of arbitrary size.
    If *pylong* cannot be converted, an :exc:`OverflowError` will be raised.  This
    is only assured to produce a usable :c:type:`void` pointer for values created
    with :c:func:`PyLong_FromVoidPtr`.
+
+   Returns NULL on error.  Use :c:func:`PyErr_Occurred` to disambiguate.


From webhook-mailer at python.org  Mon Jan 29 13:07:48 2018
From: webhook-mailer at python.org (R. David Murray)
Date: Mon, 29 Jan 2018 18:07:48 -0000
Subject: [Python-checkins] bpo-27931: Fix email address header parsing error
 (#5329)
Message-ID: <mailman.57.1517249270.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/aa218d1649690d1c1ba86a9972f7fae646bf1a8f
commit: aa218d1649690d1c1ba86a9972f7fae646bf1a8f
branch: master
author: jayyyin <jayyin11043 at hotmail.com>
committer: R. David Murray <rdmurray at bitdance.com>
date: 2018-01-29T13:07:44-05:00
summary:

bpo-27931: Fix email address header parsing error (#5329)

Correctly handle addresses whose username is an empty quoted string.

files:
A Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst
M Lib/email/_header_value_parser.py
M Lib/test/test_email/test__header_value_parser.py

diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index b34c58bf85d5..d2d1f5cb5f3d 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -430,7 +430,10 @@ def route(self):
     def addr_spec(self):
         for x in self:
             if x.token_type == 'addr-spec':
-                return x.addr_spec
+                if x.local_part:
+                    return x.addr_spec
+                else:
+                    return quote_string(x.local_part) + x.addr_spec
         else:
             return '<>'
 
@@ -1164,6 +1167,9 @@ def get_bare_quoted_string(value):
             "expected '\"' but found '{}'".format(value))
     bare_quoted_string = BareQuotedString()
     value = value[1:]
+    if value[0] == '"':
+        token, value = get_qcontent(value)
+        bare_quoted_string.append(token)
     while value and value[0] != '"':
         if value[0] in WSP:
             token, value = get_fws(value)
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index 1667617b9e46..5cdc4bcecad4 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -490,6 +490,10 @@ def test_get_bare_quoted_string_must_start_with_dquote(self):
         with self.assertRaises(errors.HeaderParseError):
             parser.get_bare_quoted_string('  "foo"')
 
+    def test_get_bare_quoted_string_only_quotes(self):
+        self._test_get_x(parser.get_bare_quoted_string,
+                         '""', '""', '', [], '')
+
     def test_get_bare_quoted_string_following_wsp_preserved(self):
         self._test_get_x(parser.get_bare_quoted_string,
              '"foo"\t bar', '"foo"', 'foo', [], '\t bar')
@@ -1467,6 +1471,19 @@ def test_get_angle_addr_empty(self):
         self.assertIsNone(angle_addr.route)
         self.assertEqual(angle_addr.addr_spec, '<>')
 
+    def test_get_angle_addr_qs_only_quotes(self):
+        angle_addr = self._test_get_x(parser.get_angle_addr,
+            '<""@example.com>',
+            '<""@example.com>',
+            '<""@example.com>',
+            [],
+            '')
+        self.assertEqual(angle_addr.token_type, 'angle-addr')
+        self.assertEqual(angle_addr.local_part, '')
+        self.assertEqual(angle_addr.domain, 'example.com')
+        self.assertIsNone(angle_addr.route)
+        self.assertEqual(angle_addr.addr_spec, '""@example.com')
+
     def test_get_angle_addr_with_cfws(self):
         angle_addr = self._test_get_x(parser.get_angle_addr,
             ' (foo) <dinsdale at example.com>(bar)',
diff --git a/Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst b/Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst
new file mode 100644
index 000000000000..7324247357a6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst
@@ -0,0 +1 @@
+Fix email address header parsing error when the username is an empty quoted string. Patch by Xiang Zhang.


From webhook-mailer at python.org  Mon Jan 29 13:31:45 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 29 Jan 2018 18:31:45 -0000
Subject: [Python-checkins] bpo-32707: Fix warnings in hamt.c (#5430)
Message-ID: <mailman.58.1517250708.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/b647d7039d396b1da71ab33b101a78b53d4e6834
commit: b647d7039d396b1da71ab33b101a78b53d4e6834
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-29T13:31:37-05:00
summary:

bpo-32707: Fix warnings in hamt.c (#5430)

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index 79c42c0f4334..38f76d11e3eb 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -620,7 +620,8 @@ hamt_node_bitmap_clone_without(PyHamtNode_Bitmap *o, uint32_t bit)
         new->b_array[i] = o->b_array[i];
     }
 
-    for (i = val_idx + 1; i < Py_SIZE(o); i++) {
+    assert(Py_SIZE(o) >= 0 && Py_SIZE(o) <= 32);
+    for (i = val_idx + 1; i < (uint32_t)Py_SIZE(o); i++) {
         Py_XINCREF(o->b_array[i]);
         new->b_array[i - 2] = o->b_array[i];
     }
@@ -920,7 +921,7 @@ hamt_node_bitmap_assoc(PyHamtNode_Bitmap *self,
 
             uint32_t key_idx = 2 * idx;
             uint32_t val_idx = key_idx + 1;
-            Py_ssize_t i;
+            uint32_t i;
 
             *added_leaf = 1;
 
@@ -947,7 +948,8 @@ hamt_node_bitmap_assoc(PyHamtNode_Bitmap *self,
 
             /* Copy all keys/values that will be after the new key/value
                we are adding. */
-            for (i = key_idx; i < Py_SIZE(self); i++) {
+            assert(Py_SIZE(self) >= 0 && Py_SIZE(self) <= 32);
+            for (i = key_idx; i < (uint32_t)Py_SIZE(self); i++) {
                 Py_XINCREF(self->b_array[i]);
                 new_node->b_array[i + 2] = self->b_array[i];
             }


From webhook-mailer at python.org  Mon Jan 29 14:31:56 2018
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 29 Jan 2018 19:31:56 -0000
Subject: [Python-checkins] bpo-32703: Fix coroutine resource warning in case
 where there's an error (GH-5410)
Message-ID: <mailman.59.1517254318.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/2a2270db9be9bdac5ffd2d50929bf905e7391a06
commit: 2a2270db9be9bdac5ffd2d50929bf905e7391a06
branch: master
author: Yury Selivanov <yury at magic.io>
committer: GitHub <noreply at github.com>
date: 2018-01-29T14:31:47-05:00
summary:

bpo-32703: Fix coroutine resource warning in case where there's an error (GH-5410)

The commit removes one unnecessary "if" clause in genobject.c.  That "if" clause was masking un-awaited coroutines warnings just to make writing unittests more convenient.

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-29-01-15-17.bpo-32703.mwrF4-.rst
M Lib/test/test_coroutines.py
M Objects/genobject.c

diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py
index 19a3444b6ad7..f3d77192ae76 100644
--- a/Lib/test/test_coroutines.py
+++ b/Lib/test/test_coroutines.py
@@ -520,34 +520,38 @@ def test_func_3(self):
         async def foo():
             raise StopIteration
 
-        with silence_coro_gc():
-            self.assertRegex(repr(foo()), '^<coroutine object.* at 0x.*>$')
+        coro = foo()
+        self.assertRegex(repr(coro), '^<coroutine object.* at 0x.*>$')
+        coro.close()
 
     def test_func_4(self):
         async def foo():
             raise StopIteration
+        coro = foo()
 
         check = lambda: self.assertRaisesRegex(
             TypeError, "'coroutine' object is not iterable")
 
         with check():
-            list(foo())
+            list(coro)
 
         with check():
-            tuple(foo())
+            tuple(coro)
 
         with check():
-            sum(foo())
+            sum(coro)
 
         with check():
-            iter(foo())
+            iter(coro)
 
-        with silence_coro_gc(), check():
-            for i in foo():
+        with check():
+            for i in coro:
                 pass
 
-        with silence_coro_gc(), check():
-            [i for i in foo()]
+        with check():
+            [i for i in coro]
+
+        coro.close()
 
     def test_func_5(self):
         @types.coroutine
@@ -560,8 +564,11 @@ def bar():
         check = lambda: self.assertRaisesRegex(
             TypeError, "'coroutine' object is not iterable")
 
+        coro = foo()
         with check():
-            for el in foo(): pass
+            for el in coro:
+                pass
+        coro.close()
 
         # the following should pass without an error
         for el in bar():
@@ -588,35 +595,53 @@ def bar():
     def test_func_7(self):
         async def bar():
             return 10
+        coro = bar()
 
         def foo():
-            yield from bar()
-
-        with silence_coro_gc(), self.assertRaisesRegex(
-            TypeError,
-            "cannot 'yield from' a coroutine object in a non-coroutine generator"):
+            yield from coro
 
+        with self.assertRaisesRegex(
+                TypeError,
+                "cannot 'yield from' a coroutine object in "
+                "a non-coroutine generator"):
             list(foo())
 
+        coro.close()
+
     def test_func_8(self):
         @types.coroutine
         def bar():
-            return (yield from foo())
+            return (yield from coro)
 
         async def foo():
             return 'spam'
 
-        self.assertEqual(run_async(bar()), ([], 'spam') )
+        coro = foo()
+        self.assertEqual(run_async(bar()), ([], 'spam'))
+        coro.close()
 
     def test_func_9(self):
-        async def foo(): pass
+        async def foo():
+            pass
 
         with self.assertWarnsRegex(
-            RuntimeWarning, "coroutine '.*test_func_9.*foo' was never awaited"):
+                RuntimeWarning,
+                r"coroutine '.*test_func_9.*foo' was never awaited"):
 
             foo()
             support.gc_collect()
 
+        with self.assertWarnsRegex(
+                RuntimeWarning,
+                r"coroutine '.*test_func_9.*foo' was never awaited"):
+
+            with self.assertRaises(TypeError):
+                # See bpo-32703.
+                for _ in foo():
+                    pass
+
+            support.gc_collect()
+
     def test_func_10(self):
         N = 0
 
@@ -674,11 +699,14 @@ def test_func_12(self):
     def test_func_13(self):
         async def g():
             pass
+
+        coro = g()
         with self.assertRaisesRegex(
-            TypeError,
-            "can't send non-None value to a just-started coroutine"):
+                TypeError,
+                "can't send non-None value to a just-started coroutine"):
+            coro.send('spam')
 
-            g().send('spam')
+        coro.close()
 
     def test_func_14(self):
         @types.coroutine
@@ -977,8 +1005,6 @@ def wrap():
             return 42
 
         async def foo():
-            b = bar()
-
             db = {'b':  lambda: wrap}
 
             class DB:
@@ -1023,19 +1049,21 @@ def ident(val):
     def test_await_12(self):
         async def coro():
             return 'spam'
+        c = coro()
 
         class Awaitable:
             def __await__(self):
-                return coro()
+                return c
 
         async def foo():
             return await Awaitable()
 
         with self.assertRaisesRegex(
-            TypeError, r"__await__\(\) returned a coroutine"):
-
+                TypeError, r"__await__\(\) returned a coroutine"):
             run_async(foo())
 
+        c.close()
+
     def test_await_13(self):
         class Awaitable:
             def __await__(self):
@@ -1991,14 +2019,15 @@ def wrap(gen):
         finally:
             with self.assertWarns(DeprecationWarning):
                 sys.set_coroutine_wrapper(None)
+            f.close()
 
         with self.assertWarns(DeprecationWarning):
             self.assertIsNone(sys.get_coroutine_wrapper())
 
         wrapped = None
-        with silence_coro_gc():
-            foo()
+        coro = foo()
         self.assertFalse(wrapped)
+        coro.close()
 
     def test_set_wrapper_2(self):
         with self.assertWarns(DeprecationWarning):
@@ -2022,11 +2051,12 @@ def wrapper(coro):
             sys.set_coroutine_wrapper(wrapper)
         try:
             with silence_coro_gc(), self.assertRaisesRegex(
-                RuntimeError,
-                r"coroutine wrapper.*\.wrapper at 0x.*attempted to "
-                r"recursively wrap .* wrap .*"):
+                    RuntimeError,
+                    r"coroutine wrapper.*\.wrapper at 0x.*attempted to "
+                    r"recursively wrap .* wrap .*"):
 
                 foo()
+
         finally:
             with self.assertWarns(DeprecationWarning):
                 sys.set_coroutine_wrapper(None)
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-29-01-15-17.bpo-32703.mwrF4-.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-29-01-15-17.bpo-32703.mwrF4-.rst
new file mode 100644
index 000000000000..cfd166b64c2b
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-29-01-15-17.bpo-32703.mwrF4-.rst	
@@ -0,0 +1,2 @@
+Fix coroutine's ResourceWarning when there's an active error set when it's
+being finalized.
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 1fdb57c8ce88..88b03c5ef313 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -44,9 +44,10 @@ _PyGen_Finalize(PyObject *self)
     PyObject *res = NULL;
     PyObject *error_type, *error_value, *error_traceback;
 
-    if (gen->gi_frame == NULL || gen->gi_frame->f_stacktop == NULL)
+    if (gen->gi_frame == NULL || gen->gi_frame->f_stacktop == NULL) {
         /* Generator isn't paused, so no need to close */
         return;
+    }
 
     if (PyAsyncGen_CheckExact(self)) {
         PyAsyncGenObject *agen = (PyAsyncGenObject*)self;
@@ -75,18 +76,18 @@ _PyGen_Finalize(PyObject *self)
        issue a RuntimeWarning. */
     if (gen->gi_code != NULL &&
         ((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE &&
-        gen->gi_frame->f_lasti == -1) {
-        if (!error_value) {
-            _PyErr_WarnUnawaitedCoroutine((PyObject *)gen);
-        }
+        gen->gi_frame->f_lasti == -1)
+    {
+        _PyErr_WarnUnawaitedCoroutine((PyObject *)gen);
     }
     else {
         res = gen_close(gen, NULL);
     }
 
     if (res == NULL) {
-        if (PyErr_Occurred())
+        if (PyErr_Occurred()) {
             PyErr_WriteUnraisable(self);
+        }
     }
     else {
         Py_DECREF(res);


From webhook-mailer at python.org  Mon Jan 29 14:34:00 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Mon, 29 Jan 2018 19:34:00 -0000
Subject: [Python-checkins] rename _imp initialization function to follow
 conventions (#5432)
Message-ID: <mailman.60.1517254441.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/c65ef772c335324deb03626cc447b89987781f27
commit: c65ef772c335324deb03626cc447b89987781f27
branch: master
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T11:33:57-08:00
summary:

rename _imp initialization function to follow conventions (#5432)

When the C imp module became _imp in 6f44d66bc491bad5b8d897a68da68e009e27829d, the initialization function should have been renamed from PyInit_imp to PyInit__imp.

files:
M Include/import.h
M Modules/config.c.in
M PC/config.c
M Python/import.c
M Python/pylifecycle.c

diff --git a/Include/import.h b/Include/import.h
index 26c4b1f18ef2..ac3fc3bd28c0 100644
--- a/Include/import.h
+++ b/Include/import.h
@@ -10,7 +10,7 @@ extern "C" {
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(_PyInitError) _PyImportZip_Init(void);
 
-PyMODINIT_FUNC PyInit_imp(void);
+PyMODINIT_FUNC PyInit__imp(void);
 #endif /* !Py_LIMITED_API */
 PyAPI_FUNC(long) PyImport_GetMagicNumber(void);
 PyAPI_FUNC(const char *) PyImport_GetMagicTag(void);
diff --git a/Modules/config.c.in b/Modules/config.c.in
index 7b77199c2e93..d69e8e88b0ca 100644
--- a/Modules/config.c.in
+++ b/Modules/config.c.in
@@ -25,7 +25,7 @@ extern "C" {
 /* -- ADDMODULE MARKER 1 -- */
 
 extern PyObject* PyMarshal_Init(void);
-extern PyObject* PyInit_imp(void);
+extern PyObject* PyInit__imp(void);
 extern PyObject* PyInit_gc(void);
 extern PyObject* PyInit__ast(void);
 extern PyObject* _PyWarnings_Init(void);
@@ -39,7 +39,7 @@ struct _inittab _PyImport_Inittab[] = {
     {"marshal", PyMarshal_Init},
 
     /* This lives in import.c */
-    {"_imp", PyInit_imp},
+    {"_imp", PyInit__imp},
 
     /* This lives in Python/Python-ast.c */
     {"_ast", PyInit__ast},
diff --git a/PC/config.c b/PC/config.c
index 91f15b5867d5..6209d7e284f3 100644
--- a/PC/config.c
+++ b/PC/config.c
@@ -74,7 +74,7 @@ extern PyObject* PyInit__opcode(void);
 /* -- ADDMODULE MARKER 1 -- */
 
 extern PyObject* PyMarshal_Init(void);
-extern PyObject* PyInit_imp(void);
+extern PyObject* PyInit__imp(void);
 
 struct _inittab _PyImport_Inittab[] = {
 
@@ -147,7 +147,7 @@ struct _inittab _PyImport_Inittab[] = {
     {"marshal", PyMarshal_Init},
 
     /* This lives it with import.c */
-    {"_imp", PyInit_imp},
+    {"_imp", PyInit__imp},
 
     /* These entries are here for sys.builtin_module_names */
     {"builtins", NULL},
diff --git a/Python/import.c b/Python/import.c
index 8d306403a29d..eb5aeac55461 100644
--- a/Python/import.c
+++ b/Python/import.c
@@ -1,4 +1,3 @@
-
 /* Module definition and import implementation */
 
 #include "Python.h"
@@ -2253,7 +2252,7 @@ static struct PyModuleDef impmodule = {
 const char *_Py_CheckHashBasedPycsMode = "default";
 
 PyMODINIT_FUNC
-PyInit_imp(void)
+PyInit__imp(void)
 {
     PyObject *m, *d;
 
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 82ab9154256f..5db586e15dff 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -312,7 +312,7 @@ initimport(PyInterpreterState *interp, PyObject *sysmod)
     Py_INCREF(interp->import_func);
 
     /* Import the _imp module */
-    impmod = PyInit_imp();
+    impmod = PyInit__imp();
     if (impmod == NULL) {
         return _Py_INIT_ERR("can't import _imp");
     }


From webhook-mailer at python.org  Mon Jan 29 15:34:48 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Mon, 29 Jan 2018 20:34:48 -0000
Subject: [Python-checkins] bpo-20104: Fix leaks and errors in new
 os.posix_spawn (GH-5418)
Message-ID: <mailman.61.1517258093.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/0cd6bca65519109a8a7862d38ba1b8924e432a16
commit: 0cd6bca65519109a8a7862d38ba1b8924e432a16
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T12:34:42-08:00
summary:

bpo-20104: Fix leaks and errors in new os.posix_spawn (GH-5418)

* Fix memory leaks and error handling in posix spawn
* Improve error handling when destroying the file_actions object
* Py_DECREF the result of PySequence_Fast on error
* Handle uninitialized pid
* Use OSError if file actions fails to initialize
* Move _file_actions to outer scope to avoid undefined behaviour
* Remove HAVE_POSIX_SPAWN define in Modules/posixmodule.c
* Unshadow exception and clean error message

files:
M Modules/posixmodule.c

diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 8b11e981f115..46f3adaf4dc3 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -176,14 +176,6 @@ corresponding Unix manual entries for more information on calls.");
 #else
 /* Unix functions that the configure script doesn't check for */
 #define HAVE_EXECV      1
-/* bpo-32705: Current Android does not have posix_spawn
- * Most likely posix_spawn will be available in next Android version (Android
- * P, API 28). Need revisit then. See
- * https://android-review.googlesource.com/c/platform/bionic/+/504842
- **/
-#ifndef __ANDROID__
-#define HAVE_POSIX_SPAWN 1
-#endif
 #define HAVE_FORK       1
 #if defined(__USLC__) && defined(__SCO_VERSION__)       /* SCO UDK Compiler */
 #define HAVE_FORK1      1
@@ -5139,17 +5131,22 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv,
 /*[clinic end generated code: output=d023521f541c709c input=0ec9f1cfdc890be5]*/
 {
     EXECV_CHAR **argvlist = NULL;
-    EXECV_CHAR **envlist;
+    EXECV_CHAR **envlist = NULL;
+    posix_spawn_file_actions_t _file_actions;
+    posix_spawn_file_actions_t *file_actionsp = NULL;
     Py_ssize_t argc, envc;
+    PyObject* result = NULL;
+    PyObject* seq = NULL;
+ 
 
     /* posix_spawn has three arguments: (path, argv, env), where
    argv is a list or tuple of strings and env is a dictionary
        like posix.environ. */
 
-    if (!PySequence_Check(argv)){
+    if (!PySequence_Check(argv)) {
         PyErr_SetString(PyExc_TypeError,
                         "posix_spawn: argv must be a tuple or list");
-        goto fail;
+        goto exit;
     }
     argc = PySequence_Size(argv);
     if (argc < 1) {
@@ -5160,40 +5157,37 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv,
     if (!PyMapping_Check(env)) {
         PyErr_SetString(PyExc_TypeError,
                         "posix_spawn: environment must be a mapping object");
-        goto fail;
+        goto exit;
     }
 
     argvlist = parse_arglist(argv, &argc);
     if (argvlist == NULL) {
-        goto fail;
+        goto exit;
     }
     if (!argvlist[0][0]) {
         PyErr_SetString(PyExc_ValueError,
             "posix_spawn: argv first element cannot be empty");
-        goto fail;
+        goto exit;
     }
 
     envlist = parse_envlist(env, &envc);
-    if (envlist == NULL)
-        goto fail;
+    if (envlist == NULL) {
+        goto exit;
+    }
 
     pid_t pid;
-    posix_spawn_file_actions_t *file_actionsp = NULL;
-    if (file_actions  != NULL && file_actions != Py_None){
-        posix_spawn_file_actions_t _file_actions;
+    if (file_actions != NULL && file_actions != Py_None) {
         if(posix_spawn_file_actions_init(&_file_actions) != 0){
-            PyErr_SetString(PyExc_TypeError,
+            PyErr_SetString(PyExc_OSError,
                             "Error initializing file actions");
-            goto fail;
+            goto exit;
         }
 
-
         file_actionsp = &_file_actions;
 
-
-        PyObject* seq = PySequence_Fast(file_actions, "file_actions must be a sequence");
-        if(seq == NULL){
-            goto fail;
+        seq = PySequence_Fast(file_actions, "file_actions must be a sequence");
+        if(seq == NULL) {
+            goto exit;
         }
         PyObject* file_actions_obj;
         PyObject* mode_obj;
@@ -5201,9 +5195,9 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv,
         for (int i = 0; i < PySequence_Fast_GET_SIZE(seq); ++i) {
             file_actions_obj = PySequence_Fast_GET_ITEM(seq, i);
 
-            if(!PySequence_Check(file_actions_obj) | !PySequence_Size(file_actions_obj)){
+            if(!PySequence_Check(file_actions_obj) | !PySequence_Size(file_actions_obj)) {
                 PyErr_SetString(PyExc_TypeError,"Each file_action element must be a non empty sequence");
-                goto fail;
+                goto exit;
             }
 
 
@@ -5215,83 +5209,103 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv,
             switch(mode) {
 
                 case POSIX_SPAWN_OPEN:
-                    if(PySequence_Size(file_actions_obj) != 5){
+                    if(PySequence_Size(file_actions_obj) != 5) {
                         PyErr_SetString(PyExc_TypeError,"A open file_action object must have 5 elements");
-                        goto fail;
+                        goto exit;
                     }
 
                     long open_fd = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 1));
                     if(PyErr_Occurred()) {
-                        goto fail;
+                        goto exit;
                     }
                     const char* open_path = PyUnicode_AsUTF8(PySequence_GetItem(file_actions_obj, 2));
-                    if(open_path == NULL){
-                        goto fail;
+                    if(open_path == NULL) {
+                        goto exit;
                     }
                     long open_oflag = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 3));
                     if(PyErr_Occurred()) {
-                        goto fail;
+                        goto exit;
                     }
                     long open_mode = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 4));
                     if(PyErr_Occurred()) {
-                        goto fail;
+                        goto exit;
+                    }
+                    if(posix_spawn_file_actions_addopen(file_actionsp, open_fd, open_path, open_oflag, open_mode)) {
+                        PyErr_SetString(PyExc_OSError,"Failed to add open file action");
+                        goto exit;
                     }
-                    posix_spawn_file_actions_addopen(file_actionsp, open_fd, open_path, open_oflag, open_mode);
+   
                     break;
 
                 case POSIX_SPAWN_CLOSE:
                     if(PySequence_Size(file_actions_obj) != 2){
                         PyErr_SetString(PyExc_TypeError,"A close file_action object must have 2 elements");
-                        goto fail;
+                        goto exit;
                     }
 
                     long close_fd = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 1));
                     if(PyErr_Occurred()) {
-                        goto fail;
+                        goto exit;
+                    }
+                    if(posix_spawn_file_actions_addclose(file_actionsp, close_fd)) {
+                        PyErr_SetString(PyExc_OSError,"Failed to add close file action");
+                        goto exit;
                     }
-                    posix_spawn_file_actions_addclose(file_actionsp, close_fd);
                     break;
 
                 case POSIX_SPAWN_DUP2:
                     if(PySequence_Size(file_actions_obj) != 3){
                         PyErr_SetString(PyExc_TypeError,"A dup2 file_action object must have 3 elements");
-                        goto fail;
+                        goto exit;
                     }
 
                     long fd1 = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 1));
                     if(PyErr_Occurred()) {
-                        goto fail;
+                        goto exit;
                     }
                     long fd2 = PyLong_AsLong(PySequence_GetItem(file_actions_obj, 2));
                     if(PyErr_Occurred()) {
-                        goto fail;
+                        goto exit;
+                    }
+                    if(posix_spawn_file_actions_adddup2(file_actionsp, fd1, fd2)) {
+                        PyErr_SetString(PyExc_OSError,"Failed to add dup2 file action");
+                        goto exit;
                     }
-                    posix_spawn_file_actions_adddup2(file_actionsp, fd1, fd2);
                     break;
 
                 default:
                     PyErr_SetString(PyExc_TypeError,"Unknown file_actions identifier");
-                    goto fail;
+                    goto exit;
             }
         }
-        Py_DECREF(seq);
-}
+    }
 
     _Py_BEGIN_SUPPRESS_IPH
-        posix_spawn(&pid, path->narrow, file_actionsp, NULL, argvlist, envlist);
-        return PyLong_FromPid(pid);
+    int err_code = posix_spawn(&pid, path->narrow, file_actionsp, NULL, argvlist, envlist);
     _Py_END_SUPPRESS_IPH
+    if(err_code) {
+        PyErr_SetString(PyExc_OSError,"posix_spawn call failed");
+        goto exit;
+    }
+    result = PyLong_FromPid(pid);
 
-    path_error(path);
+exit:
 
-    free_string_array(envlist, envc);
+    Py_XDECREF(seq);
 
-fail:
+    if(file_actionsp) {
+        posix_spawn_file_actions_destroy(file_actionsp);
+    }
+    
+    if (envlist) {
+        free_string_array(envlist, envc);
+    }
 
     if (argvlist) {
         free_string_array(argvlist, argc);
     }
-    return NULL;
+
+    return result;
 
 
 }


From webhook-mailer at python.org  Mon Jan 29 15:37:12 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 29 Jan 2018 20:37:12 -0000
Subject: [Python-checkins] bpo-31356: Add context manager to temporarily
 disable GC (GH-4224)
Message-ID: <mailman.62.1517258234.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/72a0d218dcc94a3cc409a9ef32dfcd5a7bbcb43c
commit: 72a0d218dcc94a3cc409a9ef32dfcd5a7bbcb43c
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-29T12:37:09-08:00
summary:

bpo-31356: Add context manager to temporarily disable GC (GH-4224)

files:
A Misc/NEWS.d/next/Core and Builtins/2017-11-02-00-34-42.bpo-31356.54Lb8U.rst
M Doc/library/gc.rst
M Include/internal/mem.h
M Lib/test/test_gc.py
M Modules/gcmodule.c

diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst
index 153d8fb70456..92240c760677 100644
--- a/Doc/library/gc.rst
+++ b/Doc/library/gc.rst
@@ -33,6 +33,34 @@ The :mod:`gc` module provides the following functions:
    Disable automatic garbage collection.
 
 
+.. class:: ensure_disabled()
+
+   Return a context manager object that disables the garbage collector and reenables the previous
+   state upon completion of the block. This is basically equivalent to::
+
+     from gc import enable, disable, isenabled
+
+     @contextmanager
+     def ensure_disabled():
+         was_enabled_previously = isenabled()
+         gc.disable()
+         yield
+         if was_enabled_previously:
+             gc.enable()
+
+   And lets you write code like this::
+
+     with ensure_disabled():
+         run_some_timing()
+
+     with ensure_disabled():
+         # do_something_that_has_real_time_guarantees
+         # such as a pair trade, robotic braking, etc
+
+   without needing to explicitly enable and disable the garbage collector yourself.
+   This context manager is implemented in C to assure atomicity, thread safety and speed.
+
+
 .. function:: isenabled()
 
    Returns true if automatic collection is enabled.
diff --git a/Include/internal/mem.h b/Include/internal/mem.h
index a731e30e6af7..4a84b4a78d90 100644
--- a/Include/internal/mem.h
+++ b/Include/internal/mem.h
@@ -116,6 +116,7 @@ struct _gc_runtime_state {
 
     int enabled;
     int debug;
+    long disabled_threads;
     /* linked lists of container objects */
     struct gc_generation generations[NUM_GENERATIONS];
     PyGC_Head *generation0;
diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index 904fc7d88c03..246980aa74c2 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -1,7 +1,7 @@
 import unittest
 from test.support import (verbose, refcount_test, run_unittest,
                           strip_python_stderr, cpython_only, start_threads,
-                          temp_dir, requires_type_collecting)
+                          temp_dir, requires_type_collecting,reap_threads)
 from test.support.script_helper import assert_python_ok, make_script
 
 import sys
@@ -9,6 +9,8 @@
 import gc
 import weakref
 import threading
+import warnings
+
 
 try:
     from _testcapi import with_tp_del
@@ -1007,6 +1009,73 @@ def __del__(self):
             # empty __dict__.
             self.assertEqual(x, None)
 
+    def test_ensure_disabled(self):
+        original_status = gc.isenabled()
+
+        with gc.ensure_disabled():
+            inside_status = gc.isenabled()
+
+        after_status = gc.isenabled()
+        self.assertEqual(original_status, True)
+        self.assertEqual(inside_status, False)
+        self.assertEqual(after_status, True)
+
+    def test_ensure_disabled_with_gc_disabled(self):
+        gc.disable()
+
+        original_status = gc.isenabled()
+
+        with gc.ensure_disabled():
+            inside_status = gc.isenabled()
+
+        after_status = gc.isenabled()
+        self.assertEqual(original_status, False)
+        self.assertEqual(inside_status, False)
+        self.assertEqual(after_status, False)
+
+    @reap_threads
+    def test_ensure_disabled_thread(self):
+
+        thread_original_status = None
+        thread_inside_status = None
+        thread_after_status = None
+
+        def disabling_thread():
+            nonlocal thread_original_status
+            nonlocal thread_inside_status
+            nonlocal thread_after_status
+            thread_original_status = gc.isenabled()
+
+            with gc.ensure_disabled():
+                time.sleep(0.01)
+                thread_inside_status = gc.isenabled()
+
+            thread_after_status = gc.isenabled()
+
+        original_status = gc.isenabled()
+
+        with warnings.catch_warnings(record=True) as w, gc.ensure_disabled():
+            inside_status_before_thread = gc.isenabled()
+            thread = threading.Thread(target=disabling_thread)
+            thread.start()
+            inside_status_after_thread = gc.isenabled()
+
+        after_status = gc.isenabled()
+        thread.join()
+
+        self.assertEqual(len(w), 1)
+        self.assertTrue(issubclass(w[-1].category, RuntimeWarning))
+        self.assertEqual("Garbage collector enabled while another thread is "
+                         "inside gc.ensure_enabled", str(w[-1].message))
+        self.assertEqual(original_status, True)
+        self.assertEqual(inside_status_before_thread, False)
+        self.assertEqual(thread_original_status, False)
+        self.assertEqual(thread_inside_status, True)
+        self.assertEqual(thread_after_status, False)
+        self.assertEqual(inside_status_after_thread, False)
+        self.assertEqual(after_status, True)
+
+
 def test_main():
     enabled = gc.isenabled()
     gc.disable()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-11-02-00-34-42.bpo-31356.54Lb8U.rst b/Misc/NEWS.d/next/Core and Builtins/2017-11-02-00-34-42.bpo-31356.54Lb8U.rst
new file mode 100644
index 000000000000..792f314c7a6d
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-11-02-00-34-42.bpo-31356.54Lb8U.rst	
@@ -0,0 +1,3 @@
+Add a new contextmanager to the gc module that temporarily disables the GC
+and restores the previous state. The implementation is done in C to assure
+atomicity and speed.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 8ba1093c029d..c057d25a12b0 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1067,6 +1067,10 @@ static PyObject *
 gc_enable_impl(PyObject *module)
 /*[clinic end generated code: output=45a427e9dce9155c input=81ac4940ca579707]*/
 {
+    if(_PyRuntime.gc.disabled_threads){
+        PyErr_WarnEx(PyExc_RuntimeWarning, "Garbage collector enabled while another "
+            "thread is inside gc.ensure_enabled",1);
+    }
     _PyRuntime.gc.enabled = 1;
     Py_RETURN_NONE;
 }
@@ -1508,6 +1512,102 @@ static PyMethodDef GcMethods[] = {
     {NULL,      NULL}           /* Sentinel */
 };
 
+typedef struct {
+    PyObject_HEAD
+    int previous_gc_state;
+} ensure_disabled_object;
+
+
+static void
+ensure_disabled_object_dealloc(ensure_disabled_object *m_obj)
+{
+    Py_TYPE(m_obj)->tp_free((PyObject*)m_obj);
+}
+
+static PyObject *
+ensure_disabled__enter__method(ensure_disabled_object *self, PyObject *args)
+{
+    PyGILState_STATE gstate = PyGILState_Ensure();
+    ++_PyRuntime.gc.disabled_threads;
+    self->previous_gc_state = _PyRuntime.gc.enabled;
+    gc_disable_impl(NULL);
+    PyGILState_Release(gstate);
+    Py_RETURN_NONE;
+}
+
+static PyObject *
+ensure_disabled__exit__method(ensure_disabled_object *self, PyObject *args)
+{
+    PyGILState_STATE gstate = PyGILState_Ensure();
+    --_PyRuntime.gc.disabled_threads;
+    if(self->previous_gc_state){
+        gc_enable_impl(NULL);
+    }else{
+        gc_disable_impl(NULL);
+    }
+    PyGILState_Release(gstate);
+    Py_RETURN_NONE;
+}
+
+
+
+static struct PyMethodDef ensure_disabled_object_methods[] = {
+    {"__enter__",       (PyCFunction) ensure_disabled__enter__method,      METH_NOARGS},
+    {"__exit__",        (PyCFunction) ensure_disabled__exit__method,       METH_VARARGS},
+    {NULL,         NULL}       /* sentinel */
+};
+
+static PyObject *
+new_disabled_obj(PyTypeObject *type, PyObject *args, PyObject *kwdict){
+    ensure_disabled_object *self;
+    self = (ensure_disabled_object *)type->tp_alloc(type, 0);
+    return (PyObject *) self;
+};
+
+static PyTypeObject gc_ensure_disabled_type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "gc.ensure_disabled",                       /* tp_name */
+    sizeof(ensure_disabled_object),             /* tp_size */
+    0,                                          /* tp_itemsize */
+    /* methods */
+    (destructor) ensure_disabled_object_dealloc,/* tp_dealloc */
+    0,                                          /* tp_print */
+    0,                                          /* tp_getattr */
+    0,                                          /* tp_setattr */
+    0,                                          /* tp_reserved */
+    0,                                          /* tp_repr */
+    0,                                          /* tp_as_number */
+    0,                                          /*tp_as_sequence*/
+    0,                                          /*tp_as_mapping*/
+    0,                                          /*tp_hash*/
+    0,                                          /*tp_call*/
+    0,                                          /*tp_str*/
+    PyObject_GenericGetAttr,                    /*tp_getattro*/
+    0,                                          /*tp_setattro*/
+    0,                                          /*tp_as_buffer*/
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /*tp_flags*/
+    0,                                          /*tp_doc*/
+    0,                                          /* tp_traverse */
+    0,                                          /* tp_clear */
+    0,                                          /* tp_richcompare */
+    0,                                          /* tp_weaklistoffset */
+    0,                                          /* tp_iter */
+    0,                                          /* tp_iternext */
+    ensure_disabled_object_methods,             /* tp_methods */
+    0,                                          /* tp_members */
+    0,                                          /* tp_getset */
+    0,                                          /* tp_base */
+    0,                                          /* tp_dict */
+    0,                                          /* tp_descr_get */
+    0,                                          /* tp_descr_set */
+    0,                                          /* tp_dictoffset */
+    0,                                          /* tp_init */
+    PyType_GenericAlloc,                        /* tp_alloc */
+    new_disabled_obj,                           /* tp_new */
+    PyObject_Del,                               /* tp_free */
+};
+
+
 static struct PyModuleDef gcmodule = {
     PyModuleDef_HEAD_INIT,
     "gc",              /* m_name */
@@ -1548,6 +1648,12 @@ PyInit_gc(void)
     if (PyModule_AddObject(m, "callbacks", _PyRuntime.gc.callbacks) < 0)
         return NULL;
 
+    if (PyType_Ready(&gc_ensure_disabled_type) < 0)
+        return NULL;
+    if (PyModule_AddObject(m, "ensure_disabled", (PyObject*) &gc_ensure_disabled_type) < 0)
+        return NULL;
+
+
 #define ADD_INT(NAME) if (PyModule_AddIntConstant(m, #NAME, NAME) < 0) return NULL
     ADD_INT(DEBUG_STATS);
     ADD_INT(DEBUG_COLLECTABLE);


From webhook-mailer at python.org  Mon Jan 29 16:38:02 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Mon, 29 Jan 2018 21:38:02 -0000
Subject: [Python-checkins] bpo-28134: Auto-detect socket values from file
 descriptor (#1349)
Message-ID: <mailman.63.1517261885.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/b6e43af669f61a37a29d8ff0785455108e6bc29d
commit: b6e43af669f61a37a29d8ff0785455108e6bc29d
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T22:37:58+01:00
summary:

bpo-28134: Auto-detect socket values from file descriptor (#1349)

Fix socket(fileno=fd) by auto-detecting the socket's family, type,
and proto from the file descriptor. The auto-detection can be overruled
by passing in family, type, and proto explicitly.

Without the fix, all socket except for TCP/IP over IPv4 are basically broken:

>>> s = socket.create_connection(('www.python.org', 443))
>>> s
<socket.socket fd=3, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('2003:58:bc4a:3b00:56ee:75ff:fe47:ca7b', 59730, 0, 0), raddr=('2a04:4e42:1b::223', 443, 0, 0)>
>>> socket.socket(fileno=s.fileno())
<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('2003:58:bc4a:3b00::%2550471192', 59730, 0, 2550471192), raddr=('2a04:4e42:1b:0:700c:e70b:ff7f:0%2550471192', 443, 0, 2550471192)>

Signed-off-by: Christian Heimes <christian at python.org>

files:
A Misc/NEWS.d/next/Library/2017-12-24-20-01-09.bpo-28134.HJ8Beb.rst
M Doc/library/socket.rst
M Lib/socket.py
M Lib/test/_test_multiprocessing.py
M Lib/test/test_socket.py
M Modules/socketmodule.c

diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index faa260e5d6c5..7f0d4ede7ccf 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -461,10 +461,15 @@ The following functions all create :ref:`socket objects <socket-objects>`.
    :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_``
    constants. The protocol number is usually zero and may be omitted or in the
    case where the address family is :const:`AF_CAN` the protocol should be one
-   of :const:`CAN_RAW`, :const:`CAN_BCM` or :const:`CAN_ISOTP`.  If *fileno* is specified, the other
-   arguments are ignored, causing the socket with the specified file descriptor
-   to return.  Unlike :func:`socket.fromfd`, *fileno* will return the same
-   socket and not a duplicate. This may help close a detached socket using
+   of :const:`CAN_RAW`, :const:`CAN_BCM` or :const:`CAN_ISOTP`
+
+   If *fileno* is specified, the values for *family*, *type*, and *proto* are
+   auto-detected from the specified file descriptor.  Auto-detection can be
+   overruled by calling the function with explicit *family*, *type*, or *proto*
+   arguments.  This only affects how Python represents e.g. the return value
+   of :meth:`socket.getpeername` but not the actual OS resource.  Unlike
+   :func:`socket.fromfd`, *fileno* will return the same socket and not a
+   duplicate. This may help close a detached socket using
    :meth:`socket.close()`.
 
    The newly created socket is :ref:`non-inheritable <fd_inheritance>`.
diff --git a/Lib/socket.py b/Lib/socket.py
index 2d8aee3e9047..cfa605a22ada 100644
--- a/Lib/socket.py
+++ b/Lib/socket.py
@@ -136,11 +136,18 @@ class socket(_socket.socket):
 
     __slots__ = ["__weakref__", "_io_refs", "_closed"]
 
-    def __init__(self, family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None):
+    def __init__(self, family=-1, type=-1, proto=-1, fileno=None):
         # For user code address family and type values are IntEnum members, but
         # for the underlying _socket.socket they're just integers. The
         # constructor of _socket.socket converts the given argument to an
         # integer automatically.
+        if fileno is None:
+            if family == -1:
+                family = AF_INET
+            if type == -1:
+                type = SOCK_STREAM
+            if proto == -1:
+                proto = 0
         _socket.socket.__init__(self, family, type, proto, fileno)
         self._io_refs = 0
         self._closed = False
diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index 05166b91ba83..1e497a572a76 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -4204,7 +4204,7 @@ def get_high_socket_fd(self):
 
     def close(self, fd):
         if WIN32:
-            socket.socket(fileno=fd).close()
+            socket.socket(socket.AF_INET, socket.SOCK_STREAM, fileno=fd).close()
         else:
             os.close(fd)
 
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 365da36c4210..89cf1fa52009 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -20,6 +20,7 @@
 import pickle
 import struct
 import random
+import shutil
 import string
 import _thread as thread
 import threading
@@ -1284,6 +1285,7 @@ def testSendAfterClose(self):
 
     def testCloseException(self):
         sock = socket.socket()
+        sock.bind((socket._LOCALHOST, 0))
         socket.socket(fileno=sock.fileno()).close()
         try:
             sock.close()
@@ -1636,9 +1638,11 @@ def test_uknown_socket_family_repr(self):
         ) + 1
 
         with socket.socket(
-                family=unknown_family, type=unknown_type, fileno=fd) as s:
+                family=unknown_family, type=unknown_type, proto=23,
+                fileno=fd) as s:
             self.assertEqual(s.family, unknown_family)
             self.assertEqual(s.type, unknown_type)
+            self.assertEqual(s.proto, 23)
 
     @unittest.skipUnless(hasattr(os, 'sendfile'), 'test needs os.sendfile()')
     def test__sendfile_use_sendfile(self):
@@ -1658,6 +1662,45 @@ def fileno(self):
             with self.assertRaises(TypeError):
                 sock._sendfile_use_sendfile(File(None))
 
+    def _test_socket_fileno(self, s, family, stype):
+        self.assertEqual(s.family, family)
+        self.assertEqual(s.type, stype)
+
+        fd = s.fileno()
+        s2 = socket.socket(fileno=fd)
+        self.addCleanup(s2.close)
+        # detach old fd to avoid double close
+        s.detach()
+        self.assertEqual(s2.family, family)
+        self.assertEqual(s2.type, stype)
+        self.assertEqual(s2.fileno(), fd)
+
+    def test_socket_fileno(self):
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        self.addCleanup(s.close)
+        s.bind((support.HOST, 0))
+        self._test_socket_fileno(s, socket.AF_INET, socket.SOCK_STREAM)
+
+        if hasattr(socket, "SOCK_DGRAM"):
+            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+            self.addCleanup(s.close)
+            s.bind((support.HOST, 0))
+            self._test_socket_fileno(s, socket.AF_INET, socket.SOCK_DGRAM)
+
+        if support.IPV6_ENABLED:
+            s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
+            self.addCleanup(s.close)
+            s.bind((support.HOSTv6, 0, 0, 0))
+            self._test_socket_fileno(s, socket.AF_INET6, socket.SOCK_STREAM)
+
+        if hasattr(socket, "AF_UNIX"):
+            tmpdir = tempfile.mkdtemp()
+            self.addCleanup(shutil.rmtree, tmpdir)
+            s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+            self.addCleanup(s.close)
+            s.bind(os.path.join(tmpdir, 'socket'))
+            self._test_socket_fileno(s, socket.AF_UNIX, socket.SOCK_STREAM)
+
 
 @unittest.skipUnless(HAVE_SOCKET_CAN, 'SocketCan required for this test.')
 class BasicCANTest(unittest.TestCase):
diff --git a/Misc/NEWS.d/next/Library/2017-12-24-20-01-09.bpo-28134.HJ8Beb.rst b/Misc/NEWS.d/next/Library/2017-12-24-20-01-09.bpo-28134.HJ8Beb.rst
new file mode 100644
index 000000000000..9c4c683db6e3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-24-20-01-09.bpo-28134.HJ8Beb.rst
@@ -0,0 +1,2 @@
+Sockets now auto-detect family, type and protocol from file descriptor by
+default.
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index 8744ea0be87c..8cc5d14cbb20 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -102,7 +102,8 @@ Local naming conventions:
 
 /* Socket object documentation */
 PyDoc_STRVAR(sock_doc,
-"socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None) -> socket object\n\
+"socket(family=AF_INET, type=SOCK_STREAM, proto=0) -> socket object\n\
+socket(family=-1, type=-1, proto=-1, fileno=None) -> socket object\n\
 \n\
 Open a socket of the given type.  The family argument specifies the\n\
 address family; it defaults to AF_INET.  The type argument specifies\n\
@@ -111,6 +112,9 @@ or datagram (SOCK_DGRAM) socket.  The protocol argument defaults to 0,\n\
 specifying the default protocol.  Keyword arguments are accepted.\n\
 The socket is created as non-inheritable.\n\
 \n\
+When a fileno is passed in, family, type and proto are auto-detected,\n\
+unless they are explicitly set.\n\
+\n\
 A socket object represents one endpoint of a network connection.\n\
 \n\
 Methods of socket objects (keyword arguments not allowed):\n\
@@ -4792,7 +4796,7 @@ sock_initobj(PyObject *self, PyObject *args, PyObject *kwds)
     PySocketSockObject *s = (PySocketSockObject *)self;
     PyObject *fdobj = NULL;
     SOCKET_T fd = INVALID_SOCKET;
-    int family = AF_INET, type = SOCK_STREAM, proto = 0;
+    int family = -1, type = -1, proto = -1;
     static char *keywords[] = {"family", "type", "proto", "fileno", 0};
 #ifndef MS_WINDOWS
 #ifdef SOCK_CLOEXEC
@@ -4842,9 +4846,72 @@ sock_initobj(PyObject *self, PyObject *args, PyObject *kwds)
                                 "can't use invalid socket value");
                 return -1;
             }
+
+            if (family == -1) {
+                sock_addr_t addrbuf;
+                socklen_t addrlen = sizeof(sock_addr_t);
+
+                memset(&addrbuf, 0, addrlen);
+                if (getsockname(fd, SAS2SA(&addrbuf), &addrlen) == 0) {
+                    family = SAS2SA(&addrbuf)->sa_family;
+                } else {
+#ifdef MS_WINDOWS
+                    PyErr_SetFromWindowsErrWithFilename(0, "family");
+#else
+                    PyErr_SetFromErrnoWithFilename(PyExc_OSError, "family");
+#endif
+                    return -1;
+                }
+            }
+#ifdef SO_TYPE
+            if (type == -1) {
+                int tmp;
+                socklen_t slen = sizeof(tmp);
+                if (getsockopt(fd, SOL_SOCKET, SO_TYPE, &tmp, &slen) == 0) {
+                    type = tmp;
+                } else {
+#ifdef MS_WINDOWS
+                    PyErr_SetFromWindowsErrWithFilename(0, "type");
+#else
+                    PyErr_SetFromErrnoWithFilename(PyExc_OSError, "type");
+#endif
+                    return -1;
+                }
+            }
+#else
+            type = SOCK_STREAM;
+#endif
+#ifdef SO_PROTOCOL
+            if (proto == -1) {
+                int tmp;
+                socklen_t slen = sizeof(tmp);
+                if (getsockopt(fd, SOL_SOCKET, SO_PROTOCOL, &tmp, &slen) == 0) {
+                    proto = tmp;
+                } else {
+#ifdef MS_WINDOWS
+                    PyErr_SetFromWindowsErrWithFilename(0, "protocol");
+#else
+                    PyErr_SetFromErrnoWithFilename(PyExc_OSError, "protocol");
+#endif
+                    return -1;
+                }
+            }
+#else
+            proto = 0;
+#endif
         }
     }
     else {
+        /* No fd, default to AF_INET and SOCK_STREAM */
+        if (family == -1) {
+            family = AF_INET;
+        }
+        if (type == -1) {
+            type = SOCK_STREAM;
+        }
+        if (proto == -1) {
+            proto = 0;
+        }
 #ifdef MS_WINDOWS
         /* Windows implementation */
 #ifndef WSA_FLAG_NO_HANDLE_INHERIT


From webhook-mailer at python.org  Mon Jan 29 17:28:05 2018
From: webhook-mailer at python.org (Mariatta)
Date: Mon, 29 Jan 2018 22:28:05 -0000
Subject: [Python-checkins] bpo-27931: Fix email address header parsing error
 (GH-5329) (GH-5431)
Message-ID: <mailman.64.1517264887.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/6ea75b174da0cf824e2acc5db6b53798f5f4e4f9
commit: 6ea75b174da0cf824e2acc5db6b53798f5f4e4f9
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-29T14:27:55-08:00
summary:

bpo-27931: Fix email address header parsing error (GH-5329) (GH-5431)

Correctly handle addresses whose username is an empty quoted string.
(cherry picked from commit aa218d1649690d1c1ba86a9972f7fae646bf1a8f)

Co-authored-by: jayyyin <jayyin11043 at hotmail.com>

files:
A Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst
M Lib/email/_header_value_parser.py
M Lib/test/test_email/test__header_value_parser.py

diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index 3ebbbe5383ab..b23c897e97bb 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -430,7 +430,10 @@ def route(self):
     def addr_spec(self):
         for x in self:
             if x.token_type == 'addr-spec':
-                return x.addr_spec
+                if x.local_part:
+                    return x.addr_spec
+                else:
+                    return quote_string(x.local_part) + x.addr_spec
         else:
             return '<>'
 
@@ -1165,6 +1168,9 @@ def get_bare_quoted_string(value):
             "expected '\"' but found '{}'".format(value))
     bare_quoted_string = BareQuotedString()
     value = value[1:]
+    if value[0] == '"':
+        token, value = get_qcontent(value)
+        bare_quoted_string.append(token)
     while value and value[0] != '"':
         if value[0] in WSP:
             token, value = get_fws(value)
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index 1667617b9e46..5cdc4bcecad4 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -490,6 +490,10 @@ def test_get_bare_quoted_string_must_start_with_dquote(self):
         with self.assertRaises(errors.HeaderParseError):
             parser.get_bare_quoted_string('  "foo"')
 
+    def test_get_bare_quoted_string_only_quotes(self):
+        self._test_get_x(parser.get_bare_quoted_string,
+                         '""', '""', '', [], '')
+
     def test_get_bare_quoted_string_following_wsp_preserved(self):
         self._test_get_x(parser.get_bare_quoted_string,
              '"foo"\t bar', '"foo"', 'foo', [], '\t bar')
@@ -1467,6 +1471,19 @@ def test_get_angle_addr_empty(self):
         self.assertIsNone(angle_addr.route)
         self.assertEqual(angle_addr.addr_spec, '<>')
 
+    def test_get_angle_addr_qs_only_quotes(self):
+        angle_addr = self._test_get_x(parser.get_angle_addr,
+            '<""@example.com>',
+            '<""@example.com>',
+            '<""@example.com>',
+            [],
+            '')
+        self.assertEqual(angle_addr.token_type, 'angle-addr')
+        self.assertEqual(angle_addr.local_part, '')
+        self.assertEqual(angle_addr.domain, 'example.com')
+        self.assertIsNone(angle_addr.route)
+        self.assertEqual(angle_addr.addr_spec, '""@example.com')
+
     def test_get_angle_addr_with_cfws(self):
         angle_addr = self._test_get_x(parser.get_angle_addr,
             ' (foo) <dinsdale at example.com>(bar)',
diff --git a/Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst b/Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst
new file mode 100644
index 000000000000..7324247357a6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-25-21-04-11.bpo-27931.e4r52t.rst
@@ -0,0 +1 @@
+Fix email address header parsing error when the username is an empty quoted string. Patch by Xiang Zhang.


From webhook-mailer at python.org  Mon Jan 29 19:41:13 2018
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Tue, 30 Jan 2018 00:41:13 -0000
Subject: [Python-checkins] bpo-32550. Remove the STORE_ANNOTATION bytecode.
 (GH-5181)
Message-ID: <mailman.65.1517272877.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/332cd5ee4ff42c9904c56e68a1028f383f7fc9a8
commit: 332cd5ee4ff42c9904c56e68a1028f383f7fc9a8
branch: master
author: Mark Shannon <mark at hotpy.org>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2018-01-29T16:41:04-08:00
summary:

bpo-32550. Remove the STORE_ANNOTATION bytecode. (GH-5181)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-14-12-42-17.bpo-32550.k0EK-4.rst
M Doc/library/dis.rst
M Doc/whatsnew/3.7.rst
M Include/opcode.h
M Lib/importlib/_bootstrap_external.py
M Lib/opcode.py
M Lib/test/test_dis.py
M Python/ceval.c
M Python/compile.c
M Python/importlib_external.h
M Python/opcode_targets.h

diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst
index 01d46f88fb57..48c42d119d5d 100644
--- a/Doc/library/dis.rst
+++ b/Doc/library/dis.rst
@@ -993,13 +993,6 @@ All of the following opcodes use their arguments.
    Deletes local ``co_varnames[var_num]``.
 
 
-.. opcode:: STORE_ANNOTATION (namei)
-
-   Stores TOS as ``locals()['__annotations__'][co_names[namei]] = TOS``.
-
-   .. versionadded:: 3.6
-
-
 .. opcode:: LOAD_CLOSURE (i)
 
    Pushes a reference to the cell contained in slot *i* of the cell and free
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 71070df76470..4fdbb9182bbd 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -1173,6 +1173,8 @@ CPython bytecode changes
 * Added two new opcodes: :opcode:`LOAD_METHOD` and :opcode:`CALL_METHOD`.
   (Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.)
 
+* Removed the STORE_ANNOTATION opcode.
+
 
 Other CPython implementation changes
 ------------------------------------
diff --git a/Include/opcode.h b/Include/opcode.h
index 99c3b0ef817a..fc6cbf3a7af4 100644
--- a/Include/opcode.h
+++ b/Include/opcode.h
@@ -99,7 +99,6 @@ extern "C" {
 #define LOAD_FAST               124
 #define STORE_FAST              125
 #define DELETE_FAST             126
-#define STORE_ANNOTATION        127
 #define RAISE_VARARGS           130
 #define CALL_FUNCTION           131
 #define MAKE_FUNCTION           132
diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py
index cf75719c86f5..d3f58af504d8 100644
--- a/Lib/importlib/_bootstrap_external.py
+++ b/Lib/importlib/_bootstrap_external.py
@@ -242,6 +242,7 @@ def _write_atomic(path, data, mode=0o666):
 #     Python 3.7a0  3390 (add LOAD_METHOD and CALL_METHOD opcodes)
 #     Python 3.7a0  3391 (update GET_AITER #31709)
 #     Python 3.7a0  3392 (PEP 552: Deterministic pycs)
+#     Python 3.7a0  3393 (remove  STORE_ANNOTATION opcode)
 #
 # MAGIC must change whenever the bytecode emitted by the compiler may no
 # longer be understood by older implementations of the eval loop (usually
@@ -250,7 +251,7 @@ def _write_atomic(path, data, mode=0o666):
 # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
 # in PC/launcher.c must also be updated.
 
-MAGIC_NUMBER = (3392).to_bytes(2, 'little') + b'\r\n'
+MAGIC_NUMBER = (3393).to_bytes(2, 'little') + b'\r\n'
 _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little')  # For import.c
 
 _PYCACHE = '__pycache__'
diff --git a/Lib/opcode.py b/Lib/opcode.py
index dffb38c314a1..8f45f0a666f1 100644
--- a/Lib/opcode.py
+++ b/Lib/opcode.py
@@ -169,7 +169,6 @@ def jabs_op(name, op):
 haslocal.append(125)
 def_op('DELETE_FAST', 126)      # Local variable number
 haslocal.append(126)
-name_op('STORE_ANNOTATION', 127) # Index in name list
 
 def_op('RAISE_VARARGS', 130)    # Number of raise arguments (1, 2, or 3)
 def_op('CALL_FUNCTION', 131)    # #args
diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
index 590f041aa926..ba8c6b940b04 100644
--- a/Lib/test/test_dis.py
+++ b/Lib/test/test_dis.py
@@ -226,23 +226,27 @@ def bug1333982(x=[]):
               2 LOAD_CONST               0 (1)
               4 STORE_NAME               0 (x)
               6 LOAD_NAME                1 (int)
-              8 STORE_ANNOTATION         0 (x)
-
-  3          10 LOAD_NAME                2 (fun)
-             12 LOAD_CONST               0 (1)
-             14 CALL_FUNCTION            1
-             16 STORE_ANNOTATION         3 (y)
-
-  4          18 LOAD_CONST               0 (1)
-             20 LOAD_NAME                4 (lst)
-             22 LOAD_NAME                2 (fun)
-             24 LOAD_CONST               1 (0)
-             26 CALL_FUNCTION            1
-             28 STORE_SUBSCR
-             30 LOAD_NAME                1 (int)
-             32 POP_TOP
-             34 LOAD_CONST               2 (None)
-             36 RETURN_VALUE
+              8 LOAD_NAME                2 (__annotations__)
+             10 LOAD_CONST               1 ('x')
+             12 STORE_SUBSCR
+
+  3          14 LOAD_NAME                3 (fun)
+             16 LOAD_CONST               0 (1)
+             18 CALL_FUNCTION            1
+             20 LOAD_NAME                2 (__annotations__)
+             22 LOAD_CONST               2 ('y')
+             24 STORE_SUBSCR
+
+  4          26 LOAD_CONST               0 (1)
+             28 LOAD_NAME                4 (lst)
+             30 LOAD_NAME                3 (fun)
+             32 LOAD_CONST               3 (0)
+             34 CALL_FUNCTION            1
+             36 STORE_SUBSCR
+             38 LOAD_NAME                1 (int)
+             40 POP_TOP
+             42 LOAD_CONST               4 (None)
+             44 RETURN_VALUE
 """
 
 compound_stmt_str = """\
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-14-12-42-17.bpo-32550.k0EK-4.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-14-12-42-17.bpo-32550.k0EK-4.rst
new file mode 100644
index 000000000000..9f77b94f6dd5
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-14-12-42-17.bpo-32550.k0EK-4.rst	
@@ -0,0 +1 @@
+Remove the STORE_ANNOTATION bytecode.
diff --git a/Python/ceval.c b/Python/ceval.c
index 52a42b00724e..af5eb99d6c90 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -1574,61 +1574,6 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
             DISPATCH();
         }
 
-        TARGET(STORE_ANNOTATION) {
-            _Py_IDENTIFIER(__annotations__);
-            PyObject *ann_dict;
-            PyObject *ann = POP();
-            PyObject *name = GETITEM(names, oparg);
-            int err;
-            if (f->f_locals == NULL) {
-                PyErr_Format(PyExc_SystemError,
-                             "no locals found when storing annotation");
-                Py_DECREF(ann);
-                goto error;
-            }
-            /* first try to get __annotations__ from locals... */
-            if (PyDict_CheckExact(f->f_locals)) {
-                ann_dict = _PyDict_GetItemId(f->f_locals,
-                                             &PyId___annotations__);
-                if (ann_dict == NULL) {
-                    PyErr_SetString(PyExc_NameError,
-                                    "__annotations__ not found");
-                    Py_DECREF(ann);
-                    goto error;
-                }
-                Py_INCREF(ann_dict);
-            }
-            else {
-                PyObject *ann_str = _PyUnicode_FromId(&PyId___annotations__);
-                if (ann_str == NULL) {
-                    Py_DECREF(ann);
-                    goto error;
-                }
-                ann_dict = PyObject_GetItem(f->f_locals, ann_str);
-                if (ann_dict == NULL) {
-                    if (PyErr_ExceptionMatches(PyExc_KeyError)) {
-                        PyErr_SetString(PyExc_NameError,
-                                        "__annotations__ not found");
-                    }
-                    Py_DECREF(ann);
-                    goto error;
-                }
-            }
-            /* ...if succeeded, __annotations__[name] = ann */
-            if (PyDict_CheckExact(ann_dict)) {
-                err = PyDict_SetItem(ann_dict, name, ann);
-            }
-            else {
-                err = PyObject_SetItem(ann_dict, name, ann);
-            }
-            Py_DECREF(ann_dict);
-            Py_DECREF(ann);
-            if (err != 0) {
-                goto error;
-            }
-            DISPATCH();
-        }
-
         TARGET(DELETE_SUBSCR) {
             PyObject *sub = TOP();
             PyObject *container = SECOND();
diff --git a/Python/compile.c b/Python/compile.c
index 3e8323b933f4..0dc35662749b 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -213,7 +213,7 @@ static int compiler_async_comprehension_generator(
                                       expr_ty elt, expr_ty val, int type);
 
 static PyCodeObject *assemble(struct compiler *, int addNone);
-static PyObject *__doc__;
+static PyObject *__doc__, *__annotations__;
 
 #define CAPSULE_NAME "compile.c compiler unit"
 
@@ -311,7 +311,11 @@ PyAST_CompileObject(mod_ty mod, PyObject *filename, PyCompilerFlags *flags,
         if (!__doc__)
             return NULL;
     }
-
+    if (!__annotations__) {
+        __annotations__ = PyUnicode_InternFromString("__annotations__");
+        if (!__annotations__)
+            return NULL;
+    }
     if (!compiler_init(&c))
         return NULL;
     Py_INCREF(filename);
@@ -1056,8 +1060,6 @@ stack_effect(int opcode, int oparg, int jump)
             return -1;
         case DELETE_FAST:
             return 0;
-        case STORE_ANNOTATION:
-            return -1;
 
         case RAISE_VARARGS:
             return -oparg;
@@ -4711,8 +4713,10 @@ compiler_annassign(struct compiler *c, stmt_ty s)
             else {
                 VISIT(c, expr, s->v.AnnAssign.annotation);
             }
-            /* ADDOP_N decrefs its argument */
-            ADDOP_N(c, STORE_ANNOTATION, mangled, names);
+            ADDOP_NAME(c, LOAD_NAME, __annotations__, names);
+            ADDOP_O(c, LOAD_CONST, mangled, consts);
+            Py_DECREF(mangled);
+            ADDOP(c, STORE_SUBSCR);
         }
         break;
     case Attribute_kind:
diff --git a/Python/importlib_external.h b/Python/importlib_external.h
index e298bd0c20cd..6bf189d006ab 100644
--- a/Python/importlib_external.h
+++ b/Python/importlib_external.h
@@ -243,7 +243,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,4,0,0,0,218,13,95,119,114,105,116,101,95,97,116,
     111,109,105,99,105,0,0,0,115,26,0,0,0,0,5,16,
     1,6,1,26,1,2,3,14,1,20,1,16,1,14,1,2,
-    1,14,1,14,1,6,1,114,56,0,0,0,105,64,13,0,
+    1,14,1,14,1,6,1,114,56,0,0,0,105,65,13,0,
     0,233,2,0,0,0,114,13,0,0,0,115,2,0,0,0,
     13,10,90,11,95,95,112,121,99,97,99,104,101,95,95,122,
     4,111,112,116,45,122,3,46,112,121,122,4,46,112,121,99,
@@ -348,7 +348,7 @@ const unsigned char _Py_M__importlib_external[] = {
     108,109,111,115,116,95,102,105,108,101,110,97,109,101,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,17,99,
     97,99,104,101,95,102,114,111,109,95,115,111,117,114,99,101,
-    9,1,0,0,115,48,0,0,0,0,18,8,1,6,1,6,
+    10,1,0,0,115,48,0,0,0,0,18,8,1,6,1,6,
     1,8,1,4,1,8,1,12,1,10,1,12,1,16,1,8,
     1,8,1,8,1,24,1,8,1,12,1,6,2,8,1,8,
     1,8,1,8,1,14,1,14,1,114,81,0,0,0,99,1,
@@ -422,7 +422,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,118,101,108,90,13,98,97,115,101,95,102,105,108,101,110,
     97,109,101,114,2,0,0,0,114,2,0,0,0,114,4,0,
     0,0,218,17,115,111,117,114,99,101,95,102,114,111,109,95,
-    99,97,99,104,101,54,1,0,0,115,46,0,0,0,0,9,
+    99,97,99,104,101,55,1,0,0,115,46,0,0,0,0,9,
     12,1,8,1,10,1,12,1,12,1,8,1,6,1,10,1,
     10,1,8,1,6,1,10,1,8,1,16,1,10,1,6,1,
     8,1,16,1,8,1,6,1,8,1,14,1,114,87,0,0,
@@ -456,7 +456,7 @@ const unsigned char _Py_M__importlib_external[] = {
     36,0,0,0,90,9,101,120,116,101,110,115,105,111,110,218,
     11,115,111,117,114,99,101,95,112,97,116,104,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,218,15,95,103,101,
-    116,95,115,111,117,114,99,101,102,105,108,101,88,1,0,0,
+    116,95,115,111,117,114,99,101,102,105,108,101,89,1,0,0,
     115,20,0,0,0,0,7,12,1,4,1,16,1,24,1,4,
     1,2,1,12,1,18,1,18,1,114,93,0,0,0,99,1,
     0,0,0,0,0,0,0,1,0,0,0,8,0,0,0,67,
@@ -470,7 +470,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,68,0,0,0,114,76,0,0,0,41,1,218,8,102,105,
     108,101,110,97,109,101,114,2,0,0,0,114,2,0,0,0,
     114,4,0,0,0,218,11,95,103,101,116,95,99,97,99,104,
-    101,100,107,1,0,0,115,16,0,0,0,0,1,14,1,2,
+    101,100,108,1,0,0,115,16,0,0,0,0,1,14,1,2,
     1,8,1,14,1,8,1,14,1,4,2,114,97,0,0,0,
     99,1,0,0,0,0,0,0,0,2,0,0,0,8,0,0,
     0,67,0,0,0,115,52,0,0,0,121,14,116,0,124,0,
@@ -484,7 +484,7 @@ const unsigned char _Py_M__importlib_external[] = {
     3,114,39,0,0,0,114,41,0,0,0,114,40,0,0,0,
     41,2,114,35,0,0,0,114,42,0,0,0,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,218,10,95,99,97,
-    108,99,95,109,111,100,101,119,1,0,0,115,12,0,0,0,
+    108,99,95,109,111,100,101,120,1,0,0,115,12,0,0,0,
     0,2,2,1,14,1,14,1,10,3,8,1,114,99,0,0,
     0,99,1,0,0,0,0,0,0,0,3,0,0,0,8,0,
     0,0,3,0,0,0,115,68,0,0,0,100,6,135,0,102,
@@ -521,7 +521,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,108,102,114,100,0,0,0,218,4,97,114,103,115,90,6,
     107,119,97,114,103,115,41,1,218,6,109,101,116,104,111,100,
     114,2,0,0,0,114,4,0,0,0,218,19,95,99,104,101,
-    99,107,95,110,97,109,101,95,119,114,97,112,112,101,114,139,
+    99,107,95,110,97,109,101,95,119,114,97,112,112,101,114,140,
     1,0,0,115,12,0,0,0,0,1,8,1,8,1,10,1,
     4,1,18,1,122,40,95,99,104,101,99,107,95,110,97,109,
     101,46,60,108,111,99,97,108,115,62,46,95,99,104,101,99,
@@ -539,7 +539,7 @@ const unsigned char _Py_M__importlib_external[] = {
     116,116,114,218,8,95,95,100,105,99,116,95,95,218,6,117,
     112,100,97,116,101,41,3,90,3,110,101,119,90,3,111,108,
     100,114,53,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,218,5,95,119,114,97,112,150,1,0,0,
+    114,4,0,0,0,218,5,95,119,114,97,112,151,1,0,0,
     115,8,0,0,0,0,1,10,1,10,1,22,1,122,26,95,
     99,104,101,99,107,95,110,97,109,101,46,60,108,111,99,97,
     108,115,62,46,95,119,114,97,112,41,1,78,41,3,218,10,
@@ -547,7 +547,7 @@ const unsigned char _Py_M__importlib_external[] = {
     9,78,97,109,101,69,114,114,111,114,41,3,114,104,0,0,
     0,114,105,0,0,0,114,115,0,0,0,114,2,0,0,0,
     41,1,114,104,0,0,0,114,4,0,0,0,218,11,95,99,
-    104,101,99,107,95,110,97,109,101,131,1,0,0,115,14,0,
+    104,101,99,107,95,110,97,109,101,132,1,0,0,115,14,0,
     0,0,0,8,14,7,2,1,10,1,14,2,14,5,10,1,
     114,118,0,0,0,99,2,0,0,0,0,0,0,0,5,0,
     0,0,6,0,0,0,67,0,0,0,115,60,0,0,0,124,
@@ -575,7 +575,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,218,6,108,111,97,100,101,114,218,8,112,111,114,116,105,
     111,110,115,218,3,109,115,103,114,2,0,0,0,114,2,0,
     0,0,114,4,0,0,0,218,17,95,102,105,110,100,95,109,
-    111,100,117,108,101,95,115,104,105,109,159,1,0,0,115,10,
+    111,100,117,108,101,95,115,104,105,109,160,1,0,0,115,10,
     0,0,0,0,10,14,1,16,1,4,1,22,1,114,125,0,
     0,0,99,3,0,0,0,0,0,0,0,6,0,0,0,4,
     0,0,0,67,0,0,0,115,158,0,0,0,124,0,100,1,
@@ -642,7 +642,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,90,5,109,97,103,105,99,114,77,0,0,0,114,69,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
     0,218,13,95,99,108,97,115,115,105,102,121,95,112,121,99,
-    176,1,0,0,115,28,0,0,0,0,16,12,1,8,1,16,
+    177,1,0,0,115,28,0,0,0,0,16,12,1,8,1,16,
     1,12,1,12,1,12,1,10,1,12,1,8,1,16,2,8,
     1,16,1,12,1,114,133,0,0,0,99,5,0,0,0,0,
     0,0,0,6,0,0,0,4,0,0,0,67,0,0,0,115,
@@ -696,7 +696,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,114,100,0,0,0,114,132,0,0,0,114,77,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
     23,95,118,97,108,105,100,97,116,101,95,116,105,109,101,115,
-    116,97,109,112,95,112,121,99,209,1,0,0,115,14,0,0,
+    116,97,109,112,95,112,121,99,210,1,0,0,115,14,0,0,
     0,0,19,24,1,10,1,12,1,12,1,8,1,24,1,114,
     137,0,0,0,99,4,0,0,0,0,0,0,0,4,0,0,
     0,3,0,0,0,67,0,0,0,115,38,0,0,0,124,0,
@@ -742,7 +742,7 @@ const unsigned char _Py_M__importlib_external[] = {
     104,97,115,104,114,100,0,0,0,114,132,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,18,95,
     118,97,108,105,100,97,116,101,95,104,97,115,104,95,112,121,
-    99,237,1,0,0,115,8,0,0,0,0,17,16,1,2,1,
+    99,238,1,0,0,115,8,0,0,0,0,17,16,1,2,1,
     10,1,114,139,0,0,0,99,4,0,0,0,0,0,0,0,
     5,0,0,0,5,0,0,0,67,0,0,0,115,80,0,0,
     0,116,0,160,1,124,0,161,1,125,4,116,2,124,4,116,
@@ -765,7 +765,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,100,0,0,0,114,91,0,0,0,114,92,0,0,
     0,218,4,99,111,100,101,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,17,95,99,111,109,112,105,108,101,
-    95,98,121,116,101,99,111,100,101,5,2,0,0,115,16,0,
+    95,98,121,116,101,99,111,100,101,6,2,0,0,115,16,0,
     0,0,0,2,10,1,10,1,12,1,8,1,12,1,4,2,
     10,1,114,145,0,0,0,114,60,0,0,0,99,3,0,0,
     0,0,0,0,0,4,0,0,0,5,0,0,0,67,0,0,
@@ -783,7 +783,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,218,5,109,116,105,109,101,114,136,0,0,0,114,54,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
     0,0,218,22,95,99,111,100,101,95,116,111,95,116,105,109,
-    101,115,116,97,109,112,95,112,121,99,18,2,0,0,115,12,
+    101,115,116,97,109,112,95,112,121,99,19,2,0,0,115,12,
     0,0,0,0,2,8,1,14,1,14,1,14,1,16,1,114,
     150,0,0,0,84,99,3,0,0,0,0,0,0,0,5,0,
     0,0,5,0,0,0,67,0,0,0,115,80,0,0,0,116,
@@ -802,7 +802,7 @@ const unsigned char _Py_M__importlib_external[] = {
     138,0,0,0,90,7,99,104,101,99,107,101,100,114,54,0,
     0,0,114,69,0,0,0,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,17,95,99,111,100,101,95,116,111,
-    95,104,97,115,104,95,112,121,99,28,2,0,0,115,14,0,
+    95,104,97,115,104,95,112,121,99,29,2,0,0,115,14,0,
     0,0,0,2,8,1,12,1,14,1,16,1,10,1,16,1,
     114,152,0,0,0,99,1,0,0,0,0,0,0,0,5,0,
     0,0,6,0,0,0,67,0,0,0,115,62,0,0,0,100,
@@ -829,7 +829,7 @@ const unsigned char _Py_M__importlib_external[] = {
     100,108,105,110,101,218,8,101,110,99,111,100,105,110,103,90,
     15,110,101,119,108,105,110,101,95,100,101,99,111,100,101,114,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    13,100,101,99,111,100,101,95,115,111,117,114,99,101,39,2,
+    13,100,101,99,111,100,101,95,115,111,117,114,99,101,40,2,
     0,0,115,10,0,0,0,0,5,8,1,12,1,10,1,12,
     1,114,157,0,0,0,41,2,114,122,0,0,0,218,26,115,
     117,98,109,111,100,117,108,101,95,115,101,97,114,99,104,95,
@@ -891,7 +891,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,114,161,0,0,0,90,7,100,105,114,110,97,109,101,114,
     2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,23,
     115,112,101,99,95,102,114,111,109,95,102,105,108,101,95,108,
-    111,99,97,116,105,111,110,56,2,0,0,115,62,0,0,0,
+    111,99,97,116,105,111,110,57,2,0,0,115,62,0,0,0,
     0,12,8,4,4,1,10,2,2,1,14,1,14,1,8,2,
     10,8,16,1,6,3,8,1,16,1,14,1,10,1,6,1,
     6,2,4,3,8,2,10,1,2,1,14,1,14,1,6,2,
@@ -928,7 +928,7 @@ const unsigned char _Py_M__importlib_external[] = {
     65,67,72,73,78,69,41,2,218,3,99,108,115,114,3,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
     0,218,14,95,111,112,101,110,95,114,101,103,105,115,116,114,
-    121,136,2,0,0,115,8,0,0,0,0,2,2,1,14,1,
+    121,137,2,0,0,115,8,0,0,0,0,2,2,1,14,1,
     14,1,122,36,87,105,110,100,111,119,115,82,101,103,105,115,
     116,114,121,70,105,110,100,101,114,46,95,111,112,101,110,95,
     114,101,103,105,115,116,114,121,99,2,0,0,0,0,0,0,
@@ -953,7 +953,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,116,114,121,95,107,101,121,114,3,0,0,0,90,4,104,
     107,101,121,218,8,102,105,108,101,112,97,116,104,114,2,0,
     0,0,114,2,0,0,0,114,4,0,0,0,218,16,95,115,
-    101,97,114,99,104,95,114,101,103,105,115,116,114,121,143,2,
+    101,97,114,99,104,95,114,101,103,105,115,116,114,121,144,2,
     0,0,115,22,0,0,0,0,2,6,1,8,2,6,1,6,
     1,22,1,2,1,12,1,26,1,14,1,6,1,122,38,87,
     105,110,100,111,119,115,82,101,103,105,115,116,114,121,70,105,
@@ -976,7 +976,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,116,114,178,0,0,0,114,122,0,0,0,114,168,0,0,
     0,114,166,0,0,0,114,2,0,0,0,114,2,0,0,0,
     114,4,0,0,0,218,9,102,105,110,100,95,115,112,101,99,
-    158,2,0,0,115,26,0,0,0,0,2,10,1,8,1,4,
+    159,2,0,0,115,26,0,0,0,0,2,10,1,8,1,4,
     1,2,1,12,1,14,1,6,1,16,1,14,1,6,1,8,
     1,8,1,122,31,87,105,110,100,111,119,115,82,101,103,105,
     115,116,114,121,70,105,110,100,101,114,46,102,105,110,100,95,
@@ -994,7 +994,7 @@ const unsigned char _Py_M__importlib_external[] = {
     78,41,2,114,182,0,0,0,114,122,0,0,0,41,4,114,
     172,0,0,0,114,121,0,0,0,114,35,0,0,0,114,166,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,218,11,102,105,110,100,95,109,111,100,117,108,101,174,
+    0,0,218,11,102,105,110,100,95,109,111,100,117,108,101,175,
     2,0,0,115,8,0,0,0,0,7,12,1,8,1,6,2,
     122,33,87,105,110,100,111,119,115,82,101,103,105,115,116,114,
     121,70,105,110,100,101,114,46,102,105,110,100,95,109,111,100,
@@ -1004,7 +1004,7 @@ const unsigned char _Py_M__importlib_external[] = {
     11,99,108,97,115,115,109,101,116,104,111,100,114,173,0,0,
     0,114,179,0,0,0,114,182,0,0,0,114,183,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,170,0,0,0,124,2,0,0,115,18,0,
+    4,0,0,0,114,170,0,0,0,125,2,0,0,115,18,0,
     0,0,12,5,4,3,4,2,4,2,12,7,12,15,2,1,
     12,15,2,1,114,170,0,0,0,99,0,0,0,0,0,0,
     0,0,0,0,0,0,2,0,0,0,64,0,0,0,115,48,
@@ -1039,7 +1039,7 @@ const unsigned char _Py_M__importlib_external[] = {
     121,0,0,0,114,96,0,0,0,90,13,102,105,108,101,110,
     97,109,101,95,98,97,115,101,90,9,116,97,105,108,95,110,
     97,109,101,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,161,0,0,0,193,2,0,0,115,8,0,0,0,
+    0,0,114,161,0,0,0,194,2,0,0,115,8,0,0,0,
     0,3,18,1,16,1,14,1,122,24,95,76,111,97,100,101,
     114,66,97,115,105,99,115,46,105,115,95,112,97,99,107,97,
     103,101,99,2,0,0,0,0,0,0,0,2,0,0,0,1,
@@ -1049,7 +1049,7 @@ const unsigned char _Py_M__importlib_external[] = {
     100,117,108,101,32,99,114,101,97,116,105,111,110,46,78,114,
     2,0,0,0,41,2,114,102,0,0,0,114,166,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    13,99,114,101,97,116,101,95,109,111,100,117,108,101,201,2,
+    13,99,114,101,97,116,101,95,109,111,100,117,108,101,202,2,
     0,0,115,0,0,0,0,122,27,95,76,111,97,100,101,114,
     66,97,115,105,99,115,46,99,114,101,97,116,101,95,109,111,
     100,117,108,101,99,2,0,0,0,0,0,0,0,3,0,0,
@@ -1069,7 +1069,7 @@ const unsigned char _Py_M__importlib_external[] = {
     4,101,120,101,99,114,113,0,0,0,41,3,114,102,0,0,
     0,218,6,109,111,100,117,108,101,114,144,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,11,101,
-    120,101,99,95,109,111,100,117,108,101,204,2,0,0,115,10,
+    120,101,99,95,109,111,100,117,108,101,205,2,0,0,115,10,
     0,0,0,0,2,12,1,8,1,6,1,10,1,122,25,95,
     76,111,97,100,101,114,66,97,115,105,99,115,46,101,120,101,
     99,95,109,111,100,117,108,101,99,2,0,0,0,0,0,0,
@@ -1080,14 +1080,14 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,218,17,95,108,111,97,100,95,109,111,100,117,108,
     101,95,115,104,105,109,41,2,114,102,0,0,0,114,121,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,11,108,111,97,100,95,109,111,100,117,108,101,212,2,
+    0,218,11,108,111,97,100,95,109,111,100,117,108,101,213,2,
     0,0,115,2,0,0,0,0,2,122,25,95,76,111,97,100,
     101,114,66,97,115,105,99,115,46,108,111,97,100,95,109,111,
     100,117,108,101,78,41,8,114,107,0,0,0,114,106,0,0,
     0,114,108,0,0,0,114,109,0,0,0,114,161,0,0,0,
     114,187,0,0,0,114,192,0,0,0,114,194,0,0,0,114,
     2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,185,0,0,0,188,2,0,0,115,8,0,0,
+    0,0,0,114,185,0,0,0,189,2,0,0,115,8,0,0,
     0,12,5,8,8,8,3,8,8,114,185,0,0,0,99,0,
     0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,
     0,0,0,115,74,0,0,0,101,0,90,1,100,0,90,2,
@@ -1112,7 +1112,7 @@ const unsigned char _Py_M__importlib_external[] = {
     46,10,32,32,32,32,32,32,32,32,78,41,1,114,40,0,
     0,0,41,2,114,102,0,0,0,114,35,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,10,112,
-    97,116,104,95,109,116,105,109,101,219,2,0,0,115,2,0,
+    97,116,104,95,109,116,105,109,101,220,2,0,0,115,2,0,
     0,0,0,6,122,23,83,111,117,114,99,101,76,111,97,100,
     101,114,46,112,97,116,104,95,109,116,105,109,101,99,2,0,
     0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,0,
@@ -1147,7 +1147,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,32,32,32,32,32,114,149,0,0,0,41,1,114,196,0,
     0,0,41,2,114,102,0,0,0,114,35,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,10,112,
-    97,116,104,95,115,116,97,116,115,227,2,0,0,115,2,0,
+    97,116,104,95,115,116,97,116,115,228,2,0,0,115,2,0,
     0,0,0,11,122,23,83,111,117,114,99,101,76,111,97,100,
     101,114,46,112,97,116,104,95,115,116,97,116,115,99,4,0,
     0,0,0,0,0,0,4,0,0,0,4,0,0,0,67,0,
@@ -1171,7 +1171,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,90,10,99,97,99,104,101,95,112,97,116,104,114,
     54,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
     0,0,0,218,15,95,99,97,99,104,101,95,98,121,116,101,
-    99,111,100,101,240,2,0,0,115,2,0,0,0,0,8,122,
+    99,111,100,101,241,2,0,0,115,2,0,0,0,0,8,122,
     28,83,111,117,114,99,101,76,111,97,100,101,114,46,95,99,
     97,99,104,101,95,98,121,116,101,99,111,100,101,99,3,0,
     0,0,0,0,0,0,3,0,0,0,1,0,0,0,67,0,
@@ -1188,7 +1188,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,32,32,32,32,78,114,2,0,0,0,41,3,114,102,0,
     0,0,114,35,0,0,0,114,54,0,0,0,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,114,198,0,0,0,
-    250,2,0,0,115,0,0,0,0,122,21,83,111,117,114,99,
+    251,2,0,0,115,0,0,0,0,122,21,83,111,117,114,99,
     101,76,111,97,100,101,114,46,115,101,116,95,100,97,116,97,
     99,2,0,0,0,0,0,0,0,5,0,0,0,10,0,0,
     0,67,0,0,0,115,82,0,0,0,124,0,160,0,124,1,
@@ -1208,7 +1208,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,157,0,0,0,41,5,114,102,0,0,0,114,121,0,0,
     0,114,35,0,0,0,114,155,0,0,0,218,3,101,120,99,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    10,103,101,116,95,115,111,117,114,99,101,1,3,0,0,115,
+    10,103,101,116,95,115,111,117,114,99,101,2,3,0,0,115,
     14,0,0,0,0,2,10,1,2,1,14,1,16,1,4,1,
     28,1,122,23,83,111,117,114,99,101,76,111,97,100,101,114,
     46,103,101,116,95,115,111,117,114,99,101,114,89,0,0,0,
@@ -1230,7 +1230,7 @@ const unsigned char _Py_M__importlib_external[] = {
     105,108,101,41,4,114,102,0,0,0,114,54,0,0,0,114,
     35,0,0,0,114,203,0,0,0,114,2,0,0,0,114,2,
     0,0,0,114,4,0,0,0,218,14,115,111,117,114,99,101,
-    95,116,111,95,99,111,100,101,11,3,0,0,115,4,0,0,
+    95,116,111,95,99,111,100,101,12,3,0,0,115,4,0,0,
     0,0,5,12,1,122,27,83,111,117,114,99,101,76,111,97,
     100,101,114,46,115,111,117,114,99,101,95,116,111,95,99,111,
     100,101,99,2,0,0,0,0,0,0,0,15,0,0,0,9,
@@ -1309,7 +1309,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,132,0,0,0,114,69,0,0,0,90,10,98,121,
     116,101,115,95,100,97,116,97,90,11,99,111,100,101,95,111,
     98,106,101,99,116,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,188,0,0,0,19,3,0,0,115,134,0,
+    4,0,0,0,114,188,0,0,0,20,3,0,0,115,134,0,
     0,0,0,7,10,1,4,1,4,1,4,1,4,1,4,1,
     2,1,12,1,14,1,12,2,2,1,14,1,14,1,8,2,
     12,1,2,1,14,1,14,1,6,3,2,1,8,2,2,1,
@@ -1324,7 +1324,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,196,0,0,0,114,197,0,0,0,114,199,0,0,0,
     114,198,0,0,0,114,202,0,0,0,114,206,0,0,0,114,
     188,0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,195,0,0,0,217,2,0,
+    0,0,0,114,4,0,0,0,114,195,0,0,0,218,2,0,
     0,115,14,0,0,0,8,2,8,8,8,13,8,10,8,7,
     8,10,14,8,114,195,0,0,0,99,0,0,0,0,0,0,
     0,0,0,0,0,0,4,0,0,0,0,0,0,0,115,124,
@@ -1354,7 +1354,7 @@ const unsigned char _Py_M__importlib_external[] = {
     41,2,114,100,0,0,0,114,35,0,0,0,41,3,114,102,
     0,0,0,114,121,0,0,0,114,35,0,0,0,114,2,0,
     0,0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,
-    0,110,3,0,0,115,4,0,0,0,0,3,6,1,122,19,
+    0,111,3,0,0,115,4,0,0,0,0,3,6,1,122,19,
     70,105,108,101,76,111,97,100,101,114,46,95,95,105,110,105,
     116,95,95,99,2,0,0,0,0,0,0,0,2,0,0,0,
     2,0,0,0,67,0,0,0,115,24,0,0,0,124,0,106,
@@ -1362,7 +1362,7 @@ const unsigned char _Py_M__importlib_external[] = {
     1,107,2,83,0,41,1,78,41,2,218,9,95,95,99,108,
     97,115,115,95,95,114,113,0,0,0,41,2,114,102,0,0,
     0,218,5,111,116,104,101,114,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,218,6,95,95,101,113,95,95,116,
+    0,0,114,4,0,0,0,218,6,95,95,101,113,95,95,117,
     3,0,0,115,4,0,0,0,0,1,12,1,122,17,70,105,
     108,101,76,111,97,100,101,114,46,95,95,101,113,95,95,99,
     1,0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,
@@ -1371,7 +1371,7 @@ const unsigned char _Py_M__importlib_external[] = {
     41,3,218,4,104,97,115,104,114,100,0,0,0,114,35,0,
     0,0,41,1,114,102,0,0,0,114,2,0,0,0,114,2,
     0,0,0,114,4,0,0,0,218,8,95,95,104,97,115,104,
-    95,95,120,3,0,0,115,2,0,0,0,0,1,122,19,70,
+    95,95,121,3,0,0,115,2,0,0,0,0,1,122,19,70,
     105,108,101,76,111,97,100,101,114,46,95,95,104,97,115,104,
     95,95,99,2,0,0,0,0,0,0,0,2,0,0,0,3,
     0,0,0,3,0,0,0,115,16,0,0,0,116,0,116,1,
@@ -1385,7 +1385,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,32,32,32,41,3,218,5,115,117,112,101,114,114,212,0,
     0,0,114,194,0,0,0,41,2,114,102,0,0,0,114,121,
     0,0,0,41,1,114,213,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,194,0,0,0,123,3,0,0,115,2,0,
+    4,0,0,0,114,194,0,0,0,124,3,0,0,115,2,0,
     0,0,0,10,122,22,70,105,108,101,76,111,97,100,101,114,
     46,108,111,97,100,95,109,111,100,117,108,101,99,2,0,0,
     0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,
@@ -1396,7 +1396,7 @@ const unsigned char _Py_M__importlib_external[] = {
     104,101,32,102,105,110,100,101,114,46,41,1,114,35,0,0,
     0,41,2,114,102,0,0,0,114,121,0,0,0,114,2,0,
     0,0,114,2,0,0,0,114,4,0,0,0,114,159,0,0,
-    0,135,3,0,0,115,2,0,0,0,0,3,122,23,70,105,
+    0,136,3,0,0,115,2,0,0,0,0,3,122,23,70,105,
     108,101,76,111,97,100,101,114,46,103,101,116,95,102,105,108,
     101,110,97,109,101,99,2,0,0,0,0,0,0,0,3,0,
     0,0,9,0,0,0,67,0,0,0,115,32,0,0,0,116,
@@ -1408,7 +1408,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,50,0,0,0,114,51,0,0,0,90,4,114,101,97,100,
     41,3,114,102,0,0,0,114,35,0,0,0,114,55,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,200,0,0,0,140,3,0,0,115,4,0,0,0,0,2,
+    114,200,0,0,0,141,3,0,0,115,4,0,0,0,0,2,
     14,1,122,19,70,105,108,101,76,111,97,100,101,114,46,103,
     101,116,95,100,97,116,97,99,2,0,0,0,0,0,0,0,
     2,0,0,0,3,0,0,0,67,0,0,0,115,18,0,0,
@@ -1416,7 +1416,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,83,0,41,1,78,41,1,114,161,0,0,0,41,2,114,
     102,0,0,0,114,191,0,0,0,114,2,0,0,0,114,2,
     0,0,0,114,4,0,0,0,218,19,103,101,116,95,114,101,
-    115,111,117,114,99,101,95,114,101,97,100,101,114,147,3,0,
+    115,111,117,114,99,101,95,114,101,97,100,101,114,148,3,0,
     0,115,6,0,0,0,0,2,10,1,4,1,122,30,70,105,
     108,101,76,111,97,100,101,114,46,103,101,116,95,114,101,115,
     111,117,114,99,101,95,114,101,97,100,101,114,99,2,0,0,
@@ -1429,7 +1429,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,102,0,0,0,218,8,114,101,115,111,117,114,99,101,114,
     35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
     0,0,0,218,13,111,112,101,110,95,114,101,115,111,117,114,
-    99,101,153,3,0,0,115,4,0,0,0,0,1,20,1,122,
+    99,101,154,3,0,0,115,4,0,0,0,0,1,20,1,122,
     24,70,105,108,101,76,111,97,100,101,114,46,111,112,101,110,
     95,114,101,115,111,117,114,99,101,99,2,0,0,0,0,0,
     0,0,3,0,0,0,3,0,0,0,67,0,0,0,115,38,
@@ -1442,7 +1442,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,41,3,114,102,0,0,0,114,221,0,0,0,114,35,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
     0,0,218,13,114,101,115,111,117,114,99,101,95,112,97,116,
-    104,157,3,0,0,115,8,0,0,0,0,1,10,1,4,1,
+    104,158,3,0,0,115,8,0,0,0,0,1,10,1,4,1,
     20,1,122,24,70,105,108,101,76,111,97,100,101,114,46,114,
     101,115,111,117,114,99,101,95,112,97,116,104,99,2,0,0,
     0,0,0,0,0,3,0,0,0,3,0,0,0,67,0,0,
@@ -1453,7 +1453,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,114,38,0,0,0,114,35,0,0,0,114,44,0,
     0,0,41,3,114,102,0,0,0,114,100,0,0,0,114,35,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,223,0,0,0,163,3,0,0,115,8,0,0,0,
+    0,0,114,223,0,0,0,164,3,0,0,115,8,0,0,0,
     0,1,8,1,4,1,20,1,122,22,70,105,108,101,76,111,
     97,100,101,114,46,105,115,95,114,101,115,111,117,114,99,101,
     99,1,0,0,0,0,0,0,0,1,0,0,0,5,0,0,
@@ -1463,7 +1463,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,114,114,1,0,0,0,218,7,108,105,115,116,100,105,114,
     114,38,0,0,0,114,35,0,0,0,41,1,114,102,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    218,8,99,111,110,116,101,110,116,115,169,3,0,0,115,2,
+    218,8,99,111,110,116,101,110,116,115,170,3,0,0,115,2,
     0,0,0,0,1,122,19,70,105,108,101,76,111,97,100,101,
     114,46,99,111,110,116,101,110,116,115,41,17,114,107,0,0,
     0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,0,
@@ -1473,7 +1473,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,223,0,0,0,114,228,0,0,0,90,13,95,95,
     99,108,97,115,115,99,101,108,108,95,95,114,2,0,0,0,
     114,2,0,0,0,41,1,114,213,0,0,0,114,4,0,0,
-    0,114,212,0,0,0,105,3,0,0,115,22,0,0,0,12,
+    0,114,212,0,0,0,106,3,0,0,115,22,0,0,0,12,
     5,8,6,8,4,8,3,16,12,12,5,8,7,12,6,8,
     4,8,6,8,6,114,212,0,0,0,99,0,0,0,0,0,
     0,0,0,0,0,0,0,3,0,0,0,64,0,0,0,115,
@@ -1495,7 +1495,7 @@ const unsigned char _Py_M__importlib_external[] = {
     115,116,95,109,116,105,109,101,90,7,115,116,95,115,105,122,
     101,41,3,114,102,0,0,0,114,35,0,0,0,114,211,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,197,0,0,0,177,3,0,0,115,4,0,0,0,0,
+    0,114,197,0,0,0,178,3,0,0,115,4,0,0,0,0,
     2,8,1,122,27,83,111,117,114,99,101,70,105,108,101,76,
     111,97,100,101,114,46,112,97,116,104,95,115,116,97,116,115,
     99,4,0,0,0,0,0,0,0,5,0,0,0,5,0,0,
@@ -1505,7 +1505,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,99,0,0,0,114,198,0,0,0,41,5,114,102,0,0,
     0,114,92,0,0,0,114,91,0,0,0,114,54,0,0,0,
     114,42,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,199,0,0,0,182,3,0,0,115,4,0,
+    4,0,0,0,114,199,0,0,0,183,3,0,0,115,4,0,
     0,0,0,2,8,1,122,32,83,111,117,114,99,101,70,105,
     108,101,76,111,97,100,101,114,46,95,99,97,99,104,101,95,
     98,121,116,101,99,111,100,101,105,182,1,0,0,41,1,114,
@@ -1540,7 +1540,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,231,0,0,0,218,6,112,97,114,101,110,116,114,96,0,
     0,0,114,27,0,0,0,114,23,0,0,0,114,201,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,198,0,0,0,187,3,0,0,115,42,0,0,0,0,2,
+    114,198,0,0,0,188,3,0,0,115,42,0,0,0,0,2,
     12,1,4,2,14,1,12,1,14,2,14,1,10,1,2,1,
     14,1,14,2,6,1,16,3,6,1,8,1,22,1,2,1,
     12,1,16,1,16,2,8,1,122,25,83,111,117,114,99,101,
@@ -1549,7 +1549,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,108,0,0,0,114,109,0,0,0,114,197,0,0,0,114,
     199,0,0,0,114,198,0,0,0,114,2,0,0,0,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,114,229,0,
-    0,0,173,3,0,0,115,6,0,0,0,12,4,8,5,8,
+    0,0,174,3,0,0,115,6,0,0,0,12,4,8,5,8,
     5,114,229,0,0,0,99,0,0,0,0,0,0,0,0,0,
     0,0,0,2,0,0,0,64,0,0,0,115,32,0,0,0,
     101,0,90,1,100,0,90,2,100,1,90,3,100,2,100,3,
@@ -1570,7 +1570,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,133,0,0,0,114,145,0,0,0,114,208,0,0,0,
     41,5,114,102,0,0,0,114,121,0,0,0,114,35,0,0,
     0,114,54,0,0,0,114,132,0,0,0,114,2,0,0,0,
-    114,2,0,0,0,114,4,0,0,0,114,188,0,0,0,222,
+    114,2,0,0,0,114,4,0,0,0,114,188,0,0,0,223,
     3,0,0,115,18,0,0,0,0,1,10,1,10,4,2,1,
     8,2,12,1,2,1,14,1,2,1,122,29,83,111,117,114,
     99,101,108,101,115,115,70,105,108,101,76,111,97,100,101,114,
@@ -1581,13 +1581,13 @@ const unsigned char _Py_M__importlib_external[] = {
     105,115,32,110,111,32,115,111,117,114,99,101,32,99,111,100,
     101,46,78,114,2,0,0,0,41,2,114,102,0,0,0,114,
     121,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,202,0,0,0,238,3,0,0,115,2,0,0,
+    0,0,0,114,202,0,0,0,239,3,0,0,115,2,0,0,
     0,0,2,122,31,83,111,117,114,99,101,108,101,115,115,70,
     105,108,101,76,111,97,100,101,114,46,103,101,116,95,115,111,
     117,114,99,101,78,41,6,114,107,0,0,0,114,106,0,0,
     0,114,108,0,0,0,114,109,0,0,0,114,188,0,0,0,
     114,202,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    2,0,0,0,114,4,0,0,0,114,234,0,0,0,218,3,
+    2,0,0,0,114,4,0,0,0,114,234,0,0,0,219,3,
     0,0,115,4,0,0,0,12,4,8,16,114,234,0,0,0,
     99,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,
     0,64,0,0,0,115,92,0,0,0,101,0,90,1,100,0,
@@ -1609,7 +1609,7 @@ const unsigned char _Py_M__importlib_external[] = {
     95,1,100,0,83,0,41,1,78,41,2,114,100,0,0,0,
     114,35,0,0,0,41,3,114,102,0,0,0,114,100,0,0,
     0,114,35,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,186,0,0,0,255,3,0,0,115,4,
+    114,4,0,0,0,114,186,0,0,0,0,4,0,0,115,4,
     0,0,0,0,1,6,1,122,28,69,120,116,101,110,115,105,
     111,110,70,105,108,101,76,111,97,100,101,114,46,95,95,105,
     110,105,116,95,95,99,2,0,0,0,0,0,0,0,2,0,
@@ -1618,7 +1618,7 @@ const unsigned char _Py_M__importlib_external[] = {
     1,106,1,107,2,83,0,41,1,78,41,2,114,213,0,0,
     0,114,113,0,0,0,41,2,114,102,0,0,0,114,214,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,215,0,0,0,3,4,0,0,115,4,0,0,0,0,
+    0,114,215,0,0,0,4,4,0,0,115,4,0,0,0,0,
     1,12,1,122,26,69,120,116,101,110,115,105,111,110,70,105,
     108,101,76,111,97,100,101,114,46,95,95,101,113,95,95,99,
     1,0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,
@@ -1626,7 +1626,7 @@ const unsigned char _Py_M__importlib_external[] = {
     1,116,0,124,0,106,2,131,1,65,0,83,0,41,1,78,
     41,3,114,216,0,0,0,114,100,0,0,0,114,35,0,0,
     0,41,1,114,102,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,114,217,0,0,0,7,4,0,0,
+    0,0,114,4,0,0,0,114,217,0,0,0,8,4,0,0,
     115,2,0,0,0,0,1,122,28,69,120,116,101,110,115,105,
     111,110,70,105,108,101,76,111,97,100,101,114,46,95,95,104,
     97,115,104,95,95,99,2,0,0,0,0,0,0,0,3,0,
@@ -1643,7 +1643,7 @@ const unsigned char _Py_M__importlib_external[] = {
     121,110,97,109,105,99,114,130,0,0,0,114,100,0,0,0,
     114,35,0,0,0,41,3,114,102,0,0,0,114,166,0,0,
     0,114,191,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,187,0,0,0,10,4,0,0,115,10,
+    114,4,0,0,0,114,187,0,0,0,11,4,0,0,115,10,
     0,0,0,0,2,4,1,10,1,6,1,12,1,122,33,69,
     120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,100,
     101,114,46,99,114,101,97,116,101,95,109,111,100,117,108,101,
@@ -1660,7 +1660,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,99,95,100,121,110,97,109,105,99,114,130,0,0,0,114,
     100,0,0,0,114,35,0,0,0,41,2,114,102,0,0,0,
     114,191,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,192,0,0,0,18,4,0,0,115,6,0,
+    4,0,0,0,114,192,0,0,0,19,4,0,0,115,6,0,
     0,0,0,2,14,1,6,1,122,31,69,120,116,101,110,115,
     105,111,110,70,105,108,101,76,111,97,100,101,114,46,101,120,
     101,99,95,109,111,100,117,108,101,99,2,0,0,0,0,0,
@@ -1678,7 +1678,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,41,2,114,22,0,0,0,218,6,115,117,102,102,105,
     120,41,1,218,9,102,105,108,101,95,110,97,109,101,114,2,
     0,0,0,114,4,0,0,0,250,9,60,103,101,110,101,120,
-    112,114,62,27,4,0,0,115,2,0,0,0,4,1,122,49,
+    112,114,62,28,4,0,0,115,2,0,0,0,4,1,122,49,
     69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,97,
     100,101,114,46,105,115,95,112,97,99,107,97,103,101,46,60,
     108,111,99,97,108,115,62,46,60,103,101,110,101,120,112,114,
@@ -1686,7 +1686,7 @@ const unsigned char _Py_M__importlib_external[] = {
     110,121,218,18,69,88,84,69,78,83,73,79,78,95,83,85,
     70,70,73,88,69,83,41,2,114,102,0,0,0,114,121,0,
     0,0,114,2,0,0,0,41,1,114,237,0,0,0,114,4,
-    0,0,0,114,161,0,0,0,24,4,0,0,115,6,0,0,
+    0,0,0,114,161,0,0,0,25,4,0,0,115,6,0,0,
     0,0,2,14,1,12,1,122,30,69,120,116,101,110,115,105,
     111,110,70,105,108,101,76,111,97,100,101,114,46,105,115,95,
     112,97,99,107,97,103,101,99,2,0,0,0,0,0,0,0,
@@ -1697,7 +1697,7 @@ const unsigned char _Py_M__importlib_external[] = {
     111,116,32,99,114,101,97,116,101,32,97,32,99,111,100,101,
     32,111,98,106,101,99,116,46,78,114,2,0,0,0,41,2,
     114,102,0,0,0,114,121,0,0,0,114,2,0,0,0,114,
-    2,0,0,0,114,4,0,0,0,114,188,0,0,0,30,4,
+    2,0,0,0,114,4,0,0,0,114,188,0,0,0,31,4,
     0,0,115,2,0,0,0,0,2,122,28,69,120,116,101,110,
     115,105,111,110,70,105,108,101,76,111,97,100,101,114,46,103,
     101,116,95,99,111,100,101,99,2,0,0,0,0,0,0,0,
@@ -1708,7 +1708,7 @@ const unsigned char _Py_M__importlib_external[] = {
     111,32,115,111,117,114,99,101,32,99,111,100,101,46,78,114,
     2,0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    202,0,0,0,34,4,0,0,115,2,0,0,0,0,2,122,
+    202,0,0,0,35,4,0,0,115,2,0,0,0,0,2,122,
     30,69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,
     97,100,101,114,46,103,101,116,95,115,111,117,114,99,101,99,
     2,0,0,0,0,0,0,0,2,0,0,0,1,0,0,0,
@@ -1719,7 +1719,7 @@ const unsigned char _Py_M__importlib_external[] = {
     121,32,116,104,101,32,102,105,110,100,101,114,46,41,1,114,
     35,0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    159,0,0,0,38,4,0,0,115,2,0,0,0,0,3,122,
+    159,0,0,0,39,4,0,0,115,2,0,0,0,0,3,122,
     32,69,120,116,101,110,115,105,111,110,70,105,108,101,76,111,
     97,100,101,114,46,103,101,116,95,102,105,108,101,110,97,109,
     101,78,41,14,114,107,0,0,0,114,106,0,0,0,114,108,
@@ -1728,7 +1728,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,161,0,0,0,114,188,0,0,0,114,202,0,0,0,
     114,118,0,0,0,114,159,0,0,0,114,2,0,0,0,114,
     2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,235,
-    0,0,0,247,3,0,0,115,18,0,0,0,12,8,8,4,
+    0,0,0,248,3,0,0,115,18,0,0,0,12,8,8,4,
     8,4,8,3,8,8,8,6,8,6,8,4,8,4,114,235,
     0,0,0,99,0,0,0,0,0,0,0,0,0,0,0,0,
     2,0,0,0,64,0,0,0,115,96,0,0,0,101,0,90,
@@ -1769,7 +1769,7 @@ const unsigned char _Py_M__importlib_external[] = {
     100,101,114,41,4,114,102,0,0,0,114,100,0,0,0,114,
     35,0,0,0,218,11,112,97,116,104,95,102,105,110,100,101,
     114,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,186,0,0,0,51,4,0,0,115,8,0,0,0,0,1,
+    114,186,0,0,0,52,4,0,0,115,8,0,0,0,0,1,
     6,1,6,1,14,1,122,23,95,78,97,109,101,115,112,97,
     99,101,80,97,116,104,46,95,95,105,110,105,116,95,95,99,
     1,0,0,0,0,0,0,0,4,0,0,0,3,0,0,0,
@@ -1786,7 +1786,7 @@ const unsigned char _Py_M__importlib_external[] = {
     102,0,0,0,114,233,0,0,0,218,3,100,111,116,90,2,
     109,101,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
     0,218,23,95,102,105,110,100,95,112,97,114,101,110,116,95,
-    112,97,116,104,95,110,97,109,101,115,57,4,0,0,115,8,
+    112,97,116,104,95,110,97,109,101,115,58,4,0,0,115,8,
     0,0,0,0,2,18,1,8,2,4,3,122,38,95,78,97,
     109,101,115,112,97,99,101,80,97,116,104,46,95,102,105,110,
     100,95,112,97,114,101,110,116,95,112,97,116,104,95,110,97,
@@ -1799,7 +1799,7 @@ const unsigned char _Py_M__importlib_external[] = {
     97,114,101,110,116,95,109,111,100,117,108,101,95,110,97,109,
     101,90,14,112,97,116,104,95,97,116,116,114,95,110,97,109,
     101,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,244,0,0,0,67,4,0,0,115,4,0,0,0,0,1,
+    114,244,0,0,0,68,4,0,0,115,4,0,0,0,0,1,
     12,1,122,31,95,78,97,109,101,115,112,97,99,101,80,97,
     116,104,46,95,103,101,116,95,112,97,114,101,110,116,95,112,
     97,116,104,99,1,0,0,0,0,0,0,0,3,0,0,0,
@@ -1815,7 +1815,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,90,11,112,97,114,101,110,116,95,112,97,116,104,114,166,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
     0,0,218,12,95,114,101,99,97,108,99,117,108,97,116,101,
-    71,4,0,0,115,16,0,0,0,0,2,12,1,10,1,14,
+    72,4,0,0,115,16,0,0,0,0,2,12,1,10,1,14,
     3,18,1,6,1,8,1,6,1,122,27,95,78,97,109,101,
     115,112,97,99,101,80,97,116,104,46,95,114,101,99,97,108,
     99,117,108,97,116,101,99,1,0,0,0,0,0,0,0,1,
@@ -1823,7 +1823,7 @@ const unsigned char _Py_M__importlib_external[] = {
     116,0,124,0,160,1,161,0,131,1,83,0,41,1,78,41,
     2,114,226,0,0,0,114,251,0,0,0,41,1,114,102,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,218,8,95,95,105,116,101,114,95,95,84,4,0,0,115,
+    0,218,8,95,95,105,116,101,114,95,95,85,4,0,0,115,
     2,0,0,0,0,1,122,23,95,78,97,109,101,115,112,97,
     99,101,80,97,116,104,46,95,95,105,116,101,114,95,95,99,
     3,0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,
@@ -1832,14 +1832,14 @@ const unsigned char _Py_M__importlib_external[] = {
     0,41,3,114,102,0,0,0,218,5,105,110,100,101,120,114,
     35,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
     0,0,0,218,11,95,95,115,101,116,105,116,101,109,95,95,
-    87,4,0,0,115,2,0,0,0,0,1,122,26,95,78,97,
+    88,4,0,0,115,2,0,0,0,0,1,122,26,95,78,97,
     109,101,115,112,97,99,101,80,97,116,104,46,95,95,115,101,
     116,105,116,101,109,95,95,99,1,0,0,0,0,0,0,0,
     1,0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,
     0,116,0,124,0,160,1,161,0,131,1,83,0,41,1,78,
     41,2,114,31,0,0,0,114,251,0,0,0,41,1,114,102,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,218,7,95,95,108,101,110,95,95,90,4,0,0,115,
+    0,0,218,7,95,95,108,101,110,95,95,91,4,0,0,115,
     2,0,0,0,0,1,122,22,95,78,97,109,101,115,112,97,
     99,101,80,97,116,104,46,95,95,108,101,110,95,95,99,1,
     0,0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,
@@ -1848,7 +1848,7 @@ const unsigned char _Py_M__importlib_external[] = {
     97,99,101,80,97,116,104,40,123,33,114,125,41,41,2,114,
     48,0,0,0,114,243,0,0,0,41,1,114,102,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,218,
-    8,95,95,114,101,112,114,95,95,93,4,0,0,115,2,0,
+    8,95,95,114,101,112,114,95,95,94,4,0,0,115,2,0,
     0,0,0,1,122,23,95,78,97,109,101,115,112,97,99,101,
     80,97,116,104,46,95,95,114,101,112,114,95,95,99,2,0,
     0,0,0,0,0,0,2,0,0,0,3,0,0,0,67,0,
@@ -1856,7 +1856,7 @@ const unsigned char _Py_M__importlib_external[] = {
     6,83,0,41,1,78,41,1,114,251,0,0,0,41,2,114,
     102,0,0,0,218,4,105,116,101,109,114,2,0,0,0,114,
     2,0,0,0,114,4,0,0,0,218,12,95,95,99,111,110,
-    116,97,105,110,115,95,95,96,4,0,0,115,2,0,0,0,
+    116,97,105,110,115,95,95,97,4,0,0,115,2,0,0,0,
     0,1,122,27,95,78,97,109,101,115,112,97,99,101,80,97,
     116,104,46,95,95,99,111,110,116,97,105,110,115,95,95,99,
     2,0,0,0,0,0,0,0,2,0,0,0,3,0,0,0,
@@ -1864,7 +1864,7 @@ const unsigned char _Py_M__importlib_external[] = {
     1,161,1,1,0,100,0,83,0,41,1,78,41,2,114,243,
     0,0,0,114,165,0,0,0,41,2,114,102,0,0,0,114,
     1,1,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,165,0,0,0,99,4,0,0,115,2,0,0,
+    0,0,0,114,165,0,0,0,100,4,0,0,115,2,0,0,
     0,0,1,122,21,95,78,97,109,101,115,112,97,99,101,80,
     97,116,104,46,97,112,112,101,110,100,78,41,14,114,107,0,
     0,0,114,106,0,0,0,114,108,0,0,0,114,109,0,0,
@@ -1872,7 +1872,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,251,0,0,0,114,252,0,0,0,114,254,0,0,0,114,
     255,0,0,0,114,0,1,0,0,114,2,1,0,0,114,165,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,
-    0,0,114,4,0,0,0,114,241,0,0,0,44,4,0,0,
+    0,0,114,4,0,0,0,114,241,0,0,0,45,4,0,0,
     115,20,0,0,0,12,7,8,6,8,10,8,4,8,13,8,
     3,8,3,8,3,8,3,8,3,114,241,0,0,0,99,0,
     0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,64,
@@ -1889,7 +1889,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,243,0,0,0,41,4,114,102,0,0,0,114,100,
     0,0,0,114,35,0,0,0,114,247,0,0,0,114,2,0,
     0,0,114,2,0,0,0,114,4,0,0,0,114,186,0,0,
-    0,105,4,0,0,115,2,0,0,0,0,1,122,25,95,78,
+    0,106,4,0,0,115,2,0,0,0,0,1,122,25,95,78,
     97,109,101,115,112,97,99,101,76,111,97,100,101,114,46,95,
     95,105,110,105,116,95,95,99,2,0,0,0,0,0,0,0,
     2,0,0,0,3,0,0,0,67,0,0,0,115,12,0,0,
@@ -1906,21 +1906,21 @@ const unsigned char _Py_M__importlib_external[] = {
     2,114,48,0,0,0,114,107,0,0,0,41,2,114,172,0,
     0,0,114,191,0,0,0,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,11,109,111,100,117,108,101,95,114,
-    101,112,114,108,4,0,0,115,2,0,0,0,0,7,122,28,
+    101,112,114,109,4,0,0,115,2,0,0,0,0,7,122,28,
     95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,
     46,109,111,100,117,108,101,95,114,101,112,114,99,2,0,0,
     0,0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,
     0,115,4,0,0,0,100,1,83,0,41,2,78,84,114,2,
     0,0,0,41,2,114,102,0,0,0,114,121,0,0,0,114,
     2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,161,
-    0,0,0,117,4,0,0,115,2,0,0,0,0,1,122,27,
+    0,0,0,118,4,0,0,115,2,0,0,0,0,1,122,27,
     95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,114,
     46,105,115,95,112,97,99,107,97,103,101,99,2,0,0,0,
     0,0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,
     115,4,0,0,0,100,1,83,0,41,2,78,114,30,0,0,
     0,114,2,0,0,0,41,2,114,102,0,0,0,114,121,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,202,0,0,0,120,4,0,0,115,2,0,0,0,0,
+    0,114,202,0,0,0,121,4,0,0,115,2,0,0,0,0,
     1,122,27,95,78,97,109,101,115,112,97,99,101,76,111,97,
     100,101,114,46,103,101,116,95,115,111,117,114,99,101,99,2,
     0,0,0,0,0,0,0,2,0,0,0,6,0,0,0,67,
@@ -1929,7 +1929,7 @@ const unsigned char _Py_M__importlib_external[] = {
     122,8,60,115,116,114,105,110,103,62,114,190,0,0,0,84,
     41,1,114,204,0,0,0,41,1,114,205,0,0,0,41,2,
     114,102,0,0,0,114,121,0,0,0,114,2,0,0,0,114,
-    2,0,0,0,114,4,0,0,0,114,188,0,0,0,123,4,
+    2,0,0,0,114,4,0,0,0,114,188,0,0,0,124,4,
     0,0,115,2,0,0,0,0,1,122,25,95,78,97,109,101,
     115,112,97,99,101,76,111,97,100,101,114,46,103,101,116,95,
     99,111,100,101,99,2,0,0,0,0,0,0,0,2,0,0,
@@ -1939,14 +1939,14 @@ const unsigned char _Py_M__importlib_external[] = {
     109,111,100,117,108,101,32,99,114,101,97,116,105,111,110,46,
     78,114,2,0,0,0,41,2,114,102,0,0,0,114,166,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,187,0,0,0,126,4,0,0,115,0,0,0,0,122,
+    0,114,187,0,0,0,127,4,0,0,115,0,0,0,0,122,
     30,95,78,97,109,101,115,112,97,99,101,76,111,97,100,101,
     114,46,99,114,101,97,116,101,95,109,111,100,117,108,101,99,
     2,0,0,0,0,0,0,0,2,0,0,0,1,0,0,0,
     67,0,0,0,115,4,0,0,0,100,0,83,0,41,1,78,
     114,2,0,0,0,41,2,114,102,0,0,0,114,191,0,0,
     0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,192,0,0,0,129,4,0,0,115,2,0,0,0,0,1,
+    114,192,0,0,0,130,4,0,0,115,2,0,0,0,0,1,
     122,28,95,78,97,109,101,115,112,97,99,101,76,111,97,100,
     101,114,46,101,120,101,99,95,109,111,100,117,108,101,99,2,
     0,0,0,0,0,0,0,2,0,0,0,4,0,0,0,67,
@@ -1964,7 +1964,7 @@ const unsigned char _Py_M__importlib_external[] = {
     41,4,114,116,0,0,0,114,130,0,0,0,114,243,0,0,
     0,114,193,0,0,0,41,2,114,102,0,0,0,114,121,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,194,0,0,0,132,4,0,0,115,6,0,0,0,0,
+    0,114,194,0,0,0,133,4,0,0,115,6,0,0,0,0,
     7,6,1,8,1,122,28,95,78,97,109,101,115,112,97,99,
     101,76,111,97,100,101,114,46,108,111,97,100,95,109,111,100,
     117,108,101,78,41,12,114,107,0,0,0,114,106,0,0,0,
@@ -1972,7 +1972,7 @@ const unsigned char _Py_M__importlib_external[] = {
     4,1,0,0,114,161,0,0,0,114,202,0,0,0,114,188,
     0,0,0,114,187,0,0,0,114,192,0,0,0,114,194,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,2,0,0,
-    0,114,4,0,0,0,114,3,1,0,0,104,4,0,0,115,
+    0,114,4,0,0,0,114,3,1,0,0,105,4,0,0,115,
     16,0,0,0,8,1,8,3,12,9,8,3,8,3,8,3,
     8,3,8,3,114,3,1,0,0,99,0,0,0,0,0,0,
     0,0,0,0,0,0,4,0,0,0,64,0,0,0,115,106,
@@ -2006,7 +2006,7 @@ const unsigned char _Py_M__importlib_external[] = {
     218,6,118,97,108,117,101,115,114,110,0,0,0,114,6,1,
     0,0,41,2,114,172,0,0,0,218,6,102,105,110,100,101,
     114,114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,6,1,0,0,150,4,0,0,115,6,0,0,0,0,4,
+    114,6,1,0,0,151,4,0,0,115,6,0,0,0,0,4,
     16,1,10,1,122,28,80,97,116,104,70,105,110,100,101,114,
     46,105,110,118,97,108,105,100,97,116,101,95,99,97,99,104,
     101,115,99,2,0,0,0,0,0,0,0,3,0,0,0,9,
@@ -2026,7 +2026,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,41,3,114,172,0,0,0,114,35,0,0,0,90,
     4,104,111,111,107,114,2,0,0,0,114,2,0,0,0,114,
     4,0,0,0,218,11,95,112,97,116,104,95,104,111,111,107,
-    115,158,4,0,0,115,16,0,0,0,0,3,16,1,12,1,
+    115,159,4,0,0,115,16,0,0,0,0,3,16,1,12,1,
     12,1,2,1,8,1,14,1,12,2,122,22,80,97,116,104,
     70,105,110,100,101,114,46,95,112,97,116,104,95,104,111,111,
     107,115,99,2,0,0,0,0,0,0,0,3,0,0,0,8,
@@ -2056,7 +2056,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,111,114,114,11,1,0,0,41,3,114,172,0,0,0,114,
     35,0,0,0,114,9,1,0,0,114,2,0,0,0,114,2,
     0,0,0,114,4,0,0,0,218,20,95,112,97,116,104,95,
-    105,109,112,111,114,116,101,114,95,99,97,99,104,101,171,4,
+    105,109,112,111,114,116,101,114,95,99,97,99,104,101,172,4,
     0,0,115,22,0,0,0,0,8,8,1,2,1,12,1,14,
     3,6,1,2,1,14,1,14,1,10,1,16,1,122,31,80,
     97,116,104,70,105,110,100,101,114,46,95,112,97,116,104,95,
@@ -2074,7 +2074,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,114,9,1,0,0,114,122,0,0,0,114,123,0,
     0,0,114,166,0,0,0,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,218,16,95,108,101,103,97,99,121,95,
-    103,101,116,95,115,112,101,99,193,4,0,0,115,18,0,0,
+    103,101,116,95,115,112,101,99,194,4,0,0,115,18,0,0,
     0,0,4,10,1,16,2,10,1,4,1,8,1,12,1,12,
     1,6,1,122,27,80,97,116,104,70,105,110,100,101,114,46,
     95,108,101,103,97,99,121,95,103,101,116,95,115,112,101,99,
@@ -2105,7 +2105,7 @@ const unsigned char _Py_M__importlib_external[] = {
     110,97,109,101,115,112,97,99,101,95,112,97,116,104,90,5,
     101,110,116,114,121,114,9,1,0,0,114,166,0,0,0,114,
     123,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,218,9,95,103,101,116,95,115,112,101,99,208,4,
+    0,0,0,218,9,95,103,101,116,95,115,112,101,99,209,4,
     0,0,115,40,0,0,0,0,5,4,1,10,1,14,1,2,
     1,10,1,8,1,10,1,14,2,12,1,8,1,2,1,10,
     1,4,1,6,1,8,1,8,5,14,2,12,1,6,1,122,
@@ -2133,7 +2133,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,241,0,0,0,41,6,114,172,0,0,0,114,121,0,0,
     0,114,35,0,0,0,114,181,0,0,0,114,166,0,0,0,
     114,16,1,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,182,0,0,0,240,4,0,0,115,26,0,
+    4,0,0,0,114,182,0,0,0,241,4,0,0,115,26,0,
     0,0,0,6,8,1,6,1,14,1,8,1,4,1,10,1,
     6,1,4,3,6,1,16,1,4,2,6,2,122,20,80,97,
     116,104,70,105,110,100,101,114,46,102,105,110,100,95,115,112,
@@ -2154,7 +2154,7 @@ const unsigned char _Py_M__importlib_external[] = {
     32,32,32,32,32,32,32,32,78,41,2,114,182,0,0,0,
     114,122,0,0,0,41,4,114,172,0,0,0,114,121,0,0,
     0,114,35,0,0,0,114,166,0,0,0,114,2,0,0,0,
-    114,2,0,0,0,114,4,0,0,0,114,183,0,0,0,8,
+    114,2,0,0,0,114,4,0,0,0,114,183,0,0,0,9,
     5,0,0,115,8,0,0,0,0,8,12,1,8,1,4,1,
     122,22,80,97,116,104,70,105,110,100,101,114,46,102,105,110,
     100,95,109,111,100,117,108,101,41,1,78,41,2,78,78,41,
@@ -2163,7 +2163,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,114,11,1,0,0,114,13,1,0,0,114,14,1,0,
     0,114,17,1,0,0,114,182,0,0,0,114,183,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,5,1,0,0,146,4,0,0,115,20,0,
+    4,0,0,0,114,5,1,0,0,147,4,0,0,115,20,0,
     0,0,12,4,12,8,12,13,12,22,12,15,2,1,12,31,
     2,1,12,23,2,1,114,5,1,0,0,99,0,0,0,0,
     0,0,0,0,0,0,0,0,3,0,0,0,64,0,0,0,
@@ -2207,7 +2207,7 @@ const unsigned char _Py_M__importlib_external[] = {
     2,86,0,1,0,113,2,100,0,83,0,41,1,78,114,2,
     0,0,0,41,2,114,22,0,0,0,114,236,0,0,0,41,
     1,114,122,0,0,0,114,2,0,0,0,114,4,0,0,0,
-    114,238,0,0,0,37,5,0,0,115,2,0,0,0,4,0,
+    114,238,0,0,0,38,5,0,0,115,2,0,0,0,4,0,
     122,38,70,105,108,101,70,105,110,100,101,114,46,95,95,105,
     110,105,116,95,95,46,60,108,111,99,97,108,115,62,46,60,
     103,101,110,101,120,112,114,62,114,59,0,0,0,114,89,0,
@@ -2219,7 +2219,7 @@ const unsigned char _Py_M__importlib_external[] = {
     102,0,0,0,114,35,0,0,0,218,14,108,111,97,100,101,
     114,95,100,101,116,97,105,108,115,90,7,108,111,97,100,101,
     114,115,114,168,0,0,0,114,2,0,0,0,41,1,114,122,
-    0,0,0,114,4,0,0,0,114,186,0,0,0,31,5,0,
+    0,0,0,114,4,0,0,0,114,186,0,0,0,32,5,0,
     0,115,16,0,0,0,0,4,4,1,14,1,28,1,6,2,
     10,1,6,1,8,1,122,19,70,105,108,101,70,105,110,100,
     101,114,46,95,95,105,110,105,116,95,95,99,1,0,0,0,
@@ -2229,7 +2229,7 @@ const unsigned char _Py_M__importlib_external[] = {
     101,32,100,105,114,101,99,116,111,114,121,32,109,116,105,109,
     101,46,114,89,0,0,0,78,41,1,114,20,1,0,0,41,
     1,114,102,0,0,0,114,2,0,0,0,114,2,0,0,0,
-    114,4,0,0,0,114,6,1,0,0,45,5,0,0,115,2,
+    114,4,0,0,0,114,6,1,0,0,46,5,0,0,115,2,
     0,0,0,0,2,122,28,70,105,108,101,70,105,110,100,101,
     114,46,105,110,118,97,108,105,100,97,116,101,95,99,97,99,
     104,101,115,99,2,0,0,0,0,0,0,0,3,0,0,0,
@@ -2252,7 +2252,7 @@ const unsigned char _Py_M__importlib_external[] = {
     78,41,3,114,182,0,0,0,114,122,0,0,0,114,158,0,
     0,0,41,3,114,102,0,0,0,114,121,0,0,0,114,166,
     0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,
-    0,0,114,119,0,0,0,51,5,0,0,115,8,0,0,0,
+    0,0,114,119,0,0,0,52,5,0,0,115,8,0,0,0,
     0,7,10,1,8,1,8,1,122,22,70,105,108,101,70,105,
     110,100,101,114,46,102,105,110,100,95,108,111,97,100,101,114,
     99,6,0,0,0,0,0,0,0,7,0,0,0,6,0,0,
@@ -2263,7 +2263,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,114,167,0,0,0,114,121,0,0,0,114,35,0,0,0,
     90,4,115,109,115,108,114,181,0,0,0,114,122,0,0,0,
     114,2,0,0,0,114,2,0,0,0,114,4,0,0,0,114,
-    17,1,0,0,63,5,0,0,115,6,0,0,0,0,1,10,
+    17,1,0,0,64,5,0,0,115,6,0,0,0,0,1,10,
     1,8,1,122,20,70,105,108,101,70,105,110,100,101,114,46,
     95,103,101,116,95,115,112,101,99,78,99,3,0,0,0,0,
     0,0,0,14,0,0,0,8,0,0,0,67,0,0,0,115,
@@ -2317,7 +2317,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,167,0,0,0,90,13,105,110,105,116,95,102,105,108,101,
     110,97,109,101,90,9,102,117,108,108,95,112,97,116,104,114,
     166,0,0,0,114,2,0,0,0,114,2,0,0,0,114,4,
-    0,0,0,114,182,0,0,0,68,5,0,0,115,70,0,0,
+    0,0,0,114,182,0,0,0,69,5,0,0,115,70,0,0,
     0,0,5,4,1,14,1,2,1,24,1,14,1,10,1,10,
     1,8,1,6,2,6,1,6,1,10,2,6,1,4,2,8,
     1,12,1,16,1,8,1,10,1,8,1,24,4,8,2,16,
@@ -2349,7 +2349,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,2,0,0,0,41,1,114,90,0,0,0,41,2,114,22,
     0,0,0,90,2,102,110,114,2,0,0,0,114,2,0,0,
     0,114,4,0,0,0,250,9,60,115,101,116,99,111,109,112,
-    62,145,5,0,0,115,2,0,0,0,6,0,122,41,70,105,
+    62,146,5,0,0,115,2,0,0,0,6,0,122,41,70,105,
     108,101,70,105,110,100,101,114,46,95,102,105,108,108,95,99,
     97,99,104,101,46,60,108,111,99,97,108,115,62,46,60,115,
     101,116,99,111,109,112,62,78,41,18,114,35,0,0,0,114,
@@ -2365,7 +2365,7 @@ const unsigned char _Py_M__importlib_external[] = {
     111,110,116,101,110,116,115,114,1,1,0,0,114,100,0,0,
     0,114,248,0,0,0,114,236,0,0,0,90,8,110,101,119,
     95,110,97,109,101,114,2,0,0,0,114,2,0,0,0,114,
-    4,0,0,0,114,25,1,0,0,116,5,0,0,115,34,0,
+    4,0,0,0,114,25,1,0,0,117,5,0,0,115,34,0,
     0,0,0,2,6,1,2,1,22,1,20,3,10,3,12,1,
     12,7,6,1,10,1,16,1,4,1,18,2,4,1,14,1,
     6,1,12,1,122,22,70,105,108,101,70,105,110,100,101,114,
@@ -2403,7 +2403,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,41,1,114,35,0,0,0,41,2,114,172,0,0,
     0,114,24,1,0,0,114,2,0,0,0,114,4,0,0,0,
     218,24,112,97,116,104,95,104,111,111,107,95,102,111,114,95,
-    70,105,108,101,70,105,110,100,101,114,157,5,0,0,115,6,
+    70,105,108,101,70,105,110,100,101,114,158,5,0,0,115,6,
     0,0,0,0,2,8,1,12,1,122,54,70,105,108,101,70,
     105,110,100,101,114,46,112,97,116,104,95,104,111,111,107,46,
     60,108,111,99,97,108,115,62,46,112,97,116,104,95,104,111,
@@ -2411,7 +2411,7 @@ const unsigned char _Py_M__importlib_external[] = {
     114,114,2,0,0,0,41,3,114,172,0,0,0,114,24,1,
     0,0,114,30,1,0,0,114,2,0,0,0,41,2,114,172,
     0,0,0,114,24,1,0,0,114,4,0,0,0,218,9,112,
-    97,116,104,95,104,111,111,107,147,5,0,0,115,4,0,0,
+    97,116,104,95,104,111,111,107,148,5,0,0,115,4,0,0,
     0,0,10,14,6,122,20,70,105,108,101,70,105,110,100,101,
     114,46,112,97,116,104,95,104,111,111,107,99,1,0,0,0,
     0,0,0,0,1,0,0,0,3,0,0,0,67,0,0,0,
@@ -2419,7 +2419,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,41,2,78,122,16,70,105,108,101,70,105,110,100,101,114,
     40,123,33,114,125,41,41,2,114,48,0,0,0,114,35,0,
     0,0,41,1,114,102,0,0,0,114,2,0,0,0,114,2,
-    0,0,0,114,4,0,0,0,114,0,1,0,0,165,5,0,
+    0,0,0,114,4,0,0,0,114,0,1,0,0,166,5,0,
     0,115,2,0,0,0,0,1,122,19,70,105,108,101,70,105,
     110,100,101,114,46,95,95,114,101,112,114,95,95,41,1,78,
     41,15,114,107,0,0,0,114,106,0,0,0,114,108,0,0,
@@ -2428,7 +2428,7 @@ const unsigned char _Py_M__importlib_external[] = {
     17,1,0,0,114,182,0,0,0,114,25,1,0,0,114,184,
     0,0,0,114,31,1,0,0,114,0,1,0,0,114,2,0,
     0,0,114,2,0,0,0,114,2,0,0,0,114,4,0,0,
-    0,114,18,1,0,0,22,5,0,0,115,18,0,0,0,12,
+    0,114,18,1,0,0,23,5,0,0,115,18,0,0,0,12,
     9,8,14,8,4,4,2,8,12,8,5,10,48,8,31,12,
     18,114,18,1,0,0,99,4,0,0,0,0,0,0,0,6,
     0,0,0,8,0,0,0,67,0,0,0,115,146,0,0,0,
@@ -2451,7 +2451,7 @@ const unsigned char _Py_M__importlib_external[] = {
     97,109,101,90,9,99,112,97,116,104,110,97,109,101,114,122,
     0,0,0,114,166,0,0,0,114,2,0,0,0,114,2,0,
     0,0,114,4,0,0,0,218,14,95,102,105,120,95,117,112,
-    95,109,111,100,117,108,101,171,5,0,0,115,34,0,0,0,
+    95,109,111,100,117,108,101,172,5,0,0,115,34,0,0,0,
     0,2,10,1,10,1,4,1,4,1,8,1,8,1,12,2,
     10,1,4,1,14,1,2,1,8,1,8,1,8,1,12,1,
     14,2,114,36,1,0,0,99,0,0,0,0,0,0,0,0,
@@ -2471,7 +2471,7 @@ const unsigned char _Py_M__importlib_external[] = {
     90,10,101,120,116,101,110,115,105,111,110,115,90,6,115,111,
     117,114,99,101,90,8,98,121,116,101,99,111,100,101,114,2,
     0,0,0,114,2,0,0,0,114,4,0,0,0,114,163,0,
-    0,0,194,5,0,0,115,8,0,0,0,0,5,12,1,8,
+    0,0,195,5,0,0,115,8,0,0,0,0,5,12,1,8,
     1,8,1,114,163,0,0,0,99,1,0,0,0,0,0,0,
     0,12,0,0,0,9,0,0,0,67,0,0,0,115,156,1,
     0,0,124,0,97,0,116,0,106,1,97,1,116,0,106,2,
@@ -2521,7 +2521,7 @@ const unsigned char _Py_M__importlib_external[] = {
     1,100,0,107,2,86,0,1,0,113,2,100,1,83,0,41,
     2,114,29,0,0,0,78,41,1,114,31,0,0,0,41,2,
     114,22,0,0,0,114,79,0,0,0,114,2,0,0,0,114,
-    2,0,0,0,114,4,0,0,0,114,238,0,0,0,230,5,
+    2,0,0,0,114,4,0,0,0,114,238,0,0,0,231,5,
     0,0,115,2,0,0,0,4,0,122,25,95,115,101,116,117,
     112,46,60,108,111,99,97,108,115,62,46,60,103,101,110,101,
     120,112,114,62,114,60,0,0,0,122,30,105,109,112,111,114,
@@ -2549,7 +2549,7 @@ const unsigned char _Py_M__importlib_external[] = {
     119,101,97,107,114,101,102,95,109,111,100,117,108,101,90,13,
     119,105,110,114,101,103,95,109,111,100,117,108,101,114,2,0,
     0,0,114,2,0,0,0,114,4,0,0,0,218,6,95,115,
-    101,116,117,112,205,5,0,0,115,76,0,0,0,0,8,4,
+    101,116,117,112,206,5,0,0,115,76,0,0,0,0,8,4,
     1,6,1,6,3,10,1,10,1,10,1,12,2,10,1,16,
     3,22,1,14,2,22,1,8,1,10,1,10,1,4,2,2,
     1,10,1,6,1,14,1,12,2,8,1,12,1,12,1,18,
@@ -2569,7 +2569,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,41,2,114,42,1,0,0,90,17,115,117,112,112,111,
     114,116,101,100,95,108,111,97,100,101,114,115,114,2,0,0,
     0,114,2,0,0,0,114,4,0,0,0,218,8,95,105,110,
-    115,116,97,108,108,13,6,0,0,115,8,0,0,0,0,2,
+    115,116,97,108,108,14,6,0,0,115,8,0,0,0,0,2,
     8,1,6,1,20,1,114,45,1,0,0,41,1,114,47,0,
     0,0,41,1,78,41,3,78,78,78,41,2,114,60,0,0,
     0,114,60,0,0,0,41,1,84,41,1,78,41,1,78,41,
@@ -2601,7 +2601,7 @@ const unsigned char _Py_M__importlib_external[] = {
     0,0,0,114,2,0,0,0,114,4,0,0,0,218,8,60,
     109,111,100,117,108,101,62,23,0,0,0,115,116,0,0,0,
     4,0,4,1,4,1,2,1,6,3,8,17,8,5,8,5,
-    8,6,8,12,8,10,8,9,8,5,8,7,10,22,10,126,
+    8,6,8,12,8,10,8,9,8,5,8,7,10,22,10,127,
     16,1,12,2,4,1,4,2,6,2,6,2,8,2,16,45,
     8,34,8,19,8,12,8,12,8,28,8,17,8,33,8,28,
     8,24,10,13,10,10,10,11,8,14,6,3,4,1,14,67,
diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h
index 567f8723a678..da6d032bce41 100644
--- a/Python/opcode_targets.h
+++ b/Python/opcode_targets.h
@@ -126,7 +126,7 @@ static void *opcode_targets[256] = {
     &&TARGET_LOAD_FAST,
     &&TARGET_STORE_FAST,
     &&TARGET_DELETE_FAST,
-    &&TARGET_STORE_ANNOTATION,
+    &&_unknown_opcode,
     &&_unknown_opcode,
     &&_unknown_opcode,
     &&TARGET_RAISE_VARARGS,


From webhook-mailer at python.org  Mon Jan 29 20:23:47 2018
From: webhook-mailer at python.org (Eric Snow)
Date: Tue, 30 Jan 2018 01:23:47 -0000
Subject: [Python-checkins] bpo-32604: Expose the subinterpreters C-API in a
 "private" stdlib module. (gh-1748)
Message-ID: <mailman.66.1517275427.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/7f8bfc9b9a8381ddb768421b5dd5cbd970266190
commit: 7f8bfc9b9a8381ddb768421b5dd5cbd970266190
branch: master
author: Eric Snow <ericsnowcurrently at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T18:23:44-07:00
summary:

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. (gh-1748)

The module is primarily intended for internal use in the test suite.  Building the module under Windows will come in a follow-up PR.

files:
A Lib/test/test__xxsubinterpreters.py
A Misc/NEWS.d/next/Tests/2018-01-26-21-29-09.bpo-32604.7iazNx.rst
A Modules/_xxsubinterpretersmodule.c
M Include/internal/pystate.h
M Python/pystate.c
M setup.py

diff --git a/Include/internal/pystate.h b/Include/internal/pystate.h
index 0b464bcb2e86..2b60b25c19b1 100644
--- a/Include/internal/pystate.h
+++ b/Include/internal/pystate.h
@@ -65,6 +65,79 @@ PyAPI_FUNC(_PyInitError) _PyPathConfig_Calculate(
 PyAPI_FUNC(void) _PyPathConfig_Clear(_PyPathConfig *config);
 
 
+/* interpreter state */
+
+PyAPI_FUNC(PyInterpreterState *) _PyInterpreterState_LookUpID(PY_INT64_T);
+
+
+/* cross-interpreter data */
+
+struct _xid;
+
+// _PyCrossInterpreterData is similar to Py_buffer as an effectively
+// opaque struct that holds data outside the object machinery.  This
+// is necessary to pass between interpreters in the same process.
+typedef struct _xid {
+    // data is the cross-interpreter-safe derivation of a Python object
+    // (see _PyObject_GetCrossInterpreterData).  It will be NULL if the
+    // new_object func (below) encodes the data.
+    void *data;
+    // obj is the Python object from which the data was derived.  This
+    // is non-NULL only if the data remains bound to the object in some
+    // way, such that the object must be "released" (via a decref) when
+    // the data is released.  In that case it is automatically
+    // incref'ed (to match the automatic decref when releaed).
+    PyObject *obj;
+    // interp is the ID of the owning interpreter of the original
+    // object.  It corresponds to the active interpreter when
+    // _PyObject_GetCrossInterpreterData() was called.  This should only
+    // be set by the cross-interpreter machinery.
+    //
+    // We use the ID rather than the PyInterpreterState to avoid issues
+    // with deleted interpreters.
+    int64_t interp;
+    // new_object is a function that returns a new object in the current
+    // interpreter given the data.  The resulting object (a new
+    // reference) will be equivalent to the original object.  This field
+    // is required.
+    PyObject *(*new_object)(struct _xid *);
+    // free is called when the data is released.  If it is NULL then
+    // nothing will be done to free the data.  For some types this is
+    // okay (e.g. bytes) and for those types this field should be set
+    // to NULL.  However, for most the data was allocated just for
+    // cross-interpreter use, so it must be freed when
+    // _PyCrossInterpreterData_Release is called or the memory will
+    // leak.  In that case, at the very least this field should be set
+    // to PyMem_RawFree (the default if not explicitly set to NULL).
+    // The call will happen with the original interpreter activated.
+    void (*free)(void *);
+} _PyCrossInterpreterData;
+
+typedef int (*crossinterpdatafunc)(PyObject *, _PyCrossInterpreterData *);
+PyAPI_FUNC(int) _PyObject_CheckCrossInterpreterData(PyObject *);
+
+PyAPI_FUNC(int) _PyObject_GetCrossInterpreterData(PyObject *, _PyCrossInterpreterData *);
+PyAPI_FUNC(PyObject *) _PyCrossInterpreterData_NewObject(_PyCrossInterpreterData *);
+PyAPI_FUNC(void) _PyCrossInterpreterData_Release(_PyCrossInterpreterData *);
+
+/* cross-interpreter data registry */
+
+/* For now we use a global registry of shareable classes.  An
+   alternative would be to add a tp_* slot for a class's
+   crossinterpdatafunc. It would be simpler and more efficient. */
+
+PyAPI_FUNC(int) _PyCrossInterpreterData_Register_Class(PyTypeObject *, crossinterpdatafunc);
+PyAPI_FUNC(crossinterpdatafunc) _PyCrossInterpreterData_Lookup(PyObject *);
+
+struct _xidregitem;
+
+struct _xidregitem {
+    PyTypeObject *cls;
+    crossinterpdatafunc getdata;
+    struct _xidregitem *next;
+};
+
+
 /* Full Python runtime state */
 
 typedef struct pyruntimestate {
@@ -86,6 +159,11 @@ typedef struct pyruntimestate {
            using a Python int. */
         int64_t next_id;
     } interpreters;
+    // XXX Remove this field once we have a tp_* slot.
+    struct _xidregistry {
+        PyThread_type_lock mutex;
+        struct _xidregitem *head;
+    } xidregistry;
 
 #define NEXITFUNCS 32
     void (*exitfuncs[NEXITFUNCS])(void);
diff --git a/Lib/test/test__xxsubinterpreters.py b/Lib/test/test__xxsubinterpreters.py
new file mode 100644
index 000000000000..2b170443a3b6
--- /dev/null
+++ b/Lib/test/test__xxsubinterpreters.py
@@ -0,0 +1,1118 @@
+import contextlib
+import os
+import pickle
+from textwrap import dedent, indent
+import threading
+import unittest
+
+from test import support
+from test.support import script_helper
+
+interpreters = support.import_module('_xxsubinterpreters')
+
+
+def _captured_script(script):
+    r, w = os.pipe()
+    indented = script.replace('\n', '\n                ')
+    wrapped = dedent(f"""
+        import contextlib
+        with open({w}, 'w') as chan:
+            with contextlib.redirect_stdout(chan):
+                {indented}
+        """)
+    return wrapped, open(r)
+
+
+def _run_output(interp, request, shared=None):
+    script, chan = _captured_script(request)
+    with chan:
+        interpreters.run_string(interp, script, shared)
+        return chan.read()
+
+
+ at contextlib.contextmanager
+def _running(interp):
+    r, w = os.pipe()
+    def run():
+        interpreters.run_string(interp, dedent(f"""
+            # wait for "signal"
+            with open({r}) as chan:
+                chan.read()
+            """))
+
+    t = threading.Thread(target=run)
+    t.start()
+
+    yield
+
+    with open(w, 'w') as chan:
+        chan.write('done')
+    t.join()
+
+
+class IsShareableTests(unittest.TestCase):
+
+    def test_default_shareables(self):
+        shareables = [
+                # singletons
+                None,
+                # builtin objects
+                b'spam',
+                ]
+        for obj in shareables:
+            with self.subTest(obj):
+                self.assertTrue(
+                    interpreters.is_shareable(obj))
+
+    def test_not_shareable(self):
+        class Cheese:
+            def __init__(self, name):
+                self.name = name
+            def __str__(self):
+                return self.name
+
+        class SubBytes(bytes):
+            """A subclass of a shareable type."""
+
+        not_shareables = [
+                # singletons
+                True,
+                False,
+                NotImplemented,
+                ...,
+                # builtin types and objects
+                type,
+                object,
+                object(),
+                Exception(),
+                42,
+                100.0,
+                'spam',
+                # user-defined types and objects
+                Cheese,
+                Cheese('Wensleydale'),
+                SubBytes(b'spam'),
+                ]
+        for obj in not_shareables:
+            with self.subTest(obj):
+                self.assertFalse(
+                    interpreters.is_shareable(obj))
+
+
+class TestBase(unittest.TestCase):
+
+    def tearDown(self):
+        for id in interpreters.list_all():
+            if id == 0:  # main
+                continue
+            try:
+                interpreters.destroy(id)
+            except RuntimeError:
+                pass  # already destroyed
+
+        for cid in interpreters.channel_list_all():
+            try:
+                interpreters.channel_destroy(cid)
+            except interpreters.ChannelNotFoundError:
+                pass  # already destroyed
+
+
+class ListAllTests(TestBase):
+
+    def test_initial(self):
+        main = interpreters.get_main()
+        ids = interpreters.list_all()
+        self.assertEqual(ids, [main])
+
+    def test_after_creating(self):
+        main = interpreters.get_main()
+        first = interpreters.create()
+        second = interpreters.create()
+        ids = interpreters.list_all()
+        self.assertEqual(ids, [main, first, second])
+
+    def test_after_destroying(self):
+        main = interpreters.get_main()
+        first = interpreters.create()
+        second = interpreters.create()
+        interpreters.destroy(first)
+        ids = interpreters.list_all()
+        self.assertEqual(ids, [main, second])
+
+
+class GetCurrentTests(TestBase):
+
+    def test_main(self):
+        main = interpreters.get_main()
+        cur = interpreters.get_current()
+        self.assertEqual(cur, main)
+
+    def test_subinterpreter(self):
+        main = interpreters.get_main()
+        interp = interpreters.create()
+        out = _run_output(interp, dedent("""
+            import _xxsubinterpreters as _interpreters
+            print(_interpreters.get_current())
+            """))
+        cur = int(out.strip())
+        _, expected = interpreters.list_all()
+        self.assertEqual(cur, expected)
+        self.assertNotEqual(cur, main)
+
+
+class GetMainTests(TestBase):
+
+    def test_from_main(self):
+        [expected] = interpreters.list_all()
+        main = interpreters.get_main()
+        self.assertEqual(main, expected)
+
+    def test_from_subinterpreter(self):
+        [expected] = interpreters.list_all()
+        interp = interpreters.create()
+        out = _run_output(interp, dedent("""
+            import _xxsubinterpreters as _interpreters
+            print(_interpreters.get_main())
+            """))
+        main = int(out.strip())
+        self.assertEqual(main, expected)
+
+
+class IsRunningTests(TestBase):
+
+    def test_main(self):
+        main = interpreters.get_main()
+        self.assertTrue(interpreters.is_running(main))
+
+    def test_subinterpreter(self):
+        interp = interpreters.create()
+        self.assertFalse(interpreters.is_running(interp))
+
+        with _running(interp):
+            self.assertTrue(interpreters.is_running(interp))
+        self.assertFalse(interpreters.is_running(interp))
+
+    def test_from_subinterpreter(self):
+        interp = interpreters.create()
+        out = _run_output(interp, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            if _interpreters.is_running({interp}):
+                print(True)
+            else:
+                print(False)
+            """))
+        self.assertEqual(out.strip(), 'True')
+
+    def test_already_destroyed(self):
+        interp = interpreters.create()
+        interpreters.destroy(interp)
+        with self.assertRaises(RuntimeError):
+            interpreters.is_running(interp)
+
+    def test_does_not_exist(self):
+        with self.assertRaises(RuntimeError):
+            interpreters.is_running(1_000_000)
+
+    def test_bad_id(self):
+        with self.assertRaises(RuntimeError):
+            interpreters.is_running(-1)
+
+
+class CreateTests(TestBase):
+
+    def test_in_main(self):
+        id = interpreters.create()
+
+        self.assertIn(id, interpreters.list_all())
+
+    @unittest.skip('enable this test when working on pystate.c')
+    def test_unique_id(self):
+        seen = set()
+        for _ in range(100):
+            id = interpreters.create()
+            interpreters.destroy(id)
+            seen.add(id)
+
+        self.assertEqual(len(seen), 100)
+
+    def test_in_thread(self):
+        lock = threading.Lock()
+        id = None
+        def f():
+            nonlocal id
+            id = interpreters.create()
+            lock.acquire()
+            lock.release()
+
+        t = threading.Thread(target=f)
+        with lock:
+            t.start()
+        t.join()
+        self.assertIn(id, interpreters.list_all())
+
+    def test_in_subinterpreter(self):
+        main, = interpreters.list_all()
+        id1 = interpreters.create()
+        out = _run_output(id1, dedent("""
+            import _xxsubinterpreters as _interpreters
+            id = _interpreters.create()
+            print(id)
+            """))
+        id2 = int(out.strip())
+
+        self.assertEqual(set(interpreters.list_all()), {main, id1, id2})
+
+    def test_in_threaded_subinterpreter(self):
+        main, = interpreters.list_all()
+        id1 = interpreters.create()
+        id2 = None
+        def f():
+            nonlocal id2
+            out = _run_output(id1, dedent("""
+                import _xxsubinterpreters as _interpreters
+                id = _interpreters.create()
+                print(id)
+                """))
+            id2 = int(out.strip())
+
+        t = threading.Thread(target=f)
+        t.start()
+        t.join()
+
+        self.assertEqual(set(interpreters.list_all()), {main, id1, id2})
+
+    def test_after_destroy_all(self):
+        before = set(interpreters.list_all())
+        # Create 3 subinterpreters.
+        ids = []
+        for _ in range(3):
+            id = interpreters.create()
+            ids.append(id)
+        # Now destroy them.
+        for id in ids:
+            interpreters.destroy(id)
+        # Finally, create another.
+        id = interpreters.create()
+        self.assertEqual(set(interpreters.list_all()), before | {id})
+
+    def test_after_destroy_some(self):
+        before = set(interpreters.list_all())
+        # Create 3 subinterpreters.
+        id1 = interpreters.create()
+        id2 = interpreters.create()
+        id3 = interpreters.create()
+        # Now destroy 2 of them.
+        interpreters.destroy(id1)
+        interpreters.destroy(id3)
+        # Finally, create another.
+        id = interpreters.create()
+        self.assertEqual(set(interpreters.list_all()), before | {id, id2})
+
+
+class DestroyTests(TestBase):
+
+    def test_one(self):
+        id1 = interpreters.create()
+        id2 = interpreters.create()
+        id3 = interpreters.create()
+        self.assertIn(id2, interpreters.list_all())
+        interpreters.destroy(id2)
+        self.assertNotIn(id2, interpreters.list_all())
+        self.assertIn(id1, interpreters.list_all())
+        self.assertIn(id3, interpreters.list_all())
+
+    def test_all(self):
+        before = set(interpreters.list_all())
+        ids = set()
+        for _ in range(3):
+            id = interpreters.create()
+            ids.add(id)
+        self.assertEqual(set(interpreters.list_all()), before | ids)
+        for id in ids:
+            interpreters.destroy(id)
+        self.assertEqual(set(interpreters.list_all()), before)
+
+    def test_main(self):
+        main, = interpreters.list_all()
+        with self.assertRaises(RuntimeError):
+            interpreters.destroy(main)
+
+        def f():
+            with self.assertRaises(RuntimeError):
+                interpreters.destroy(main)
+
+        t = threading.Thread(target=f)
+        t.start()
+        t.join()
+
+    def test_already_destroyed(self):
+        id = interpreters.create()
+        interpreters.destroy(id)
+        with self.assertRaises(RuntimeError):
+            interpreters.destroy(id)
+
+    def test_does_not_exist(self):
+        with self.assertRaises(RuntimeError):
+            interpreters.destroy(1_000_000)
+
+    def test_bad_id(self):
+        with self.assertRaises(RuntimeError):
+            interpreters.destroy(-1)
+
+    def test_from_current(self):
+        main, = interpreters.list_all()
+        id = interpreters.create()
+        script = dedent("""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.destroy({})
+            """).format(id)
+
+        with self.assertRaises(RuntimeError):
+            interpreters.run_string(id, script)
+        self.assertEqual(set(interpreters.list_all()), {main, id})
+
+    def test_from_sibling(self):
+        main, = interpreters.list_all()
+        id1 = interpreters.create()
+        id2 = interpreters.create()
+        script = dedent("""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.destroy({})
+            """).format(id2)
+        interpreters.run_string(id1, script)
+
+        self.assertEqual(set(interpreters.list_all()), {main, id1})
+
+    def test_from_other_thread(self):
+        id = interpreters.create()
+        def f():
+            interpreters.destroy(id)
+
+        t = threading.Thread(target=f)
+        t.start()
+        t.join()
+
+    def test_still_running(self):
+        main, = interpreters.list_all()
+        interp = interpreters.create()
+        with _running(interp):
+            with self.assertRaises(RuntimeError):
+                interpreters.destroy(interp)
+            self.assertTrue(interpreters.is_running(interp))
+
+
+class RunStringTests(TestBase):
+
+    SCRIPT = dedent("""
+        with open('{}', 'w') as out:
+            out.write('{}')
+        """)
+    FILENAME = 'spam'
+
+    def setUp(self):
+        super().setUp()
+        self.id = interpreters.create()
+        self._fs = None
+
+    def tearDown(self):
+        if self._fs is not None:
+            self._fs.close()
+        super().tearDown()
+
+    @property
+    def fs(self):
+        if self._fs is None:
+            self._fs = FSFixture(self)
+        return self._fs
+
+    def test_success(self):
+        script, file = _captured_script('print("it worked!", end="")')
+        with file:
+            interpreters.run_string(self.id, script)
+            out = file.read()
+
+        self.assertEqual(out, 'it worked!')
+
+    def test_in_thread(self):
+        script, file = _captured_script('print("it worked!", end="")')
+        with file:
+            def f():
+                interpreters.run_string(self.id, script)
+
+            t = threading.Thread(target=f)
+            t.start()
+            t.join()
+            out = file.read()
+
+        self.assertEqual(out, 'it worked!')
+
+    def test_create_thread(self):
+        script, file = _captured_script("""
+            import threading
+            def f():
+                print('it worked!', end='')
+
+            t = threading.Thread(target=f)
+            t.start()
+            t.join()
+            """)
+        with file:
+            interpreters.run_string(self.id, script)
+            out = file.read()
+
+        self.assertEqual(out, 'it worked!')
+
+    @unittest.skipUnless(hasattr(os, 'fork'), "test needs os.fork()")
+    def test_fork(self):
+        import tempfile
+        with tempfile.NamedTemporaryFile('w+') as file:
+            file.write('')
+            file.flush()
+
+            expected = 'spam spam spam spam spam'
+            script = dedent(f"""
+                # (inspired by Lib/test/test_fork.py)
+                import os
+                pid = os.fork()
+                if pid == 0:  # child
+                    with open('{file.name}', 'w') as out:
+                        out.write('{expected}')
+                    # Kill the unittest runner in the child process.
+                    os._exit(1)
+                else:
+                    SHORT_SLEEP = 0.1
+                    import time
+                    for _ in range(10):
+                        spid, status = os.waitpid(pid, os.WNOHANG)
+                        if spid == pid:
+                            break
+                        time.sleep(SHORT_SLEEP)
+                    assert(spid == pid)
+                """)
+            interpreters.run_string(self.id, script)
+
+            file.seek(0)
+            content = file.read()
+            self.assertEqual(content, expected)
+
+    def test_already_running(self):
+        with _running(self.id):
+            with self.assertRaises(RuntimeError):
+                interpreters.run_string(self.id, 'print("spam")')
+
+    def test_does_not_exist(self):
+        id = 0
+        while id in interpreters.list_all():
+            id += 1
+        with self.assertRaises(RuntimeError):
+            interpreters.run_string(id, 'print("spam")')
+
+    def test_error_id(self):
+        with self.assertRaises(RuntimeError):
+            interpreters.run_string(-1, 'print("spam")')
+
+    def test_bad_id(self):
+        with self.assertRaises(TypeError):
+            interpreters.run_string('spam', 'print("spam")')
+
+    def test_bad_script(self):
+        with self.assertRaises(TypeError):
+            interpreters.run_string(self.id, 10)
+
+    def test_bytes_for_script(self):
+        with self.assertRaises(TypeError):
+            interpreters.run_string(self.id, b'print("spam")')
+
+    @contextlib.contextmanager
+    def assert_run_failed(self, exctype, msg=None):
+        with self.assertRaises(interpreters.RunFailedError) as caught:
+            yield
+        if msg is None:
+            self.assertEqual(str(caught.exception).split(':')[0],
+                             str(exctype))
+        else:
+            self.assertEqual(str(caught.exception),
+                             "{}: {}".format(exctype, msg))
+
+    def test_invalid_syntax(self):
+        with self.assert_run_failed(SyntaxError):
+            # missing close paren
+            interpreters.run_string(self.id, 'print("spam"')
+
+    def test_failure(self):
+        with self.assert_run_failed(Exception, 'spam'):
+            interpreters.run_string(self.id, 'raise Exception("spam")')
+
+    def test_SystemExit(self):
+        with self.assert_run_failed(SystemExit, '42'):
+            interpreters.run_string(self.id, 'raise SystemExit(42)')
+
+    def test_sys_exit(self):
+        with self.assert_run_failed(SystemExit):
+            interpreters.run_string(self.id, dedent("""
+                import sys
+                sys.exit()
+                """))
+
+        with self.assert_run_failed(SystemExit, '42'):
+            interpreters.run_string(self.id, dedent("""
+                import sys
+                sys.exit(42)
+                """))
+
+    def test_with_shared(self):
+        r, w = os.pipe()
+
+        shared = {
+                'spam': b'ham',
+                'eggs': b'-1',
+                'cheddar': None,
+                }
+        script = dedent(f"""
+            eggs = int(eggs)
+            spam = 42
+            result = spam + eggs
+
+            ns = dict(vars())
+            del ns['__builtins__']
+            import pickle
+            with open({w}, 'wb') as chan:
+                pickle.dump(ns, chan)
+            """)
+        interpreters.run_string(self.id, script, shared)
+        with open(r, 'rb') as chan:
+            ns = pickle.load(chan)
+
+        self.assertEqual(ns['spam'], 42)
+        self.assertEqual(ns['eggs'], -1)
+        self.assertEqual(ns['result'], 41)
+        self.assertIsNone(ns['cheddar'])
+
+    def test_shared_overwrites(self):
+        interpreters.run_string(self.id, dedent("""
+            spam = 'eggs'
+            ns1 = dict(vars())
+            del ns1['__builtins__']
+            """))
+
+        shared = {'spam': b'ham'}
+        script = dedent(f"""
+            ns2 = dict(vars())
+            del ns2['__builtins__']
+        """)
+        interpreters.run_string(self.id, script, shared)
+
+        r, w = os.pipe()
+        script = dedent(f"""
+            ns = dict(vars())
+            del ns['__builtins__']
+            import pickle
+            with open({w}, 'wb') as chan:
+                pickle.dump(ns, chan)
+            """)
+        interpreters.run_string(self.id, script)
+        with open(r, 'rb') as chan:
+            ns = pickle.load(chan)
+
+        self.assertEqual(ns['ns1']['spam'], 'eggs')
+        self.assertEqual(ns['ns2']['spam'], b'ham')
+        self.assertEqual(ns['spam'], b'ham')
+
+    def test_shared_overwrites_default_vars(self):
+        r, w = os.pipe()
+
+        shared = {'__name__': b'not __main__'}
+        script = dedent(f"""
+            spam = 42
+
+            ns = dict(vars())
+            del ns['__builtins__']
+            import pickle
+            with open({w}, 'wb') as chan:
+                pickle.dump(ns, chan)
+            """)
+        interpreters.run_string(self.id, script, shared)
+        with open(r, 'rb') as chan:
+            ns = pickle.load(chan)
+
+        self.assertEqual(ns['__name__'], b'not __main__')
+
+    def test_main_reused(self):
+        r, w = os.pipe()
+        interpreters.run_string(self.id, dedent(f"""
+            spam = True
+
+            ns = dict(vars())
+            del ns['__builtins__']
+            import pickle
+            with open({w}, 'wb') as chan:
+                pickle.dump(ns, chan)
+            del ns, pickle, chan
+            """))
+        with open(r, 'rb') as chan:
+            ns1 = pickle.load(chan)
+
+        r, w = os.pipe()
+        interpreters.run_string(self.id, dedent(f"""
+            eggs = False
+
+            ns = dict(vars())
+            del ns['__builtins__']
+            import pickle
+            with open({w}, 'wb') as chan:
+                pickle.dump(ns, chan)
+            """))
+        with open(r, 'rb') as chan:
+            ns2 = pickle.load(chan)
+
+        self.assertIn('spam', ns1)
+        self.assertNotIn('eggs', ns1)
+        self.assertIn('eggs', ns2)
+        self.assertIn('spam', ns2)
+
+    def test_execution_namespace_is_main(self):
+        r, w = os.pipe()
+
+        script = dedent(f"""
+            spam = 42
+
+            ns = dict(vars())
+            ns['__builtins__'] = str(ns['__builtins__'])
+            import pickle
+            with open({w}, 'wb') as chan:
+                pickle.dump(ns, chan)
+            """)
+        interpreters.run_string(self.id, script)
+        with open(r, 'rb') as chan:
+            ns = pickle.load(chan)
+
+        ns.pop('__builtins__')
+        ns.pop('__loader__')
+        self.assertEqual(ns, {
+            '__name__': '__main__',
+            '__annotations__': {},
+            '__doc__': None,
+            '__package__': None,
+            '__spec__': None,
+            'spam': 42,
+            })
+
+    def test_still_running_at_exit(self):
+        script = dedent(f"""
+        from textwrap import dedent
+        import threading
+        import _xxsubinterpreters as _interpreters
+        def f():
+            _interpreters.run_string(id, dedent('''
+                import time
+                # Give plenty of time for the main interpreter to finish.
+                time.sleep(1_000_000)
+                '''))
+
+        t = threading.Thread(target=f)
+        t.start()
+        """)
+        with support.temp_dir() as dirname:
+            filename = script_helper.make_script(dirname, 'interp', script)
+            with script_helper.spawn_python(filename) as proc:
+                retcode = proc.wait()
+
+        self.assertEqual(retcode, 0)
+
+
+class ChannelIDTests(TestBase):
+
+    def test_default_kwargs(self):
+        cid = interpreters._channel_id(10, force=True)
+
+        self.assertEqual(int(cid), 10)
+        self.assertEqual(cid.end, 'both')
+
+    def test_with_kwargs(self):
+        cid = interpreters._channel_id(10, send=True, force=True)
+        self.assertEqual(cid.end, 'send')
+
+        cid = interpreters._channel_id(10, send=True, recv=False, force=True)
+        self.assertEqual(cid.end, 'send')
+
+        cid = interpreters._channel_id(10, recv=True, force=True)
+        self.assertEqual(cid.end, 'recv')
+
+        cid = interpreters._channel_id(10, recv=True, send=False, force=True)
+        self.assertEqual(cid.end, 'recv')
+
+        cid = interpreters._channel_id(10, send=True, recv=True, force=True)
+        self.assertEqual(cid.end, 'both')
+
+    def test_coerce_id(self):
+        cid = interpreters._channel_id('10', force=True)
+        self.assertEqual(int(cid), 10)
+
+        cid = interpreters._channel_id(10.0, force=True)
+        self.assertEqual(int(cid), 10)
+
+        class Int(str):
+            def __init__(self, value):
+                self._value = value
+            def __int__(self):
+                return self._value
+
+        cid = interpreters._channel_id(Int(10), force=True)
+        self.assertEqual(int(cid), 10)
+
+    def test_bad_id(self):
+        ids = [-1, 2**64, "spam"]
+        for cid in ids:
+            with self.subTest(cid):
+                with self.assertRaises(ValueError):
+                    interpreters._channel_id(cid)
+
+        with self.assertRaises(TypeError):
+            interpreters._channel_id(object())
+
+    def test_bad_kwargs(self):
+        with self.assertRaises(ValueError):
+            interpreters._channel_id(10, send=False, recv=False)
+
+    def test_does_not_exist(self):
+        cid = interpreters.channel_create()
+        with self.assertRaises(interpreters.ChannelNotFoundError):
+            interpreters._channel_id(int(cid) + 1)  # unforced
+
+    def test_repr(self):
+        cid = interpreters._channel_id(10, force=True)
+        self.assertEqual(repr(cid), 'ChannelID(10)')
+
+        cid = interpreters._channel_id(10, send=True, force=True)
+        self.assertEqual(repr(cid), 'ChannelID(10, send=True)')
+
+        cid = interpreters._channel_id(10, recv=True, force=True)
+        self.assertEqual(repr(cid), 'ChannelID(10, recv=True)')
+
+        cid = interpreters._channel_id(10, send=True, recv=True, force=True)
+        self.assertEqual(repr(cid), 'ChannelID(10)')
+
+    def test_equality(self):
+        cid1 = interpreters.channel_create()
+        cid2 = interpreters._channel_id(int(cid1))
+        cid3 = interpreters.channel_create()
+
+        self.assertTrue(cid1 == cid1)
+        self.assertTrue(cid1 == cid2)
+        self.assertTrue(cid1 == int(cid1))
+        self.assertFalse(cid1 == cid3)
+
+        self.assertFalse(cid1 != cid1)
+        self.assertFalse(cid1 != cid2)
+        self.assertTrue(cid1 != cid3)
+
+
+class ChannelTests(TestBase):
+
+    def test_sequential_ids(self):
+        before = interpreters.channel_list_all()
+        id1 = interpreters.channel_create()
+        id2 = interpreters.channel_create()
+        id3 = interpreters.channel_create()
+        after = interpreters.channel_list_all()
+
+        self.assertEqual(id2, int(id1) + 1)
+        self.assertEqual(id3, int(id2) + 1)
+        self.assertEqual(set(after) - set(before), {id1, id2, id3})
+
+    def test_ids_global(self):
+        id1 = interpreters.create()
+        out = _run_output(id1, dedent("""
+            import _xxsubinterpreters as _interpreters
+            cid = _interpreters.channel_create()
+            print(int(cid))
+            """))
+        cid1 = int(out.strip())
+
+        id2 = interpreters.create()
+        out = _run_output(id2, dedent("""
+            import _xxsubinterpreters as _interpreters
+            cid = _interpreters.channel_create()
+            print(int(cid))
+            """))
+        cid2 = int(out.strip())
+
+        self.assertEqual(cid2, int(cid1) + 1)
+
+    ####################
+
+    def test_drop_single_user(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_drop_interpreter(cid, send=True, recv=True)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'eggs')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_drop_multiple_users(self):
+        cid = interpreters.channel_create()
+        id1 = interpreters.create()
+        id2 = interpreters.create()
+        interpreters.run_string(id1, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.channel_send({int(cid)}, b'spam')
+            """))
+        out = _run_output(id2, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            obj = _interpreters.channel_recv({int(cid)})
+            _interpreters.channel_drop_interpreter({int(cid)})
+            print(repr(obj))
+            """))
+        interpreters.run_string(id1, dedent(f"""
+            _interpreters.channel_drop_interpreter({int(cid)})
+            """))
+
+        self.assertEqual(out.strip(), "b'spam'")
+
+    def test_drop_no_kwargs(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_drop_interpreter(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'eggs')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_drop_multiple_times(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_drop_interpreter(cid, send=True, recv=True)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_drop_interpreter(cid, send=True, recv=True)
+
+    def test_drop_with_unused_items(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_send(cid, b'ham')
+        interpreters.channel_drop_interpreter(cid, send=True, recv=True)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_drop_never_used(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_drop_interpreter(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'spam')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_drop_by_unassociated_interp(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interp = interpreters.create()
+        interpreters.run_string(interp, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.channel_drop_interpreter({int(cid)})
+            """))
+        obj = interpreters.channel_recv(cid)
+        interpreters.channel_drop_interpreter(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'eggs')
+        self.assertEqual(obj, b'spam')
+
+    def test_drop_close_if_unassociated(self):
+        cid = interpreters.channel_create()
+        interp = interpreters.create()
+        interpreters.run_string(interp, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            obj = _interpreters.channel_send({int(cid)}, b'spam')
+            _interpreters.channel_drop_interpreter({int(cid)})
+            """))
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_drop_partially(self):
+        # XXX Is partial close too wierd/confusing?
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, None)
+        interpreters.channel_recv(cid)
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_drop_interpreter(cid, send=True)
+        obj = interpreters.channel_recv(cid)
+
+        self.assertEqual(obj, b'spam')
+
+    def test_drop_used_multiple_times_by_single_user(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_drop_interpreter(cid, send=True, recv=True)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'eggs')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    ####################
+
+    def test_close_single_user(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_close(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'eggs')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_close_multiple_users(self):
+        cid = interpreters.channel_create()
+        id1 = interpreters.create()
+        id2 = interpreters.create()
+        interpreters.run_string(id1, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.channel_send({int(cid)}, b'spam')
+            """))
+        interpreters.run_string(id2, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.channel_recv({int(cid)})
+            """))
+        interpreters.channel_close(cid)
+        with self.assertRaises(interpreters.RunFailedError) as cm:
+            interpreters.run_string(id1, dedent(f"""
+                _interpreters.channel_send({int(cid)}, b'spam')
+                """))
+        self.assertIn('ChannelClosedError', str(cm.exception))
+        with self.assertRaises(interpreters.RunFailedError) as cm:
+            interpreters.run_string(id2, dedent(f"""
+                _interpreters.channel_send({int(cid)}, b'spam')
+                """))
+        self.assertIn('ChannelClosedError', str(cm.exception))
+
+    def test_close_multiple_times(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_close(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_close(cid)
+
+    def test_close_with_unused_items(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_send(cid, b'ham')
+        interpreters.channel_close(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_close_never_used(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_close(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'spam')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    def test_close_by_unassociated_interp(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interp = interpreters.create()
+        interpreters.run_string(interp, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.channel_close({int(cid)})
+            """))
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_close(cid)
+
+    def test_close_used_multiple_times_by_single_user(self):
+        cid = interpreters.channel_create()
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_send(cid, b'spam')
+        interpreters.channel_recv(cid)
+        interpreters.channel_close(cid)
+
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_send(cid, b'eggs')
+        with self.assertRaises(interpreters.ChannelClosedError):
+            interpreters.channel_recv(cid)
+
+    ####################
+
+    def test_send_recv_main(self):
+        cid = interpreters.channel_create()
+        orig = b'spam'
+        interpreters.channel_send(cid, orig)
+        obj = interpreters.channel_recv(cid)
+
+        self.assertEqual(obj, orig)
+        self.assertIsNot(obj, orig)
+
+    def test_send_recv_same_interpreter(self):
+        id1 = interpreters.create()
+        out = _run_output(id1, dedent("""
+            import _xxsubinterpreters as _interpreters
+            cid = _interpreters.channel_create()
+            orig = b'spam'
+            _interpreters.channel_send(cid, orig)
+            obj = _interpreters.channel_recv(cid)
+            assert obj is not orig
+            assert obj == orig
+            """))
+
+    def test_send_recv_different_interpreters(self):
+        cid = interpreters.channel_create()
+        id1 = interpreters.create()
+        out = _run_output(id1, dedent(f"""
+            import _xxsubinterpreters as _interpreters
+            _interpreters.channel_send({int(cid)}, b'spam')
+            """))
+        obj = interpreters.channel_recv(cid)
+
+        self.assertEqual(obj, b'spam')
+
+    def test_send_not_found(self):
+        with self.assertRaises(interpreters.ChannelNotFoundError):
+            interpreters.channel_send(10, b'spam')
+
+    def test_recv_not_found(self):
+        with self.assertRaises(interpreters.ChannelNotFoundError):
+            interpreters.channel_recv(10)
+
+    def test_recv_empty(self):
+        cid = interpreters.channel_create()
+        with self.assertRaises(interpreters.ChannelEmptyError):
+            interpreters.channel_recv(cid)
+
+    def test_run_string_arg(self):
+        cid = interpreters.channel_create()
+        interp = interpreters.create()
+
+        out = _run_output(interp, dedent("""
+            import _xxsubinterpreters as _interpreters
+            print(cid.end)
+            _interpreters.channel_send(cid, b'spam')
+            """),
+            dict(cid=cid.send))
+        obj = interpreters.channel_recv(cid)
+
+        self.assertEqual(obj, b'spam')
+        self.assertEqual(out.strip(), 'send')
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/Misc/NEWS.d/next/Tests/2018-01-26-21-29-09.bpo-32604.7iazNx.rst b/Misc/NEWS.d/next/Tests/2018-01-26-21-29-09.bpo-32604.7iazNx.rst
new file mode 100644
index 000000000000..f5472f9fe23f
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-01-26-21-29-09.bpo-32604.7iazNx.rst
@@ -0,0 +1,4 @@
+Add a new "_xxsubinterpreters" extension module that exposes the existing
+subinterpreter C-API and a new cross-interpreter data sharing mechanism. The
+module is primarily intended for more thorough testing of the existing
+subinterpreter support.
diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c
new file mode 100644
index 000000000000..ed79a13d4a40
--- /dev/null
+++ b/Modules/_xxsubinterpretersmodule.c
@@ -0,0 +1,2061 @@
+
+/* interpreters module */
+/* low-level access to interpreter primitives */
+
+#include "Python.h"
+#include "frameobject.h"
+#include "internal/pystate.h"
+
+
+static PyInterpreterState *
+_get_current(void)
+{
+    PyThreadState *tstate = PyThreadState_Get();
+    // PyThreadState_Get() aborts if lookup fails, so we don't need
+    // to check the result for NULL.
+    return tstate->interp;
+}
+
+static int64_t
+_coerce_id(PyObject *id)
+{
+    id = PyNumber_Long(id);
+    if (id == NULL) {
+        if (PyErr_ExceptionMatches(PyExc_TypeError)) {
+            PyErr_SetString(PyExc_TypeError,
+                            "'id' must be a non-negative int");
+        }
+        else {
+            PyErr_SetString(PyExc_ValueError,
+                            "'id' must be a non-negative int");
+        }
+        return -1;
+    }
+    long long cid = PyLong_AsLongLong(id);
+    if (cid == -1 && PyErr_Occurred() != NULL) {
+        PyErr_SetString(PyExc_ValueError,
+                        "'id' must be a non-negative int");
+        return -1;
+    }
+    if (cid < 0) {
+        PyErr_SetString(PyExc_ValueError,
+                        "'id' must be a non-negative int");
+        return -1;
+    }
+    if (cid > INT64_MAX) {
+        PyErr_SetString(PyExc_ValueError,
+                        "'id' too large (must be 64-bit int)");
+        return -1;
+    }
+    return cid;
+}
+
+/* data-sharing-specific code ***********************************************/
+
+typedef struct _shareditem {
+    Py_UNICODE *name;
+    Py_ssize_t namelen;
+    _PyCrossInterpreterData data;
+} _shareditem;
+
+void
+_sharedns_clear(_shareditem *shared)
+{
+    for (_shareditem *item=shared; item->name != NULL; item += 1) {
+        _PyCrossInterpreterData_Release(&item->data);
+    }
+}
+
+static _shareditem *
+_get_shared_ns(PyObject *shareable, Py_ssize_t *lenp)
+{
+    if (shareable == NULL || shareable == Py_None) {
+        *lenp = 0;
+        return NULL;
+    }
+    Py_ssize_t len = PyDict_Size(shareable);
+    *lenp = len;
+    if (len == 0) {
+        return NULL;
+    }
+
+    _shareditem *shared = PyMem_NEW(_shareditem, len+1);
+    if (shared == NULL) {
+        return NULL;
+    }
+    for (Py_ssize_t i=0; i < len; i++) {
+        *(shared + i) = (_shareditem){0};
+    }
+    Py_ssize_t pos = 0;
+    for (Py_ssize_t i=0; i < len; i++) {
+        PyObject *key, *value;
+        if (PyDict_Next(shareable, &pos, &key, &value) == 0) {
+            break;
+        }
+        _shareditem *item = shared + i;
+
+        if (_PyObject_GetCrossInterpreterData(value, &item->data) != 0) {
+            break;
+        }
+        item->name = PyUnicode_AsUnicodeAndSize(key, &item->namelen);
+        if (item->name == NULL) {
+            _PyCrossInterpreterData_Release(&item->data);
+            break;
+        }
+        (item + 1)->name = NULL;  // Mark the next one as the last.
+    }
+    if (PyErr_Occurred()) {
+        _sharedns_clear(shared);
+        PyMem_Free(shared);
+        return NULL;
+    }
+    return shared;
+}
+
+static int
+_shareditem_apply(_shareditem *item, PyObject *ns)
+{
+    PyObject *name = PyUnicode_FromUnicode(item->name, item->namelen);
+    if (name == NULL) {
+        return 1;
+    }
+    PyObject *value = _PyCrossInterpreterData_NewObject(&item->data);
+    if (value == NULL) {
+        Py_DECREF(name);
+        return 1;
+    }
+    int res = PyDict_SetItem(ns, name, value);
+    Py_DECREF(name);
+    Py_DECREF(value);
+    return res;
+}
+
+// Ultimately we'd like to preserve enough information about the
+// exception and traceback that we could re-constitute (or at least
+// simulate, a la traceback.TracebackException), and even chain, a copy
+// of the exception in the calling interpreter.
+
+typedef struct _sharedexception {
+    char *msg;
+} _sharedexception;
+
+static _sharedexception *
+_get_shared_exception(void)
+{
+    _sharedexception *err = PyMem_NEW(_sharedexception, 1);
+    if (err == NULL) {
+        return NULL;
+    }
+    PyObject *exc;
+    PyObject *value;
+    PyObject *tb;
+    PyErr_Fetch(&exc, &value, &tb);
+    PyObject *msg;
+    if (value == NULL) {
+        msg = PyUnicode_FromFormat("%S", exc);
+    }
+    else {
+        msg = PyUnicode_FromFormat("%S: %S", exc, value);
+    }
+    if (msg == NULL) {
+        err->msg = "unable to format exception";
+        return err;
+    }
+    err->msg = (char *)PyUnicode_AsUTF8(msg);
+    if (err->msg == NULL) {
+        err->msg = "unable to encode exception";
+    }
+    return err;
+}
+
+static PyObject * RunFailedError;
+
+static int
+interp_exceptions_init(PyObject *ns)
+{
+    // XXX Move the exceptions into per-module memory?
+
+    // An uncaught exception came out of interp_run_string().
+    RunFailedError = PyErr_NewException("_xxsubinterpreters.RunFailedError",
+                                        PyExc_RuntimeError, NULL);
+    if (RunFailedError == NULL) {
+        return -1;
+    }
+    if (PyDict_SetItemString(ns, "RunFailedError", RunFailedError) != 0) {
+        return -1;
+    }
+
+    return 0;
+}
+
+static void
+_apply_shared_exception(_sharedexception *exc)
+{
+    PyErr_SetString(RunFailedError, exc->msg);
+}
+
+/* channel-specific code */
+
+static PyObject *ChannelError;
+static PyObject *ChannelNotFoundError;
+static PyObject *ChannelClosedError;
+static PyObject *ChannelEmptyError;
+
+static int
+channel_exceptions_init(PyObject *ns)
+{
+    // XXX Move the exceptions into per-module memory?
+
+    // A channel-related operation failed.
+    ChannelError = PyErr_NewException("_xxsubinterpreters.ChannelError",
+                                      PyExc_RuntimeError, NULL);
+    if (ChannelError == NULL) {
+        return -1;
+    }
+    if (PyDict_SetItemString(ns, "ChannelError", ChannelError) != 0) {
+        return -1;
+    }
+
+    // An operation tried to use a channel that doesn't exist.
+    ChannelNotFoundError = PyErr_NewException(
+            "_xxsubinterpreters.ChannelNotFoundError", ChannelError, NULL);
+    if (ChannelNotFoundError == NULL) {
+        return -1;
+    }
+    if (PyDict_SetItemString(ns, "ChannelNotFoundError", ChannelNotFoundError) != 0) {
+        return -1;
+    }
+
+    // An operation tried to use a closed channel.
+    ChannelClosedError = PyErr_NewException(
+            "_xxsubinterpreters.ChannelClosedError", ChannelError, NULL);
+    if (ChannelClosedError == NULL) {
+        return -1;
+    }
+    if (PyDict_SetItemString(ns, "ChannelClosedError", ChannelClosedError) != 0) {
+        return -1;
+    }
+
+    // An operation tried to pop from an empty channel.
+    ChannelEmptyError = PyErr_NewException(
+            "_xxsubinterpreters.ChannelEmptyError", ChannelError, NULL);
+    if (ChannelEmptyError == NULL) {
+        return -1;
+    }
+    if (PyDict_SetItemString(ns, "ChannelEmptyError", ChannelEmptyError) != 0) {
+        return -1;
+    }
+
+    return 0;
+}
+
+struct _channelend;
+
+typedef struct _channelend {
+    struct _channelend *next;
+    int64_t interp;
+    int open;
+} _channelend;
+
+static _channelend *
+_channelend_new(int64_t interp)
+{
+    _channelend *end = PyMem_NEW(_channelend, 1);
+    if (end == NULL) {
+        return NULL;
+    }
+
+    end->next = NULL;
+    end->interp = interp;
+
+    end->open = 1;
+
+    return end;
+}
+
+static void
+_channelend_free_all(_channelend *end) {
+    while (end != NULL) {
+        _channelend *last = end;
+        end = end->next;
+        PyMem_Free(last);
+    }
+}
+
+static _channelend *
+_channelend_find(_channelend *first, int64_t interp, _channelend **pprev)
+{
+    _channelend *prev = NULL;
+    _channelend *end = first;
+    while (end != NULL) {
+        if (end->interp == interp) {
+            break;
+        }
+        prev = end;
+        end = end->next;
+    }
+    if (pprev != NULL) {
+        *pprev = prev;
+    }
+    return end;
+}
+
+struct _channelitem;
+
+typedef struct _channelitem {
+    _PyCrossInterpreterData *data;
+    struct _channelitem *next;
+} _channelitem;
+
+struct _channel;
+
+typedef struct _channel {
+    PyThread_type_lock mutex;
+
+    int open;
+
+    int64_t count;
+    _channelitem *first;
+    _channelitem *last;
+
+    // Note that the list entries are never removed for interpreter
+    // for which the channel is closed.  This should be a problem in
+    // practice.  Also, a channel isn't automatically closed when an
+    // interpreter is destroyed.
+    int64_t numsendopen;
+    int64_t numrecvopen;
+    _channelend *send;
+    _channelend *recv;
+} _PyChannelState;
+
+static _PyChannelState *
+_channel_new(void)
+{
+    _PyChannelState *chan = PyMem_NEW(_PyChannelState, 1);
+    if (chan == NULL) {
+        return NULL;
+    }
+    chan->mutex = PyThread_allocate_lock();
+    if (chan->mutex == NULL) {
+        PyMem_Free(chan);
+        PyErr_SetString(ChannelError,
+                        "can't initialize mutex for new channel");
+        return NULL;
+    }
+
+    chan->open = 1;
+
+    chan->count = 0;
+    chan->first = NULL;
+    chan->last = NULL;
+
+    chan->numsendopen = 0;
+    chan->numrecvopen = 0;
+    chan->send = NULL;
+    chan->recv = NULL;
+
+    return chan;
+}
+
+static _channelend *
+_channel_add_end(_PyChannelState *chan, _channelend *prev, int64_t interp,
+                 int send)
+{
+    _channelend *end = _channelend_new(interp);
+    if (end == NULL) {
+        return NULL;
+    }
+
+    if (prev == NULL) {
+        if (send) {
+            chan->send = end;
+        }
+        else {
+            chan->recv = end;
+        }
+    }
+    else {
+        prev->next = end;
+    }
+    if (send) {
+        chan->numsendopen += 1;
+    }
+    else {
+        chan->numrecvopen += 1;
+    }
+    return end;
+}
+
+static _channelend *
+_channel_associate_end(_PyChannelState *chan, int64_t interp, int send)
+{
+    if (!chan->open) {
+        PyErr_SetString(ChannelClosedError, "channel closed");
+        return NULL;
+    }
+
+    _channelend *prev;
+    _channelend *end = _channelend_find(send ? chan->send : chan->recv,
+                                        interp, &prev);
+    if (end != NULL) {
+        if (!end->open) {
+            PyErr_SetString(ChannelClosedError, "channel already closed");
+            return NULL;
+        }
+        // already associated
+        return end;
+    }
+    return _channel_add_end(chan, prev, interp, send);
+}
+
+static void
+_channel_close_channelend(_PyChannelState *chan, _channelend *end, int send)
+{
+    end->open = 0;
+    if (send) {
+        chan->numsendopen -= 1;
+    }
+    else {
+        chan->numrecvopen -= 1;
+    }
+}
+
+static int
+_channel_close_interpreter(_PyChannelState *chan, int64_t interp, int which)
+{
+    PyThread_acquire_lock(chan->mutex, WAIT_LOCK);
+
+    int res = -1;
+    if (!chan->open) {
+        PyErr_SetString(ChannelClosedError, "channel already closed");
+        goto done;
+    }
+
+    _channelend *prev;
+    _channelend *end;
+    if (which >= 0) {  // send/both
+        end = _channelend_find(chan->send, interp, &prev);
+        if (end == NULL) {
+            // never associated so add it
+            end = _channel_add_end(chan, prev, interp, 1);
+            if (end == NULL) {
+                goto done;
+            }
+        }
+        _channel_close_channelend(chan, end, 1);
+    }
+    if (which <= 0) {  // recv/both
+        end = _channelend_find(chan->recv, interp, &prev);
+        if (end == NULL) {
+            // never associated so add it
+            end = _channel_add_end(chan, prev, interp, 0);
+            if (end == NULL) {
+                goto done;
+            }
+        }
+        _channel_close_channelend(chan, end, 0);
+    }
+
+    if (chan->numsendopen == 0 && chan->numrecvopen == 0) {
+        if (chan->send != NULL || chan->recv != NULL) {
+            chan->open = 0;
+        }
+    }
+
+    res = 0;
+done:
+    PyThread_release_lock(chan->mutex);
+    return res;
+}
+
+static int
+_channel_close_all(_PyChannelState *chan)
+{
+    int res = -1;
+    PyThread_acquire_lock(chan->mutex, WAIT_LOCK);
+
+    if (!chan->open) {
+        PyErr_SetString(ChannelClosedError, "channel already closed");
+        goto done;
+    }
+
+    chan->open = 0;
+
+    // We *could* also just leave these in place, since we've marked
+    // the channel as closed already.
+
+    // Ensure all the "send"-associated interpreters are closed.
+    _channelend *end;
+    for (end = chan->send; end != NULL; end = end->next) {
+        _channel_close_channelend(chan, end, 1);
+    }
+
+    // Ensure all the "recv"-associated interpreters are closed.
+    for (end = chan->recv; end != NULL; end = end->next) {
+        _channel_close_channelend(chan, end, 0);
+    }
+
+    res = 0;
+done:
+    PyThread_release_lock(chan->mutex);
+    return res;
+}
+
+static int
+_channel_add(_PyChannelState *chan, int64_t interp,
+             _PyCrossInterpreterData *data)
+{
+    int res = -1;
+
+    PyThread_acquire_lock(chan->mutex, WAIT_LOCK);
+    if (_channel_associate_end(chan, interp, 1) == NULL) {
+        goto done;
+    }
+
+    _channelitem *item = PyMem_NEW(_channelitem, 1);
+    if (item == NULL) {
+        goto done;
+    }
+    item->data = data;
+    item->next = NULL;
+
+    chan->count += 1;
+    if (chan->first == NULL) {
+        chan->first = item;
+    }
+    chan->last = item;
+
+    res = 0;
+done:
+    PyThread_release_lock(chan->mutex);
+    return res;
+}
+
+static _PyCrossInterpreterData *
+_channel_next(_PyChannelState *chan, int64_t interp)
+{
+    _PyCrossInterpreterData *data = NULL;
+    PyThread_acquire_lock(chan->mutex, WAIT_LOCK);
+    if (_channel_associate_end(chan, interp, 0) == NULL) {
+        goto done;
+    }
+
+    _channelitem *item = chan->first;
+    if (item == NULL) {
+        goto done;
+    }
+    chan->first = item->next;
+    if (chan->last == item) {
+        chan->last = NULL;
+    }
+    chan->count -= 1;
+
+    data = item->data;
+    PyMem_Free(item);
+
+done:
+    PyThread_release_lock(chan->mutex);
+    return data;
+}
+
+static void
+_channel_clear(_PyChannelState *chan)
+{
+    _channelitem *item = chan->first;
+    while (item != NULL) {
+        _PyCrossInterpreterData_Release(item->data);
+        PyMem_Free(item->data);
+        _channelitem *last = item;
+        item = item->next;
+        PyMem_Free(last);
+    }
+    chan->first = NULL;
+    chan->last = NULL;
+}
+
+static void
+_channel_free(_PyChannelState *chan)
+{
+    PyThread_acquire_lock(chan->mutex, WAIT_LOCK);
+    _channel_clear(chan);
+    _channelend_free_all(chan->send);
+    _channelend_free_all(chan->recv);
+    PyThread_release_lock(chan->mutex);
+
+    PyThread_free_lock(chan->mutex);
+    PyMem_Free(chan);
+}
+
+struct _channelref;
+
+typedef struct _channelref {
+    int64_t id;
+    _PyChannelState *chan;
+    struct _channelref *next;
+    Py_ssize_t objcount;
+} _channelref;
+
+static _channelref *
+_channelref_new(int64_t id, _PyChannelState *chan)
+{
+    _channelref *ref = PyMem_NEW(_channelref, 1);
+    if (ref == NULL) {
+        return NULL;
+    }
+    ref->id = id;
+    ref->chan = chan;
+    ref->next = NULL;
+    ref->objcount = 0;
+    return ref;
+}
+
+static _channelref *
+_channelref_find(_channelref *first, int64_t id, _channelref **pprev)
+{
+    _channelref *prev = NULL;
+    _channelref *ref = first;
+    while (ref != NULL) {
+        if (ref->id == id) {
+            break;
+        }
+        prev = ref;
+        ref = ref->next;
+    }
+    if (pprev != NULL) {
+        *pprev = prev;
+    }
+    return ref;
+}
+
+typedef struct _channels {
+    PyThread_type_lock mutex;
+    _channelref *head;
+    int64_t numopen;
+    int64_t next_id;
+} _channels;
+
+static int
+_channels_init(_channels *channels)
+{
+    if (channels->mutex == NULL) {
+        channels->mutex = PyThread_allocate_lock();
+        if (channels->mutex == NULL) {
+            PyMem_Free(channels);
+            PyErr_SetString(ChannelError,
+                            "can't initialize mutex for channel management");
+            return -1;
+        }
+    }
+    channels->head = NULL;
+    channels->numopen = 0;
+    channels->next_id = 0;
+    return 0;
+}
+
+static int64_t
+_channels_next_id(_channels *channels)  // needs lock
+{
+    int64_t id = channels->next_id;
+    if (id < 0) {
+        /* overflow */
+        PyErr_SetString(ChannelError,
+                        "failed to get a channel ID");
+        return -1;
+    }
+    channels->next_id += 1;
+    return id;
+}
+
+static _PyChannelState *
+_channels_lookup(_channels *channels, int64_t id, PyThread_type_lock *pmutex)
+{
+    _PyChannelState *chan = NULL;
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+    if (pmutex != NULL) {
+        *pmutex = NULL;
+    }
+
+    _channelref *ref = _channelref_find(channels->head, id, NULL);
+    if (ref == NULL) {
+        PyErr_Format(ChannelNotFoundError, "channel %d not found", id);
+        goto done;
+    }
+    if (ref->chan == NULL || !ref->chan->open) {
+        PyErr_Format(ChannelClosedError, "channel %d closed", id);
+        goto done;
+    }
+
+    if (pmutex != NULL) {
+        // The mutex will be closed by the caller.
+        *pmutex = channels->mutex;
+    }
+
+    chan = ref->chan;
+done:
+    if (pmutex == NULL || *pmutex == NULL) {
+        PyThread_release_lock(channels->mutex);
+    }
+    return chan;
+}
+
+static int64_t
+_channels_add(_channels *channels, _PyChannelState *chan)
+{
+    int64_t cid = -1;
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+
+    // Create a new ref.
+    int64_t id = _channels_next_id(channels);
+    if (id < 0) {
+        goto done;
+    }
+    _channelref *ref = _channelref_new(id, chan);
+    if (ref == NULL) {
+        goto done;
+    }
+
+    // Add it to the list.
+    // We assume that the channel is a new one (not already in the list).
+    ref->next = channels->head;
+    channels->head = ref;
+    channels->numopen += 1;
+
+    cid = id;
+done:
+    PyThread_release_lock(channels->mutex);
+    return cid;
+}
+
+static int
+_channels_close(_channels *channels, int64_t cid, _PyChannelState **pchan)
+{
+    int res = -1;
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+    if (pchan != NULL) {
+        *pchan = NULL;
+    }
+
+    _channelref *ref = _channelref_find(channels->head, cid, NULL);
+    if (ref == NULL) {
+        PyErr_Format(ChannelNotFoundError, "channel %d not found", cid);
+        goto done;
+    }
+
+    if (ref->chan == NULL) {
+        PyErr_Format(ChannelClosedError, "channel %d closed", cid);
+        goto done;
+    }
+    else {
+        if (_channel_close_all(ref->chan) != 0) {
+            goto done;
+        }
+        if (pchan != NULL) {
+            *pchan = ref->chan;
+        }
+        ref->chan = NULL;
+    }
+
+    res = 0;
+done:
+    PyThread_release_lock(channels->mutex);
+    return res;
+}
+
+static void
+_channels_remove_ref(_channels *channels, _channelref *ref, _channelref *prev,
+                     _PyChannelState **pchan)
+{
+    if (ref == channels->head) {
+        channels->head = ref->next;
+    }
+    else {
+        prev->next = ref->next;
+    }
+    channels->numopen -= 1;
+
+    if (pchan != NULL) {
+        *pchan = ref->chan;
+    }
+    PyMem_Free(ref);
+}
+
+static int
+_channels_remove(_channels *channels, int64_t id, _PyChannelState **pchan)
+{
+    int res = -1;
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+
+    if (pchan != NULL) {
+        *pchan = NULL;
+    }
+
+    _channelref *prev = NULL;
+    _channelref *ref = _channelref_find(channels->head, id, &prev);
+    if (ref == NULL) {
+        PyErr_Format(ChannelNotFoundError, "channel %d not found", id);
+        goto done;
+    }
+
+    _channels_remove_ref(channels, ref, prev, pchan);
+
+    res = 0;
+done:
+    PyThread_release_lock(channels->mutex);
+    return res;
+}
+
+static int
+_channels_add_id_object(_channels *channels, int64_t id)
+{
+    int res = -1;
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+
+    _channelref *ref = _channelref_find(channels->head, id, NULL);
+    if (ref == NULL) {
+        PyErr_Format(ChannelNotFoundError, "channel %d not found", id);
+        goto done;
+    }
+    ref->objcount += 1;
+
+    res = 0;
+done:
+    PyThread_release_lock(channels->mutex);
+    return res;
+}
+
+static void
+_channels_drop_id_object(_channels *channels, int64_t id)
+{
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+
+    _channelref *prev = NULL;
+    _channelref *ref = _channelref_find(channels->head, id, &prev);
+    if (ref == NULL) {
+        // Already destroyed.
+        goto done;
+    }
+    ref->objcount -= 1;
+
+    // Destroy if no longer used.
+    if (ref->objcount == 0) {
+        _PyChannelState *chan = NULL;
+        _channels_remove_ref(channels, ref, prev, &chan);
+        if (chan != NULL) {
+            _channel_free(chan);
+        }
+    }
+
+done:
+    PyThread_release_lock(channels->mutex);
+}
+
+int64_t *
+_channels_list_all(_channels *channels, int64_t *count)
+{
+    int64_t *cids = NULL;
+    PyThread_acquire_lock(channels->mutex, WAIT_LOCK);
+    int64_t numopen = channels->numopen;
+    if (numopen >= PY_SSIZE_T_MAX) {
+        PyErr_SetString(PyExc_RuntimeError, "too many channels open");
+        goto done;
+    }
+    int64_t *ids = PyMem_NEW(int64_t, (Py_ssize_t)(channels->numopen));
+    if (ids == NULL) {
+        goto done;
+    }
+    _channelref *ref = channels->head;
+    for (int64_t i=0; ref != NULL; ref = ref->next, i++) {
+        ids[i] = ref->id;
+    }
+    *count = channels->numopen;
+
+    cids = ids;
+done:
+    PyThread_release_lock(channels->mutex);
+    return cids;
+}
+
+/* "high"-level channel-related functions */
+
+static int64_t
+_channel_create(_channels *channels)
+{
+    _PyChannelState *chan = _channel_new();
+    if (chan == NULL) {
+        return -1;
+    }
+    int64_t id = _channels_add(channels, chan);
+    if (id < 0) {
+        _channel_free(chan);
+        return -1;
+    }
+    return id;
+}
+
+static int
+_channel_destroy(_channels *channels, int64_t id)
+{
+    _PyChannelState *chan = NULL;
+    if (_channels_remove(channels, id, &chan) != 0) {
+        return -1;
+    }
+    if (chan != NULL) {
+        _channel_free(chan);
+    }
+    return 0;
+}
+
+static int
+_channel_send(_channels *channels, int64_t id, PyObject *obj)
+{
+    PyInterpreterState *interp = _get_current();
+    if (interp == NULL) {
+        return -1;
+    }
+
+    // Look up the channel.
+    PyThread_type_lock mutex = NULL;
+    _PyChannelState *chan = _channels_lookup(channels, id, &mutex);
+    if (chan == NULL) {
+        return -1;
+    }
+    // Past this point we are responsible for releasing the mutex.
+
+    // Convert the object to cross-interpreter data.
+    _PyCrossInterpreterData *data = PyMem_NEW(_PyCrossInterpreterData, 1);
+    if (data == NULL) {
+        PyThread_release_lock(mutex);
+        return -1;
+    }
+    if (_PyObject_GetCrossInterpreterData(obj, data) != 0) {
+        PyThread_release_lock(mutex);
+        return -1;
+    }
+
+    // Add the data to the channel.
+    int res = _channel_add(chan, interp->id, data);
+    PyThread_release_lock(mutex);
+    if (res != 0) {
+        _PyCrossInterpreterData_Release(data);
+        PyMem_Free(data);
+        return -1;
+    }
+
+    return 0;
+}
+
+static PyObject *
+_channel_recv(_channels *channels, int64_t id)
+{
+    PyInterpreterState *interp = _get_current();
+    if (interp == NULL) {
+        return NULL;
+    }
+
+    // Look up the channel.
+    PyThread_type_lock mutex = NULL;
+    _PyChannelState *chan = _channels_lookup(channels, id, &mutex);
+    if (chan == NULL) {
+        return NULL;
+    }
+    // Past this point we are responsible for releasing the mutex.
+
+    // Pop off the next item from the channel.
+    _PyCrossInterpreterData *data = _channel_next(chan, interp->id);
+    PyThread_release_lock(mutex);
+    if (data == NULL) {
+        PyErr_Format(ChannelEmptyError, "channel %d is empty", id);
+        return NULL;
+    }
+
+    // Convert the data back to an object.
+    PyObject *obj = _PyCrossInterpreterData_NewObject(data);
+    if (obj == NULL) {
+        return NULL;
+    }
+    _PyCrossInterpreterData_Release(data);
+
+    return obj;
+}
+
+static int
+_channel_drop(_channels *channels, int64_t id, int send, int recv)
+{
+    PyInterpreterState *interp = _get_current();
+    if (interp == NULL) {
+        return -1;
+    }
+
+    // Look up the channel.
+    PyThread_type_lock mutex = NULL;
+    _PyChannelState *chan = _channels_lookup(channels, id, &mutex);
+    if (chan == NULL) {
+        return -1;
+    }
+    // Past this point we are responsible for releasing the mutex.
+
+    // Close one or both of the two ends.
+    int res =_channel_close_interpreter(chan, interp->id, send-recv);
+    PyThread_release_lock(mutex);
+    return res;
+}
+
+static int
+_channel_close(_channels *channels, int64_t id)
+{
+    return _channels_close(channels, id, NULL);
+}
+
+/* ChannelID class */
+
+#define CHANNEL_SEND 1
+#define CHANNEL_RECV -1
+
+static PyTypeObject ChannelIDtype;
+
+typedef struct channelid {
+    PyObject_HEAD
+    int64_t id;
+    int end;
+    _channels *channels;
+} channelid;
+
+static channelid *
+newchannelid(PyTypeObject *cls, int64_t cid, int end, _channels *channels,
+             int force)
+{
+    channelid *self = PyObject_New(channelid, cls);
+    if (self == NULL) {
+        return NULL;
+    }
+    self->id = cid;
+    self->end = end;
+    self->channels = channels;
+
+    if (_channels_add_id_object(channels, cid) != 0) {
+        if (force && PyErr_ExceptionMatches(ChannelNotFoundError)) {
+            PyErr_Clear();
+        }
+        else {
+            Py_DECREF((PyObject *)self);
+            return NULL;
+        }
+    }
+
+    return self;
+}
+
+static _channels * _global_channels(void);
+
+static PyObject *
+channelid_new(PyTypeObject *cls, PyObject *args, PyObject *kwds)
+{
+    static char *kwlist[] = {"id", "send", "recv", "force", NULL};
+    PyObject *id;
+    int send = -1;
+    int recv = -1;
+    int force = 0;
+    if (!PyArg_ParseTupleAndKeywords(args, kwds,
+                                     "O|$ppp:ChannelID.__init__", kwlist,
+                                     &id, &send, &recv, &force))
+        return NULL;
+
+    // Coerce and check the ID.
+    int64_t cid;
+    if (PyObject_TypeCheck(id, &ChannelIDtype)) {
+        cid = ((channelid *)id)->id;
+    }
+    else {
+        cid = _coerce_id(id);
+        if (cid < 0) {
+            return NULL;
+        }
+    }
+
+    // Handle "send" and "recv".
+    if (send == 0 && recv == 0) {
+        PyErr_SetString(PyExc_ValueError,
+                        "'send' and 'recv' cannot both be False");
+        return NULL;
+    }
+    int end = 0;
+    if (send == 1) {
+        if (recv == 0 || recv == -1) {
+            end = CHANNEL_SEND;
+        }
+    }
+    else if (recv == 1) {
+        end = CHANNEL_RECV;
+    }
+
+    return (PyObject *)newchannelid(cls, cid, end, _global_channels(), force);
+}
+
+static void
+channelid_dealloc(PyObject *v)
+{
+    int64_t cid = ((channelid *)v)->id;
+    _channels *channels = ((channelid *)v)->channels;
+    Py_TYPE(v)->tp_free(v);
+
+    _channels_drop_id_object(channels, cid);
+}
+
+static PyObject *
+channelid_repr(PyObject *self)
+{
+    PyTypeObject *type = Py_TYPE(self);
+    const char *name = _PyType_Name(type);
+
+    channelid *cid = (channelid *)self;
+    const char *fmt;
+    if (cid->end == CHANNEL_SEND) {
+        fmt = "%s(%d, send=True)";
+    }
+    else if (cid->end == CHANNEL_RECV) {
+        fmt = "%s(%d, recv=True)";
+    }
+    else {
+        fmt = "%s(%d)";
+    }
+    return PyUnicode_FromFormat(fmt, name, cid->id);
+}
+
+PyObject *
+channelid_int(PyObject *self)
+{
+    channelid *cid = (channelid *)self;
+    return PyLong_FromLongLong(cid->id);
+}
+
+static PyNumberMethods channelid_as_number = {
+     0,                        /* nb_add */
+     0,                        /* nb_subtract */
+     0,                        /* nb_multiply */
+     0,                        /* nb_remainder */
+     0,                        /* nb_divmod */
+     0,                        /* nb_power */
+     0,                        /* nb_negative */
+     0,                        /* nb_positive */
+     0,                        /* nb_absolute */
+     0,                        /* nb_bool */
+     0,                        /* nb_invert */
+     0,                        /* nb_lshift */
+     0,                        /* nb_rshift */
+     0,                        /* nb_and */
+     0,                        /* nb_xor */
+     0,                        /* nb_or */
+     (unaryfunc)channelid_int, /* nb_int */
+     0,                        /* nb_reserved */
+     0,                        /* nb_float */
+
+     0,                        /* nb_inplace_add */
+     0,                        /* nb_inplace_subtract */
+     0,                        /* nb_inplace_multiply */
+     0,                        /* nb_inplace_remainder */
+     0,                        /* nb_inplace_power */
+     0,                        /* nb_inplace_lshift */
+     0,                        /* nb_inplace_rshift */
+     0,                        /* nb_inplace_and */
+     0,                        /* nb_inplace_xor */
+     0,                        /* nb_inplace_or */
+
+     0,                        /* nb_floor_divide */
+     0,                        /* nb_true_divide */
+     0,                        /* nb_inplace_floor_divide */
+     0,                        /* nb_inplace_true_divide */
+
+     (unaryfunc)channelid_int, /* nb_index */
+};
+
+static Py_hash_t
+channelid_hash(PyObject *self)
+{
+    channelid *cid = (channelid *)self;
+    PyObject *id = PyLong_FromLongLong(cid->id);
+    if (id == NULL) {
+        return -1;
+    }
+    return PyObject_Hash(id);
+}
+
+static PyObject *
+channelid_richcompare(PyObject *self, PyObject *other, int op)
+{
+    if (op != Py_EQ && op != Py_NE) {
+        Py_RETURN_NOTIMPLEMENTED;
+    }
+
+    if (!PyObject_TypeCheck(self, &ChannelIDtype)) {
+        Py_RETURN_NOTIMPLEMENTED;
+    }
+
+    channelid *cid = (channelid *)self;
+    int equal;
+    if (PyObject_TypeCheck(other, &ChannelIDtype)) {
+        channelid *othercid = (channelid *)other;
+        if (cid->end != othercid->end) {
+            equal = 0;
+        }
+        else {
+            equal = (cid->id == othercid->id);
+        }
+    }
+    else {
+        other = PyNumber_Long(other);
+        if (other == NULL) {
+            PyErr_Clear();
+            Py_RETURN_NOTIMPLEMENTED;
+        }
+        int64_t othercid = PyLong_AsLongLong(other);
+        // XXX decref other here?
+        if (othercid == -1 && PyErr_Occurred() != NULL) {
+            return NULL;
+        }
+        if (othercid < 0 || othercid > INT64_MAX) {
+            equal = 0;
+        }
+        else {
+            equal = (cid->id == othercid);
+        }
+    }
+
+    if ((op == Py_EQ && equal) || (op == Py_NE && !equal)) {
+        Py_RETURN_TRUE;
+    }
+    Py_RETURN_FALSE;
+}
+
+struct _channelid_xid {
+    int64_t id;
+    int end;
+};
+
+static PyObject *
+_channelid_from_xid(_PyCrossInterpreterData *data)
+{
+    struct _channelid_xid *xid = (struct _channelid_xid *)data->data;
+    return (PyObject *)newchannelid(&ChannelIDtype, xid->id, xid->end,
+                                    _global_channels(), 0);
+}
+
+static int
+_channelid_shared(PyObject *obj, _PyCrossInterpreterData *data)
+{
+    struct _channelid_xid *xid = PyMem_NEW(struct _channelid_xid, 1);
+    if (xid == NULL) {
+        return -1;
+    }
+    xid->id = ((channelid *)obj)->id;
+    xid->end = ((channelid *)obj)->end;
+
+    data->data = xid;
+    data->obj = obj;
+    data->new_object = _channelid_from_xid;
+    data->free = PyMem_Free;
+    return 0;
+}
+
+static PyObject *
+channelid_end(PyObject *self, void *end)
+{
+    int force = 1;
+    channelid *cid = (channelid *)self;
+    if (end != NULL) {
+        return (PyObject *)newchannelid(Py_TYPE(self), cid->id, *(int *)end,
+                                        cid->channels, force);
+    }
+
+    if (cid->end == CHANNEL_SEND) {
+        return PyUnicode_InternFromString("send");
+    }
+    if (cid->end == CHANNEL_RECV) {
+        return PyUnicode_InternFromString("recv");
+    }
+    return PyUnicode_InternFromString("both");
+}
+
+static int _channelid_end_send = CHANNEL_SEND;
+static int _channelid_end_recv = CHANNEL_RECV;
+
+static PyGetSetDef channelid_getsets[] = {
+    {"end", (getter)channelid_end, NULL,
+     PyDoc_STR("'send', 'recv', or 'both'")},
+    {"send", (getter)channelid_end, NULL,
+     PyDoc_STR("the 'send' end of the channel"), &_channelid_end_send},
+    {"recv", (getter)channelid_end, NULL,
+     PyDoc_STR("the 'recv' end of the channel"), &_channelid_end_recv},
+    {NULL}
+};
+
+PyDoc_STRVAR(channelid_doc,
+"A channel ID identifies a channel and may be used as an int.");
+
+static PyTypeObject ChannelIDtype = {
+    PyVarObject_HEAD_INIT(&PyType_Type, 0)
+    "_xxsubinterpreters.ChannelID", /* tp_name */
+    sizeof(channelid),              /* tp_size */
+    0,                              /* tp_itemsize */
+    (destructor)channelid_dealloc,  /* tp_dealloc */
+    0,                              /* tp_print */
+    0,                              /* tp_getattr */
+    0,                              /* tp_setattr */
+    0,                              /* tp_as_async */
+    (reprfunc)channelid_repr,       /* tp_repr */
+    &channelid_as_number,           /* tp_as_number */
+    0,                              /* tp_as_sequence */
+    0,                              /* tp_as_mapping */
+    channelid_hash,                 /* tp_hash */
+    0,                              /* tp_call */
+    0,                              /* tp_str */
+    0,                              /* tp_getattro */
+    0,                              /* tp_setattro */
+    0,                              /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
+        Py_TPFLAGS_LONG_SUBCLASS,   /* tp_flags */
+    channelid_doc,                  /* tp_doc */
+    0,                              /* tp_traverse */
+    0,                              /* tp_clear */
+    channelid_richcompare,          /* tp_richcompare */
+    0,                              /* tp_weaklistoffset */
+    0,                              /* tp_iter */
+    0,                              /* tp_iternext */
+    0,                              /* tp_methods */
+    0,                              /* tp_members */
+    channelid_getsets,              /* tp_getset */
+    0,                              /* tp_base */
+    0,                              /* tp_dict */
+    0,                              /* tp_descr_get */
+    0,                              /* tp_descr_set */
+    0,                              /* tp_dictoffset */
+    0,                              /* tp_init */
+    0,                              /* tp_alloc */
+    // Note that we do not set tp_new to channelid_new.  Instead we
+    // set it to NULL, meaning it cannot be instantiated from Python
+    // code.  We do this because there is a strong relationship between
+    // channel IDs and the channel lifecycle, so this limitation avoids
+    // related complications.
+    NULL,                           /* tp_new */
+};
+
+/* interpreter-specific functions *******************************************/
+
+static PyInterpreterState *
+_look_up(PyObject *requested_id)
+{
+    long long id = PyLong_AsLongLong(requested_id);
+    if (id == -1 && PyErr_Occurred() != NULL) {
+        return NULL;
+    }
+    assert(id <= INT64_MAX);
+    return _PyInterpreterState_LookUpID(id);
+}
+
+static PyObject *
+_get_id(PyInterpreterState *interp)
+{
+    PY_INT64_T id = PyInterpreterState_GetID(interp);
+    if (id < 0) {
+        return NULL;
+    }
+    return PyLong_FromLongLong(id);
+}
+
+static int
+_is_running(PyInterpreterState *interp)
+{
+    PyThreadState *tstate = PyInterpreterState_ThreadHead(interp);
+    if (PyThreadState_Next(tstate) != NULL) {
+        PyErr_SetString(PyExc_RuntimeError,
+                        "interpreter has more than one thread");
+        return -1;
+    }
+    PyFrameObject *frame = tstate->frame;
+    if (frame == NULL) {
+        if (PyErr_Occurred() != NULL) {
+            return -1;
+        }
+        return 0;
+    }
+    return (int)(frame->f_executing);
+}
+
+static int
+_ensure_not_running(PyInterpreterState *interp)
+{
+    int is_running = _is_running(interp);
+    if (is_running < 0) {
+        return -1;
+    }
+    if (is_running) {
+        PyErr_Format(PyExc_RuntimeError, "interpreter already running");
+        return -1;
+    }
+    return 0;
+}
+
+static int
+_run_script(PyInterpreterState *interp, const char *codestr,
+            _shareditem *shared, Py_ssize_t num_shared,
+            _sharedexception **exc)
+{
+    assert(num_shared >= 0);
+    PyObject *main_mod = PyMapping_GetItemString(interp->modules, "__main__");
+    if (main_mod == NULL) {
+        goto error;
+    }
+    PyObject *ns = PyModule_GetDict(main_mod);  // borrowed
+    Py_DECREF(main_mod);
+    if (ns == NULL) {
+        goto error;
+    }
+    Py_INCREF(ns);
+
+    // Apply the cross-interpreter data.
+    if (shared != NULL) {
+        for (Py_ssize_t i=0; i < num_shared; i++) {
+            _shareditem *item = &shared[i];
+            if (_shareditem_apply(item, ns) != 0) {
+                Py_DECREF(ns);
+                goto error;
+            }
+        }
+    }
+
+    // Run the string (see PyRun_SimpleStringFlags).
+    PyObject *result = PyRun_StringFlags(codestr, Py_file_input, ns, ns, NULL);
+    Py_DECREF(ns);
+    if (result == NULL) {
+        goto error;
+    }
+    else {
+        Py_DECREF(result);  // We throw away the result.
+    }
+
+    return 0;
+
+error:
+    *exc = _get_shared_exception();
+    PyErr_Clear();
+    return -1;
+}
+
+static int
+_run_script_in_interpreter(PyInterpreterState *interp, const char *codestr,
+                           PyObject *shareables)
+{
+    if (_ensure_not_running(interp) < 0) {
+        return -1;
+    }
+
+    Py_ssize_t num_shared = -1;
+    _shareditem *shared = _get_shared_ns(shareables, &num_shared);
+    if (shared == NULL && PyErr_Occurred()) {
+        return -1;
+    }
+
+    // Switch to interpreter.
+    PyThreadState *tstate = PyInterpreterState_ThreadHead(interp);
+    PyThreadState *save_tstate = PyThreadState_Swap(tstate);
+
+    // Run the script.
+    _sharedexception *exc = NULL;
+    int result = _run_script(interp, codestr, shared, num_shared, &exc);
+
+    // Switch back.
+    if (save_tstate != NULL) {
+        PyThreadState_Swap(save_tstate);
+    }
+
+    // Propagate any exception out to the caller.
+    if (exc != NULL) {
+        _apply_shared_exception(exc);
+        PyMem_Free(exc);
+    }
+    else if (result != 0) {
+        // We were unable to allocate a shared exception.
+        PyErr_NoMemory();
+    }
+
+    if (shared != NULL) {
+        _sharedns_clear(shared);
+        PyMem_Free(shared);
+    }
+
+    return result;
+}
+
+
+/* module level code ********************************************************/
+
+/* globals is the process-global state for the module.  It holds all
+   the data that we need to share between interpreters, so it cannot
+   hold PyObject values. */
+static struct globals {
+    _channels channels;
+} _globals = {{0}};
+
+static int
+_init_globals(void)
+{
+    if (_channels_init(&_globals.channels) != 0) {
+        return -1;
+    }
+    return 0;
+}
+
+static _channels *
+_global_channels(void) {
+    return &_globals.channels;
+}
+
+static PyObject *
+interp_create(PyObject *self, PyObject *args)
+{
+    if (!PyArg_UnpackTuple(args, "create", 0, 0)) {
+        return NULL;
+    }
+
+    // Create and initialize the new interpreter.
+    PyThreadState *tstate, *save_tstate;
+    save_tstate = PyThreadState_Swap(NULL);
+    tstate = Py_NewInterpreter();
+    PyThreadState_Swap(save_tstate);
+    if (tstate == NULL) {
+        /* Since no new thread state was created, there is no exception to
+           propagate; raise a fresh one after swapping in the old thread
+           state. */
+        PyErr_SetString(PyExc_RuntimeError, "interpreter creation failed");
+        return NULL;
+    }
+    return _get_id(tstate->interp);
+}
+
+PyDoc_STRVAR(create_doc,
+"create() -> ID\n\
+\n\
+Create a new interpreter and return a unique generated ID.");
+
+
+static PyObject *
+interp_destroy(PyObject *self, PyObject *args)
+{
+    PyObject *id;
+    if (!PyArg_UnpackTuple(args, "destroy", 1, 1, &id)) {
+        return NULL;
+    }
+    if (!PyLong_Check(id)) {
+        PyErr_SetString(PyExc_TypeError, "ID must be an int");
+        return NULL;
+    }
+
+    // Look up the interpreter.
+    PyInterpreterState *interp = _look_up(id);
+    if (interp == NULL) {
+        return NULL;
+    }
+
+    // Ensure we don't try to destroy the current interpreter.
+    PyInterpreterState *current = _get_current();
+    if (current == NULL) {
+        return NULL;
+    }
+    if (interp == current) {
+        PyErr_SetString(PyExc_RuntimeError,
+                        "cannot destroy the current interpreter");
+        return NULL;
+    }
+
+    // Ensure the interpreter isn't running.
+    /* XXX We *could* support destroying a running interpreter but
+       aren't going to worry about it for now. */
+    if (_ensure_not_running(interp) < 0) {
+        return NULL;
+    }
+
+    // Destroy the interpreter.
+    //PyInterpreterState_Delete(interp);
+    PyThreadState *tstate, *save_tstate;
+    tstate = PyInterpreterState_ThreadHead(interp);
+    save_tstate = PyThreadState_Swap(tstate);
+    Py_EndInterpreter(tstate);
+    PyThreadState_Swap(save_tstate);
+
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(destroy_doc,
+"destroy(ID)\n\
+\n\
+Destroy the identified interpreter.\n\
+\n\
+Attempting to destroy the current interpreter results in a RuntimeError.\n\
+So does an unrecognized ID.");
+
+
+static PyObject *
+interp_list_all(PyObject *self)
+{
+    PyObject *ids, *id;
+    PyInterpreterState *interp;
+
+    ids = PyList_New(0);
+    if (ids == NULL) {
+        return NULL;
+    }
+
+    interp = PyInterpreterState_Head();
+    while (interp != NULL) {
+        id = _get_id(interp);
+        if (id == NULL) {
+            Py_DECREF(ids);
+            return NULL;
+        }
+        // insert at front of list
+        if (PyList_Insert(ids, 0, id) < 0) {
+            Py_DECREF(ids);
+            return NULL;
+        }
+
+        interp = PyInterpreterState_Next(interp);
+    }
+
+    return ids;
+}
+
+PyDoc_STRVAR(list_all_doc,
+"list_all() -> [ID]\n\
+\n\
+Return a list containing the ID of every existing interpreter.");
+
+
+static PyObject *
+interp_get_current(PyObject *self)
+{
+    PyInterpreterState *interp =_get_current();
+    if (interp == NULL) {
+        return NULL;
+    }
+    return _get_id(interp);
+}
+
+PyDoc_STRVAR(get_current_doc,
+"get_current() -> ID\n\
+\n\
+Return the ID of current interpreter.");
+
+
+static PyObject *
+interp_get_main(PyObject *self)
+{
+    // Currently, 0 is always the main interpreter.
+    return PyLong_FromLongLong(0);
+}
+
+PyDoc_STRVAR(get_main_doc,
+"get_main() -> ID\n\
+\n\
+Return the ID of main interpreter.");
+
+
+static PyObject *
+interp_run_string(PyObject *self, PyObject *args)
+{
+    PyObject *id, *code;
+    PyObject *shared = NULL;
+    if (!PyArg_UnpackTuple(args, "run_string", 2, 3, &id, &code, &shared)) {
+        return NULL;
+    }
+    if (!PyLong_Check(id)) {
+        PyErr_SetString(PyExc_TypeError, "first arg (ID) must be an int");
+        return NULL;
+    }
+    if (!PyUnicode_Check(code)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "second arg (code) must be a string");
+        return NULL;
+    }
+
+    // Look up the interpreter.
+    PyInterpreterState *interp = _look_up(id);
+    if (interp == NULL) {
+        return NULL;
+    }
+
+    // Extract code.
+    Py_ssize_t size;
+    const char *codestr = PyUnicode_AsUTF8AndSize(code, &size);
+    if (codestr == NULL) {
+        return NULL;
+    }
+    if (strlen(codestr) != (size_t)size) {
+        PyErr_SetString(PyExc_ValueError,
+                        "source code string cannot contain null bytes");
+        return NULL;
+    }
+
+    // Run the code in the interpreter.
+    if (_run_script_in_interpreter(interp, codestr, shared) != 0) {
+        return NULL;
+    }
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(run_string_doc,
+"run_string(ID, sourcetext)\n\
+\n\
+Execute the provided string in the identified interpreter.\n\
+\n\
+See PyRun_SimpleStrings.");
+
+
+static PyObject *
+object_is_shareable(PyObject *self, PyObject *args)
+{
+    PyObject *obj;
+    if (!PyArg_UnpackTuple(args, "is_shareable", 1, 1, &obj)) {
+        return NULL;
+    }
+    if (_PyObject_CheckCrossInterpreterData(obj) == 0) {
+        Py_RETURN_TRUE;
+    }
+    PyErr_Clear();
+    Py_RETURN_FALSE;
+}
+
+PyDoc_STRVAR(is_shareable_doc,
+"is_shareable(obj) -> bool\n\
+\n\
+Return True if the object's data may be shared between interpreters and\n\
+False otherwise.");
+
+
+static PyObject *
+interp_is_running(PyObject *self, PyObject *args)
+{
+    PyObject *id;
+    if (!PyArg_UnpackTuple(args, "is_running", 1, 1, &id)) {
+        return NULL;
+    }
+    if (!PyLong_Check(id)) {
+        PyErr_SetString(PyExc_TypeError, "ID must be an int");
+        return NULL;
+    }
+
+    PyInterpreterState *interp = _look_up(id);
+    if (interp == NULL) {
+        return NULL;
+    }
+    int is_running = _is_running(interp);
+    if (is_running < 0) {
+        return NULL;
+    }
+    if (is_running) {
+        Py_RETURN_TRUE;
+    }
+    Py_RETURN_FALSE;
+}
+
+PyDoc_STRVAR(is_running_doc,
+"is_running(id) -> bool\n\
+\n\
+Return whether or not the identified interpreter is running.");
+
+static PyObject *
+channel_create(PyObject *self)
+{
+    int64_t cid = _channel_create(&_globals.channels);
+    if (cid < 0) {
+        return NULL;
+    }
+    PyObject *id = (PyObject *)newchannelid(&ChannelIDtype, cid, 0,
+                                            &_globals.channels, 0);
+    if (id == NULL) {
+        if (_channel_destroy(&_globals.channels, cid) != 0) {
+            // XXX issue a warning?
+        }
+        return NULL;
+    }
+    assert(((channelid *)id)->channels != NULL);
+    return id;
+}
+
+PyDoc_STRVAR(channel_create_doc,
+"channel_create() -> ID\n\
+\n\
+Create a new cross-interpreter channel and return a unique generated ID.");
+
+static PyObject *
+channel_destroy(PyObject *self, PyObject *args)
+{
+    PyObject *id;
+    if (!PyArg_UnpackTuple(args, "channel_destroy", 1, 1, &id)) {
+        return NULL;
+    }
+    int64_t cid = _coerce_id(id);
+    if (cid < 0) {
+        return NULL;
+    }
+
+    if (_channel_destroy(&_globals.channels, cid) != 0) {
+        return NULL;
+    }
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(channel_destroy_doc,
+"channel_destroy(ID)\n\
+\n\
+Close and finalize the channel.  Afterward attempts to use the channel\n\
+will behave as though it never existed.");
+
+static PyObject *
+channel_list_all(PyObject *self)
+{
+    int64_t count = 0;
+    int64_t *cids = _channels_list_all(&_globals.channels, &count);
+    if (cids == NULL) {
+        if (count == 0) {
+            return PyList_New(0);
+        }
+        return NULL;
+    }
+    PyObject *ids = PyList_New((Py_ssize_t)count);
+    if (ids == NULL) {
+        // XXX free cids
+        return NULL;
+    }
+    for (int64_t i=0; i < count; cids++, i++) {
+        PyObject *id = (PyObject *)newchannelid(&ChannelIDtype, *cids, 0,
+                                                &_globals.channels, 0);
+        if (id == NULL) {
+            Py_DECREF(ids);
+            ids = NULL;
+            break;
+        }
+        PyList_SET_ITEM(ids, i, id);
+    }
+    // XXX free cids
+    return ids;
+}
+
+PyDoc_STRVAR(channel_list_all_doc,
+"channel_list_all() -> [ID]\n\
+\n\
+Return the list of all IDs for active channels.");
+
+static PyObject *
+channel_send(PyObject *self, PyObject *args)
+{
+    PyObject *id;
+    PyObject *obj;
+    if (!PyArg_UnpackTuple(args, "channel_send", 2, 2, &id, &obj)) {
+        return NULL;
+    }
+    int64_t cid = _coerce_id(id);
+    if (cid < 0) {
+        return NULL;
+    }
+
+    if (_channel_send(&_globals.channels, cid, obj) != 0) {
+        return NULL;
+    }
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(channel_send_doc,
+"channel_send(ID, obj)\n\
+\n\
+Add the object's data to the channel's queue.");
+
+static PyObject *
+channel_recv(PyObject *self, PyObject *args)
+{
+    PyObject *id;
+    if (!PyArg_UnpackTuple(args, "channel_recv", 1, 1, &id)) {
+        return NULL;
+    }
+    int64_t cid = _coerce_id(id);
+    if (cid < 0) {
+        return NULL;
+    }
+
+    return _channel_recv(&_globals.channels, cid);
+}
+
+PyDoc_STRVAR(channel_recv_doc,
+"channel_recv(ID) -> obj\n\
+\n\
+Return a new object from the data at the from of the channel's queue.");
+
+static PyObject *
+channel_close(PyObject *self, PyObject *args, PyObject *kwds)
+{
+    PyObject *id;
+    if (!PyArg_UnpackTuple(args, "channel_recv", 1, 1, &id)) {
+        return NULL;
+    }
+    int64_t cid = _coerce_id(id);
+    if (cid < 0) {
+        return NULL;
+    }
+
+    if (_channel_close(&_globals.channels, cid) != 0) {
+        return NULL;
+    }
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(channel_close_doc,
+"channel_close(ID)\n\
+\n\
+Close the channel for all interpreters.  Once the channel's ID has\n\
+no more ref counts the channel will be destroyed.");
+
+static PyObject *
+channel_drop_interpreter(PyObject *self, PyObject *args, PyObject *kwds)
+{
+    // Note that only the current interpreter is affected.
+    static char *kwlist[] = {"id", "send", "recv"};
+    PyObject *id;
+    int send = -1;
+    int recv = -1;
+    if (!PyArg_ParseTupleAndKeywords(args, kwds,
+                                     "O|$pp:channel_drop_interpreter", kwlist,
+                                     &id, &send, &recv))
+        return NULL;
+
+    int64_t cid = _coerce_id(id);
+    if (cid < 0) {
+        return NULL;
+    }
+    if (send < 0 && recv < 0) {
+        send = 1;
+        recv = 1;
+    }
+    else {
+        if (send < 0) {
+            send = 0;
+        }
+        if (recv < 0) {
+            recv = 0;
+        }
+    }
+    if (_channel_drop(&_globals.channels, cid, send, recv) != 0) {
+        return NULL;
+    }
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(channel_drop_interpreter_doc,
+"channel_drop_interpreter(ID, *, send=None, recv=None)\n\
+\n\
+Close the channel for the current interpreter.  'send' and 'recv'\n\
+(bool) may be used to indicate the ends to close.  By default both\n\
+ends are closed.  Closing an already closed end is a noop.");
+
+static PyObject *
+channel__channel_id(PyObject *self, PyObject *args, PyObject *kwds)
+{
+    return channelid_new(&ChannelIDtype, args, kwds);
+}
+
+static PyMethodDef module_functions[] = {
+    {"create",                    (PyCFunction)interp_create,
+     METH_VARARGS, create_doc},
+    {"destroy",                   (PyCFunction)interp_destroy,
+     METH_VARARGS, destroy_doc},
+    {"list_all",                  (PyCFunction)interp_list_all,
+     METH_NOARGS, list_all_doc},
+    {"get_current",               (PyCFunction)interp_get_current,
+     METH_NOARGS, get_current_doc},
+    {"get_main",                  (PyCFunction)interp_get_main,
+     METH_NOARGS, get_main_doc},
+    {"is_running",                (PyCFunction)interp_is_running,
+     METH_VARARGS, is_running_doc},
+    {"run_string",                (PyCFunction)interp_run_string,
+     METH_VARARGS, run_string_doc},
+
+    {"is_shareable",              (PyCFunction)object_is_shareable,
+     METH_VARARGS, is_shareable_doc},
+
+    {"channel_create",            (PyCFunction)channel_create,
+     METH_NOARGS, channel_create_doc},
+    {"channel_destroy",           (PyCFunction)channel_destroy,
+     METH_VARARGS, channel_destroy_doc},
+    {"channel_list_all",          (PyCFunction)channel_list_all,
+     METH_NOARGS, channel_list_all_doc},
+    {"channel_send",              (PyCFunction)channel_send,
+     METH_VARARGS, channel_send_doc},
+    {"channel_recv",              (PyCFunction)channel_recv,
+     METH_VARARGS, channel_recv_doc},
+    {"channel_close",             (PyCFunction)channel_close,
+     METH_VARARGS, channel_close_doc},
+    {"channel_drop_interpreter",  (PyCFunction)channel_drop_interpreter,
+     METH_VARARGS | METH_KEYWORDS, channel_drop_interpreter_doc},
+    {"_channel_id",               (PyCFunction)channel__channel_id,
+     METH_VARARGS | METH_KEYWORDS, NULL},
+
+    {NULL,                        NULL}           /* sentinel */
+};
+
+
+/* initialization function */
+
+PyDoc_STRVAR(module_doc,
+"This module provides primitive operations to manage Python interpreters.\n\
+The 'interpreters' module provides a more convenient interface.");
+
+static struct PyModuleDef interpretersmodule = {
+    PyModuleDef_HEAD_INIT,
+    "_xxsubinterpreters",  /* m_name */
+    module_doc,            /* m_doc */
+    -1,                    /* m_size */
+    module_functions,      /* m_methods */
+    NULL,                  /* m_slots */
+    NULL,                  /* m_traverse */
+    NULL,                  /* m_clear */
+    NULL                   /* m_free */
+};
+
+
+PyMODINIT_FUNC
+PyInit__xxsubinterpreters(void)
+{
+    if (_init_globals() != 0) {
+        return NULL;
+    }
+
+    /* Initialize types */
+    ChannelIDtype.tp_base = &PyLong_Type;
+    if (PyType_Ready(&ChannelIDtype) != 0) {
+        return NULL;
+    }
+
+    /* Create the module */
+    PyObject *module = PyModule_Create(&interpretersmodule);
+    if (module == NULL) {
+        return NULL;
+    }
+
+    /* Add exception types */
+    PyObject *ns = PyModule_GetDict(module);  // borrowed
+    if (interp_exceptions_init(ns) != 0) {
+        return NULL;
+    }
+    if (channel_exceptions_init(ns) != 0) {
+        return NULL;
+    }
+
+    /* Add other types */
+    Py_INCREF(&ChannelIDtype);
+    if (PyDict_SetItemString(ns, "ChannelID", (PyObject *)&ChannelIDtype) != 0) {
+        return NULL;
+    }
+
+    if (_PyCrossInterpreterData_Register_Class(&ChannelIDtype, _channelid_shared)) {
+        return NULL;
+    }
+
+    return module;
+}
diff --git a/Python/pystate.c b/Python/pystate.c
index 909d831465d4..a474549a8c73 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -54,8 +54,13 @@ _PyRuntimeState_Init_impl(_PyRuntimeState *runtime)
     if (runtime->interpreters.mutex == NULL) {
         return _Py_INIT_ERR("Can't initialize threads for interpreter");
     }
-
     runtime->interpreters.next_id = -1;
+
+    runtime->xidregistry.mutex = PyThread_allocate_lock();
+    if (runtime->xidregistry.mutex == NULL) {
+        return _Py_INIT_ERR("Can't initialize threads for cross-interpreter data registry");
+    }
+
     return _Py_INIT_OK();
 }
 
@@ -166,6 +171,7 @@ PyInterpreterState_New(void)
         /* overflow or Py_Initialize() not called! */
         PyErr_SetString(PyExc_RuntimeError,
                         "failed to get an interpreter ID");
+        /* XXX deallocate! */
         interp = NULL;
     } else {
         interp->id = _PyRuntime.interpreters.next_id;
@@ -256,6 +262,28 @@ PyInterpreterState_GetID(PyInterpreterState *interp)
 }
 
 
+PyInterpreterState *
+_PyInterpreterState_LookUpID(PY_INT64_T requested_id)
+{
+    if (requested_id < 0)
+        goto error;
+
+    PyInterpreterState *interp = PyInterpreterState_Head();
+    while (interp != NULL) {
+        PY_INT64_T id = PyInterpreterState_GetID(interp);
+        if (id < 0)
+            return NULL;
+        if (requested_id == id)
+            return interp;
+        interp = PyInterpreterState_Next(interp);
+    }
+
+error:
+    PyErr_Format(PyExc_RuntimeError,
+                 "unrecognized interpreter ID %lld", requested_id);
+    return NULL;
+}
+
 /* Default implementation for _PyThreadState_GetFrame */
 static struct _frame *
 threadstate_getframe(PyThreadState *self)
@@ -1024,6 +1052,251 @@ PyGILState_Release(PyGILState_STATE oldstate)
 }
 
 
+/**************************/
+/* cross-interpreter data */
+/**************************/
+
+/* cross-interpreter data */
+
+crossinterpdatafunc _PyCrossInterpreterData_Lookup(PyObject *);
+
+/* This is a separate func from _PyCrossInterpreterData_Lookup in order
+   to keep the registry code separate. */
+static crossinterpdatafunc
+_lookup_getdata(PyObject *obj)
+{
+    crossinterpdatafunc getdata = _PyCrossInterpreterData_Lookup(obj);
+    if (getdata == NULL && PyErr_Occurred() == 0)
+        PyErr_Format(PyExc_ValueError,
+                     "%S does not support cross-interpreter data", obj);
+    return getdata;
+}
+
+int
+_PyObject_CheckCrossInterpreterData(PyObject *obj)
+{
+    crossinterpdatafunc getdata = _lookup_getdata(obj);
+    if (getdata == NULL) {
+        return -1;
+    }
+    return 0;
+}
+
+static int
+_check_xidata(_PyCrossInterpreterData *data)
+{
+    // data->data can be anything, including NULL, so we don't check it.
+
+    // data->obj may be NULL, so we don't check it.
+
+    if (data->interp < 0) {
+        PyErr_SetString(PyExc_SystemError, "missing interp");
+        return -1;
+    }
+
+    if (data->new_object == NULL) {
+        PyErr_SetString(PyExc_SystemError, "missing new_object func");
+        return -1;
+    }
+
+    // data->free may be NULL, so we don't check it.
+
+    return 0;
+}
+
+int
+_PyObject_GetCrossInterpreterData(PyObject *obj, _PyCrossInterpreterData *data)
+{
+    PyThreadState *tstate = PyThreadState_Get();
+    // PyThreadState_Get() aborts if lookup fails, so we don't need
+    // to check the result for NULL.
+    PyInterpreterState *interp = tstate->interp;
+
+    // Reset data before re-populating.
+    *data = (_PyCrossInterpreterData){0};
+    data->free = PyMem_RawFree;  // Set a default that may be overridden.
+
+    // Call the "getdata" func for the object.
+    Py_INCREF(obj);
+    crossinterpdatafunc getdata = _lookup_getdata(obj);
+    if (getdata == NULL) {
+        Py_DECREF(obj);
+        return -1;
+    }
+    int res = getdata(obj, data);
+    Py_DECREF(obj);
+    if (res != 0) {
+        return -1;
+    }
+
+    // Fill in the blanks and validate the result.
+    Py_XINCREF(data->obj);
+    data->interp = interp->id;
+    if (_check_xidata(data) != 0) {
+        _PyCrossInterpreterData_Release(data);
+        return -1;
+    }
+
+    return 0;
+}
+
+void
+_PyCrossInterpreterData_Release(_PyCrossInterpreterData *data)
+{
+    if (data->data == NULL && data->obj == NULL) {
+        // Nothing to release!
+        return;
+    }
+
+    // Switch to the original interpreter.
+    PyInterpreterState *interp = _PyInterpreterState_LookUpID(data->interp);
+    if (interp == NULL) {
+        // The intepreter was already destroyed.
+        if (data->free != NULL) {
+            // XXX Someone leaked some memory...
+        }
+        return;
+    }
+    PyThreadState *tstate = PyInterpreterState_ThreadHead(interp);
+    PyThreadState *save_tstate = PyThreadState_Swap(tstate);
+
+    // "Release" the data and/or the object.
+    if (data->free != NULL) {
+        data->free(data->data);
+    }
+    Py_XDECREF(data->obj);
+
+    // Switch back.
+    if (save_tstate != NULL)
+        PyThreadState_Swap(save_tstate);
+}
+
+PyObject *
+_PyCrossInterpreterData_NewObject(_PyCrossInterpreterData *data)
+{
+    return data->new_object(data);
+}
+
+/* registry of {type -> crossinterpdatafunc} */
+
+/* For now we use a global registry of shareable classes.  An
+   alternative would be to add a tp_* slot for a class's
+   crossinterpdatafunc. It would be simpler and more efficient. */
+
+static int
+_register_xidata(PyTypeObject *cls, crossinterpdatafunc getdata)
+{
+    // Note that we effectively replace already registered classes
+    // rather than failing.
+    struct _xidregitem *newhead = PyMem_RawMalloc(sizeof(struct _xidregitem));
+    if (newhead == NULL)
+        return -1;
+    newhead->cls = cls;
+    newhead->getdata = getdata;
+    newhead->next = _PyRuntime.xidregistry.head;
+    _PyRuntime.xidregistry.head = newhead;
+    return 0;
+}
+
+static void _register_builtins_for_crossinterpreter_data(void);
+
+int
+_PyCrossInterpreterData_Register_Class(PyTypeObject *cls,
+                                       crossinterpdatafunc getdata)
+{
+    if (!PyType_Check(cls)) {
+        PyErr_Format(PyExc_ValueError, "only classes may be registered");
+        return -1;
+    }
+    if (getdata == NULL) {
+        PyErr_Format(PyExc_ValueError, "missing 'getdata' func");
+        return -1;
+    }
+
+    // Make sure the class isn't ever deallocated.
+    Py_INCREF((PyObject *)cls);
+
+    PyThread_acquire_lock(_PyRuntime.xidregistry.mutex, WAIT_LOCK);
+    if (_PyRuntime.xidregistry.head == NULL) {
+        _register_builtins_for_crossinterpreter_data();
+    }
+    int res = _register_xidata(cls, getdata);
+    PyThread_release_lock(_PyRuntime.xidregistry.mutex);
+    return res;
+}
+
+crossinterpdatafunc
+_PyCrossInterpreterData_Lookup(PyObject *obj)
+{
+    PyObject *cls = PyObject_Type(obj);
+    crossinterpdatafunc getdata = NULL;
+    PyThread_acquire_lock(_PyRuntime.xidregistry.mutex, WAIT_LOCK);
+    struct _xidregitem *cur = _PyRuntime.xidregistry.head;
+    if (cur == NULL) {
+        _register_builtins_for_crossinterpreter_data();
+        cur = _PyRuntime.xidregistry.head;
+    }
+    for(; cur != NULL; cur = cur->next) {
+        if (cur->cls == (PyTypeObject *)cls) {
+            getdata = cur->getdata;
+            break;
+        }
+    }
+    PyThread_release_lock(_PyRuntime.xidregistry.mutex);
+    return getdata;
+}
+
+/* cross-interpreter data for builtin types */
+
+static PyObject *
+_new_bytes_object(_PyCrossInterpreterData *data)
+{
+    return PyBytes_FromString((char *)(data->data));
+}
+
+static int
+_bytes_shared(PyObject *obj, _PyCrossInterpreterData *data)
+{
+    data->data = (void *)(PyBytes_AS_STRING(obj));
+    data->obj = obj;  // Will be "released" (decref'ed) when data released.
+    data->new_object = _new_bytes_object;
+    data->free = NULL;  // Do not free the data (it belongs to the object).
+    return 0;
+}
+
+static PyObject *
+_new_none_object(_PyCrossInterpreterData *data)
+{
+    // XXX Singleton refcounts are problematic across interpreters...
+    Py_INCREF(Py_None);
+    return Py_None;
+}
+
+static int
+_none_shared(PyObject *obj, _PyCrossInterpreterData *data)
+{
+    data->data = NULL;
+    // data->obj remains NULL
+    data->new_object = _new_none_object;
+    data->free = NULL;  // There is nothing to free.
+    return 0;
+}
+
+static void
+_register_builtins_for_crossinterpreter_data(void)
+{
+    // None
+    if (_register_xidata((PyTypeObject *)PyObject_Type(Py_None), _none_shared) != 0) {
+        Py_FatalError("could not register None for cross-interpreter sharing");
+    }
+
+    // bytes
+    if (_register_xidata(&PyBytes_Type, _bytes_shared) != 0) {
+        Py_FatalError("could not register bytes for cross-interpreter sharing");
+    }
+}
+
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/setup.py b/setup.py
index ba0a7624cfcd..6f9a39719b43 100644
--- a/setup.py
+++ b/setup.py
@@ -755,6 +755,10 @@ def detect_modules(self):
             ['_xxtestfuzz/_xxtestfuzz.c', '_xxtestfuzz/fuzzer.c'])
         )
 
+        # Python interface to subinterpreter C-API.
+        exts.append(Extension('_xxsubinterpreters', ['_xxsubinterpretersmodule.c'],
+                              define_macros=[('Py_BUILD_CORE', '')]))
+
         #
         # Here ends the simple stuff.  From here on, modules need certain
         # libraries, are platform-specific, or present other surprises.


From webhook-mailer at python.org  Mon Jan 29 21:03:04 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 02:03:04 -0000
Subject: [Python-checkins] replace dynamic import with 'exec' with
 importlib.import_module (#5433)
Message-ID: <mailman.67.1517277784.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/77526f05fa788d6fb12f2121fe6b96c130d9b717
commit: 77526f05fa788d6fb12f2121fe6b96c130d9b717
branch: master
author: Benjamin Peterson <benjamin at python.org>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T18:03:01-08:00
summary:

replace dynamic import with 'exec' with importlib.import_module (#5433)

files:
M Lib/test/test_hashlib.py

diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py
index e4e5280dc8ea..15fc22b02d77 100644
--- a/Lib/test/test_hashlib.py
+++ b/Lib/test/test_hashlib.py
@@ -9,6 +9,7 @@
 import array
 from binascii import unhexlify
 import hashlib
+import importlib
 import itertools
 import os
 import sys
@@ -83,11 +84,11 @@ class HashLibTestCase(unittest.TestCase):
     def _conditional_import_module(self, module_name):
         """Import a module and return a reference to it or None on failure."""
         try:
-            exec('import '+module_name)
-        except ImportError as error:
+            return importlib.import_module(module_name)
+        except ModuleNotFoundError as error:
             if self._warn_on_extension_import:
                 warnings.warn('Did a C extension fail to compile? %s' % error)
-        return locals().get(module_name)
+        return None
 
     def __init__(self, *args, **kwargs):
         algorithms = set()


From webhook-mailer at python.org  Mon Jan 29 22:36:10 2018
From: webhook-mailer at python.org (Mariatta)
Date: Tue, 30 Jan 2018 03:36:10 -0000
Subject: [Python-checkins] bpo-32701: Clarify the quotetabs flag in quopri
 documentation (GH-5401)
Message-ID: <mailman.68.1517283373.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/9424dcbb3e20a26dfdd81659303b989f7d3da044
commit: 9424dcbb3e20a26dfdd81659303b989f7d3da044
branch: master
author: Julien Palard <julien at palard.fr>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-29T19:36:06-08:00
summary:

bpo-32701: Clarify the quotetabs flag in quopri documentation (GH-5401)

files:
M Doc/library/quopri.rst

diff --git a/Doc/library/quopri.rst b/Doc/library/quopri.rst
index a3f94a0ad066..86717c00c3c1 100644
--- a/Doc/library/quopri.rst
+++ b/Doc/library/quopri.rst
@@ -34,9 +34,10 @@ sending a graphics file.
 
    Encode the contents of the *input* file and write the resulting quoted-printable
    data to the *output* file. *input* and *output* must be
-   :term:`binary file objects <file object>`. *quotetabs*, a flag which controls
-   whether to encode embedded spaces and tabs must be provideda and when true it
-   encodes such embedded whitespace, and when false it leaves them unencoded.
+   :term:`binary file objects <file object>`. *quotetabs*, a
+   non-optional flag which controls whether to encode embedded spaces
+   and tabs; when true it encodes such embedded whitespace, and when
+   false it leaves them unencoded.
    Note that spaces and tabs appearing at the end of lines are always encoded,
    as per :rfc:`1521`.  *header* is a flag which controls if spaces are encoded
    as underscores as per :rfc:`1522`.


From webhook-mailer at python.org  Mon Jan 29 23:04:18 2018
From: webhook-mailer at python.org (Eric Snow)
Date: Tue, 30 Jan 2018 04:04:18 -0000
Subject: [Python-checkins] bpo-32604: NULL-terminate kwlist in
 channel_drop_interpreter(). (gh-5437)
Message-ID: <mailman.69.1517285062.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/83e64c8a544028ae677af2a0bc268dbe1c11cc3a
commit: 83e64c8a544028ae677af2a0bc268dbe1c11cc3a
branch: master
author: Eric Snow <ericsnowcurrently at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-29T21:04:15-07:00
summary:

bpo-32604: NULL-terminate kwlist in channel_drop_interpreter(). (gh-5437)

files:
M Modules/_xxsubinterpretersmodule.c

diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c
index ed79a13d4a40..d2b5f26fae1d 100644
--- a/Modules/_xxsubinterpretersmodule.c
+++ b/Modules/_xxsubinterpretersmodule.c
@@ -1916,7 +1916,7 @@ static PyObject *
 channel_drop_interpreter(PyObject *self, PyObject *args, PyObject *kwds)
 {
     // Note that only the current interpreter is affected.
-    static char *kwlist[] = {"id", "send", "recv"};
+    static char *kwlist[] = {"id", "send", "recv", NULL};
     PyObject *id;
     int send = -1;
     int recv = -1;


From webhook-mailer at python.org  Mon Jan 29 23:16:24 2018
From: webhook-mailer at python.org (Mariatta)
Date: Tue, 30 Jan 2018 04:16:24 -0000
Subject: [Python-checkins] bpo-32701: Clarify the quotetabs flag in quopri
 documentation (GH-5401) (GH-5438)
Message-ID: <mailman.70.1517285785.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/04f99ba9d7186278eaf072e9a62c03aa63a8a6bb
commit: 04f99ba9d7186278eaf072e9a62c03aa63a8a6bb
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-29T20:16:21-08:00
summary:

bpo-32701: Clarify the quotetabs flag in quopri documentation (GH-5401) (GH-5438)

(cherry picked from commit 9424dcbb3e20a26dfdd81659303b989f7d3da044)

Co-authored-by: Julien Palard <julien at palard.fr>

files:
M Doc/library/quopri.rst

diff --git a/Doc/library/quopri.rst b/Doc/library/quopri.rst
index a3f94a0ad066..86717c00c3c1 100644
--- a/Doc/library/quopri.rst
+++ b/Doc/library/quopri.rst
@@ -34,9 +34,10 @@ sending a graphics file.
 
    Encode the contents of the *input* file and write the resulting quoted-printable
    data to the *output* file. *input* and *output* must be
-   :term:`binary file objects <file object>`. *quotetabs*, a flag which controls
-   whether to encode embedded spaces and tabs must be provideda and when true it
-   encodes such embedded whitespace, and when false it leaves them unencoded.
+   :term:`binary file objects <file object>`. *quotetabs*, a
+   non-optional flag which controls whether to encode embedded spaces
+   and tabs; when true it encodes such embedded whitespace, and when
+   false it leaves them unencoded.
    Note that spaces and tabs appearing at the end of lines are always encoded,
    as per :rfc:`1521`.  *header* is a flag which controls if spaces are encoded
    as underscores as per :rfc:`1522`.


From webhook-mailer at python.org  Tue Jan 30 00:27:42 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 05:27:42 -0000
Subject: [Python-checkins] bpo-25942: make subprocess more graceful on ^C
 (GH-5026)
Message-ID: <mailman.71.1517290064.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/f4d644f36ffb6cb11b34bfcf533c14cfaebf709a
commit: f4d644f36ffb6cb11b34bfcf533c14cfaebf709a
branch: master
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T21:27:39-08:00
summary:

bpo-25942: make subprocess more graceful on ^C (GH-5026)

Do not allow receiving a SIGINT to cause the subprocess module to trigger an
immediate SIGKILL of the child process.  SIGINT is normally sent to all child
processes by the OS at the same time already as was the established normal
behavior in 2.7 and 3.2.  This behavior change was introduced during the fix to https://bugs.python.org/issue12494 and is generally surprising to command line
tool users who expect other tools launched in child processes to get their own
SIGINT and do their own cleanup.

In Python 3.3-3.6 subprocess.call and subprocess.run would immediately
SIGKILL the child process upon receiving a SIGINT (which raises a
KeyboardInterrupt).  We now give the child a small amount of time to
exit gracefully before resorting to a SIGKILL.

This is also the case for subprocess.Popen.__exit__ which would
previously block indefinitely waiting for the child to die.  This was
hidden from many users by virtue of subprocess.call and subprocess.run
sending the signal immediately.

Behavior change: subprocess.Popen.__exit__ will not block indefinitely
when the exiting exception is a KeyboardInterrupt.  This is done for
user friendliness as people expect their ^C to actually happen.  This
could cause occasional orphaned Popen objects when not using `call` or
`run` with a child process that hasn't exited.

Refactoring involved: The Popen.wait method deals with the
KeyboardInterrupt second chance, existing platform specific internals
have been renamed to _wait().
Also fixes comment typos.

files:
A Misc/NEWS.d/next/Library/2017-12-27-20-15-51.bpo-25942.Giyr8v.rst
M Lib/subprocess.py
M Lib/test/test_subprocess.py

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index db6342fa4912..f69159e3aadc 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -304,9 +304,9 @@ def call(*popenargs, timeout=None, **kwargs):
     with Popen(*popenargs, **kwargs) as p:
         try:
             return p.wait(timeout=timeout)
-        except:
+        except:  # Including KeyboardInterrupt, wait handled that.
             p.kill()
-            p.wait()
+            # We don't call p.wait() again as p.__exit__ does that for us.
             raise
 
 
@@ -450,9 +450,9 @@ def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
             stdout, stderr = process.communicate()
             raise TimeoutExpired(process.args, timeout, output=stdout,
                                  stderr=stderr)
-        except:
+        except:  # Including KeyboardInterrupt, communicate handled that.
             process.kill()
-            process.wait()
+            # We don't call process.wait() as .__exit__ does that for us.
             raise
         retcode = process.poll()
         if check and retcode:
@@ -714,6 +714,11 @@ def __init__(self, args, bufsize=-1, executable=None,
 
         self.text_mode = encoding or errors or text or universal_newlines
 
+        # How long to resume waiting on a child after the first ^C.
+        # There is no right value for this.  The purpose is to be polite
+        # yet remain good for interactive users trying to exit a tool.
+        self._sigint_wait_secs = 0.25  # 1/xkcd221.getRandomNumber()
+
         self._closed_child_pipe_fds = False
 
         try:
@@ -787,7 +792,7 @@ def _translate_newlines(self, data, encoding, errors):
     def __enter__(self):
         return self
 
-    def __exit__(self, type, value, traceback):
+    def __exit__(self, exc_type, value, traceback):
         if self.stdout:
             self.stdout.close()
         if self.stderr:
@@ -796,6 +801,22 @@ def __exit__(self, type, value, traceback):
             if self.stdin:
                 self.stdin.close()
         finally:
+            if exc_type == KeyboardInterrupt:
+                # https://bugs.python.org/issue25942
+                # In the case of a KeyboardInterrupt we assume the SIGINT
+                # was also already sent to our child processes.  We can't
+                # block indefinitely as that is not user friendly.
+                # If we have not already waited a brief amount of time in
+                # an interrupted .wait() or .communicate() call, do so here
+                # for consistency.
+                if self._sigint_wait_secs > 0:
+                    try:
+                        self._wait(timeout=self._sigint_wait_secs)
+                    except TimeoutExpired:
+                        pass
+                self._sigint_wait_secs = 0  # Note that this has been done.
+                return  # resume the KeyboardInterrupt
+
             # Wait for the process to terminate, to avoid zombies.
             self.wait()
 
@@ -804,7 +825,7 @@ def __del__(self, _maxsize=sys.maxsize, _warn=warnings.warn):
             # We didn't get to successfully create a child process.
             return
         if self.returncode is None:
-            # Not reading subprocess exit status creates a zombi process which
+            # Not reading subprocess exit status creates a zombie process which
             # is only destroyed at the parent python process exit
             _warn("subprocess %s is still running" % self.pid,
                   ResourceWarning, source=self)
@@ -889,6 +910,21 @@ def communicate(self, input=None, timeout=None):
 
             try:
                 stdout, stderr = self._communicate(input, endtime, timeout)
+            except KeyboardInterrupt:
+                # https://bugs.python.org/issue25942
+                # See the detailed comment in .wait().
+                if timeout is not None:
+                    sigint_timeout = min(self._sigint_wait_secs,
+                                         self._remaining_time(endtime))
+                else:
+                    sigint_timeout = self._sigint_wait_secs
+                self._sigint_wait_secs = 0  # nothing else should wait.
+                try:
+                    self._wait(timeout=sigint_timeout)
+                except TimeoutExpired:
+                    pass
+                raise  # resume the KeyboardInterrupt
+
             finally:
                 self._communication_started = True
 
@@ -919,6 +955,30 @@ def _check_timeout(self, endtime, orig_timeout):
             raise TimeoutExpired(self.args, orig_timeout)
 
 
+    def wait(self, timeout=None):
+        """Wait for child process to terminate; returns self.returncode."""
+        if timeout is not None:
+            endtime = _time() + timeout
+        try:
+            return self._wait(timeout=timeout)
+        except KeyboardInterrupt:
+            # https://bugs.python.org/issue25942
+            # The first keyboard interrupt waits briefly for the child to
+            # exit under the common assumption that it also received the ^C
+            # generated SIGINT and will exit rapidly.
+            if timeout is not None:
+                sigint_timeout = min(self._sigint_wait_secs,
+                                     self._remaining_time(endtime))
+            else:
+                sigint_timeout = self._sigint_wait_secs
+            self._sigint_wait_secs = 0  # nothing else should wait.
+            try:
+                self._wait(timeout=sigint_timeout)
+            except TimeoutExpired:
+                pass
+            raise  # resume the KeyboardInterrupt
+
+
     if _mswindows:
         #
         # Windows methods
@@ -1127,16 +1187,16 @@ def _internal_poll(self, _deadstate=None,
             return self.returncode
 
 
-        def wait(self, timeout=None):
-            """Wait for child process to terminate.  Returns returncode
-            attribute."""
+        def _wait(self, timeout):
+            """Internal implementation of wait() on Windows."""
             if timeout is None:
                 timeout_millis = _winapi.INFINITE
             else:
                 timeout_millis = int(timeout * 1000)
             if self.returncode is None:
+                # API note: Returns immediately if timeout_millis == 0.
                 result = _winapi.WaitForSingleObject(self._handle,
-                                                    timeout_millis)
+                                                     timeout_millis)
                 if result == _winapi.WAIT_TIMEOUT:
                     raise TimeoutExpired(self.args, timeout)
                 self.returncode = _winapi.GetExitCodeProcess(self._handle)
@@ -1498,9 +1558,8 @@ def _try_wait(self, wait_flags):
             return (pid, sts)
 
 
-        def wait(self, timeout=None):
-            """Wait for child process to terminate.  Returns returncode
-            attribute."""
+        def _wait(self, timeout):
+            """Internal implementation of wait() on POSIX."""
             if self.returncode is not None:
                 return self.returncode
 
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 2e2721e5e998..dd63818f3b2f 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -2921,6 +2921,71 @@ def test_terminate_dead(self):
         self._kill_dead_process('terminate')
 
 class MiscTests(unittest.TestCase):
+
+    class RecordingPopen(subprocess.Popen):
+        """A Popen that saves a reference to each instance for testing."""
+        instances_created = []
+
+        def __init__(self, *args, **kwargs):
+            super().__init__(*args, **kwargs)
+            self.instances_created.append(self)
+
+    @mock.patch.object(subprocess.Popen, "_communicate")
+    def _test_keyboardinterrupt_no_kill(self, popener, mock__communicate,
+                                        **kwargs):
+        """Fake a SIGINT happening during Popen._communicate() and ._wait().
+
+        This avoids the need to actually try and get test environments to send
+        and receive signals reliably across platforms.  The net effect of a ^C
+        happening during a blocking subprocess execution which we want to clean
+        up from is a KeyboardInterrupt coming out of communicate() or wait().
+        """
+
+        mock__communicate.side_effect = KeyboardInterrupt
+        try:
+            with mock.patch.object(subprocess.Popen, "_wait") as mock__wait:
+                # We patch out _wait() as no signal was involved so the
+                # child process isn't actually going to exit rapidly.
+                mock__wait.side_effect = KeyboardInterrupt
+                with mock.patch.object(subprocess, "Popen",
+                                       self.RecordingPopen):
+                    with self.assertRaises(KeyboardInterrupt):
+                        popener([sys.executable, "-c",
+                                 "import time\ntime.sleep(9)\nimport sys\n"
+                                 "sys.stderr.write('\\n!runaway child!\\n')"],
+                                stdout=subprocess.DEVNULL, **kwargs)
+                for call in mock__wait.call_args_list[1:]:
+                    self.assertNotEqual(
+                            call, mock.call(timeout=None),
+                            "no open-ended wait() after the first allowed: "
+                            f"{mock__wait.call_args_list}")
+                sigint_calls = []
+                for call in mock__wait.call_args_list:
+                    if call == mock.call(timeout=0.25):  # from Popen.__init__
+                        sigint_calls.append(call)
+                self.assertLessEqual(mock__wait.call_count, 2,
+                                     msg=mock__wait.call_args_list)
+                self.assertEqual(len(sigint_calls), 1,
+                                 msg=mock__wait.call_args_list)
+        finally:
+            # cleanup the forgotten (due to our mocks) child process
+            process = self.RecordingPopen.instances_created.pop()
+            process.kill()
+            process.wait()
+            self.assertEqual([], self.RecordingPopen.instances_created)
+
+    def test_call_keyboardinterrupt_no_kill(self):
+        self._test_keyboardinterrupt_no_kill(subprocess.call, timeout=6.282)
+
+    def test_run_keyboardinterrupt_no_kill(self):
+        self._test_keyboardinterrupt_no_kill(subprocess.run, timeout=6.282)
+
+    def test_context_manager_keyboardinterrupt_no_kill(self):
+        def popen_via_context_manager(*args, **kwargs):
+            with subprocess.Popen(*args, **kwargs) as unused_process:
+                raise KeyboardInterrupt  # Test how __exit__ handles ^C.
+        self._test_keyboardinterrupt_no_kill(popen_via_context_manager)
+
     def test_getoutput(self):
         self.assertEqual(subprocess.getoutput('echo xyzzy'), 'xyzzy')
         self.assertEqual(subprocess.getstatusoutput('echo xyzzy'),
diff --git a/Misc/NEWS.d/next/Library/2017-12-27-20-15-51.bpo-25942.Giyr8v.rst b/Misc/NEWS.d/next/Library/2017-12-27-20-15-51.bpo-25942.Giyr8v.rst
new file mode 100644
index 000000000000..b898345b2b32
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-27-20-15-51.bpo-25942.Giyr8v.rst
@@ -0,0 +1,6 @@
+The subprocess module is now more graceful when handling a Ctrl-C
+KeyboardInterrupt during subprocess.call, subprocess.run, or a Popen context
+manager.  It now waits a short amount of time for the child (presumed to
+have also gotten the SIGINT) to exit, before continuing the
+KeyboardInterrupt exception handling.  This still includes a SIGKILL in the
+call() and run() APIs, but at least the child had a chance first.


From webhook-mailer at python.org  Tue Jan 30 00:27:44 2018
From: webhook-mailer at python.org (Nick Coghlan)
Date: Tue, 30 Jan 2018 05:27:44 -0000
Subject: [Python-checkins] [3.6] Revert "bpo-32690: Preserve order of
 locals() (GH-5379) (#5390)"
Message-ID: <mailman.72.1517290067.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/05f91a42cd9575ef338a67bd04855b44f6ac20c2
commit: 05f91a42cd9575ef338a67bd04855b44f6ac20c2
branch: 3.6
author: Nick Coghlan <ncoghlan at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-30T15:27:41+10:00
summary:

[3.6] Revert "bpo-32690: Preserve order of locals() (GH-5379) (#5390)"

This reverts commit 9105879bfd7133ecbac67f3e9c0bacf6e477de5a
in order to keep the behaviour of locals() consistent between
3.6.4 and 3.6.5+.

files:
D Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst
M Objects/frameobject.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst
deleted file mode 100644
index 1663b96cb927..000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2018-01-28-09-26-07.bpo-32690.8i9g5P.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-The locals() dictionary now displays in the lexical order that variables
-were defined.  Previously, the order was reversed.
diff --git a/Objects/frameobject.c b/Objects/frameobject.c
index ca1fbfa1a28f..62f9f34c8ebf 100644
--- a/Objects/frameobject.c
+++ b/Objects/frameobject.c
@@ -776,7 +776,7 @@ map_to_dict(PyObject *map, Py_ssize_t nmap, PyObject *dict, PyObject **values,
     assert(PyTuple_Check(map));
     assert(PyDict_Check(dict));
     assert(PyTuple_Size(map) >= nmap);
-    for (j=0; j < nmap; j++) {
+    for (j = nmap; --j >= 0; ) {
         PyObject *key = PyTuple_GET_ITEM(map, j);
         PyObject *value = values[j];
         assert(PyUnicode_Check(key));
@@ -829,7 +829,7 @@ dict_to_map(PyObject *map, Py_ssize_t nmap, PyObject *dict, PyObject **values,
     assert(PyTuple_Check(map));
     assert(PyDict_Check(dict));
     assert(PyTuple_Size(map) >= nmap);
-    for (j=0; j < nmap; j++) {
+    for (j = nmap; --j >= 0; ) {
         PyObject *key = PyTuple_GET_ITEM(map, j);
         PyObject *value = PyObject_GetItem(dict, key);
         assert(PyUnicode_Check(key));


From webhook-mailer at python.org  Tue Jan 30 00:54:10 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 05:54:10 -0000
Subject: [Python-checkins] bpo-21417: Add compresslevel= to the zipfile
 module (GH-5385)
Message-ID: <mailman.73.1517291652.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/ce237c7d58ba207575cdfb0195a58a6407fbf717
commit: ce237c7d58ba207575cdfb0195a58a6407fbf717
branch: master
author: Bo Bayles <bbayles at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T21:54:07-08:00
summary:

bpo-21417: Add compresslevel= to the zipfile module (GH-5385)

This allows the compression level to be specified when writing zipfiles
(for the entire file *and* overridden on a per-file basis).

Contributed by Bo Bayles

files:
A Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst
M Doc/library/zipfile.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst
index 5b8c776ed648..d58efe0b4175 100644
--- a/Doc/library/zipfile.rst
+++ b/Doc/library/zipfile.rst
@@ -130,10 +130,12 @@ ZipFile Objects
 ---------------
 
 
-.. class:: ZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=True)
+.. class:: ZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=True, \
+                   compresslevel=None)
 
    Open a ZIP file, where *file* can be a path to a file (a string), a
    file-like object or a :term:`path-like object`.
+
    The *mode* parameter should be ``'r'`` to read an existing
    file, ``'w'`` to truncate and write a new file, ``'a'`` to append to an
    existing file, or ``'x'`` to exclusively create and write a new file.
@@ -145,16 +147,27 @@ ZipFile Objects
    adding a ZIP archive to another file (such as :file:`python.exe`).  If
    *mode* is ``'a'`` and the file does not exist at all, it is created.
    If *mode* is ``'r'`` or ``'a'``, the file should be seekable.
+
    *compression* is the ZIP compression method to use when writing the archive,
    and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`,
    :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized
-   values will cause :exc:`NotImplementedError` to be raised.  If :const:`ZIP_DEFLATED`,
-   :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponding module
-   (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available, :exc:`RuntimeError`
-   is raised. The default is :const:`ZIP_STORED`.  If *allowZip64* is
-   ``True`` (the default) zipfile will create ZIP files that use the ZIP64
-   extensions when the zipfile is larger than 4 GiB. If it is  false :mod:`zipfile`
-   will raise an exception when the ZIP file would require ZIP64 extensions.
+   values will cause :exc:`NotImplementedError` to be raised.  If
+   :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified
+   but the corresponding module (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not
+   available, :exc:`RuntimeError` is raised. The default is :const:`ZIP_STORED`.
+
+   If *allowZip64* is ``True`` (the default) zipfile will create ZIP files that
+   use the ZIP64 extensions when the zipfile is larger than 4 GiB. If it is
+   ``false`` :mod:`zipfile` will raise an exception when the ZIP file would
+   require ZIP64 extensions.
+
+   The *compresslevel* parameter controls the compression level to use when
+   writing files to the archive.
+   When using :const:`ZIP_STORED` or :const:`ZIP_LZMA` it has no effect.
+   When using :const:`ZIP_DEFLATED` integers ``0`` through ``9`` are accepted
+   (see :class:`zlib <zlib.compressobj>` for more information).
+   When using :const:`ZIP_BZIP2` integers ``1`` through ``9`` are accepted
+   (see :class:`bz2 <bz2.BZ2File>` for more information).
 
    If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then
    :meth:`closed <close>` without adding any files to the archive, the appropriate
@@ -187,6 +200,9 @@ ZipFile Objects
    .. versionchanged:: 3.6.2
       The *file* parameter accepts a :term:`path-like object`.
 
+   .. versionchanged:: 3.7
+      Add the *compresslevel* parameter.
+
 
 .. method:: ZipFile.close()
 
@@ -351,13 +367,15 @@ ZipFile Objects
       :exc:`ValueError`.  Previously, a :exc:`RuntimeError` was raised.
 
 
-.. method:: ZipFile.write(filename, arcname=None, compress_type=None)
+.. method:: ZipFile.write(filename, arcname=None, compress_type=None, \
+                          compresslevel=None)
 
    Write the file named *filename* to the archive, giving it the archive name
    *arcname* (by default, this will be the same as *filename*, but without a drive
    letter and with leading path separators removed).  If given, *compress_type*
    overrides the value given for the *compression* parameter to the constructor for
-   the new entry.
+   the new entry. Similarly, *compresslevel* will override the constructor if
+   given.
    The archive must be open with mode ``'w'``, ``'x'`` or ``'a'``.
 
    .. note::
@@ -383,7 +401,8 @@ ZipFile Objects
       a :exc:`RuntimeError` was raised.
 
 
-.. method:: ZipFile.writestr(zinfo_or_arcname, data[, compress_type])
+.. method:: ZipFile.writestr(zinfo_or_arcname, data, compress_type=None, \
+                             compresslevel=None)
 
    Write the string *data* to the archive; *zinfo_or_arcname* is either the file
    name it will be given in the archive, or a :class:`ZipInfo` instance.  If it's
@@ -393,7 +412,8 @@ ZipFile Objects
 
    If given, *compress_type* overrides the value given for the *compression*
    parameter to the constructor for the new entry, or in the *zinfo_or_arcname*
-   (if that is a :class:`ZipInfo` instance).
+   (if that is a :class:`ZipInfo` instance). Similarly, *compresslevel* will
+   override the constructor if given.
 
    .. note::
 
diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index 3bc867ea51c9..94db858a1517 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -53,9 +53,10 @@ def setUp(self):
         with open(TESTFN, "wb") as fp:
             fp.write(self.data)
 
-    def make_test_archive(self, f, compression):
+    def make_test_archive(self, f, compression, compresslevel=None):
+        kwargs = {'compression': compression, 'compresslevel': compresslevel}
         # Create the ZIP archive
-        with zipfile.ZipFile(f, "w", compression) as zipfp:
+        with zipfile.ZipFile(f, "w", **kwargs) as zipfp:
             zipfp.write(TESTFN, "another.name")
             zipfp.write(TESTFN, TESTFN)
             zipfp.writestr("strfile", self.data)
@@ -63,8 +64,8 @@ def make_test_archive(self, f, compression):
                 for line in self.line_gen:
                     f.write(line)
 
-    def zip_test(self, f, compression):
-        self.make_test_archive(f, compression)
+    def zip_test(self, f, compression, compresslevel=None):
+        self.make_test_archive(f, compression, compresslevel)
 
         # Read the ZIP archive
         with zipfile.ZipFile(f, "r", compression) as zipfp:
@@ -297,6 +298,22 @@ def test_writestr_compression(self):
         info = zipfp.getinfo('b.txt')
         self.assertEqual(info.compress_type, self.compression)
 
+    def test_writestr_compresslevel(self):
+        zipfp = zipfile.ZipFile(TESTFN2, "w", compresslevel=1)
+        zipfp.writestr("a.txt", "hello world", compress_type=self.compression)
+        zipfp.writestr("b.txt", "hello world", compress_type=self.compression,
+                       compresslevel=2)
+
+        # Compression level follows the constructor.
+        a_info = zipfp.getinfo('a.txt')
+        self.assertEqual(a_info.compress_type, self.compression)
+        self.assertEqual(a_info._compresslevel, 1)
+
+        # Compression level is overridden.
+        b_info = zipfp.getinfo('b.txt')
+        self.assertEqual(b_info.compress_type, self.compression)
+        self.assertEqual(b_info._compresslevel, 2)
+
     def test_read_return_size(self):
         # Issue #9837: ZipExtFile.read() shouldn't return more bytes
         # than requested.
@@ -370,6 +387,21 @@ def test_repr(self):
                 self.assertIn('[closed]', repr(zipopen))
             self.assertIn('[closed]', repr(zipfp))
 
+    def test_compresslevel_basic(self):
+        for f in get_files(self):
+            self.zip_test(f, self.compression, compresslevel=9)
+
+    def test_per_file_compresslevel(self):
+        """Check that files within a Zip archive can have different
+        compression levels."""
+        with zipfile.ZipFile(TESTFN2, "w", compresslevel=1) as zipfp:
+            zipfp.write(TESTFN, 'compress_1')
+            zipfp.write(TESTFN, 'compress_9', compresslevel=9)
+            one_info = zipfp.getinfo('compress_1')
+            nine_info = zipfp.getinfo('compress_9')
+            self.assertEqual(one_info._compresslevel, 1)
+            self.assertEqual(nine_info._compresslevel, 9)
+
     def tearDown(self):
         unlink(TESTFN)
         unlink(TESTFN2)
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 37ce3281e092..f9db45f58a2b 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -295,6 +295,7 @@ class ZipInfo (object):
         'filename',
         'date_time',
         'compress_type',
+        '_compresslevel',
         'comment',
         'extra',
         'create_system',
@@ -334,6 +335,7 @@ def __init__(self, filename="NoName", date_time=(1980,1,1,0,0,0)):
 
         # Standard values:
         self.compress_type = ZIP_STORED # Type of compression for the file
+        self._compresslevel = None      # Level for the compressor
         self.comment = b""              # Comment for each file
         self.extra = b""                # ZIP extra data
         if sys.platform == 'win32':
@@ -654,12 +656,16 @@ def _check_compression(compression):
         raise NotImplementedError("That compression method is not supported")
 
 
-def _get_compressor(compress_type):
+def _get_compressor(compress_type, compresslevel=None):
     if compress_type == ZIP_DEFLATED:
-        return zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION,
-                                zlib.DEFLATED, -15)
+        if compresslevel is not None:
+            return zlib.compressobj(compresslevel, zlib.DEFLATED, -15)
+        return zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, -15)
     elif compress_type == ZIP_BZIP2:
+        if compresslevel is not None:
+            return bz2.BZ2Compressor(compresslevel)
         return bz2.BZ2Compressor()
+    # compresslevel is ignored for ZIP_LZMA
     elif compress_type == ZIP_LZMA:
         return LZMACompressor()
     else:
@@ -963,7 +969,8 @@ def __init__(self, zf, zinfo, zip64):
         self._zinfo = zinfo
         self._zip64 = zip64
         self._zipfile = zf
-        self._compressor = _get_compressor(zinfo.compress_type)
+        self._compressor = _get_compressor(zinfo.compress_type,
+                                           zinfo._compresslevel)
         self._file_size = 0
         self._compress_size = 0
         self._crc = 0
@@ -1035,7 +1042,8 @@ def close(self):
 class ZipFile:
     """ Class with methods to open, read, write, close, list zip files.
 
-    z = ZipFile(file, mode="r", compression=ZIP_STORED, allowZip64=True)
+    z = ZipFile(file, mode="r", compression=ZIP_STORED, allowZip64=True,
+                compresslevel=None)
 
     file: Either the path to the file, or a file-like object.
           If it is a path, the file will be opened and closed by ZipFile.
@@ -1046,13 +1054,19 @@ class ZipFile:
     allowZip64: if True ZipFile will create files with ZIP64 extensions when
                 needed, otherwise it will raise an exception when this would
                 be necessary.
+    compresslevel: None (default for the given compression type) or an integer
+                   specifying the level to pass to the compressor.
+                   When using ZIP_STORED or ZIP_LZMA this keyword has no effect.
+                   When using ZIP_DEFLATED integers 0 through 9 are accepted.
+                   When using ZIP_BZIP2 integers 1 through 9 are accepted.
 
     """
 
     fp = None                   # Set here since __del__ checks it
     _windows_illegal_name_trans_table = None
 
-    def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True):
+    def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True,
+                 compresslevel=None):
         """Open the ZIP file with mode read 'r', write 'w', exclusive create 'x',
         or append 'a'."""
         if mode not in ('r', 'w', 'x', 'a'):
@@ -1066,6 +1080,7 @@ def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True):
         self.NameToInfo = {}    # Find file info given name
         self.filelist = []      # List of ZipInfo instances for archive
         self.compression = compression  # Method of compression
+        self.compresslevel = compresslevel
         self.mode = mode
         self.pwd = None
         self._comment = b''
@@ -1342,6 +1357,7 @@ def open(self, name, mode="r", pwd=None, *, force_zip64=False):
         elif mode == 'w':
             zinfo = ZipInfo(name)
             zinfo.compress_type = self.compression
+            zinfo._compresslevel = self.compresslevel
         else:
             # Get info object for name
             zinfo = self.getinfo(name)
@@ -1575,7 +1591,8 @@ def _writecheck(self, zinfo):
                 raise LargeZipFile(requires_zip64 +
                                    " would require ZIP64 extensions")
 
-    def write(self, filename, arcname=None, compress_type=None):
+    def write(self, filename, arcname=None,
+              compress_type=None, compresslevel=None):
         """Put the bytes from filename into the archive under the name
         arcname."""
         if not self.fp:
@@ -1597,6 +1614,11 @@ def write(self, filename, arcname=None, compress_type=None):
             else:
                 zinfo.compress_type = self.compression
 
+            if compresslevel is not None:
+                zinfo._compresslevel = compresslevel
+            else:
+                zinfo._compresslevel = self.compresslevel
+
         if zinfo.is_dir():
             with self._lock:
                 if self._seekable:
@@ -1617,7 +1639,8 @@ def write(self, filename, arcname=None, compress_type=None):
             with open(filename, "rb") as src, self.open(zinfo, 'w') as dest:
                 shutil.copyfileobj(src, dest, 1024*8)
 
-    def writestr(self, zinfo_or_arcname, data, compress_type=None):
+    def writestr(self, zinfo_or_arcname, data,
+                 compress_type=None, compresslevel=None):
         """Write a file into the archive.  The contents is 'data', which
         may be either a 'str' or a 'bytes' instance; if it is a 'str',
         it is encoded as UTF-8 first.
@@ -1629,6 +1652,7 @@ def writestr(self, zinfo_or_arcname, data, compress_type=None):
             zinfo = ZipInfo(filename=zinfo_or_arcname,
                             date_time=time.localtime(time.time())[:6])
             zinfo.compress_type = self.compression
+            zinfo._compresslevel = self.compresslevel
             if zinfo.filename[-1] == '/':
                 zinfo.external_attr = 0o40775 << 16   # drwxrwxr-x
                 zinfo.external_attr |= 0x10           # MS-DOS directory flag
@@ -1648,6 +1672,9 @@ def writestr(self, zinfo_or_arcname, data, compress_type=None):
         if compress_type is not None:
             zinfo.compress_type = compress_type
 
+        if compresslevel is not None:
+            zinfo._compresslevel = compresslevel
+
         zinfo.file_size = len(data)            # Uncompressed size
         with self._lock:
             with self.open(zinfo, mode='w') as dest:
diff --git a/Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst b/Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst
new file mode 100644
index 000000000000..50207a0e4c33
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-28-07-55-10.bpo-21417.JFnV99.rst
@@ -0,0 +1 @@
+Added support for setting the compression level for zipfile.ZipFile.


From webhook-mailer at python.org  Tue Jan 30 01:02:11 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 06:02:11 -0000
Subject: [Python-checkins] [3.6] replace dynamic import with 'exec' with
 importlib.import_module (GH-5433) (GH-5440)
Message-ID: <mailman.74.1517292132.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/88fa79a10a340439ed021fbc4988b6f3e73b2391
commit: 88fa79a10a340439ed021fbc4988b6f3e73b2391
branch: 3.6
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T22:02:09-08:00
summary:

[3.6] replace dynamic import with 'exec' with importlib.import_module (GH-5433) (GH-5440)

(cherry picked from commit 77526f05fa788d6fb12f2121fe6b96c130d9b717)

files:
M Lib/test/test_hashlib.py

diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py
index 981748892efb..28613496d111 100644
--- a/Lib/test/test_hashlib.py
+++ b/Lib/test/test_hashlib.py
@@ -9,6 +9,7 @@
 import array
 from binascii import unhexlify
 import hashlib
+import importlib
 import itertools
 import os
 import sys
@@ -86,11 +87,11 @@ class HashLibTestCase(unittest.TestCase):
     def _conditional_import_module(self, module_name):
         """Import a module and return a reference to it or None on failure."""
         try:
-            exec('import '+module_name)
-        except ImportError as error:
+            return importlib.import_module(module_name)
+        except ModuleNotFoundError as error:
             if self._warn_on_extension_import:
                 warnings.warn('Did a C extension fail to compile? %s' % error)
-        return locals().get(module_name)
+        return None
 
     def __init__(self, *args, **kwargs):
         algorithms = set()


From webhook-mailer at python.org  Tue Jan 30 01:04:39 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 06:04:39 -0000
Subject: [Python-checkins] bpo-32681: Fix an uninitialized variable in the C
 implementation of os.dup2 (GH-5346)
Message-ID: <mailman.75.1517292282.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/3d86e484de6334fe16cbab512744597bd0de4e80
commit: 3d86e484de6334fe16cbab512744597bd0de4e80
branch: master
author: St?phane Wirtel <stephane at wirtel.be>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T22:04:36-08:00
summary:

bpo-32681: Fix an uninitialized variable in the C implementation of os.dup2 (GH-5346)

See https://bugs.python.org/issue32441 for where this was introduced.

files:
A Misc/NEWS.d/next/C API/2018-01-26-17-29-29.bpo-32681.N1ruWa.rst
M Modules/posixmodule.c

diff --git a/Misc/NEWS.d/next/C API/2018-01-26-17-29-29.bpo-32681.N1ruWa.rst b/Misc/NEWS.d/next/C API/2018-01-26-17-29-29.bpo-32681.N1ruWa.rst
new file mode 100644
index 000000000000..1506ec797c4b
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2018-01-26-17-29-29.bpo-32681.N1ruWa.rst	
@@ -0,0 +1,2 @@
+Fix uninitialized variable 'res' in the C implementation of os.dup2. Patch
+by St?phane Wirtel
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 46f3adaf4dc3..4a1c9f398a07 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -8012,7 +8012,7 @@ static int
 os_dup2_impl(PyObject *module, int fd, int fd2, int inheritable)
 /*[clinic end generated code: output=bc059d34a73404d1 input=c3cddda8922b038d]*/
 {
-    int res;
+    int res = 0;
 #if defined(HAVE_DUP3) && \
     !(defined(HAVE_FCNTL_H) && defined(F_DUP2FD_CLOEXEC))
     /* dup3() is available on Linux 2.6.27+ and glibc 2.9 */


From webhook-mailer at python.org  Tue Jan 30 01:14:21 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 06:14:21 -0000
Subject: [Python-checkins] closes bpo-32721: do not fail test_hashlib if
 _md5 isn't available (GH-5441)
Message-ID: <mailman.76.1517292862.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/95441809ef77a8df5e14601ade6c054ef7114c02
commit: 95441809ef77a8df5e14601ade6c054ef7114c02
branch: master
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-29T22:14:17-08:00
summary:

closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441)

files:
A Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst
M Lib/test/test_hashlib.py

diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py
index 15fc22b02d77..751f74813d35 100644
--- a/Lib/test/test_hashlib.py
+++ b/Lib/test/test_hashlib.py
@@ -193,7 +193,7 @@ def test_get_builtin_constructor(self):
         try:
             import _md5
         except ImportError:
-            pass
+            self.skipTest("_md5 module not available")
         # This forces an ImportError for "import _md5" statements
         sys.modules['_md5'] = None
         # clear the cache
diff --git a/Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst b/Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst
new file mode 100644
index 000000000000..f4bf6aa1fc4a
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst
@@ -0,0 +1 @@
+Fix test_hashlib to not fail if the _md5 module is not built.


From webhook-mailer at python.org  Tue Jan 30 01:40:41 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 06:40:41 -0000
Subject: [Python-checkins] bpo-32102 Add "capture_output=True" to
 subprocess.run (GH-5149)
Message-ID: <mailman.77.1517294445.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/ce0f33d04528fcafc673a8707871f8430d8f7ce8
commit: ce0f33d04528fcafc673a8707871f8430d8f7ce8
branch: master
author: Bo Bayles <bbayles at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T22:40:39-08:00
summary:

bpo-32102 Add "capture_output=True" to subprocess.run (GH-5149)

Add "capture_output=True" option to subprocess.run, this is equivalent to
setting stdout=PIPE, stderr=PIPE but is much more readable.

files:
A Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst
M Doc/library/subprocess.rst
M Lib/subprocess.py
M Lib/test/test_subprocess.py
M Misc/ACKS

diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
index af96f41ef40a..86f3e06318f4 100644
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -47,12 +47,14 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
    The arguments shown above are merely the most common ones, described below
    in :ref:`frequently-used-arguments` (hence the use of keyword-only notation
    in the abbreviated signature). The full function signature is largely the
-   same as that of the :class:`Popen` constructor - apart from *timeout*,
-   *input* and *check*, all the arguments to this function are passed through to
-   that interface.
+   same as that of the :class:`Popen` constructor - most of the arguments to
+   this function are passed through to that interface. (*timeout*,  *input*,
+   *check*, and *capture_output* are not.)
 
-   This does not capture stdout or stderr by default. To do so, pass
-   :data:`PIPE` for the *stdout* and/or *stderr* arguments.
+   If *capture_output* is true, stdout and stderr will be captured.
+   When used, the internal :class:`Popen` object is automatically created with
+   ``stdout=PIPE`` and ``stderr=PIPE``. The *stdout* and *stderr* arguments may
+   not be used as well.
 
    The *timeout* argument is passed to :meth:`Popen.communicate`. If the timeout
    expires, the child process will be killed and waited for.  The
@@ -86,9 +88,9 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
         ...
       subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
 
-      >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
+      >>> subprocess.run(["ls", "-l", "/dev/null"], capture_output=True)
       CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
-      stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')
+      stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n', stderr=b'')
 
    .. versionadded:: 3.5
 
@@ -98,7 +100,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
 
    .. versionchanged:: 3.7
 
-      Added the *text* parameter, as a more understandable alias of *universal_newlines*
+      Added the *text* parameter, as a more understandable alias of *universal_newlines*.
+      Added the *capture_output* parameter.
 
 .. class:: CompletedProcess
 
diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index f69159e3aadc..93635ee61f7e 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -409,7 +409,8 @@ def check_returncode(self):
                                      self.stderr)
 
 
-def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
+def run(*popenargs,
+        input=None, capture_output=False, timeout=None, check=False, **kwargs):
     """Run command with arguments and return a CompletedProcess instance.
 
     The returned instance will have attributes args, returncode, stdout and
@@ -442,6 +443,13 @@ def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
             raise ValueError('stdin and input arguments may not both be used.')
         kwargs['stdin'] = PIPE
 
+    if capture_output:
+        if ('stdout' in kwargs) or ('stderr' in kwargs):
+            raise ValueError('stdout and stderr arguments may not be used '
+                             'with capture_output.')
+        kwargs['stdout'] = PIPE
+        kwargs['stderr'] = PIPE
+
     with Popen(*popenargs, **kwargs) as process:
         try:
             stdout, stderr = process.communicate(input, timeout=timeout)
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index dd63818f3b2f..eee24bbe37cf 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -1475,6 +1475,38 @@ def test_run_kwargs(self):
                              env=newenv)
         self.assertEqual(cp.returncode, 33)
 
+    def test_capture_output(self):
+        cp = self.run_python(("import sys;"
+                              "sys.stdout.write('BDFL'); "
+                              "sys.stderr.write('FLUFL')"),
+                             capture_output=True)
+        self.assertIn(b'BDFL', cp.stdout)
+        self.assertIn(b'FLUFL', cp.stderr)
+
+    def test_stdout_with_capture_output_arg(self):
+        # run() refuses to accept 'stdout' with 'capture_output'
+        tf = tempfile.TemporaryFile()
+        self.addCleanup(tf.close)
+        with self.assertRaises(ValueError,
+            msg=("Expected ValueError when stdout and capture_output "
+                 "args supplied.")) as c:
+            output = self.run_python("print('will not be run')",
+                                      capture_output=True, stdout=tf)
+        self.assertIn('stdout', c.exception.args[0])
+        self.assertIn('capture_output', c.exception.args[0])
+
+    def test_stderr_with_capture_output_arg(self):
+        # run() refuses to accept 'stderr' with 'capture_output'
+        tf = tempfile.TemporaryFile()
+        self.addCleanup(tf.close)
+        with self.assertRaises(ValueError,
+            msg=("Expected ValueError when stderr and capture_output "
+                 "args supplied.")) as c:
+            output = self.run_python("print('will not be run')",
+                                      capture_output=True, stderr=tf)
+        self.assertIn('stderr', c.exception.args[0])
+        self.assertIn('capture_output', c.exception.args[0])
+
 
 @unittest.skipIf(mswindows, "POSIX specific tests")
 class POSIXProcessTestCase(BaseTestCase):
diff --git a/Misc/ACKS b/Misc/ACKS
index bfcdd33b8ed7..3fdfa3041f87 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -108,6 +108,7 @@ Michael R Bax
 Anthony Baxter
 Mike Bayer
 Samuel L. Bayer
+Bo Bayles
 Tommy Beadle
 Donald Beaudry
 David Beazley
diff --git a/Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst b/Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst
new file mode 100644
index 000000000000..cd4d0b5a8bd6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-10-18-04-21.bpo-32102.9-CZgD.rst
@@ -0,0 +1 @@
+New argument ``capture_output`` for subprocess.run


From webhook-mailer at python.org  Tue Jan 30 02:01:35 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 07:01:35 -0000
Subject: [Python-checkins] closes bpo-32721: do not fail test_hashlib if
 _md5 isn't available (GH-5441) (GH-5442)
Message-ID: <mailman.78.1517295696.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/6b2bbcc4cca414f35f67caa4674f59f41ff638ea
commit: 6b2bbcc4cca414f35f67caa4674f59f41ff638ea
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2018-01-29T23:01:32-08:00
summary:

closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441) (GH-5442)

(cherry picked from commit 95441809ef77a8df5e14601ade6c054ef7114c02)

Co-authored-by: Benjamin Peterson <benjamin at python.org>

files:
A Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst
M Lib/test/test_hashlib.py

diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py
index 28613496d111..177143e01dc7 100644
--- a/Lib/test/test_hashlib.py
+++ b/Lib/test/test_hashlib.py
@@ -196,7 +196,7 @@ def test_get_builtin_constructor(self):
         try:
             import _md5
         except ImportError:
-            pass
+            self.skipTest("_md5 module not available")
         # This forces an ImportError for "import _md5" statements
         sys.modules['_md5'] = None
         # clear the cache
diff --git a/Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst b/Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst
new file mode 100644
index 000000000000..f4bf6aa1fc4a
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-01-29-21-30-44.bpo-32721.2Bebm1.rst
@@ -0,0 +1 @@
+Fix test_hashlib to not fail if the _md5 module is not built.


From webhook-mailer at python.org  Tue Jan 30 02:13:06 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 07:13:06 -0000
Subject: [Python-checkins] closes bpo-30117: fix lib2to3 ParserIdempotency
 test (GH-1242)
Message-ID: <mailman.79.1517296389.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/14e976e00e65bf343ba0fca016c3c9132a843daf
commit: 14e976e00e65bf343ba0fca016c3c9132a843daf
branch: master
author: Eric Appelt <eric.appelt at gmail.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2018-01-29T23:13:03-08:00
summary:

closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)

Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.

    Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

    If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.

files:
M Lib/lib2to3/tests/support.py
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/tests/support.py b/Lib/lib2to3/tests/support.py
index ae7cfe8ee274..fe084e8903fc 100644
--- a/Lib/lib2to3/tests/support.py
+++ b/Lib/lib2to3/tests/support.py
@@ -15,7 +15,13 @@
 proj_dir = os.path.normpath(os.path.join(test_dir, ".."))
 grammar_path = os.path.join(test_dir, "..", "Grammar.txt")
 grammar = pgen2_driver.load_grammar(grammar_path)
+grammar_no_print_statement = pgen2_driver.load_grammar(grammar_path)
+del grammar_no_print_statement.keywords["print"]
 driver = pgen2_driver.Driver(grammar, convert=pytree.convert)
+driver_no_print_statement = pgen2_driver.Driver(
+    grammar_no_print_statement,
+    convert=pytree.convert
+)
 
 def parse_string(string):
     return driver.parse_string(reformat(string), debug=True)
diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index ed1ef8364639..15f5b63f1f1e 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -8,7 +8,7 @@
 
 # Testing imports
 from . import support
-from .support import driver
+from .support import driver, driver_no_print_statement
 from test.support import verbose
 
 # Python imports
@@ -413,8 +413,6 @@ class TestParserIdempotency(support.TestCase):
 
     """A cut-down version of pytree_idempotency.py."""
 
-    # Issue 13125
-    @unittest.expectedFailure
     def test_all_project_files(self):
         for filepath in support.all_project_files():
             with open(filepath, "rb") as fp:
@@ -425,12 +423,13 @@ def test_all_project_files(self):
                 source = fp.read()
             try:
                 tree = driver.parse_string(source)
-            except ParseError as err:
-                if verbose > 0:
-                    warnings.warn('ParseError on file %s (%s)' % (filepath, err))
-                continue
+            except ParseError:
+                try:
+                    tree = driver_no_print_statement.parse_string(source)
+                except ParseError as err:
+                    self.fail('ParseError on file %s (%s)' % (filepath, err))
             new = str(tree)
-            x = diff(filepath, new)
+            x = diff(filepath, new, encoding=encoding)
             if x:
                 self.fail("Idempotency failed: %s" % filepath)
 
@@ -481,9 +480,9 @@ def test_trailing_comma_after_generator_expression_argument_works(self):
         self.validate("set(x for x in [],)")
 
 
-def diff(fn, result):
+def diff(fn, result, encoding='utf-8'):
     try:
-        with open('@', 'w') as f:
+        with open('@', 'w', encoding=encoding, newline='\n') as f:
             f.write(str(result))
         fn = fn.replace('"', '\\"')
         return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))


From webhook-mailer at python.org  Tue Jan 30 02:27:31 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 07:27:31 -0000
Subject: [Python-checkins] bpo-31961: subprocess now accepts path-like args
 (GH-4329)
Message-ID: <mailman.80.1517297253.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/dd42cb71f2cb02f3a32f016137b12a146bc0d0e2
commit: dd42cb71f2cb02f3a32f016137b12a146bc0d0e2
branch: master
author: Anders Lorentsen <Phaqui at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-29T23:27:28-08:00
summary:

bpo-31961: subprocess now accepts path-like args (GH-4329)

Allow os.PathLike args in subprocess APIs.

files:
A Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst
M Doc/library/subprocess.rst
M Lib/subprocess.py
M Lib/test/test_subprocess.py

diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
index 86f3e06318f4..27d4288f67b3 100644
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -335,12 +335,12 @@ functions.
    the class uses the Windows ``CreateProcess()`` function.  The arguments to
    :class:`Popen` are as follows.
 
-   *args* should be a sequence of program arguments or else a single string.
-   By default, the program to execute is the first item in *args* if *args* is
-   a sequence.  If *args* is a string, the interpretation is
-   platform-dependent and described below.  See the *shell* and *executable*
-   arguments for additional differences from the default behavior.  Unless
-   otherwise stated, it is recommended to pass *args* as a sequence.
+   *args* should be a sequence of program arguments or else a single string or
+   :term:`path-like object`. By default, the program to execute is the first
+   item in *args* if *args* is a sequence. If *args* is a string, the
+   interpretation is platform-dependent and described below.  See the *shell*
+   and *executable* arguments for additional differences from the default
+   behavior.  Unless otherwise stated, it is recommended to pass *args* as a sequence.
 
    On POSIX, if *args* is a string, the string is interpreted as the name or
    path of the program to execute.  However, this can only be done if not
@@ -551,6 +551,10 @@ functions.
       Popen destructor now emits a :exc:`ResourceWarning` warning if the child
       process is still running.
 
+   .. versionchanged:: 3.7
+      *args*, or the first element of *args* if *args* is a sequence, can now
+      be a :term:`path-like object`.
+
 
 Exceptions
 ^^^^^^^^^^
diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index 93635ee61f7e..2723bc9e4274 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -1097,7 +1097,12 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
             assert not pass_fds, "pass_fds not supported on Windows."
 
             if not isinstance(args, str):
-                args = list2cmdline(args)
+                try:
+                    args = os.fsdecode(args)  # os.PathLike -> str
+                except TypeError:  # not an os.PathLike, must be a sequence.
+                    args = list(args)
+                    args[0] = os.fsdecode(args[0])  # os.PathLike -> str
+                    args = list2cmdline(args)
 
             # Process startup details
             if startupinfo is None:
@@ -1369,7 +1374,10 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
             if isinstance(args, (str, bytes)):
                 args = [args]
             else:
-                args = list(args)
+                try:
+                    args = list(args)
+                except TypeError:  # os.PathLike instead of a sequence?
+                    args = [os.fsencode(args)]  # os.PathLike -> [str]
 
             if shell:
                 # On Android the default shell is at '/system/bin/sh'.
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index eee24bbe37cf..858a70131205 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -1475,6 +1475,37 @@ def test_run_kwargs(self):
                              env=newenv)
         self.assertEqual(cp.returncode, 33)
 
+    def test_run_with_pathlike_path(self):
+        # bpo-31961: test run(pathlike_object)
+        class Path:
+            def __fspath__(self):
+                # the name of a command that can be run without
+                # any argumenets that exit fast
+                return 'dir' if mswindows else 'ls'
+
+        path = Path()
+        if mswindows:
+            res = subprocess.run(path, stdout=subprocess.DEVNULL, shell=True)
+        else:
+            res = subprocess.run(path, stdout=subprocess.DEVNULL)
+
+        self.assertEqual(res.returncode, 0)
+
+    def test_run_with_pathlike_path_and_arguments(self):
+        # bpo-31961: test run([pathlike_object, 'additional arguments'])
+        class Path:
+            def __fspath__(self):
+                # the name of a command that can be run without
+                # any argumenets that exits fast
+                return sys.executable
+
+        path = Path()
+
+        args = [path, '-c', 'import sys; sys.exit(57)']
+        res = subprocess.run(args)
+
+        self.assertEqual(res.returncode, 57)
+
     def test_capture_output(self):
         cp = self.run_python(("import sys;"
                               "sys.stdout.write('BDFL'); "
diff --git a/Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst b/Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst
new file mode 100644
index 000000000000..611f4e978f1a
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-11-08-03-38-20.bpo-31961.x5Sv0R.rst
@@ -0,0 +1,4 @@
+The *args* argument of subprocess.Popen can now be a
+:term:`path-like object`. If *args* is given as a
+sequence, it's first element can now be a
+:term:`path-like object` as well.


From webhook-mailer at python.org  Tue Jan 30 02:55:49 2018
From: webhook-mailer at python.org (Christian Heimes)
Date: Tue, 30 Jan 2018 07:55:49 -0000
Subject: [Python-checkins] bpo-28134: Ignore proto in unknown socket test
 (GH-5435)
Message-ID: <mailman.81.1517298951.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/2e0ecde8d74f5fc0e3e3e39216975cc70efc4796
commit: 2e0ecde8d74f5fc0e3e3e39216975cc70efc4796
branch: master
author: Christian Heimes <christian at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-30T08:55:46+01:00
summary:

bpo-28134: Ignore proto in unknown socket test (GH-5435)

Band-aid for macOS: Some platforms seem to ignore unknown protocols.

Signed-off-by: Christian Heimes <christian at python.org>

files:
M Lib/test/test_socket.py

diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 89cf1fa52009..2851922bd0c5 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1618,7 +1618,7 @@ def test_socket_consistent_sock_type(self):
             self.assertEqual(s.type, socket.SOCK_STREAM)
 
     @unittest.skipIf(os.name == 'nt', 'Will not work on Windows')
-    def test_uknown_socket_family_repr(self):
+    def test_unknown_socket_family_repr(self):
         # Test that when created with a family that's not one of the known
         # AF_*/SOCK_* constants, socket.family just returns the number.
         #
@@ -1642,7 +1642,8 @@ def test_uknown_socket_family_repr(self):
                 fileno=fd) as s:
             self.assertEqual(s.family, unknown_family)
             self.assertEqual(s.type, unknown_type)
-            self.assertEqual(s.proto, 23)
+            # some OS like macOS ignore proto
+            self.assertIn(s.proto, {0, 23})
 
     @unittest.skipUnless(hasattr(os, 'sendfile'), 'test needs os.sendfile()')
     def test__sendfile_use_sendfile(self):


From webhook-mailer at python.org  Tue Jan 30 03:51:43 2018
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Tue, 30 Jan 2018 08:51:43 -0000
Subject: [Python-checkins] bpo-22908: Add seek and tell functionality to
 ZipExtFile (GH-4966)
Message-ID: <mailman.82.1517302305.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/066df4fd454d6ff9be66e80b2a65995b10af174f
commit: 066df4fd454d6ff9be66e80b2a65995b10af174f
branch: master
author: John Jolly <john.jolly at gmail.com>
committer: Gregory P. Smith <greg at krypto.org>
date: 2018-01-30T00:51:35-08:00
summary:

bpo-22908: Add seek and tell functionality to ZipExtFile (GH-4966)

This allows for nested zip files, tar files within zip files, zip files within tar files, etc.

Contributed by: John Jolly

files:
A Misc/NEWS.d/next/Library/2017-12-21-22-00-11.bpo-22908.cVm89I.rst
M Doc/library/zipfile.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst
index d58efe0b4175..7c9a8c802254 100644
--- a/Doc/library/zipfile.rst
+++ b/Doc/library/zipfile.rst
@@ -246,9 +246,9 @@ ZipFile Objects
    With *mode* ``'r'`` the file-like object
    (``ZipExtFile``) is read-only and provides the following methods:
    :meth:`~io.BufferedIOBase.read`, :meth:`~io.IOBase.readline`,
-   :meth:`~io.IOBase.readlines`, :meth:`__iter__`,
-   :meth:`~iterator.__next__`.  These objects can operate independently of
-   the ZipFile.
+   :meth:`~io.IOBase.readlines`, :meth:`~io.IOBase.seek`,
+   :meth:`~io.IOBase.tell`, :meth:`__iter__`, :meth:`~iterator.__next__`.
+   These objects can operate independently of the ZipFile.
 
    With ``mode='w'``, a writable file handle is returned, which supports the
    :meth:`~io.BufferedIOBase.write` method.  While a writable file handle is open,
diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index 94db858a1517..61c3e349a69e 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1628,6 +1628,40 @@ def test_open_conflicting_handles(self):
             self.assertEqual(zipf.read('baz'), msg3)
             self.assertEqual(zipf.namelist(), ['foo', 'bar', 'baz'])
 
+    def test_seek_tell(self):
+        # Test seek functionality
+        txt = b"Where's Bruce?"
+        bloc = txt.find(b"Bruce")
+        # Check seek on a file
+        with zipfile.ZipFile(TESTFN, "w") as zipf:
+            zipf.writestr("foo.txt", txt)
+        with zipfile.ZipFile(TESTFN, "r") as zipf:
+            with zipf.open("foo.txt", "r") as fp:
+                fp.seek(bloc, os.SEEK_SET)
+                self.assertEqual(fp.tell(), bloc)
+                fp.seek(-bloc, os.SEEK_CUR)
+                self.assertEqual(fp.tell(), 0)
+                fp.seek(bloc, os.SEEK_CUR)
+                self.assertEqual(fp.tell(), bloc)
+                self.assertEqual(fp.read(5), txt[bloc:bloc+5])
+                fp.seek(0, os.SEEK_END)
+                self.assertEqual(fp.tell(), len(txt))
+        # Check seek on memory file
+        data = io.BytesIO()
+        with zipfile.ZipFile(data, mode="w") as zipf:
+            zipf.writestr("foo.txt", txt)
+        with zipfile.ZipFile(data, mode="r") as zipf:
+            with zipf.open("foo.txt", "r") as fp:
+                fp.seek(bloc, os.SEEK_SET)
+                self.assertEqual(fp.tell(), bloc)
+                fp.seek(-bloc, os.SEEK_CUR)
+                self.assertEqual(fp.tell(), 0)
+                fp.seek(bloc, os.SEEK_CUR)
+                self.assertEqual(fp.tell(), bloc)
+                self.assertEqual(fp.read(5), txt[bloc:bloc+5])
+                fp.seek(0, os.SEEK_END)
+                self.assertEqual(fp.tell(), len(txt))
+
     def tearDown(self):
         unlink(TESTFN)
         unlink(TESTFN2)
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index f9db45f58a2b..5df7b1bf75b9 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -696,6 +696,18 @@ def __init__(self, file, pos, close, lock, writing):
         self._close = close
         self._lock = lock
         self._writing = writing
+        self.seekable = file.seekable
+        self.tell = file.tell
+
+    def seek(self, offset, whence=0):
+        with self._lock:
+            if self.writing():
+                raise ValueError("Can't reposition in the ZIP file while "
+                        "there is an open writing handle on it. "
+                        "Close the writing handle before trying to read.")
+            self._file.seek(self._pos)
+            self._pos = self._file.tell()
+            return self._pos
 
     def read(self, n=-1):
         with self._lock:
@@ -746,6 +758,9 @@ class ZipExtFile(io.BufferedIOBase):
     # Read from compressed files in 4k blocks.
     MIN_READ_SIZE = 4096
 
+    # Chunk size to read during seek
+    MAX_SEEK_READ = 1 << 24
+
     def __init__(self, fileobj, mode, zipinfo, decrypter=None,
                  close_fileobj=False):
         self._fileobj = fileobj
@@ -778,6 +793,17 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
         else:
             self._expected_crc = None
 
+        self._seekable = False
+        try:
+            if fileobj.seekable():
+                self._orig_compress_start = fileobj.tell()
+                self._orig_compress_size = zipinfo.compress_size
+                self._orig_file_size = zipinfo.file_size
+                self._orig_start_crc = self._running_crc
+                self._seekable = True
+        except AttributeError:
+            pass
+
     def __repr__(self):
         result = ['<%s.%s' % (self.__class__.__module__,
                               self.__class__.__qualname__)]
@@ -963,6 +989,62 @@ def close(self):
         finally:
             super().close()
 
+    def seekable(self):
+        return self._seekable
+
+    def seek(self, offset, whence=0):
+        if not self._seekable:
+            raise io.UnsupportedOperation("underlying stream is not seekable")
+        curr_pos = self.tell()
+        if whence == 0: # Seek from start of file
+            new_pos = offset
+        elif whence == 1: # Seek from current position
+            new_pos = curr_pos + offset
+        elif whence == 2: # Seek from EOF
+            new_pos = self._orig_file_size + offset
+        else:
+            raise ValueError("whence must be os.SEEK_SET (0), "
+                             "os.SEEK_CUR (1), or os.SEEK_END (2)")
+
+        if new_pos > self._orig_file_size:
+            new_pos = self._orig_file_size
+
+        if new_pos < 0:
+            new_pos = 0
+
+        read_offset = new_pos - curr_pos
+        buff_offset = read_offset + self._offset
+
+        if buff_offset >= 0 and buff_offset < len(self._readbuffer):
+            # Just move the _offset index if the new position is in the _readbuffer
+            self._offset = buff_offset
+            read_offset = 0
+        elif read_offset < 0:
+            # Position is before the current position. Reset the ZipExtFile
+
+            self._fileobj.seek(self._orig_compress_start)
+            self._running_crc = self._orig_start_crc
+            self._compress_left = self._orig_compress_size
+            self._left = self._orig_file_size
+            self._readbuffer = b''
+            self._offset = 0
+            self._decompressor = zipfile._get_decompressor(self._compress_type)
+            self._eof = False
+            read_offset = new_pos
+
+        while read_offset > 0:
+            read_len = min(self.MAX_SEEK_READ, read_offset)
+            self.read(read_len)
+            read_offset -= read_len
+
+        return self.tell()
+
+    def tell(self):
+        if not self._seekable:
+            raise io.UnsupportedOperation("underlying stream is not seekable")
+        filepos = self._orig_file_size - self._left - len(self._readbuffer) + self._offset
+        return filepos
+
 
 class _ZipWriteFile(io.BufferedIOBase):
     def __init__(self, zf, zinfo, zip64):
diff --git a/Misc/NEWS.d/next/Library/2017-12-21-22-00-11.bpo-22908.cVm89I.rst b/Misc/NEWS.d/next/Library/2017-12-21-22-00-11.bpo-22908.cVm89I.rst
new file mode 100644
index 000000000000..4f3cc0166019
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-21-22-00-11.bpo-22908.cVm89I.rst
@@ -0,0 +1,2 @@
+Added seek and tell to the ZipExtFile class. This only works if the file
+object used to open the zipfile is seekable.


From solipsis at pitrou.net  Tue Jan 30 04:09:11 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 30 Jan 2018 09:09:11 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=8
Message-ID: <20180130090911.1.6A3D76D762913741@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_asyncio leaked [0, 0, 3] memory blocks, sum=3
test_collections leaked [7, -7, 1] memory blocks, sum=1
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_fork leaked [0, -2, 2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogjpD75O', '--timeout', '7200']

From webhook-mailer at python.org  Tue Jan 30 04:34:40 2018
From: webhook-mailer at python.org (Andrew Svetlov)
Date: Tue, 30 Jan 2018 09:34:40 -0000
Subject: [Python-checkins] bpo-32724: Fix references to commands in
 Doc/pdb.rst (GH-5444)
Message-ID: <mailman.83.1517304883.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/67adb31a416864f853772c3e74a06caeadc9b6f1
commit: 67adb31a416864f853772c3e74a06caeadc9b6f1
branch: master
author: St?phane Wirtel <stephane at wirtel.be>
committer: Andrew Svetlov <andrew.svetlov at gmail.com>
date: 2018-01-30T11:34:33+02:00
summary:

bpo-32724: Fix references to commands in Doc/pdb.rst (GH-5444)

Some commands are specified in the documentation, but there is no direct
references.

files:
A Misc/NEWS.d/next/Documentation/2018-01-30-09-00-19.bpo-32724.qPIaM-.rst
M Doc/library/pdb.rst

diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst
index e81c195782d2..c6720cb59974 100644
--- a/Doc/library/pdb.rst
+++ b/Doc/library/pdb.rst
@@ -332,16 +332,19 @@ by the local file.
       (com) end
       (Pdb)
 
-   To remove all commands from a breakpoint, type commands and follow it
+   To remove all commands from a breakpoint, type ``commands`` and follow it
    immediately with ``end``; that is, give no commands.
 
-   With no *bpnumber* argument, commands refers to the last breakpoint set.
+   With no *bpnumber* argument, ``commands`` refers to the last breakpoint set.
 
    You can use breakpoint commands to start your program up again.  Simply use
-   the continue command, or step, or any other command that resumes execution.
+   the :pdbcmd:`continue` command, or :pdbcmd:`step`,
+   or any other command that resumes execution.
 
-   Specifying any command resuming execution (currently continue, step, next,
-   return, jump, quit and their abbreviations) terminates the command list (as if
+   Specifying any command resuming execution
+   (currently :pdbcmd:`continue`, :pdbcmd:`step`, :pdbcmd:`next`,
+   :pdbcmd:`return`, :pdbcmd:`jump`, :pdbcmd:`quit` and their abbreviations)
+   terminates the command :pdbcmd:`list` (as if
    that command was immediately followed by end). This is because any time you
    resume execution (even with a simple next or step), you may encounter another
    breakpoint?which could have its own command list, leading to ambiguities about
diff --git a/Misc/NEWS.d/next/Documentation/2018-01-30-09-00-19.bpo-32724.qPIaM-.rst b/Misc/NEWS.d/next/Documentation/2018-01-30-09-00-19.bpo-32724.qPIaM-.rst
new file mode 100644
index 000000000000..ddef6327de13
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2018-01-30-09-00-19.bpo-32724.qPIaM-.rst
@@ -0,0 +1,2 @@
+Add references to some commands in the documentation of Pdb. Patch by
+St?phane Wirtel


From webhook-mailer at python.org  Tue Jan 30 05:14:15 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 30 Jan 2018 10:14:15 -0000
Subject: [Python-checkins] bpo-28440: Don't add
 /Library/Python/3.x/site-packages to sys.path (#5445)
Message-ID: <mailman.84.1517307257.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/763f094b1f0ce2a07768828e78afe9687e9ad3bb
commit: 763f094b1f0ce2a07768828e78afe9687e9ad3bb
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-30T05:14:09-05:00
summary:

bpo-28440: Don't add /Library/Python/3.x/site-packages to sys.path (#5445)

No longer add /Library/Python/3.x/site-packages, the Apple-supplied
system Python site-packages directory, to sys.path for macOS framework
builds in case Apple ships a version of Python 3. A similar change
was made earlier to Python 2.7 where it was found that the coupling
between the system Python and a user-installed framework Python often
caused confusion or pip install failures.

files:
A Misc/NEWS.d/next/macOS/2018-01-30-04-40-12.bpo-28440.W_BUWU.rst
M Lib/site.py
M Lib/test/test_site.py

diff --git a/Lib/site.py b/Lib/site.py
index 7dc1b041c192..950e7038a505 100644
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -340,11 +340,6 @@ def getsitepackages(prefixes=None):
         else:
             sitepackages.append(prefix)
             sitepackages.append(os.path.join(prefix, "lib", "site-packages"))
-        # for framework builds *only* we add the standard Apple locations.
-        if sys.platform == "darwin" and sys._framework:
-            sitepackages.append(
-                os.path.join("/Library", sys._framework,
-                             '%d.%d' % sys.version_info[:2], "site-packages"))
     return sitepackages
 
 def addsitepackages(known_paths, prefixes=None):
diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
index 266adf01e04d..b6648d67ca68 100644
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -261,20 +261,8 @@ def test_getusersitepackages(self):
     def test_getsitepackages(self):
         site.PREFIXES = ['xoxo']
         dirs = site.getsitepackages()
-
-        if (sys.platform == "darwin" and
-            sysconfig.get_config_var("PYTHONFRAMEWORK")):
-            # OS X framework builds
-            site.PREFIXES = ['Python.framework']
-            dirs = site.getsitepackages()
-            self.assertEqual(len(dirs), 2)
-            wanted = os.path.join('/Library',
-                                  sysconfig.get_config_var("PYTHONFRAMEWORK"),
-                                  '%d.%d' % sys.version_info[:2],
-                                  'site-packages')
-            self.assertEqual(dirs[1], wanted)
-        elif os.sep == '/':
-            # OS X non-framework builds, Linux, FreeBSD, etc
+        if os.sep == '/':
+            # OS X, Linux, FreeBSD, etc
             self.assertEqual(len(dirs), 1)
             wanted = os.path.join('xoxo', 'lib',
                                   'python%d.%d' % sys.version_info[:2],
diff --git a/Misc/NEWS.d/next/macOS/2018-01-30-04-40-12.bpo-28440.W_BUWU.rst b/Misc/NEWS.d/next/macOS/2018-01-30-04-40-12.bpo-28440.W_BUWU.rst
new file mode 100644
index 000000000000..e973e0957755
--- /dev/null
+++ b/Misc/NEWS.d/next/macOS/2018-01-30-04-40-12.bpo-28440.W_BUWU.rst
@@ -0,0 +1,2 @@
+No longer add /Library/Python/3.x/site-packages to sys.path for macOS
+framework builds to avoid future conflicts.


From webhook-mailer at python.org  Tue Jan 30 06:18:57 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 30 Jan 2018 11:18:57 -0000
Subject: [Python-checkins] bpo-28914: Fix compilation of select on Android
 (#5447)
Message-ID: <mailman.85.1517311140.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/b8d90328ad2abc9d6d2c9b8ce769fb82b18bbc10
commit: b8d90328ad2abc9d6d2c9b8ce769fb82b18bbc10
branch: master
author: Victor Stinner <victor.stinner at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-30T12:18:54+01:00
summary:

bpo-28914: Fix compilation of select on Android (#5447)

EPOLL_CLOEXEC is not defined on Android.

Co-Authored-By: Wataru Matsumoto <sxsns243 at gmail.com>

files:
M Modules/selectmodule.c

diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c
index f2f5cc86cdbe..3b7713abab36 100644
--- a/Modules/selectmodule.c
+++ b/Modules/selectmodule.c
@@ -1307,10 +1307,13 @@ pyepoll_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
         PyErr_SetString(PyExc_ValueError, "negative sizehint");
         return NULL;
     }
+
+#ifdef HAVE_EPOLL_CREATE1
     if (flags && flags != EPOLL_CLOEXEC) {
         PyErr_SetString(PyExc_OSError, "invalid flags");
         return NULL;
     }
+#endif
 
     return newPyEpoll_Object(type, sizehint, -1);
 }


From webhook-mailer at python.org  Tue Jan 30 07:42:19 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 30 Jan 2018 12:42:19 -0000
Subject: [Python-checkins] bpo-32726: macOS installer and framework
 enhancements and changes for 3.7.0 (GH-5448)
Message-ID: <mailman.86.1517316141.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/8c9bb72e8b6474ae722f7d6521cfb3cb2e26e3c5
commit: 8c9bb72e8b6474ae722f7d6521cfb3cb2e26e3c5
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-30T07:42:14-05:00
summary:

bpo-32726: macOS installer and framework enhancements and changes for 3.7.0 (GH-5448)

This issue covers various changes for the macOS installers provided via python.org for 3.7.0.

- Provide a provisional new installer variant for macOS 10.9 and later systems with 64-bit (x86_64) architecture only.  Apple has made it known that future versions of macOS will only fully support 64-bit executables and some other third-party software suppliers have chosen 10.9 as their oldest supported system.
 
- Support **Tcl/Tk 8.6** with the 10.9 installer variant.
 
- Upgrade **OpenSSL** to 1.1.0g and **SQLite** to 3.22.0.
 
- The compiler name used for the interpreter build and for modules built with **Distutils / pip** is now _gcc_ rather than _gcc-4.2_. And extension module builds will no longer try to force use of an old SDK if present.

files:
A Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst
D Mac/BuildScript/openssl_sdk_makedepend.patch
M Mac/BuildScript/build-installer.py
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf
M configure
M configure.ac

diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
index bff4eae31b4b..8e2e104a0b7e 100755
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -1,30 +1,39 @@
 #!/usr/bin/env python
 """
-This script is used to build "official" universal installers on Mac OS X.
-It requires at least Mac OS X 10.5, Xcode 3, and the 10.4u SDK for
-32-bit builds.  64-bit or four-way universal builds require at least
-OS X 10.5 and the 10.5 SDK.
+This script is used to build "official" universal installers on macOS.
+
+NEW for 3.7.0:
+- support Intel 64-bit-only () and 32-bit-only installer builds
+- use external Tcl/Tk 8.6 for 10.9+ builds
+- deprecate use of explicit SDK (--sdk-path=) since all but the oldest
+  versions of Xcode support implicit setting of an SDK via environment
+  variables (SDKROOT and friends, see the xcrun man page for more info).
+  The SDK stuff was primarily needed for building universal installers
+  for 10.4; so as of 3.7.0, building installers for 10.4 is no longer
+  supported with build-installer.
+- use generic "gcc" as compiler (CC env var) rather than "gcc-4.2"
+
+TODO:
+- support SDKROOT and DEVELOPER_DIR xcrun env variables
+- test with 10.5 and 10.4 and determine support status
 
 Please ensure that this script keeps working with Python 2.5, to avoid
-bootstrap issues (/usr/bin/python is Python 2.5 on OSX 10.5).  Sphinx,
-which is used to build the documentation, currently requires at least
-Python 2.4.  However, as of Python 3.4.1, Doc builds require an external
-sphinx-build and the current versions of Sphinx now require at least
-Python 2.6.
-
-In addition to what is supplied with OS X 10.5+ and Xcode 3+, the script
-requires an installed third-party version of
-Tcl/Tk 8.4 (for OS X 10.4 and 10.5 deployment targets) or Tcl/TK 8.5
-(for 10.6 or later) installed in /Library/Frameworks.  When installed,
+bootstrap issues (/usr/bin/python is Python 2.5 on OSX 10.5).  Doc builds
+use current versions of Sphinx and require a reasonably current python3.
+Sphinx and dependencies are installed into a venv using the python3's pip
+so will fetch them from PyPI if necessary.  Since python3 is now used for
+Sphinx, build-installer.py should also be converted to use python3!
+
+build-installer currently requires an installed third-party version of
+Tcl/Tk 8.4 (for OS X 10.4 and 10.5 deployment targets), Tcl/TK 8.5
+(for 10.6 or later), or Tcl/TK 8.6 (for 10.9 or later)
+installed in /Library/Frameworks.  When installed,
 the Python built by this script will attempt to dynamically link first to
 Tcl and Tk frameworks in /Library/Frameworks if available otherwise fall
 back to the ones in /System/Library/Framework.  For the build, we recommend
-installing the most recent ActiveTcl 8.4 or 8.5 version.
-
-32-bit-only installer builds are still possible on OS X 10.4 with Xcode 2.5
-and the installation of additional components, such as a newer Python
-(2.5 is needed for Python parser updates) and for the documentation
-build either svn (pre-3.4.1) or sphinx-build (3.4.1 and later).
+installing the most recent ActiveTcl 8.6. 8.5, or 8.4 version, depending
+on the deployment target.  The actual version linked to depends on the
+path of /Library/Frameworks/{Tcl,Tk}.framework/Versions/Current.
 
 Usage: see USAGE variable in the script.
 """
@@ -111,32 +120,19 @@ def getFullVersion():
 DEPSRC = os.path.join(WORKDIR, 'third-party')
 DEPSRC = os.path.expanduser('~/Universal/other-sources')
 
-# Location of the preferred SDK
-
-### There are some issues with the SDK selection below here,
-### The resulting binary doesn't work on all platforms that
-### it should. Always default to the 10.4u SDK until that
-### issue is resolved.
-###
-##if int(os.uname()[2].split('.')[0]) == 8:
-##    # Explicitly use the 10.4u (universal) SDK when
-##    # building on 10.4, the system headers are not
-##    # useable for a universal build
-##    SDKPATH = "/Developer/SDKs/MacOSX10.4u.sdk"
-##else:
-##    SDKPATH = "/"
-
-SDKPATH = "/Developer/SDKs/MacOSX10.4u.sdk"
-
 universal_opts_map = { '32-bit': ('i386', 'ppc',),
                        '64-bit': ('x86_64', 'ppc64',),
                        'intel':  ('i386', 'x86_64'),
+                       'intel-32':  ('i386',),
+                       'intel-64':  ('x86_64',),
                        '3-way':  ('ppc', 'i386', 'x86_64'),
                        'all':    ('i386', 'ppc', 'x86_64', 'ppc64',) }
 default_target_map = {
         '64-bit': '10.5',
         '3-way': '10.5',
         'intel': '10.5',
+        'intel-32': '10.4',
+        'intel-64': '10.5',
         'all': '10.5',
 }
 
@@ -154,19 +150,18 @@ def getFullVersion():
         ))))
 
 # $MACOSX_DEPLOYMENT_TARGET -> minimum OS X level
-DEPTARGET = '10.3'
+DEPTARGET = '10.5'
 
 def getDeptargetTuple():
     return tuple([int(n) for n in DEPTARGET.split('.')[0:2]])
 
 def getTargetCompilers():
     target_cc_map = {
-        '10.3': ('gcc-4.0', 'g++-4.0'),
         '10.4': ('gcc-4.0', 'g++-4.0'),
-        '10.5': ('gcc-4.2', 'g++-4.2'),
-        '10.6': ('gcc-4.2', 'g++-4.2'),
+        '10.5': ('gcc', 'g++'),
+        '10.6': ('gcc', 'g++'),
     }
-    return target_cc_map.get(DEPTARGET, ('clang', 'clang++') )
+    return target_cc_map.get(DEPTARGET, ('gcc', 'gcc++') )
 
 CC, CXX = getTargetCompilers()
 
@@ -180,9 +175,9 @@ def getTargetCompilers():
     -b DIR
     --build-dir=DIR:     Create build here (default: %(WORKDIR)r)
     --third-party=DIR:   Store third-party sources here (default: %(DEPSRC)r)
-    --sdk-path=DIR:      Location of the SDK (default: %(SDKPATH)r)
+    --sdk-path=DIR:      Location of the SDK (deprecated, use SDKROOT env variable)
     --src-dir=DIR:       Location of the Python sources (default: %(SRCDIR)r)
-    --dep-target=10.n    OS X deployment target (default: %(DEPTARGET)r)
+    --dep-target=10.n    macOS deployment target (default: %(DEPTARGET)r)
     --universal-archs=x  universal architectures (options: %(UNIVERSALOPTS)r, default: %(UNIVERSALARCHS)r)
 """)% globals()
 
@@ -213,12 +208,9 @@ def library_recipes():
 
     result.extend([
           dict(
-              name="OpenSSL 1.0.2m",
-              url="https://www.openssl.org/source/openssl-1.0.2m.tar.gz",
-              checksum='10e9e37f492094b9ef296f68f24a7666',
-              patches=[
-                  "openssl_sdk_makedepend.patch",
-                   ],
+              name="OpenSSL 1.1.0g",
+              url="https://www.openssl.org/source/openssl-1.1.0g.tar.gz",
+              checksum='ba5f1b8b835b88cadbce9b35ed9531a6',
               buildrecipe=build_universal_openssl,
               configure=None,
               install=None,
@@ -315,9 +307,9 @@ def library_recipes():
                   ),
           ),
           dict(
-              name="SQLite 3.21.0",
-              url="https://www.sqlite.org/2017/sqlite-autoconf-3210000.tar.gz",
-              checksum='7913de4c3126ba3c24689cb7a199ea31',
+              name="SQLite 3.22.0",
+              url="https://www.sqlite.org/2018/sqlite-autoconf-3220000.tar.gz",
+              checksum='96b5648d542e8afa6ab7ffb8db8ddc3d',
               extra_cflags=('-Os '
                             '-DSQLITE_ENABLE_FTS5 '
                             '-DSQLITE_ENABLE_FTS4 '
@@ -343,11 +335,10 @@ def library_recipes():
               url="http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz",
               checksum='00b516f4704d4a7cb50a1d97e6e8e15b',
               configure=None,
-              install='make install CC=%s CXX=%s, PREFIX=%s/usr/local/ CFLAGS="-arch %s -isysroot %s"'%(
+              install='make install CC=%s CXX=%s, PREFIX=%s/usr/local/ CFLAGS="-arch %s"'%(
                   CC, CXX,
                   shellQuote(os.path.join(WORKDIR, 'libraries')),
                   ' -arch '.join(ARCHLIST),
-                  SDKPATH,
               ),
           ),
           dict(
@@ -355,11 +346,10 @@ def library_recipes():
               url="http://www.gzip.org/zlib/zlib-1.2.3.tar.gz",
               checksum='debc62758716a169df9f62e6ab2bc634',
               configure=None,
-              install='make install CC=%s CXX=%s, prefix=%s/usr/local/ CFLAGS="-arch %s -isysroot %s"'%(
+              install='make install CC=%s CXX=%s, prefix=%s/usr/local/ CFLAGS="-arch %s"'%(
                   CC, CXX,
                   shellQuote(os.path.join(WORKDIR, 'libraries')),
                   ' -arch '.join(ARCHLIST),
-                  SDKPATH,
               ),
           ),
           dict(
@@ -406,8 +396,7 @@ def pkg_recipes():
             source="/Library/Frameworks/Python.framework",
             readme="""\
                 This package installs Python.framework, that is the python
-                interpreter and the standard library. This also includes Python
-                wrappers for lots of Mac OS X API's.
+                interpreter and the standard library.
             """,
             postflight="scripts/postflight.framework",
             selected='selected',
@@ -484,24 +473,6 @@ def pkg_recipes():
         ),
     ]
 
-    if getDeptargetTuple() < (10, 4) and not PYTHON_3:
-        result.append(
-            dict(
-                name="PythonSystemFixes",
-                long_name="Fix system Python",
-                readme="""\
-                    This package updates the system python installation on
-                    Mac OS X 10.3 to ensure that you can build new python extensions
-                    using that copy of python after installing this version.
-                    """,
-                postflight="../Tools/fixapplepython23.py",
-                topdir="/Library/Frameworks/Python.framework",
-                source="/empty-dir",
-                required=False,
-                selected=unselected_for_python3,
-            )
-        )
-
     return result
 
 def fatal(msg):
@@ -566,41 +537,33 @@ def checkEnvironment():
     Check that we're running on a supported system.
     """
 
-    if sys.version_info[0:2] < (2, 4):
-        fatal("This script must be run with Python 2.4 or later")
+    if sys.version_info[0:2] < (2, 5):
+        fatal("This script must be run with Python 2.5 (or later)")
 
     if platform.system() != 'Darwin':
-        fatal("This script should be run on a Mac OS X 10.4 (or later) system")
+        fatal("This script should be run on a macOS 10.5 (or later) system")
 
     if int(platform.release().split('.')[0]) < 8:
-        fatal("This script should be run on a Mac OS X 10.4 (or later) system")
-
-    if not os.path.exists(SDKPATH):
-        fatal("Please install the latest version of Xcode and the %s SDK"%(
-            os.path.basename(SDKPATH[:-4])))
+        fatal("This script should be run on a macOS 10.5 (or later) system")
 
     # Because we only support dynamic load of only one major/minor version of
     # Tcl/Tk, ensure:
-    # 1. there are no user-installed frameworks of Tcl/Tk with version
-    #       higher than the Apple-supplied system version in
-    #       SDKROOT/System/Library/Frameworks
-    # 2. there is a user-installed framework (usually ActiveTcl) in (or linked
-    #       in) SDKROOT/Library/Frameworks with the same version as the system
-    #       version. This allows users to choose to install a newer patch level.
+    # 1. there is a user-installed framework (usually ActiveTcl) in (or linked
+    #       in) SDKROOT/Library/Frameworks.  As of Python 3.7.0, we no longer
+    #       enforce that the version of the user-installed framework also
+    #       exists in the system-supplied Tcl/Tk frameworks.  Time to support
+    #       Tcl/Tk 8.6 even if Apple does not.
 
     frameworks = {}
     for framework in ['Tcl', 'Tk']:
         fwpth = 'Library/Frameworks/%s.framework/Versions/Current' % framework
-        sysfw = os.path.join(SDKPATH, 'System', fwpth)
-        libfw = os.path.join(SDKPATH, fwpth)
+        libfw = os.path.join('/', fwpth)
         usrfw = os.path.join(os.getenv('HOME'), fwpth)
-        frameworks[framework] = os.readlink(sysfw)
+        frameworks[framework] = os.readlink(libfw)
         if not os.path.exists(libfw):
             fatal("Please install a link to a current %s %s as %s so "
                     "the user can override the system framework."
                     % (framework, frameworks[framework], libfw))
-        if os.readlink(libfw) != os.readlink(sysfw):
-            fatal("Version of %s must match %s" % (libfw, sysfw) )
         if os.path.exists(usrfw):
             fatal("Please rename %s to avoid possible dynamic load issues."
                     % usrfw)
@@ -608,6 +571,10 @@ def checkEnvironment():
     if frameworks['Tcl'] != frameworks['Tk']:
         fatal("The Tcl and Tk frameworks are not the same version.")
 
+    print(" -- Building with Tcl/Tk %s frameworks"
+                % frameworks['Tk'])
+    print("")
+
     # add files to check after build
     EXPECTED_SHARED_LIBS['_tkinter.so'] = [
             "/Library/Frameworks/Tcl.framework/Versions/%s/Tcl"
@@ -644,7 +611,7 @@ def parseOptions(args=None):
     """
     Parse arguments and update global settings.
     """
-    global WORKDIR, DEPSRC, SDKPATH, SRCDIR, DEPTARGET
+    global WORKDIR, DEPSRC, SRCDIR, DEPTARGET
     global UNIVERSALOPTS, UNIVERSALARCHS, ARCHLIST, CC, CXX
     global FW_VERSION_PREFIX
     global FW_SSL_DIRECTORY
@@ -677,7 +644,7 @@ def parseOptions(args=None):
             DEPSRC=v
 
         elif k in ('--sdk-path',):
-            SDKPATH=v
+            print(" WARNING: --sdk-path is no longer supported")
 
         elif k in ('--src-dir',):
             SRCDIR=v
@@ -693,7 +660,7 @@ def parseOptions(args=None):
                 if deptarget is None:
                     # Select alternate default deployment
                     # target
-                    DEPTARGET = default_target_map.get(v, '10.3')
+                    DEPTARGET = default_target_map.get(v, '10.5')
             else:
                 raise NotImplementedError(v)
 
@@ -702,7 +669,6 @@ def parseOptions(args=None):
 
     SRCDIR=os.path.abspath(SRCDIR)
     WORKDIR=os.path.abspath(WORKDIR)
-    SDKPATH=os.path.abspath(SDKPATH)
     DEPSRC=os.path.abspath(DEPSRC)
 
     CC, CXX = getTargetCompilers()
@@ -713,7 +679,6 @@ def parseOptions(args=None):
     print("-- Settings:")
     print("   * Source directory:    %s" % SRCDIR)
     print("   * Build directory:     %s" % WORKDIR)
-    print("   * SDK location:        %s" % SDKPATH)
     print("   * Third-party source:  %s" % DEPSRC)
     print("   * Deployment target:   %s" % DEPTARGET)
     print("   * Universal archs:     %s" % str(ARCHLIST))
@@ -837,17 +802,13 @@ def build_openssl_arch(archbase, arch):
             "ppc64": ["darwin64-ppc-cc"],
         }
         configure_opts = [
-            "no-krb5",
             "no-idea",
             "no-mdc2",
             "no-rc5",
             "no-zlib",
-            "enable-tlsext",
-            "no-ssl2",
             "no-ssl3",
             # "enable-unit-test",
             "shared",
-            "--install_prefix=%s"%shellQuote(archbase),
             "--prefix=%s"%os.path.join("/", *FW_VERSION_PREFIX),
             "--openssldir=%s"%os.path.join("/", *FW_SSL_DIRECTORY),
         ]
@@ -855,9 +816,9 @@ def build_openssl_arch(archbase, arch):
             configure_opts.append("no-asm")
         runCommand(" ".join(["perl", "Configure"]
                         + arch_opts[arch] + configure_opts))
-        runCommand("make depend OSX_SDK=%s" % SDKPATH)
-        runCommand("make all OSX_SDK=%s" % SDKPATH)
-        runCommand("make install_sw OSX_SDK=%s" % SDKPATH)
+        runCommand("make depend")
+        runCommand("make all")
+        runCommand("make install_sw DESTDIR=%s"%shellQuote(archbase))
         # runCommand("make test")
         return
 
@@ -1016,27 +977,24 @@ def buildRecipe(recipe, basedir, archList):
 
         if recipe.get('useLDFlags', 1):
             configure_args.extend([
-                "CFLAGS=%s-mmacosx-version-min=%s -arch %s -isysroot %s "
+                "CFLAGS=%s-mmacosx-version-min=%s -arch %s "
                             "-I%s/usr/local/include"%(
                         recipe.get('extra_cflags', ''),
                         DEPTARGET,
                         ' -arch '.join(archList),
-                        shellQuote(SDKPATH)[1:-1],
                         shellQuote(basedir)[1:-1],),
-                "LDFLAGS=-mmacosx-version-min=%s -isysroot %s -L%s/usr/local/lib -arch %s"%(
+                "LDFLAGS=-mmacosx-version-min=%s -L%s/usr/local/lib -arch %s"%(
                     DEPTARGET,
-                    shellQuote(SDKPATH)[1:-1],
                     shellQuote(basedir)[1:-1],
                     ' -arch '.join(archList)),
             ])
         else:
             configure_args.extend([
-                "CFLAGS=%s-mmacosx-version-min=%s -arch %s -isysroot %s "
+                "CFLAGS=%s-mmacosx-version-min=%s -arch %s "
                             "-I%s/usr/local/include"%(
                         recipe.get('extra_cflags', ''),
                         DEPTARGET,
                         ' -arch '.join(archList),
-                        shellQuote(SDKPATH)[1:-1],
                         shellQuote(basedir)[1:-1],),
             ])
 
@@ -1114,10 +1072,6 @@ def buildPython():
     curdir = os.getcwd()
     os.chdir(buildDir)
 
-    # Not sure if this is still needed, the original build script
-    # claims that parts of the install assume python.exe exists.
-    os.symlink('python', os.path.join(buildDir, 'python.exe'))
-
     # Extract the version from the configure file, needed to calculate
     # several paths.
     version = getVersion()
@@ -1128,13 +1082,13 @@ def buildPython():
     os.environ['DYLD_LIBRARY_PATH'] = os.path.join(WORKDIR,
                                         'libraries', 'usr', 'local', 'lib')
     print("Running configure...")
-    runCommand("%s -C --enable-framework --enable-universalsdk=%s "
+    runCommand("%s -C --enable-framework --enable-universalsdk=/ "
                "--with-universal-archs=%s "
                "%s "
                "%s "
                "LDFLAGS='-g -L%s/libraries/usr/local/lib' "
                "CFLAGS='-g -I%s/libraries/usr/local/include' 2>&1"%(
-        shellQuote(os.path.join(SRCDIR, 'configure')), shellQuote(SDKPATH),
+        shellQuote(os.path.join(SRCDIR, 'configure')),
         UNIVERSALARCHS,
         (' ', '--with-computed-gotos ')[PYTHON_3],
         (' ', '--without-ensurepip ')[PYTHON_3],
diff --git a/Mac/BuildScript/openssl_sdk_makedepend.patch b/Mac/BuildScript/openssl_sdk_makedepend.patch
deleted file mode 100644
index 0caac0a64c1e..000000000000
--- a/Mac/BuildScript/openssl_sdk_makedepend.patch
+++ /dev/null
@@ -1,40 +0,0 @@
-# HG changeset patch
-#
-# 	using openssl 1.0.2k
-#
-# - support building with an OS X SDK
-
-diff Configure
-
-diff --git a/Configure b/Configure
---- a/Configure
-+++ b/Configure
-@@ -642,12 +642,12 @@
- 
- ##### MacOS X (a.k.a. Rhapsody or Darwin) setup
- "rhapsody-ppc-cc","cc:-O3 -DB_ENDIAN::(unknown):MACOSX_RHAPSODY::BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${no_asm}::",
--"darwin-ppc-cc","cc:-arch ppc -O3 -DB_ENDIAN -Wa,-force_cpusubtype_ALL::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc32_asm}:osx32:dlfcn:darwin-shared:-fPIC -fno-common:-arch ppc -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
--"darwin64-ppc-cc","cc:-arch ppc64 -O3 -DB_ENDIAN::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:SIXTY_FOUR_BIT_LONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc64_asm}:osx64:dlfcn:darwin-shared:-fPIC -fno-common:-arch ppc64 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
--"darwin-i386-cc","cc:-arch i386 -O3 -fomit-frame-pointer -DL_ENDIAN::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:BN_LLONG RC4_INT RC4_CHUNK DES_UNROLL BF_PTR:".eval{my $asm=$x86_asm;$asm=~s/cast\-586\.o//;$asm}.":macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch i386 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
--"debug-darwin-i386-cc","cc:-arch i386 -g3 -DL_ENDIAN::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:BN_LLONG RC4_INT RC4_CHUNK DES_UNROLL BF_PTR:${x86_asm}:macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch i386 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
--"darwin64-x86_64-cc","cc:-arch x86_64 -O3 -DL_ENDIAN -Wall::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:SIXTY_FOUR_BIT_LONG RC4_CHUNK DES_INT DES_UNROLL:".eval{my $asm=$x86_64_asm;$asm=~s/rc4\-[^:]+//;$asm}.":macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch x86_64 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
--"debug-darwin64-x86_64-cc","cc:-arch x86_64 -ggdb -g2 -O0 -DL_ENDIAN -Wall::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:SIXTY_FOUR_BIT_LONG RC4_CHUNK DES_INT DES_UNROLL:".eval{my $asm=$x86_64_asm;$asm=~s/rc4\-[^:]+//;$asm}.":macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch x86_64 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-+"darwin-ppc-cc","cc:-arch ppc -isysroot \$(OSX_SDK) -O3 -DB_ENDIAN -Wa,-force_cpusubtype_ALL::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc32_asm}:osx32:dlfcn:darwin-shared:-fPIC -fno-common:-arch ppc -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-+"darwin64-ppc-cc","cc:-arch ppc64 -isysroot \$(OSX_SDK) -O3 -DB_ENDIAN::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:SIXTY_FOUR_BIT_LONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc64_asm}:osx64:dlfcn:darwin-shared:-fPIC -fno-common:-arch ppc64 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-+"darwin-i386-cc","cc:-arch i386 -isysroot \$(OSX_SDK) -O3 -fomit-frame-pointer -DL_ENDIAN::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:BN_LLONG RC4_INT RC4_CHUNK DES_UNROLL BF_PTR:".eval{my $asm=$x86_asm;$asm=~s/cast\-586\.o//;$asm}.":macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch i386 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-+"debug-darwin-i386-cc","cc:-arch i386 -isysroot \$(OSX_SDK) -g3 -DL_ENDIAN::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:BN_LLONG RC4_INT RC4_CHUNK DES_UNROLL BF_PTR:${x86_asm}:macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch i386 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-+"darwin64-x86_64-cc","cc:-arch x86_64 -isysroot \$(OSX_SDK) -O3 -DL_ENDIAN -Wall::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:SIXTY_FOUR_BIT_LONG RC4_CHUNK DES_INT DES_UNROLL:".eval{my $asm=$x86_64_asm;$asm=~s/rc4\-[^:]+//;$asm}.":macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch x86_64 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-+"debug-darwin64-x86_64-cc","cc:-arch x86_64 -isysroot \$(OSX_SDK) -ggdb -g2 -O0 -DL_ENDIAN -Wall::-D_REENTRANT:MACOSX:-Wl,-search_paths_first%:SIXTY_FOUR_BIT_LONG RC4_CHUNK DES_INT DES_UNROLL:".eval{my $asm=$x86_64_asm;$asm=~s/rc4\-[^:]+//;$asm}.":macosx:dlfcn:darwin-shared:-fPIC -fno-common:-arch x86_64 -dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
- "debug-darwin-ppc-cc","cc:-DBN_DEBUG -DREF_CHECK -DCONF_DEBUG -DCRYPTO_MDEBUG -DB_ENDIAN -g -Wall -O::-D_REENTRANT:MACOSX::BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc32_asm}:osx32:dlfcn:darwin-shared:-fPIC:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
- # iPhoneOS/iOS
- "iphoneos-cross","llvm-gcc:-O3 -isysroot \$(CROSS_TOP)/SDKs/\$(CROSS_SDK) -fomit-frame-pointer -fno-common::-D_REENTRANT:iOS:-Wl,-search_paths_first%:BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${no_asm}:dlfcn:darwin-shared:-fPIC -fno-common:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib",
-@@ -1728,8 +1728,7 @@
- 		s/^AR=\s*ar/AR= $ar/;
- 		s/^RANLIB=.*/RANLIB= $ranlib/;
- 		s/^RC=.*/RC= $windres/;
--		s/^MAKEDEPPROG=.*$/MAKEDEPPROG= $cc/ if $cc eq "gcc";
--		s/^MAKEDEPPROG=.*$/MAKEDEPPROG= $cc/ if $ecc eq "gcc" || $ecc eq "clang";
-+		s/^MAKEDEPPROG=.*$/MAKEDEPPROG= $cc/;
- 		}
- 	s/^CFLAG=.*$/CFLAG= $cflags/;
- 	s/^DEPFLAG=.*$/DEPFLAG=$depflags/;
diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index dba96b6068c6..1eea1e291bda 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1504\cocoasubrtf830
+{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf200
 {\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
 {\*\expandedcolortbl;;}
@@ -10,21 +10,37 @@
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
 
 \b \cf0 NOTE:
-\b0   This is an early developer preview of the next feature release of Python 3.  Many features are yet to be added and existing features may change substantially or be deleted prior to the feature code freeze.\
+\b0   This is a beta preview of the next feature release of Python 3.  Existing features may change or be deleted prior to the release code freeze.\
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 \cf0 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\b \cf0 \ul \ulc0 Certificate verification and OpenSSL\
+\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 3.7.0b1]
+\b0 \ulnone \
+\
+For Python.3.7, python.org currently provides two installer variants for download: one that installs a 
+\i 64-bit/32-bit Intel
+\i0  Python capable of running on 
+\i macOS 10.6 (Snow Leopard)
+\i0  or later; and one that installs a 
+\i 64-bit-only 
+\i0 Python capable of running on 
+\i macOS 10.9 (Mavericks)
+\i0  or later.  This ReadMe was installed with the 
+\i $MACOSX_DEPLOYMENT_TARGET
+\i0  variant.  Use the 10.6 variant if you are running on macOS 10.6 through 10.8 or if you want to produce standalone applications that can run on systems from 10.6.  If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant.  The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself.  The list of these libraries varies by installer variant and is included at the end of the License.rtf file.
+\b \ul \
+\
+Certificate verification and OpenSSL \{UPDATED in 3.7.0b1]\
 
 \b0 \ulnone \
-This variant of Python 3.7 includes its own private copy of OpenSSL 1.0.2.  The deprecated Apple-supplied OpenSSL libraries are no longer used.  This means that the trust certificates in system and user keychains managed by the 
+This variant of Python 3.7 includes its own private copy of OpenSSL 1.1.0.  The deprecated Apple-supplied OpenSSL libraries are no longer used.  This means that the trust certificates in system and user keychains managed by the 
 \i Keychain Access 
 \i0 application and the 
 \i security
 \i0  command line utility are no longer used as defaults by the Python 
 \f1 ssl
-\f0  module.  For this preview release, a sample command script is included in 
+\f0  module.  For this beta release, a sample command script is included in 
 \f1 /Applications/Python 3.7
 \f0  to install a curated bundle of default root certificates from the third-party 
 \f1 certifi
@@ -37,14 +53,14 @@ The bundled
 \f0  included with this installer has its own default certificate store for verifying download connections.\
 \
 
-\b \ul Update your version of Tcl/Tk to use IDLE or other Tk applications
+\b \ul Update your version of Tcl/Tk to use IDLE or other Tk applications [CHANGED in 3.7.0b1] 
 \b0 \ulnone \
 \
 To use IDLE or other programs that use the Tkinter graphical user interface toolkit, you need to install a newer third-party version of the 
 \i Tcl/Tk
 \i0  frameworks.  Visit {\field{\*\fldinst{HYPERLINK "https://www.python.org/download/mac/tcltk/"}}{\fldrslt https://www.python.org/download/mac/tcltk/}} for current information about supported and recommended versions of 
 \i Tcl/Tk
-\i0  for this version of Python and of Mac OS X.  For this developer preview, the installer is still linked with Tcl/Tk 8.5.\
+\i0  for this version of Python and of Mac OS X.  For 3.7.0b1, the 10.9 installer variant is linked with Tcl/Tk 8.6 which you must install separately.   The 10.6 installer links /with Tcl/Tk 8.5 which you should also download and install.\
 
 \b \ul \
 Other changes\
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 528e2513af77..18f3290608ec 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1504\cocoasubrtf830
+{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf200
 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
 {\colortbl;\red255\green255\blue255;}
 {\*\expandedcolortbl;;}
@@ -19,7 +19,11 @@
 \
 
 \b NOTE:
-\b0   This is an early developer preview of the next feature release of Python 3.  Many features are yet to be added and existing features may change substantially or be deleted prior to the feature code freeze.\
+\b0   This is an beta preview of the next feature release of Python 3.  New features may change substantially or be deleted prior to the final code freeze.\
+\
+
+\b NEW in 3.7.0b1: 
+\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), Tcl/Tk 8.6 support in the 10.9+ variant, OpenSSL 1.1.0g, and more!\
 \
 
 \b IMPORTANT:
diff --git a/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst b/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst
new file mode 100644
index 000000000000..bb7ebf1a62d2
--- /dev/null
+++ b/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst
@@ -0,0 +1,4 @@
+Provide an additional, more modern macOS installer variant that supports
+macOS 10.9+ systems in 64-bit mode only.  Upgrade the supplied third-party
+libraries to OpenSSL 1.1.0g and to SQLite 3.22.0.  The 10.9+ installer now
+links with Tcl/Tk 8.6.
diff --git a/configure b/configure
index caa86677ea11..f75acf68ce2c 100755
--- a/configure
+++ b/configure
@@ -1489,7 +1489,8 @@ Optional Packages:
   --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
   --with-universal-archs=ARCH
                           select architectures for universal build ("32-bit",
-                          "64-bit", "3-way", "intel", "intel-32", or "all")
+                          "64-bit", "3-way", "intel", "intel-32", "intel-64",
+                          or "all")
   --with-framework-name=FRAMEWORK
                           specify an alternate name of the framework built
                           with --enable-framework
@@ -7224,6 +7225,11 @@ $as_echo "$CC" >&6; }
                LIPO_32BIT_FLAGS=""
                ARCH_RUN_32BIT=""
                ;;
+            intel-64)
+               UNIVERSAL_ARCH_FLAGS="-arch x86_64"
+               LIPO_32BIT_FLAGS=""
+               ARCH_RUN_32BIT="true"
+               ;;
             3-way)
                UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch x86_64"
                LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386"
@@ -7234,11 +7240,14 @@ $as_echo "$CC" >&6; }
                ;;
             esac
 
-            CFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${CFLAGS}"
-            LDFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${LDFLAGS}"
             if test "${UNIVERSALSDK}" != "/"
             then
+                CFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${CFLAGS}"
+                LDFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${LDFLAGS}"
                 CPPFLAGS="-isysroot ${UNIVERSALSDK} ${CPPFLAGS}"
+            else
+                CFLAGS="${UNIVERSAL_ARCH_FLAGS} ${CFLAGS}"
+                LDFLAGS="${UNIVERSAL_ARCH_FLAGS} ${LDFLAGS}"
             fi
         fi
 
diff --git a/configure.ac b/configure.ac
index af10a6d3cad5..525e572c1fff 100644
--- a/configure.ac
+++ b/configure.ac
@@ -212,7 +212,7 @@ fi
 AC_SUBST(LIPO_32BIT_FLAGS)
 AC_MSG_CHECKING(for --with-universal-archs)
 AC_ARG_WITH(universal-archs,
-    AS_HELP_STRING([--with-universal-archs=ARCH], [select architectures for universal build ("32-bit", "64-bit", "3-way", "intel", "intel-32", or "all")]),
+    AS_HELP_STRING([--with-universal-archs=ARCH], [select architectures for universal build ("32-bit", "64-bit", "3-way", "intel", "intel-32", "intel-64", or "all")]),
 [
 	UNIVERSAL_ARCHS="$withval"
 ],
@@ -1768,6 +1768,11 @@ yes)
                LIPO_32BIT_FLAGS=""
                ARCH_RUN_32BIT=""
                ;;
+            intel-64)
+               UNIVERSAL_ARCH_FLAGS="-arch x86_64"
+               LIPO_32BIT_FLAGS=""
+               ARCH_RUN_32BIT="true"
+               ;;
             3-way)
                UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch x86_64"
                LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386"
@@ -1778,11 +1783,14 @@ yes)
                ;;
             esac
 
-            CFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${CFLAGS}"
-            LDFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${LDFLAGS}"
             if test "${UNIVERSALSDK}" != "/"
             then
+                CFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${CFLAGS}"
+                LDFLAGS="${UNIVERSAL_ARCH_FLAGS} -isysroot ${UNIVERSALSDK} ${LDFLAGS}"
                 CPPFLAGS="-isysroot ${UNIVERSALSDK} ${CPPFLAGS}"
+            else
+                CFLAGS="${UNIVERSAL_ARCH_FLAGS} ${CFLAGS}"
+                LDFLAGS="${UNIVERSAL_ARCH_FLAGS} ${LDFLAGS}"
             fi
         fi
 


From webhook-mailer at python.org  Tue Jan 30 12:03:19 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 17:03:19 -0000
Subject: [Python-checkins] compare with difflib not diff(1) (GH-5450)
Message-ID: <mailman.87.1517331799.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/1e17d4aaff5c7ca972bab437949d2bb51c5b30f7
commit: 1e17d4aaff5c7ca972bab437949d2bb51c5b30f7
branch: master
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-30T09:03:12-08:00
summary:

compare with difflib not diff(1) (GH-5450)

files:
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index 15f5b63f1f1e..0cbba26bec04 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -12,6 +12,7 @@
 from test.support import verbose
 
 # Python imports
+import difflib
 import importlib
 import operator
 import os
@@ -429,8 +430,8 @@ def test_all_project_files(self):
                 except ParseError as err:
                     self.fail('ParseError on file %s (%s)' % (filepath, err))
             new = str(tree)
-            x = diff(filepath, new, encoding=encoding)
-            if x:
+            if new != source:
+                print(diff_texts(source, new, filepath))
                 self.fail("Idempotency failed: %s" % filepath)
 
     def test_extended_unpacking(self):
@@ -480,17 +481,12 @@ def test_trailing_comma_after_generator_expression_argument_works(self):
         self.validate("set(x for x in [],)")
 
 
-def diff(fn, result, encoding='utf-8'):
-    try:
-        with open('@', 'w', encoding=encoding, newline='\n') as f:
-            f.write(str(result))
-        fn = fn.replace('"', '\\"')
-        return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))
-    finally:
-        try:
-            os.remove("@")
-        except OSError:
-            pass
+def diff_texts(a, b, filename):
+    a = a.splitlines()
+    b = b.splitlines()
+    return difflib.unified_diff(a, b, filename, filename,
+                                "(original)", "(reserialized)",
+                                lineterm="")
 
 
 if __name__ == '__main__':


From webhook-mailer at python.org  Tue Jan 30 13:23:21 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 18:23:21 -0000
Subject: [Python-checkins] [3.6] closes bpo-30117: fix lib2to3
 ParserIdempotency test (GH-1242) (GH-5443)
Message-ID: <mailman.88.1517336604.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/a23a2c555c4187f349276fe2f2ceffa953d0afe9
commit: a23a2c555c4187f349276fe2f2ceffa953d0afe9
branch: 3.6
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-30T10:23:17-08:00
summary:

[3.6] closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242) (GH-5443)

Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.

    Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

    If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change..
(cherry picked from commit 14e976e00e65bf343ba0fca016c3c9132a843daf)

files:
M Lib/lib2to3/tests/support.py
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/tests/support.py b/Lib/lib2to3/tests/support.py
index ae7cfe8ee274..fe084e8903fc 100644
--- a/Lib/lib2to3/tests/support.py
+++ b/Lib/lib2to3/tests/support.py
@@ -15,7 +15,13 @@
 proj_dir = os.path.normpath(os.path.join(test_dir, ".."))
 grammar_path = os.path.join(test_dir, "..", "Grammar.txt")
 grammar = pgen2_driver.load_grammar(grammar_path)
+grammar_no_print_statement = pgen2_driver.load_grammar(grammar_path)
+del grammar_no_print_statement.keywords["print"]
 driver = pgen2_driver.Driver(grammar, convert=pytree.convert)
+driver_no_print_statement = pgen2_driver.Driver(
+    grammar_no_print_statement,
+    convert=pytree.convert
+)
 
 def parse_string(string):
     return driver.parse_string(reformat(string), debug=True)
diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index 0a139507b8de..9d444c204aa6 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -8,7 +8,7 @@
 
 # Testing imports
 from . import support
-from .support import driver
+from .support import driver, driver_no_print_statement
 from test.support import verbose
 
 # Python imports
@@ -413,8 +413,6 @@ class TestParserIdempotency(support.TestCase):
 
     """A cut-down version of pytree_idempotency.py."""
 
-    # Issue 13125
-    @unittest.expectedFailure
     def test_all_project_files(self):
         for filepath in support.all_project_files():
             with open(filepath, "rb") as fp:
@@ -425,12 +423,13 @@ def test_all_project_files(self):
                 source = fp.read()
             try:
                 tree = driver.parse_string(source)
-            except ParseError as err:
-                if verbose > 0:
-                    warnings.warn('ParseError on file %s (%s)' % (filepath, err))
-                continue
+            except ParseError:
+                try:
+                    tree = driver_no_print_statement.parse_string(source)
+                except ParseError as err:
+                    self.fail('ParseError on file %s (%s)' % (filepath, err))
             new = str(tree)
-            x = diff(filepath, new)
+            x = diff(filepath, new, encoding=encoding)
             if x:
                 self.fail("Idempotency failed: %s" % filepath)
 
@@ -474,9 +473,9 @@ def test_multiline_str_literals(self):
         self.validate(s)
 
 
-def diff(fn, result):
+def diff(fn, result, encoding='utf-8'):
     try:
-        with open('@', 'w') as f:
+        with open('@', 'w', encoding=encoding, newline='\n') as f:
             f.write(str(result))
         fn = fn.replace('"', '\\"')
         return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))


From webhook-mailer at python.org  Tue Jan 30 14:31:16 2018
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 30 Jan 2018 19:31:16 -0000
Subject: [Python-checkins] [3.6] compare with difflib not diff(1) (GH-5450)
 (GH-5453)
Message-ID: <mailman.89.1517340677.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/eb126eddbd7542ac9d7cd2736116aee2e0bd03dd
commit: eb126eddbd7542ac9d7cd2736116aee2e0bd03dd
branch: 3.6
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2018-01-30T11:31:10-08:00
summary:

[3.6] compare with difflib not diff(1) (GH-5450) (GH-5453)

Co-authored-by: Benjamin Peterson <benjamin at python.org>

files:
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index 9d444c204aa6..3fc499522454 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -12,6 +12,7 @@
 from test.support import verbose
 
 # Python imports
+import difflib
 import importlib
 import operator
 import os
@@ -429,8 +430,8 @@ def test_all_project_files(self):
                 except ParseError as err:
                     self.fail('ParseError on file %s (%s)' % (filepath, err))
             new = str(tree)
-            x = diff(filepath, new, encoding=encoding)
-            if x:
+            if new != source:
+                print(diff_texts(source, new, filepath))
                 self.fail("Idempotency failed: %s" % filepath)
 
     def test_extended_unpacking(self):
@@ -473,14 +474,9 @@ def test_multiline_str_literals(self):
         self.validate(s)
 
 
-def diff(fn, result, encoding='utf-8'):
-    try:
-        with open('@', 'w', encoding=encoding, newline='\n') as f:
-            f.write(str(result))
-        fn = fn.replace('"', '\\"')
-        return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))
-    finally:
-        try:
-            os.remove("@")
-        except OSError:
-            pass
+def diff_texts(a, b, filename):
+    a = a.splitlines()
+    b = b.splitlines()
+    return difflib.unified_diff(a, b, filename, filename,
+                                "(original)", "(reserialized)",
+                                lineterm="")


From webhook-mailer at python.org  Tue Jan 30 19:03:09 2018
From: webhook-mailer at python.org (R. David Murray)
Date: Wed, 31 Jan 2018 00:03:09 -0000
Subject: [Python-checkins] bpo-32727: smtplib's SMTP.send_message behaves
 differently with from_addr and to_addrs (#5451)
Message-ID: <mailman.90.1517356992.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/8d83e4ba7823827bcbc119db887004d5c3a63dc6
commit: 8d83e4ba7823827bcbc119db887004d5c3a63dc6
branch: master
author: St?phane Wirtel <stephane at wirtel.be>
committer: R. David Murray <rdmurray at bitdance.com>
date: 2018-01-30T19:02:51-05:00
summary:

bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs (#5451)

Do not pass the name field in the 'from' address in the SMTP envelope.

files:
A Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst
M Lib/smtplib.py
M Lib/test/test_smtplib.py

diff --git a/Lib/smtplib.py b/Lib/smtplib.py
index 5e422b704ad4..b679875fd2c5 100755
--- a/Lib/smtplib.py
+++ b/Lib/smtplib.py
@@ -933,6 +933,7 @@ def send_message(self, msg, from_addr=None, to_addrs=None,
             from_addr = (msg[header_prefix + 'Sender']
                            if (header_prefix + 'Sender') in msg
                            else msg[header_prefix + 'From'])
+            from_addr = email.utils.getaddresses([from_addr])[0][1]
         if to_addrs is None:
             addr_fields = [f for f in (msg[header_prefix + 'To'],
                                        msg[header_prefix + 'Bcc'],
diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py
index 040ad4e05962..7991174fb5d7 100644
--- a/Lib/test/test_smtplib.py
+++ b/Lib/test/test_smtplib.py
@@ -825,6 +825,7 @@ class SimSMTPServer(smtpd.SMTPServer):
 
     def __init__(self, *args, **kw):
         self._extra_features = []
+        self._addresses = {}
         smtpd.SMTPServer.__init__(self, *args, **kw)
 
     def handle_accepted(self, conn, addr):
@@ -833,7 +834,8 @@ def handle_accepted(self, conn, addr):
             decode_data=self._decode_data)
 
     def process_message(self, peer, mailfrom, rcpttos, data):
-        pass
+        self._addresses['from'] = mailfrom
+        self._addresses['tos'] = rcpttos
 
     def add_feature(self, feature):
         self._extra_features.append(feature)
@@ -1072,6 +1074,21 @@ def test_send_unicode_without_SMTPUTF8(self):
         self.assertRaises(UnicodeEncodeError, smtp.sendmail, 'Alice', 'B?b', '')
         self.assertRaises(UnicodeEncodeError, smtp.mail, '?lice')
 
+    def test_name_field_not_included_in_envelop_addresses(self):
+        smtp = smtplib.SMTP(
+            HOST, self.port, local_hostname='localhost', timeout=3
+        )
+        self.addCleanup(smtp.close)
+
+        message = EmailMessage()
+        message['From'] = email.utils.formataddr(('Micha?l', 'michael at example.com'))
+        message['To'] = email.utils.formataddr(('Ren?', 'rene at example.com'))
+
+        self.assertDictEqual(smtp.send_message(message), {})
+
+        self.assertEqual(self.serv._addresses['from'], 'michael at example.com')
+        self.assertEqual(self.serv._addresses['tos'], ['rene at example.com'])
+
 
 class SimSMTPUTF8Server(SimSMTPServer):
 
diff --git a/Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst b/Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst
new file mode 100644
index 000000000000..22c219636de2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst
@@ -0,0 +1 @@
+Do not include name field in SMTP envelope from address. Patch by St?phane Wirtel


From solipsis at pitrou.net  Wed Jan 31 04:08:52 2018
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 31 Jan 2018 09:08:52 +0000
Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-3
Message-ID: <20180131090852.1.7C31A4C894340A32@psf.io>

results for 4243df51fe43 on branch "default"
--------------------------------------------

test_collections leaked [-7, 8, -7] memory blocks, sum=-6
test_functools leaked [0, 3, 1] memory blocks, sum=4
test_multiprocessing_forkserver leaked [-2, 2, -1] memory blocks, sum=-1
test_multiprocessing_spawn leaked [0, -2, 2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogsse3Qi', '--timeout', '7200']

From webhook-mailer at python.org  Wed Jan 31 05:17:17 2018
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 31 Jan 2018 10:17:17 -0000
Subject: [Python-checkins] bpo-30693: zip+tarfile: sort directory listing
 (#2263)
Message-ID: <mailman.91.1517393841.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/84521047e413d7d1150aaa1c333580b683b3f4b1
commit: 84521047e413d7d1150aaa1c333580b683b3f4b1
branch: master
author: Bernhard M. Wiedemann <githubbmw at lsmod.de>
committer: Victor Stinner <victor.stinner at gmail.com>
date: 2018-01-31T11:17:10+01:00
summary:

bpo-30693: zip+tarfile: sort directory listing (#2263)

tarfile and zipfile now sort directory listing to generate tar and zip archives
in a more reproducible way.

See also https://reproducible-builds.org/docs/stable-inputs/ on that topic.

files:
A Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ7.rst
A Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ8.rst
M Doc/library/tarfile.rst
M Doc/library/zipfile.rst
M Lib/tarfile.py
M Lib/test/test_tarfile.py
M Lib/zipfile.py

diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst
index 2450716a1d91..9cd07158e7f6 100644
--- a/Doc/library/tarfile.rst
+++ b/Doc/library/tarfile.rst
@@ -451,7 +451,8 @@ be finalized; only the internally used file object will be closed. See the
    (directory, fifo, symbolic link, etc.). If given, *arcname* specifies an
    alternative name for the file in the archive. Directories are added
    recursively by default. This can be avoided by setting *recursive* to
-   :const:`False`.  If *filter* is given, it
+   :const:`False`. Recursion adds entries in sorted order.
+   If *filter* is given, it
    should be a function that takes a :class:`TarInfo` object argument and
    returns the changed :class:`TarInfo` object. If it instead returns
    :const:`None` the :class:`TarInfo` object will be excluded from the
@@ -460,6 +461,9 @@ be finalized; only the internally used file object will be closed. See the
    .. versionchanged:: 3.2
       Added the *filter* parameter.
 
+   .. versionchanged:: 3.7
+      Recursion adds entries in sorted order.
+
 
 .. method:: TarFile.addfile(tarinfo, fileobj=None)
 
diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst
index 7c9a8c802254..c0f2a89a3a17 100644
--- a/Doc/library/zipfile.rst
+++ b/Doc/library/zipfile.rst
@@ -491,7 +491,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the
       :file:`\*.pyc` are added at the top level.  If the directory is a
       package directory, then all :file:`\*.pyc` are added under the package
       name as a file path, and if any subdirectories are package directories,
-      all of these are added recursively.
+      all of these are added recursively in sorted order.
 
       *basename* is intended for internal use only.
 
@@ -524,6 +524,9 @@ The :class:`PyZipFile` constructor takes the same parameters as the
       .. versionchanged:: 3.6.2
          The *pathname* parameter accepts a :term:`path-like object`.
 
+      .. versionchanged:: 3.7
+         Recursion sorts directory entries.
+
 
 .. _zipinfo-objects:
 
diff --git a/Lib/tarfile.py b/Lib/tarfile.py
index 0b8d31f85cf3..a24ee42abf82 100755
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -1943,7 +1943,7 @@ def add(self, name, arcname=None, recursive=True, *, filter=None):
         elif tarinfo.isdir():
             self.addfile(tarinfo)
             if recursive:
-                for f in os.listdir(name):
+                for f in sorted(os.listdir(name)):
                     self.add(os.path.join(name, f), os.path.join(arcname, f),
                             recursive, filter=filter)
 
diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py
index 179cbc6dfffc..8ef4294921b2 100644
--- a/Lib/test/test_tarfile.py
+++ b/Lib/test/test_tarfile.py
@@ -1129,6 +1129,30 @@ def test_directory_size(self):
         finally:
             support.rmdir(path)
 
+    # mock the following:
+    #  os.listdir: so we know that files are in the wrong order
+    @unittest.mock.patch('os.listdir')
+    def test_ordered_recursion(self, mock_listdir):
+        path = os.path.join(TEMPDIR, "directory")
+        os.mkdir(path)
+        open(os.path.join(path, "1"), "a").close()
+        open(os.path.join(path, "2"), "a").close()
+        mock_listdir.return_value = ["2", "1"]
+        try:
+            tar = tarfile.open(tmpname, self.mode)
+            try:
+                tar.add(path)
+                paths = []
+                for m in tar.getmembers():
+                    paths.append(os.path.split(m.name)[-1])
+                self.assertEqual(paths, ["directory", "1", "2"]);
+            finally:
+                tar.close()
+        finally:
+            support.unlink(os.path.join(path, "1"))
+            support.unlink(os.path.join(path, "2"))
+            support.rmdir(path)
+
     def test_gettarinfo_pathlike_name(self):
         with tarfile.open(tmpname, self.mode) as tar:
             path = pathlib.Path(TEMPDIR) / "file"
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 5df7b1bf75b9..b90b60f72e2b 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -1940,7 +1940,7 @@ def writepy(self, pathname, basename="", filterfunc=None):
                 if self.debug:
                     print("Adding", arcname)
                 self.write(fname, arcname)
-                dirlist = os.listdir(pathname)
+                dirlist = sorted(os.listdir(pathname))
                 dirlist.remove("__init__.py")
                 # Add all *.py files and package subdirectories
                 for filename in dirlist:
@@ -1965,7 +1965,7 @@ def writepy(self, pathname, basename="", filterfunc=None):
                 # This is NOT a package directory, add its files at top level
                 if self.debug:
                     print("Adding files from directory", pathname)
-                for filename in os.listdir(pathname):
+                for filename in sorted(os.listdir(pathname)):
                     path = os.path.join(pathname, filename)
                     root, ext = os.path.splitext(filename)
                     if ext == ".py":
@@ -2116,7 +2116,7 @@ def addToZip(zf, path, zippath):
             elif os.path.isdir(path):
                 if zippath:
                     zf.write(path, zippath)
-                for nm in os.listdir(path):
+                for nm in sorted(os.listdir(path)):
                     addToZip(zf,
                              os.path.join(path, nm), os.path.join(zippath, nm))
             # else: ignore
diff --git a/Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ7.rst b/Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ7.rst
new file mode 100644
index 000000000000..9c895c53de12
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ7.rst
@@ -0,0 +1 @@
+The ZipFile class now recurses directories in a reproducible way.
diff --git a/Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ8.rst b/Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ8.rst
new file mode 100644
index 000000000000..a622e7ed6e5d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-11-27-15-09-49.bpo-30693.yC4mJ8.rst
@@ -0,0 +1 @@
+The TarFile class now recurses directories in a reproducible way.


From webhook-mailer at python.org  Wed Jan 31 07:48:15 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Wed, 31 Jan 2018 12:48:15 -0000
Subject: [Python-checkins] bpo-32583: Fix possible crashing in builtin
 Unicode decoders (#5325)
Message-ID: <mailman.92.1517402898.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/2c7fd46e11333ef5e5cce34212f7d087694f3658
commit: 2c7fd46e11333ef5e5cce34212f7d087694f3658
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-31T20:48:05+08:00
summary:

bpo-32583: Fix possible crashing in builtin Unicode decoders (#5325)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash.

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst
M Lib/test/test_codeccallbacks.py
M Objects/unicodeobject.c

diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
index 0c066e633fa8..e2e7463389ef 100644
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1044,6 +1044,58 @@ def mutating(exc):
             for (encoding, data) in baddata:
                 self.assertEqual(data.decode(encoding, "test.mutating"), "\u4242")
 
+    # issue32583
+    def test_crashing_decode_handler(self):
+        # better generating one more character to fill the extra space slot
+        # so in debug build it can steadily fail
+        def forward_shorter_than_end(exc):
+            if isinstance(exc, UnicodeDecodeError):
+                # size one character, 0 < forward < exc.end
+                return ('\ufffd', exc.start+1)
+            else:
+                raise TypeError("don't know how to handle %r" % exc)
+        codecs.register_error(
+            "test.forward_shorter_than_end", forward_shorter_than_end)
+
+        self.assertEqual(
+            b'\xd8\xd8\xd8\xd8\xd8\x00\x00\x00'.decode(
+                'utf-16-le', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\ufffd\xd8\x00'
+        )
+        self.assertEqual(
+            b'\xd8\xd8\xd8\xd8\x00\xd8\x00\x00'.decode(
+                'utf-16-be', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\ufffd\xd8\x00'
+        )
+        self.assertEqual(
+            b'\x11\x11\x11\x11\x11\x00\x00\x00\x00\x00\x00'.decode(
+                'utf-32-le', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\u1111\x00'
+        )
+        self.assertEqual(
+            b'\x11\x11\x11\x00\x00\x11\x11\x00\x00\x00\x00'.decode(
+                'utf-32-be', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\u1111\x00'
+        )
+
+        def replace_with_long(exc):
+            if isinstance(exc, UnicodeDecodeError):
+                exc.object = b"\x00" * 8
+                return ('\ufffd', exc.start)
+            else:
+                raise TypeError("don't know how to handle %r" % exc)
+        codecs.register_error("test.replace_with_long", replace_with_long)
+
+        self.assertEqual(
+            b'\x00'.decode('utf-16', 'test.replace_with_long'),
+            '\ufffd\x00\x00\x00\x00'
+        )
+        self.assertEqual(
+            b'\x00'.decode('utf-32', 'test.replace_with_long'),
+            '\ufffd\x00\x00'
+        )
+
+
     def test_fake_error_class(self):
         handlers = [
             codecs.strict_errors,
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst
new file mode 100644
index 000000000000..45f1d043f9d6
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst	
@@ -0,0 +1,2 @@
+Fix possible crashing in builtin Unicode decoders caused by write
+out-of-bound errors when using customized decode error handlers.
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 775bd15cf6d1..3d9e09d7fab1 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4190,7 +4190,10 @@ unicode_decode_call_errorhandler_writer(
     Py_ssize_t insize;
     Py_ssize_t newpos;
     Py_ssize_t replen;
+    Py_ssize_t remain;
     PyObject *inputobj = NULL;
+    int need_to_grow = 0;
+    const char *new_inptr;
 
     if (*errorHandler == NULL) {
         *errorHandler = PyCodec_LookupError(errors);
@@ -4221,6 +4224,7 @@ unicode_decode_call_errorhandler_writer(
     inputobj = PyUnicodeDecodeError_GetObject(*exceptionObject);
     if (!inputobj)
         goto onError;
+    remain = *inend - *input - *endinpos;
     *input = PyBytes_AS_STRING(inputobj);
     insize = PyBytes_GET_SIZE(inputobj);
     *inend = *input + insize;
@@ -4238,6 +4242,19 @@ unicode_decode_call_errorhandler_writer(
     replen = PyUnicode_GET_LENGTH(repunicode);
     if (replen > 1) {
         writer->min_length += replen - 1;
+        need_to_grow = 1;
+    }
+    new_inptr = *input + newpos;
+    if (*inend - new_inptr > remain) {
+        /* We don't know the decoding algorithm here so we make the worst
+           assumption that one byte decodes to one unicode character.
+           If unfortunately one byte could decode to more unicode characters,
+           the decoder may write out-of-bound then.  Is it possible for the
+           algorithms using this function? */
+        writer->min_length += *inend - new_inptr - remain;
+        need_to_grow = 1;
+    }
+    if (need_to_grow) {
         writer->overallocate = 1;
         if (_PyUnicodeWriter_Prepare(writer, writer->min_length,
                             PyUnicode_MAX_CHAR_VALUE(repunicode)) == -1)
@@ -4247,7 +4264,7 @@ unicode_decode_call_errorhandler_writer(
         goto onError;
 
     *endinpos = newpos;
-    *inptr = *input + newpos;
+    *inptr = new_inptr;
 
     /* we made it! */
     Py_DECREF(restuple);
@@ -5572,7 +5589,8 @@ PyUnicode_DecodeUTF16Stateful(const char *s,
 #endif
 
     /* Note: size will always be longer than the resulting Unicode
-       character count */
+       character count normally.  Error handler will take care of
+       resizing when needed. */
     _PyUnicodeWriter_Init(&writer);
     writer.min_length = (e - q + 1) / 2;
     if (_PyUnicodeWriter_Prepare(&writer, writer.min_length, 127) == -1)


From webhook-mailer at python.org  Wed Jan 31 08:34:20 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Wed, 31 Jan 2018 13:34:20 -0000
Subject: [Python-checkins] [3.6] bpo-32583: Fix possible crashing in builtin
 Unicode decoders (GH-5325) (#5459)
Message-ID: <mailman.93.1517405664.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/ea94fce6960d90fffeeda131e31024617912d231
commit: ea94fce6960d90fffeeda131e31024617912d231
branch: 3.6
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-31T21:34:17+08:00
summary:

[3.6] bpo-32583: Fix possible crashing in builtin Unicode decoders (GH-5325) (#5459)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash..
(cherry picked from commit 2c7fd46e11333ef5e5cce34212f7d087694f3658)

files:
A Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst
M Lib/test/test_codeccallbacks.py
M Objects/unicodeobject.c

diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
index 0c066e633fa8..e2e7463389ef 100644
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1044,6 +1044,58 @@ def mutating(exc):
             for (encoding, data) in baddata:
                 self.assertEqual(data.decode(encoding, "test.mutating"), "\u4242")
 
+    # issue32583
+    def test_crashing_decode_handler(self):
+        # better generating one more character to fill the extra space slot
+        # so in debug build it can steadily fail
+        def forward_shorter_than_end(exc):
+            if isinstance(exc, UnicodeDecodeError):
+                # size one character, 0 < forward < exc.end
+                return ('\ufffd', exc.start+1)
+            else:
+                raise TypeError("don't know how to handle %r" % exc)
+        codecs.register_error(
+            "test.forward_shorter_than_end", forward_shorter_than_end)
+
+        self.assertEqual(
+            b'\xd8\xd8\xd8\xd8\xd8\x00\x00\x00'.decode(
+                'utf-16-le', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\ufffd\xd8\x00'
+        )
+        self.assertEqual(
+            b'\xd8\xd8\xd8\xd8\x00\xd8\x00\x00'.decode(
+                'utf-16-be', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\ufffd\xd8\x00'
+        )
+        self.assertEqual(
+            b'\x11\x11\x11\x11\x11\x00\x00\x00\x00\x00\x00'.decode(
+                'utf-32-le', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\u1111\x00'
+        )
+        self.assertEqual(
+            b'\x11\x11\x11\x00\x00\x11\x11\x00\x00\x00\x00'.decode(
+                'utf-32-be', 'test.forward_shorter_than_end'),
+            '\ufffd\ufffd\ufffd\u1111\x00'
+        )
+
+        def replace_with_long(exc):
+            if isinstance(exc, UnicodeDecodeError):
+                exc.object = b"\x00" * 8
+                return ('\ufffd', exc.start)
+            else:
+                raise TypeError("don't know how to handle %r" % exc)
+        codecs.register_error("test.replace_with_long", replace_with_long)
+
+        self.assertEqual(
+            b'\x00'.decode('utf-16', 'test.replace_with_long'),
+            '\ufffd\x00\x00\x00\x00'
+        )
+        self.assertEqual(
+            b'\x00'.decode('utf-32', 'test.replace_with_long'),
+            '\ufffd\x00\x00'
+        )
+
+
     def test_fake_error_class(self):
         handlers = [
             codecs.strict_errors,
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst b/Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst
new file mode 100644
index 000000000000..45f1d043f9d6
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2018-01-26-21-20-21.bpo-32583.Fh3fau.rst	
@@ -0,0 +1,2 @@
+Fix possible crashing in builtin Unicode decoders caused by write
+out-of-bound errors when using customized decode error handlers.
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 9ccd06eebb48..a246756d9412 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4429,7 +4429,10 @@ unicode_decode_call_errorhandler_writer(
     Py_ssize_t insize;
     Py_ssize_t newpos;
     Py_ssize_t replen;
+    Py_ssize_t remain;
     PyObject *inputobj = NULL;
+    int need_to_grow = 0;
+    const char *new_inptr;
 
     if (*errorHandler == NULL) {
         *errorHandler = PyCodec_LookupError(errors);
@@ -4463,6 +4466,7 @@ unicode_decode_call_errorhandler_writer(
     if (!PyBytes_Check(inputobj)) {
         PyErr_Format(PyExc_TypeError, "exception attribute object must be bytes");
     }
+    remain = *inend - *input - *endinpos;
     *input = PyBytes_AS_STRING(inputobj);
     insize = PyBytes_GET_SIZE(inputobj);
     *inend = *input + insize;
@@ -4482,6 +4486,19 @@ unicode_decode_call_errorhandler_writer(
     replen = PyUnicode_GET_LENGTH(repunicode);
     if (replen > 1) {
         writer->min_length += replen - 1;
+        need_to_grow = 1;
+    }
+    new_inptr = *input + newpos;
+    if (*inend - new_inptr > remain) {
+        /* We don't know the decoding algorithm here so we make the worst
+           assumption that one byte decodes to one unicode character.
+           If unfortunately one byte could decode to more unicode characters,
+           the decoder may write out-of-bound then.  Is it possible for the
+           algorithms using this function? */
+        writer->min_length += *inend - new_inptr - remain;
+        need_to_grow = 1;
+    }
+    if (need_to_grow) {
         writer->overallocate = 1;
         if (_PyUnicodeWriter_Prepare(writer, writer->min_length,
                             PyUnicode_MAX_CHAR_VALUE(repunicode)) == -1)
@@ -4491,7 +4508,7 @@ unicode_decode_call_errorhandler_writer(
         goto onError;
 
     *endinpos = newpos;
-    *inptr = *input + newpos;
+    *inptr = new_inptr;
 
     /* we made it! */
     Py_XDECREF(restuple);
@@ -5663,7 +5680,8 @@ PyUnicode_DecodeUTF16Stateful(const char *s,
 #endif
 
     /* Note: size will always be longer than the resulting Unicode
-       character count */
+       character count normally.  Error handler will take care of
+       resizing when needed. */
     _PyUnicodeWriter_Init(&writer);
     writer.min_length = (e - q + 1) / 2;
     if (_PyUnicodeWriter_Prepare(&writer, writer.min_length, 127) == -1)


From webhook-mailer at python.org  Wed Jan 31 10:50:53 2018
From: webhook-mailer at python.org (Xiang Zhang)
Date: Wed, 31 Jan 2018 15:50:53 -0000
Subject: [Python-checkins] Make site documentation more clear (#5461)
Message-ID: <mailman.94.1517413854.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/a29ddf407831c9103e4df4850df6d67b4352f4f9
commit: a29ddf407831c9103e4df4850df6d67b4352f4f9
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2018-01-31T23:50:50+08:00
summary:

Make site documentation more clear (#5461)

Mention only ImportError caused by importing sitecustomize.py/usercustomize.py
themselves will be silently ignored.

files:
M Doc/library/site.rst

diff --git a/Doc/library/site.rst b/Doc/library/site.rst
index 43daf790b77c..ae408133a7ce 100644
--- a/Doc/library/site.rst
+++ b/Doc/library/site.rst
@@ -97,12 +97,12 @@ not mentioned in either path configuration file.
 After these path manipulations, an attempt is made to import a module named
 :mod:`sitecustomize`, which can perform arbitrary site-specific customizations.
 It is typically created by a system administrator in the site-packages
-directory.  If this import fails with an :exc:`ImportError` exception, it is
-silently ignored.  If Python is started without output streams available, as
+directory.  If this import fails with an :exc:`ImportError` or its subclass
+exception, and the exception's :attr:`name` attribute equals to ``'sitecustomize'``,
+it is silently ignored.  If Python is started without output streams available, as
 with :file:`pythonw.exe` on Windows (which is used by default to start IDLE),
-attempted output from :mod:`sitecustomize` is ignored. Any exception other
-than :exc:`ImportError` causes a silent and perhaps mysterious failure of the
-process.
+attempted output from :mod:`sitecustomize` is ignored.  Any other exception
+causes a silent and perhaps mysterious failure of the process.
 
 .. index:: module: usercustomize
 
@@ -110,7 +110,9 @@ After this, an attempt is made to import a module named :mod:`usercustomize`,
 which can perform arbitrary user-specific customizations, if
 :data:`ENABLE_USER_SITE` is true.  This file is intended to be created in the
 user site-packages directory (see below), which is part of ``sys.path`` unless
-disabled by :option:`-s`.  An :exc:`ImportError` will be silently ignored.
+disabled by :option:`-s`.  If this import fails with an :exc:`ImportError` or
+its subclass exception, and the exception's :attr:`name` attribute equals to
+``'usercustomize'``, it is silently ignored.
 
 Note that for some non-Unix systems, ``sys.prefix`` and ``sys.exec_prefix`` are
 empty, and the path manipulations are skipped; however the import of


From webhook-mailer at python.org  Wed Jan 31 12:19:38 2018
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 31 Jan 2018 17:19:38 -0000
Subject: [Python-checkins] bpo-25988: Emit a warning when use or import ABCs
 from 'collections'. (#5460)
Message-ID: <mailman.95.1517419180.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/c66f9f8d3909f588c251957d499599a1680e2320
commit: c66f9f8d3909f588c251957d499599a1680e2320
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2018-01-31T19:19:33+02:00
summary:

bpo-25988: Emit a warning when use or import ABCs from 'collections'. (#5460)

files:
M Lib/collections/__init__.py
M Modules/_decimal/_decimal.c

diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 8aeee359cd62..f0b41fd9d10d 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -17,16 +17,7 @@
 __all__ = ['deque', 'defaultdict', 'namedtuple', 'UserDict', 'UserList',
             'UserString', 'Counter', 'OrderedDict', 'ChainMap']
 
-# For backwards compatibility, continue to make the collections ABCs
-# through Python 3.6 available through the collections module.
-# Note, no new collections ABCs were added in Python 3.7
 import _collections_abc
-from _collections_abc import (AsyncGenerator, AsyncIterable, AsyncIterator,
-    Awaitable, ByteString, Callable, Collection, Container, Coroutine,
-    Generator, Hashable, ItemsView, Iterable, Iterator, KeysView, Mapping,
-    MappingView, MutableMapping, MutableSequence, MutableSet, Reversible,
-    Sequence, Set, Sized, ValuesView)
-
 from operator import itemgetter as _itemgetter, eq as _eq
 from keyword import iskeyword as _iskeyword
 import sys as _sys
@@ -40,7 +31,7 @@
 except ImportError:
     pass
 else:
-    MutableSequence.register(deque)
+    _collections_abc.MutableSequence.register(deque)
 
 try:
     from _collections import defaultdict
@@ -48,22 +39,37 @@
     pass
 
 
+def __getattr__(name):
+    # For backwards compatibility, continue to make the collections ABCs
+    # through Python 3.6 available through the collections module.
+    # Note, no new collections ABCs were added in Python 3.7
+    if name in _collections_abc.__all__:
+        obj = getattr(_collections_abc, name)
+        import warnings
+        warnings.warn("Using or importing the ABCs from 'collections' instead "
+                      "of from 'collections.abc' is deprecated, "
+                      "and in 3.8 it will stop working",
+                      DeprecationWarning, stacklevel=2)
+        globals()[name] = obj
+        return obj
+    raise AttributeError(f'module {__name__!r} has no attribute {name!r}')
+
 ################################################################################
 ### OrderedDict
 ################################################################################
 
-class _OrderedDictKeysView(KeysView):
+class _OrderedDictKeysView(_collections_abc.KeysView):
 
     def __reversed__(self):
         yield from reversed(self._mapping)
 
-class _OrderedDictItemsView(ItemsView):
+class _OrderedDictItemsView(_collections_abc.ItemsView):
 
     def __reversed__(self):
         for key in reversed(self._mapping):
             yield (key, self._mapping[key])
 
-class _OrderedDictValuesView(ValuesView):
+class _OrderedDictValuesView(_collections_abc.ValuesView):
 
     def __reversed__(self):
         for key in reversed(self._mapping):
@@ -215,7 +221,7 @@ def __sizeof__(self):
         size += sizeof(self.__root) * n         # proxy objects
         return size
 
-    update = __update = MutableMapping.update
+    update = __update = _collections_abc.MutableMapping.update
 
     def keys(self):
         "D.keys() -> a set-like object providing a view on D's keys"
@@ -229,7 +235,7 @@ def values(self):
         "D.values() -> an object providing a view on D's values"
         return _OrderedDictValuesView(self)
 
-    __ne__ = MutableMapping.__ne__
+    __ne__ = _collections_abc.MutableMapping.__ne__
 
     __marker = object()
 
@@ -636,7 +642,7 @@ def update(*args, **kwds):
             raise TypeError('expected at most 1 arguments, got %d' % len(args))
         iterable = args[0] if args else None
         if iterable is not None:
-            if isinstance(iterable, Mapping):
+            if isinstance(iterable, _collections_abc.Mapping):
                 if self:
                     self_get = self.get
                     for elem, count in iterable.items():
@@ -673,7 +679,7 @@ def subtract(*args, **kwds):
         iterable = args[0] if args else None
         if iterable is not None:
             self_get = self.get
-            if isinstance(iterable, Mapping):
+            if isinstance(iterable, _collections_abc.Mapping):
                 for elem, count in iterable.items():
                     self[elem] = self_get(elem, 0) - count
             else:
@@ -875,7 +881,7 @@ def __iand__(self, other):
 ###  ChainMap
 ########################################################################
 
-class ChainMap(MutableMapping):
+class ChainMap(_collections_abc.MutableMapping):
     ''' A ChainMap groups multiple dicts (or other mappings) together
     to create a single, updateable view.
 
@@ -983,7 +989,7 @@ def clear(self):
 ### UserDict
 ################################################################################
 
-class UserDict(MutableMapping):
+class UserDict(_collections_abc.MutableMapping):
 
     # Start by filling-out the abstract methods
     def __init__(*args, **kwargs):
@@ -1050,7 +1056,7 @@ def fromkeys(cls, iterable, value=None):
 ### UserList
 ################################################################################
 
-class UserList(MutableSequence):
+class UserList(_collections_abc.MutableSequence):
     """A more or less complete user-defined wrapper around list objects."""
     def __init__(self, initlist=None):
         self.data = []
@@ -1123,7 +1129,7 @@ def extend(self, other):
 ### UserString
 ################################################################################
 
-class UserString(Sequence):
+class UserString(_collections_abc.Sequence):
     def __init__(self, seq):
         if isinstance(seq, str):
             self.data = seq
diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c
index fddb39ef652a..5bce780cb7fb 100644
--- a/Modules/_decimal/_decimal.c
+++ b/Modules/_decimal/_decimal.c
@@ -5521,6 +5521,7 @@ PyInit__decimal(void)
     PyObject *numbers = NULL;
     PyObject *Number = NULL;
     PyObject *collections = NULL;
+    PyObject *collections_abc = NULL;
     PyObject *MutableMapping = NULL;
     PyObject *obj = NULL;
     DecCondMap *cm;
@@ -5595,7 +5596,8 @@ PyInit__decimal(void)
     Py_CLEAR(obj);
 
     /* MutableMapping */
-    ASSIGN_PTR(MutableMapping, PyObject_GetAttrString(collections,
+    ASSIGN_PTR(collections_abc, PyImport_ImportModule("collections.abc"));
+    ASSIGN_PTR(MutableMapping, PyObject_GetAttrString(collections_abc,
                                                       "MutableMapping"));
     /* Create SignalDict type */
     ASSIGN_PTR(PyDecSignalDict_Type,
@@ -5606,6 +5608,7 @@ PyInit__decimal(void)
 
     /* Done with collections, MutableMapping */
     Py_CLEAR(collections);
+    Py_CLEAR(collections_abc);
     Py_CLEAR(MutableMapping);
 
 
@@ -5765,6 +5768,7 @@ PyInit__decimal(void)
     Py_CLEAR(Number); /* GCOV_NOT_REACHED */
     Py_CLEAR(Rational); /* GCOV_NOT_REACHED */
     Py_CLEAR(collections); /* GCOV_NOT_REACHED */
+    Py_CLEAR(collections_abc); /* GCOV_NOT_REACHED */
     Py_CLEAR(MutableMapping); /* GCOV_NOT_REACHED */
     Py_CLEAR(SignalTuple); /* GCOV_NOT_REACHED */
     Py_CLEAR(DecimalTuple); /* GCOV_NOT_REACHED */


From webhook-mailer at python.org  Wed Jan 31 15:54:12 2018
From: webhook-mailer at python.org (R. David Murray)
Date: Wed, 31 Jan 2018 20:54:12 -0000
Subject: [Python-checkins] bpo-32727: smtplib's SMTP.send_message behaves
 differently with from_addr and to_addrs (GH-5451) (#5455)
Message-ID: <mailman.96.1517432055.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/85a92d00bd278022cc00fda6b124c06f614c5ebb
commit: 85a92d00bd278022cc00fda6b124c06f614c5ebb
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: R. David Murray <rdmurray at bitdance.com>
date: 2018-01-31T15:54:09-05:00
summary:

bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs (GH-5451) (#5455)

Do not pass the name field in the 'from' address in the SMTP envelope.
(cherry picked from commit 8d83e4ba7823827bcbc119db887004d5c3a63dc6)

Co-authored-by: St?phane Wirtel <stephane at wirtel.be>

files:
A Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst
M Lib/smtplib.py
M Lib/test/test_smtplib.py

diff --git a/Lib/smtplib.py b/Lib/smtplib.py
index 5e422b704ad4..b679875fd2c5 100755
--- a/Lib/smtplib.py
+++ b/Lib/smtplib.py
@@ -933,6 +933,7 @@ def send_message(self, msg, from_addr=None, to_addrs=None,
             from_addr = (msg[header_prefix + 'Sender']
                            if (header_prefix + 'Sender') in msg
                            else msg[header_prefix + 'From'])
+            from_addr = email.utils.getaddresses([from_addr])[0][1]
         if to_addrs is None:
             addr_fields = [f for f in (msg[header_prefix + 'To'],
                                        msg[header_prefix + 'Bcc'],
diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py
index dd3a6eeca69c..67b3b478f76f 100644
--- a/Lib/test/test_smtplib.py
+++ b/Lib/test/test_smtplib.py
@@ -818,6 +818,7 @@ class SimSMTPServer(smtpd.SMTPServer):
 
     def __init__(self, *args, **kw):
         self._extra_features = []
+        self._addresses = {}
         smtpd.SMTPServer.__init__(self, *args, **kw)
 
     def handle_accepted(self, conn, addr):
@@ -826,7 +827,8 @@ def handle_accepted(self, conn, addr):
             decode_data=self._decode_data)
 
     def process_message(self, peer, mailfrom, rcpttos, data):
-        pass
+        self._addresses['from'] = mailfrom
+        self._addresses['tos'] = rcpttos
 
     def add_feature(self, feature):
         self._extra_features.append(feature)
@@ -1066,6 +1068,21 @@ def test_send_unicode_without_SMTPUTF8(self):
         self.assertRaises(UnicodeEncodeError, smtp.sendmail, 'Alice', 'B?b', '')
         self.assertRaises(UnicodeEncodeError, smtp.mail, '?lice')
 
+    def test_name_field_not_included_in_envelop_addresses(self):
+        smtp = smtplib.SMTP(
+            HOST, self.port, local_hostname='localhost', timeout=3
+        )
+        self.addCleanup(smtp.close)
+
+        message = EmailMessage()
+        message['From'] = email.utils.formataddr(('Micha?l', 'michael at example.com'))
+        message['To'] = email.utils.formataddr(('Ren?', 'rene at example.com'))
+
+        self.assertDictEqual(smtp.send_message(message), {})
+
+        self.assertEqual(self.serv._addresses['from'], 'michael at example.com')
+        self.assertEqual(self.serv._addresses['tos'], ['rene at example.com'])
+
 
 class SimSMTPUTF8Server(SimSMTPServer):
 
diff --git a/Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst b/Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst
new file mode 100644
index 000000000000..22c219636de2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-01-30-17-46-18.bpo-32727.aHVsRC.rst
@@ -0,0 +1 @@
+Do not include name field in SMTP envelope from address. Patch by St?phane Wirtel


From webhook-mailer at python.org  Wed Jan 31 16:38:00 2018
From: webhook-mailer at python.org (Mariatta)
Date: Wed, 31 Jan 2018 21:38:00 -0000
Subject: [Python-checkins] bpo-32735: Fix typo in f-strings datetime format
 specifier example (GH-5464)
Message-ID: <mailman.97.1517434683.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/b299393cc372f3ecbef4304f8eaa4c7838e975ca
commit: b299393cc372f3ecbef4304f8eaa4c7838e975ca
branch: master
author: Cheryl Sabella <cheryl.sabella at gmail.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-31T16:37:51-05:00
summary:

bpo-32735: Fix typo in f-strings datetime format specifier example (GH-5464)

The f-string example for using datetime format specifier does not match the given output.
Changed the format from %b to %B so it matches the output of "January".

files:
M Doc/reference/lexical_analysis.rst

diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst
index 95b0f53a83e6..c6d96ca612b8 100644
--- a/Doc/reference/lexical_analysis.rst
+++ b/Doc/reference/lexical_analysis.rst
@@ -676,7 +676,7 @@ Some examples of formatted string literals::
    >>> f"result: {value:{width}.{precision}}"  # nested fields
    'result:      12.35'
    >>> today = datetime(year=2017, month=1, day=27)
-   >>> f"{today:%b %d, %Y}"  # using date format specifier
+   >>> f"{today:%B %d, %Y}"  # using date format specifier
    'January 27, 2017'
    >>> number = 1024
    >>> f"{number:#0x}"  # using integer format specifier


From webhook-mailer at python.org  Wed Jan 31 17:17:11 2018
From: webhook-mailer at python.org (Mariatta)
Date: Wed, 31 Jan 2018 22:17:11 -0000
Subject: [Python-checkins] bpo-32735: Fix typo in f-strings datetime format
 specifier example (GH-5464) (GH-5465)
Message-ID: <mailman.98.1517437033.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/581ce25e1ffa374e62547ef266b6326bee0c54e5
commit: 581ce25e1ffa374e62547ef266b6326bee0c54e5
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Mariatta <Mariatta at users.noreply.github.com>
date: 2018-01-31T17:17:02-05:00
summary:

bpo-32735: Fix typo in f-strings datetime format specifier example (GH-5464) (GH-5465)

The f-string example for using datetime format specifier does not match the given output.
Changed the format from %b to %B so it matches the output of "January".
(cherry picked from commit b299393cc372f3ecbef4304f8eaa4c7838e975ca)

Co-authored-by: Cheryl Sabella <cheryl.sabella at gmail.com>

files:
M Doc/reference/lexical_analysis.rst

diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst
index 7ad83947e04a..6d093dcc2a22 100644
--- a/Doc/reference/lexical_analysis.rst
+++ b/Doc/reference/lexical_analysis.rst
@@ -676,7 +676,7 @@ Some examples of formatted string literals::
    >>> f"result: {value:{width}.{precision}}"  # nested fields
    'result:      12.35'
    >>> today = datetime(year=2017, month=1, day=27)
-   >>> f"{today:%b %d, %Y}"  # using date format specifier
+   >>> f"{today:%B %d, %Y}"  # using date format specifier
    'January 27, 2017'
    >>> number = 1024
    >>> f"{number:#0x}"  # using integer format specifier


From webhook-mailer at python.org  Wed Jan 31 18:33:40 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Wed, 31 Jan 2018 23:33:40 -0000
Subject: [Python-checkins] bpo-32726: Build macOS 10.9+ installer with
 private copy of Tcl/Tk 8.6
Message-ID: <mailman.100.1517441623.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/1ca2ffdc532622ef001418585c146e6b0f31027f
commit: 1ca2ffdc532622ef001418585c146e6b0f31027f
branch: master
author: Ned Deily <nad at python.org>
committer: Ned Deily <nad at python.org>
date: 2018-01-31T17:14:12-05:00
summary:

bpo-32726: Build macOS 10.9+ installer with private copy of Tcl/Tk 8.6

files:
D Mac/BuildScript/issue19373_tk_8_5_15_source.patch
M Mac/BuildScript/build-installer.py
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf
M Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst

diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
index 8e2e104a0b7e..1a7080c18f1b 100755
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -189,6 +189,11 @@ def getTargetCompilers():
 #                       '/Library/Frameworks/Tk.framework/Versions/8.5/Tk']
 EXPECTED_SHARED_LIBS = {}
 
+# Are we building and linking with our own copy of Tcl/TK?
+#   For now, do so if deployment target is 10.9+.
+def internalTk():
+    return getDeptargetTuple() >= (10, 9)
+
 # List of names of third party software built with this installer.
 # The names will be inserted into the rtf version of the License.
 THIRD_PARTY_LIBS = []
@@ -217,13 +222,12 @@ def library_recipes():
           ),
     ])
 
-#   Disable for now
-    if False:   # if getDeptargetTuple() > (10, 5):
+    if internalTk():
         result.extend([
           dict(
-              name="Tcl 8.5.15",
-              url="ftp://ftp.tcl.tk/pub/tcl//tcl8_5/tcl8.5.15-src.tar.gz",
-              checksum='f3df162f92c69b254079c4d0af7a690f',
+              name="Tcl 8.6.7",
+              url="ftp://ftp.tcl.tk/pub/tcl//tcl8_6/tcl8.6.7-src.tar.gz",
+              checksum='5673aaf45b5de5d8dd80bb3daaeb8838',
               buildDir="unix",
               configure_pre=[
                     '--enable-shared',
@@ -233,16 +237,13 @@ def library_recipes():
               useLDFlags=False,
               install='make TCL_LIBRARY=%(TCL_LIBRARY)s && make install TCL_LIBRARY=%(TCL_LIBRARY)s DESTDIR=%(DESTDIR)s'%{
                   "DESTDIR": shellQuote(os.path.join(WORKDIR, 'libraries')),
-                  "TCL_LIBRARY": shellQuote('/Library/Frameworks/Python.framework/Versions/%s/lib/tcl8.5'%(getVersion())),
+                  "TCL_LIBRARY": shellQuote('/Library/Frameworks/Python.framework/Versions/%s/lib/tcl8.6'%(getVersion())),
                   },
               ),
           dict(
-              name="Tk 8.5.15",
-              url="ftp://ftp.tcl.tk/pub/tcl//tcl8_5/tk8.5.15-src.tar.gz",
-              checksum='55b8e33f903210a4e1c8bce0f820657f',
-              patches=[
-                  "issue19373_tk_8_5_15_source.patch",
-                   ],
+              name="Tk 8.6.7",
+              url="ftp://ftp.tcl.tk/pub/tcl//tcl8_6/tk8.6.7-src.tar.gz",
+              checksum='46ea9c0165c515d87393700f4891ab6f',
               buildDir="unix",
               configure_pre=[
                     '--enable-aqua',
@@ -253,8 +254,8 @@ def library_recipes():
               useLDFlags=False,
               install='make TCL_LIBRARY=%(TCL_LIBRARY)s TK_LIBRARY=%(TK_LIBRARY)s && make install TCL_LIBRARY=%(TCL_LIBRARY)s TK_LIBRARY=%(TK_LIBRARY)s DESTDIR=%(DESTDIR)s'%{
                   "DESTDIR": shellQuote(os.path.join(WORKDIR, 'libraries')),
-                  "TCL_LIBRARY": shellQuote('/Library/Frameworks/Python.framework/Versions/%s/lib/tcl8.5'%(getVersion())),
-                  "TK_LIBRARY": shellQuote('/Library/Frameworks/Python.framework/Versions/%s/lib/tk8.5'%(getVersion())),
+                  "TCL_LIBRARY": shellQuote('/Library/Frameworks/Python.framework/Versions/%s/lib/tcl8.6'%(getVersion())),
+                  "TK_LIBRARY": shellQuote('/Library/Frameworks/Python.framework/Versions/%s/lib/tk8.6'%(getVersion())),
                   },
                 ),
         ])
@@ -547,42 +548,45 @@ def checkEnvironment():
         fatal("This script should be run on a macOS 10.5 (or later) system")
 
     # Because we only support dynamic load of only one major/minor version of
+    # Tcl/Tk, if we are not using building and using our own private copy of
     # Tcl/Tk, ensure:
     # 1. there is a user-installed framework (usually ActiveTcl) in (or linked
     #       in) SDKROOT/Library/Frameworks.  As of Python 3.7.0, we no longer
     #       enforce that the version of the user-installed framework also
     #       exists in the system-supplied Tcl/Tk frameworks.  Time to support
     #       Tcl/Tk 8.6 even if Apple does not.
-
-    frameworks = {}
-    for framework in ['Tcl', 'Tk']:
-        fwpth = 'Library/Frameworks/%s.framework/Versions/Current' % framework
-        libfw = os.path.join('/', fwpth)
-        usrfw = os.path.join(os.getenv('HOME'), fwpth)
-        frameworks[framework] = os.readlink(libfw)
-        if not os.path.exists(libfw):
-            fatal("Please install a link to a current %s %s as %s so "
-                    "the user can override the system framework."
-                    % (framework, frameworks[framework], libfw))
-        if os.path.exists(usrfw):
-            fatal("Please rename %s to avoid possible dynamic load issues."
-                    % usrfw)
-
-    if frameworks['Tcl'] != frameworks['Tk']:
-        fatal("The Tcl and Tk frameworks are not the same version.")
-
-    print(" -- Building with Tcl/Tk %s frameworks"
-                % frameworks['Tk'])
+    if not internalTk():
+        frameworks = {}
+        for framework in ['Tcl', 'Tk']:
+            fwpth = 'Library/Frameworks/%s.framework/Versions/Current' % framework
+            libfw = os.path.join('/', fwpth)
+            usrfw = os.path.join(os.getenv('HOME'), fwpth)
+            frameworks[framework] = os.readlink(libfw)
+            if not os.path.exists(libfw):
+                fatal("Please install a link to a current %s %s as %s so "
+                        "the user can override the system framework."
+                        % (framework, frameworks[framework], libfw))
+            if os.path.exists(usrfw):
+                fatal("Please rename %s to avoid possible dynamic load issues."
+                        % usrfw)
+
+        if frameworks['Tcl'] != frameworks['Tk']:
+            fatal("The Tcl and Tk frameworks are not the same version.")
+
+        print(" -- Building with external Tcl/Tk %s frameworks"
+                    % frameworks['Tk'])
+
+        # add files to check after build
+        EXPECTED_SHARED_LIBS['_tkinter.so'] = [
+                "/Library/Frameworks/Tcl.framework/Versions/%s/Tcl"
+                    % frameworks['Tcl'],
+                "/Library/Frameworks/Tk.framework/Versions/%s/Tk"
+                    % frameworks['Tk'],
+                ]
+    else:
+        print(" -- Building private copy of Tcl/Tk")
     print("")
 
-    # add files to check after build
-    EXPECTED_SHARED_LIBS['_tkinter.so'] = [
-            "/Library/Frameworks/Tcl.framework/Versions/%s/Tcl"
-                % frameworks['Tcl'],
-            "/Library/Frameworks/Tk.framework/Versions/%s/Tk"
-                % frameworks['Tk'],
-            ]
-
     # Remove inherited environment variables which might influence build
     environ_var_prefixes = ['CPATH', 'C_INCLUDE_', 'DYLD_', 'LANG', 'LC_',
                             'LD_', 'LIBRARY_', 'PATH', 'PYTHON']
@@ -1086,12 +1090,18 @@ def buildPython():
                "--with-universal-archs=%s "
                "%s "
                "%s "
+               "%s "
+               "%s "
                "LDFLAGS='-g -L%s/libraries/usr/local/lib' "
                "CFLAGS='-g -I%s/libraries/usr/local/include' 2>&1"%(
         shellQuote(os.path.join(SRCDIR, 'configure')),
         UNIVERSALARCHS,
         (' ', '--with-computed-gotos ')[PYTHON_3],
         (' ', '--without-ensurepip ')[PYTHON_3],
+        (' ', "--with-tcltk-includes='-I%s/libraries/usr/local/include'"%(
+                            shellQuote(WORKDIR)[1:-1],))[internalTk()],
+        (' ', "--with-tcltk-libs='-L%s/libraries/usr/local/lib -ltcl8.6 -ltk8.6'"%(
+                            shellQuote(WORKDIR)[1:-1],))[internalTk()],
         shellQuote(WORKDIR)[1:-1],
         shellQuote(WORKDIR)[1:-1]))
 
@@ -1126,14 +1136,22 @@ def buildPython():
     del os.environ['DYLD_LIBRARY_PATH']
     print("Copying required shared libraries")
     if os.path.exists(os.path.join(WORKDIR, 'libraries', 'Library')):
-        runCommand("mv %s/* %s"%(
-            shellQuote(os.path.join(
+        build_lib_dir = os.path.join(
                 WORKDIR, 'libraries', 'Library', 'Frameworks',
-                'Python.framework', 'Versions', getVersion(),
-                'lib')),
-            shellQuote(os.path.join(WORKDIR, '_root', 'Library', 'Frameworks',
-                'Python.framework', 'Versions', getVersion(),
-                'lib'))))
+                'Python.framework', 'Versions', getVersion(), 'lib')
+        fw_lib_dir = os.path.join(
+                WORKDIR, '_root', 'Library', 'Frameworks',
+                'Python.framework', 'Versions', getVersion(), 'lib')
+        if internalTk():
+            # move Tcl and Tk pkgconfig files
+            runCommand("mv %s/pkgconfig/* %s/pkgconfig"%(
+                        shellQuote(build_lib_dir),
+                        shellQuote(fw_lib_dir) ))
+            runCommand("rm -r %s/pkgconfig"%(
+                        shellQuote(build_lib_dir), ))
+        runCommand("mv %s/* %s"%(
+                    shellQuote(build_lib_dir),
+                    shellQuote(fw_lib_dir) ))
 
     frmDir = os.path.join(rootDir, 'Library', 'Frameworks', 'Python.framework')
     frmDirVersioned = os.path.join(frmDir, 'Versions', version)
diff --git a/Mac/BuildScript/issue19373_tk_8_5_15_source.patch b/Mac/BuildScript/issue19373_tk_8_5_15_source.patch
deleted file mode 100644
index de5d08e9d6c0..000000000000
--- a/Mac/BuildScript/issue19373_tk_8_5_15_source.patch
+++ /dev/null
@@ -1,13 +0,0 @@
-Issue #19373: Patch to Tk 8.5.15 to correct refresh problem on OS x 10.9.
-From upstream checkin https://core.tcl.tk/tk/info/5a5abf71f9
-
---- tk8.5.15/macosx/tkMacOSXDraw.c	2013-09-16 09:41:21.000000000 -0700
-+++ Tk_Source_Code-5a5abf71f9fdb0da/macosx/tkMacOSXDraw.c	2013-10-27 13:27:00.000000000 -0700
-@@ -1688,6 +1688,7 @@
- {
-     if (dcPtr->context) {
- 	CGContextSynchronize(dcPtr->context);
-+	[[dcPtr->view window] setViewsNeedDisplay:YES];
- 	[[dcPtr->view window] enableFlushWindow];
- 	if (dcPtr->focusLocked) {
- 	    [dcPtr->view unlockFocus];
diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index 1eea1e291bda..cb005af7f051 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -31,6 +31,20 @@ For Python.3.7, python.org currently provides two installer variants for downloa
 \i0  variant.  Use the 10.6 variant if you are running on macOS 10.6 through 10.8 or if you want to produce standalone applications that can run on systems from 10.6.  If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant.  The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself.  The list of these libraries varies by installer variant and is included at the end of the License.rtf file.
 \b \ul \
 \
+Using IDLE or other Tk applications [NEW/CHANGED in 3.7.0b1] 
+\b0 \ulnone \
+\
+The 10.9+ installer variant comes with its own private version of Tcl/Tk 8.6. It does not use system-supplied or third-party supplied versions of Tcl/Tk.\
+\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
+\cf0 For the 10.6+ variant, you continue to need to install a newer third-party version of the 
+\i Tcl/Tk
+\i0  8.5 (not 8.6) frameworks to use IDLE or other programs that use the Tkinter graphical user interface toolkit.  Visit {\field{\*\fldinst{HYPERLINK "https://www.python.org/download/mac/tcltk/"}}{\fldrslt https://www.python.org/download/mac/tcltk/}} for current information about supported and recommended versions of 
+\i Tcl/Tk
+\i0  for this version of Python and of macOS.\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
+
+\b \cf0 \ul \ulc0 \
 Certificate verification and OpenSSL \{UPDATED in 3.7.0b1]\
 
 \b0 \ulnone \
@@ -53,17 +67,7 @@ The bundled
 \f0  included with this installer has its own default certificate store for verifying download connections.\
 \
 
-\b \ul Update your version of Tcl/Tk to use IDLE or other Tk applications [CHANGED in 3.7.0b1] 
-\b0 \ulnone \
-\
-To use IDLE or other programs that use the Tkinter graphical user interface toolkit, you need to install a newer third-party version of the 
-\i Tcl/Tk
-\i0  frameworks.  Visit {\field{\*\fldinst{HYPERLINK "https://www.python.org/download/mac/tcltk/"}}{\fldrslt https://www.python.org/download/mac/tcltk/}} for current information about supported and recommended versions of 
-\i Tcl/Tk
-\i0  for this version of Python and of Mac OS X.  For 3.7.0b1, the 10.9 installer variant is linked with Tcl/Tk 8.6 which you must install separately.   The 10.6 installer links /with Tcl/Tk 8.5 which you should also download and install.\
-
-\b \ul \
-Other changes\
+\b \ul Other changes\
 
 \b0 \ulnone \
 For other changes in this release, see the 
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 18f3290608ec..4908fed0eaa5 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -23,7 +23,7 @@
 \
 
 \b NEW in 3.7.0b1: 
-\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), Tcl/Tk 8.6 support in the 10.9+ variant, OpenSSL 1.1.0g, and more!\
+\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), built-in Tcl/Tk 8.6 support in the 10.9+ variant (no additional third-party downloads!), OpenSSL 1.1.0g, and more!\
 \
 
 \b IMPORTANT:
diff --git a/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst b/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst
index bb7ebf1a62d2..6bdf0f12e972 100644
--- a/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst
+++ b/Misc/NEWS.d/next/macOS/2018-01-30-07-13-10.bpo-32726.tcARLK.rst
@@ -1,4 +1,4 @@
 Provide an additional, more modern macOS installer variant that supports
 macOS 10.9+ systems in 64-bit mode only.  Upgrade the supplied third-party
 libraries to OpenSSL 1.1.0g and to SQLite 3.22.0.  The 10.9+ installer now
-links with Tcl/Tk 8.6.
+links with and supplies its own copy of Tcl/Tk 8.6.


From webhook-mailer at python.org  Wed Jan 31 22:12:54 2018
From: webhook-mailer at python.org (Ned Deily)
Date: Thu, 01 Feb 2018 03:12:54 -0000
Subject: [Python-checkins] Update docs template for 3.8
Message-ID: <mailman.102.1517454776.2812.python-checkins@python.org>

https://github.com/python/cpython/commit/2fc65c648ea39e5fe6ced8ad229d7a445bf2a7d1
commit: 2fc65c648ea39e5fe6ced8ad229d7a445bf2a7d1
branch: 3.5
author: Ned Deily <nad at python.org>
committer: Ned Deily <nad at python.org>
date: 2018-01-31T22:11:02-05:00
summary:

Update docs template for 3.8

files:
M Doc/tools/static/version_switch.js
M Doc/tools/templates/indexsidebar.html

diff --git a/Doc/tools/static/version_switch.js b/Doc/tools/static/version_switch.js
index e8889ebfc41e..018609e926c1 100644
--- a/Doc/tools/static/version_switch.js
+++ b/Doc/tools/static/version_switch.js
@@ -2,6 +2,7 @@
   'use strict';
 
   var all_versions = {
+    '3.8': 'dev (3.8)',
     '3.7': 'dev (3.7)',
     '3.6': '3.6',
     '3.5': '3.5',
diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index b7bcd74c787e..7c0fe00655e4 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -2,9 +2,10 @@ <h3>{% trans %}Download{% endtrans %}</h3>
 <p><a href="{{ pathto('download') }}">{% trans %}Download these documents{% endtrans %}</a></p>
 <h3>{% trans %}Docs for other versions{% endtrans %}</h3>
 <ul>
-  <li><a href="https://docs.python.org/2.7/">{% trans %}Python 2.7 (stable){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (in development){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (pre-release){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (stable){% endtrans %}</a></li>
-  <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (in development){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/2.7/">{% trans %}Python 2.7 (stable){% endtrans %}</a></li>
   <li><a href="https://www.python.org/doc/versions/">{% trans %}Old versions{% endtrans %}</a></li>
 </ul>