Mailman 3 September 2020 - Python-checkins

bpo-41700: Skip test if the locale is not supported (GH-22081)
by Dong-hee Na 04 Sep '20

04 Sep '20

https://github.com/python/cpython/commit/54a66ade2067c373d31003ad260e1b7d14… commit: 54a66ade2067c373d31003ad260e1b7d14c81564 branch: master author: Dong-hee Na <donghee.na92(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-04T17:47:40+09:00 summary: bpo-41700: Skip test if the locale is not supported (GH-22081) files: M Lib/test/test_c_locale_coercion.py diff --git a/Lib/test/test_c_locale_coercion.py b/Lib/test/test_c_locale_coercion.py index 8340a9eb2ea3a..fcc85992345db 100644 --- a/Lib/test/test_c_locale_coercion.py +++ b/Lib/test/test_c_locale_coercion.py @@ -407,7 +407,10 @@ def test_PYTHONCOERCECLOCALE_set_to_one(self): # skip the test if the LC_CTYPE locale is C or coerced old_loc = locale.setlocale(locale.LC_CTYPE, None) self.addCleanup(locale.setlocale, locale.LC_CTYPE, old_loc) - loc = locale.setlocale(locale.LC_CTYPE, "") + try: + loc = locale.setlocale(locale.LC_CTYPE, "") + except locale.Error as e: + self.skipTest(str(e)) if loc == "C": self.skipTest("test requires LC_CTYPE locale different than C") if loc in TARGET_LOCALES :

1 0

bpo-39603: Prevent header injection in http methods (GH-18485) (#21946)
by Victor Stinner 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/524b8de630036a29ca340bc2ae6fd6dc7d… commit: 524b8de630036a29ca340bc2ae6fd6dc7dda8f40 branch: 3.5 author: Victor Stinner <vstinner(a)python.org> committer: GitHub <noreply(a)github.com> date: 2020-09-03T17:54:15-07:00 summary: bpo-39603: Prevent header injection in http methods (GH-18485) (#21946) reject control chars in http method in http.client.putrequest to prevent http header injection (cherry picked from commit 8ca8a2e8fb068863c1138f07e3098478ef8be12e) Co-authored-by: AMIR <31338382+amiremohamadi(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst M Lib/http/client.py M Lib/test/test_httplib.py diff --git a/Lib/http/client.py b/Lib/http/client.py index 5d5d7a7f707b4..f7efe151f5b04 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -151,6 +151,10 @@ # _is_allowed_url_pchars_re = re.compile(r"^[/!$&'()*+,;=:@%a-zA-Z0-9._~-]+$") # We are more lenient for assumed real world compatibility purposes. +# These characters are not allowed within HTTP method names +# to prevent http header injection. +_contains_disallowed_method_pchar_re = re.compile('[\x00-\x1f]') + # We always set the Content-Length header for these methods because some # servers will otherwise respond with a 411 _METHODS_EXPECTING_BODY = {'PATCH', 'POST', 'PUT'} @@ -985,6 +989,8 @@ def putrequest(self, method, url, skip_host=False, else: raise CannotSendRequest(self.__state) + self._validate_method(method) + # Save the method for use later in the response phase self._method = method @@ -1075,6 +1081,16 @@ def _encode_request(self, request): # ASCII also helps prevent CVE-2019-9740. return request.encode('ascii') + def _validate_method(self, method): + """Validate a method name for putrequest.""" + # prevent http header injection + match = _contains_disallowed_method_pchar_re.search(method) + if match: + raise ValueError( + "method can't contain control characters. %r " + "(found at least %r)" + % (method, match.group())) + def _validate_path(self, url): """Validate a url for putrequest.""" # Prevent CVE-2019-9740. diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py index a8e7a30af1a9f..b1cd08b23a4b1 100644 --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -344,6 +344,28 @@ def test_invalid_headers(self): conn.putheader(name, value) +class HttpMethodTests(TestCase): + def test_invalid_method_names(self): + methods = ( + 'GET\r', + 'POST\n', + 'PUT\n\r', + 'POST\nValue', + 'POST\nHOST:abc', + 'GET\nrHost:abc\n', + 'POST\rRemainder:\r', + 'GET\rHOST:\n', + '\nPUT' + ) + + for method in methods: + with self.assertRaisesRegex( + ValueError, "method can't contain control characters"): + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn.request(method=method, url="/") + + class BasicTest(TestCase): def test_status_lines(self): # Test HTTP status lines @@ -1783,8 +1805,8 @@ def test_tunnel_debuglog(self): @support.reap_threads def test_main(verbose=None): - support.run_unittest(HeaderTests, OfflineTest, BasicTest, TimeoutTest, - PersistenceTest, + support.run_unittest(HeaderTests, OfflineTest, HttpMethodTests, + BasicTest, TimeoutTest, PersistenceTest, HTTPSTest, RequestBodyTest, SourceAddressTest, HTTPResponseTest, ExtendedReadTest, ExtendedReadTestChunked, TunnelTests) diff --git a/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst new file mode 100644 index 0000000000000..990affc3edd9d --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst @@ -0,0 +1,2 @@ +Prevent http header injection by rejecting control characters in +http.client.putrequest(...).

1 0

bpo-39010: Improve test shutdown (GH-22066) (#22083)
by Miss Islington (bot) 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/40e2444c364eede59f193979df5a02ed95… commit: 40e2444c364eede59f193979df5a02ed958f56e9 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T13:54:36-07:00 summary: bpo-39010: Improve test shutdown (GH-22066) (#22083) Simply closing the event loop isn't enough to avoid warnings. If we don't also shut down the event loop's default executor, it sometimes logs a "dangling thread" warning. Follow-up to GH-22017 (cherry picked from commit be435ae2b064dc64f04475bec632862e1dbf605f) Co-authored-by: Ben Darnell <ben(a)bendarnell.com> Co-authored-by: Ben Darnell <ben(a)bendarnell.com> files: M Lib/test/test_asyncio/test_windows_events.py diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py index 33388a87d48f3..f276cd205a2f8 100644 --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -225,10 +225,18 @@ def test_read_self_pipe_restart(self): self.loop.run_forever() self.loop.stop() self.loop.run_forever() - # If we don't wait for f to complete here, we may get another - # warning logged about a thread that didn't shut down cleanly. + + # Shut everything down cleanly. This is an important part of the + # test - in issue 39010, the error occurred during loop.close(), + # so we want to close the loop during the test instead of leaving + # it for tearDown. + # + # First wait for f to complete to avoid a "future's result was never + # retrieved" error. self.loop.run_until_complete(f) - self.loop.close() + # Now shut down the loop itself (self.close_loop also shuts down the + # loop's default executor). + self.close_loop(self.loop) self.assertFalse(self.loop.call_exception_handler.called)

1 0

bpo-41696: Fix handling of debug mode in asyncio.run (GH-22069) (#22072)
by Miss Islington (bot) 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/1f5f12737755f246fee83a54c600779ad7… commit: 1f5f12737755f246fee83a54c600779ad76934a4 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T13:54:06-07:00 summary: bpo-41696: Fix handling of debug mode in asyncio.run (GH-22069) (#22072) * bpo-41696: Fix handling of debug mode in asyncio.run This allows PYTHONASYNCIODEBUG or -X dev to enable asyncio debug mode when using asyncio.run * 📜🤖 Added by blurb_it. Co-authored-by: hauntsaninja <> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot](a)users.noreply.github.com> (cherry picked from commit 0770ad948cb6d9f7f6c4002efd83e27c27069808) Co-authored-by: Shantanu <12621235+hauntsaninja(a)users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2020-09-03-01-35-32.bpo-41696.zkYGre.rst M Lib/asyncio/runners.py M Lib/test/test_asyncio/test_runners.py diff --git a/Lib/asyncio/runners.py b/Lib/asyncio/runners.py index 2e37e18b45fc1..d2fa8b7f13861 100644 --- a/Lib/asyncio/runners.py +++ b/Lib/asyncio/runners.py @@ -5,7 +5,7 @@ from . import tasks -def run(main, *, debug=False): +def run(main, *, debug=None): """Execute the coroutine and return the result. This function runs the passed coroutine, taking care of @@ -39,7 +39,8 @@ async def main(): loop = events.new_event_loop() try: events.set_event_loop(loop) - loop.set_debug(debug) + if debug is not None: + loop.set_debug(debug) return loop.run_until_complete(main) finally: try: diff --git a/Lib/test/test_asyncio/test_runners.py b/Lib/test/test_asyncio/test_runners.py index 3b58ddee443ad..b9ae02dc3c04e 100644 --- a/Lib/test/test_asyncio/test_runners.py +++ b/Lib/test/test_asyncio/test_runners.py @@ -87,6 +87,9 @@ async def main(expected): asyncio.run(main(False)) asyncio.run(main(True), debug=True) + with mock.patch('asyncio.coroutines._is_debug_mode', lambda: True): + asyncio.run(main(True)) + asyncio.run(main(False), debug=False) def test_asyncio_run_from_running_loop(self): async def main(): diff --git a/Misc/NEWS.d/next/Library/2020-09-03-01-35-32.bpo-41696.zkYGre.rst b/Misc/NEWS.d/next/Library/2020-09-03-01-35-32.bpo-41696.zkYGre.rst new file mode 100644 index 0000000000000..67bbbb857f18c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-09-03-01-35-32.bpo-41696.zkYGre.rst @@ -0,0 +1 @@ +Fix handling of debug mode in :func:`asyncio.run`. This allows setting ``PYTHONASYNCIODEBUG`` or ``-X dev`` to enable asyncio debug mode when using :func:`asyncio.run`. \ No newline at end of file

1 0

bpo-39010: Fix errors logged on proactor loop restart (GH-22017) (#22035)
by Miss Islington (bot) 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/a986b061a3cd4661eb9f857e2936291f18… commit: a986b061a3cd4661eb9f857e2936291f1847bd15 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T12:38:34-07:00 summary: bpo-39010: Fix errors logged on proactor loop restart (GH-22017) (#22035) Stopping and restarting a proactor event loop on windows can lead to spurious errors logged (ConnectionResetError while reading from the self pipe). This fixes the issue by ensuring that we don't attempt to start multiple copies of the self-pipe reading loop. (cherry picked from commit ea5a6363c3f8cc90b7c0cc573922b10f296073b6) Co-authored-by: Ben Darnell <ben(a)bendarnell.com> Co-authored-by: Ben Darnell <ben(a)bendarnell.com> files: A Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst M Lib/asyncio/proactor_events.py M Lib/asyncio/windows_events.py M Lib/test/test_asyncio/test_proactor_events.py M Lib/test/test_asyncio/test_windows_events.py diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py index 830d8edc32f90..54e4deb2c349c 100644 --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -766,6 +766,14 @@ def _loop_self_reading(self, f=None): try: if f is not None: f.result() # may raise + if self._self_reading_future is not f: + # When we scheduled this Future, we assigned it to + # _self_reading_future. If it's not there now, something has + # tried to cancel the loop while this callback was still in the + # queue (see windows_events.ProactorEventLoop.run_forever). In + # that case stop here instead of continuing to schedule a new + # iteration. + return f = self._proactor.recv(self._ssock, 4096) except exceptions.CancelledError: # _close_self_pipe() has been called, stop waiting for data diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py index 12e87abfbf2ea..a5ad6fc423fbe 100644 --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -318,8 +318,12 @@ def run_forever(self): if self._self_reading_future is not None: ov = self._self_reading_future._ov self._self_reading_future.cancel() - # self_reading_future was just cancelled so it will never be signalled - # Unregister it otherwise IocpProactor.close will wait for it forever + # self_reading_future was just cancelled so if it hasn't been + # finished yet, it never will be (it's possible that it has + # already finished and its callback is waiting in the queue, + # where it could still happen if the event loop is restarted). + # Unregister it otherwise IocpProactor.close will wait for it + # forever if ov is not None: self._proactor._unregister(ov) self._self_reading_future = None diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py index 007039a7cdf5d..e4809c303a71d 100644 --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -736,6 +736,7 @@ def test_loop_self_reading(self): def test_loop_self_reading_fut(self): fut = mock.Mock() + self.loop._self_reading_future = fut self.loop._loop_self_reading(fut) self.assertTrue(fut.result.called) self.proactor.recv.assert_called_with(self.ssock, 4096) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py index 6b005702c9be7..33388a87d48f3 100644 --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -211,6 +211,26 @@ def test_wait_for_handle_cancel(self): fut.cancel() fut.cancel() + def test_read_self_pipe_restart(self): + # Regression test for https://bugs.python.org/issue39010 + # Previously, restarting a proactor event loop in certain states + # would lead to spurious ConnectionResetErrors being logged. + self.loop.call_exception_handler = mock.Mock() + # Start an operation in another thread so that the self-pipe is used. + # This is theoretically timing-dependent (the task in the executor + # must complete before our start/stop cycles), but in practice it + # seems to work every time. + f = self.loop.run_in_executor(None, lambda: None) + self.loop.stop() + self.loop.run_forever() + self.loop.stop() + self.loop.run_forever() + # If we don't wait for f to complete here, we may get another + # warning logged about a thread that didn't shut down cleanly. + self.loop.run_until_complete(f) + self.loop.close() + self.assertFalse(self.loop.call_exception_handler.called) + class WinPolicyTests(test_utils.TestCase): diff --git a/Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst b/Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst new file mode 100644 index 0000000000000..0d9015b490e4c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst @@ -0,0 +1,2 @@ +Restarting a ``ProactorEventLoop`` on Windows no longer logs spurious +``ConnectionResetErrors``.

1 0

[doc] Update documentation on logging optimization. (GH-22075)
by Vinay Sajip 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/cdbff3527c5b10e79761ad25316b46cd07… commit: cdbff3527c5b10e79761ad25316b46cd079304bc branch: master author: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> committer: GitHub <noreply(a)github.com> date: 2020-09-03T19:44:12+01:00 summary: [doc] Update documentation on logging optimization. (GH-22075) files: M Doc/howto/logging.rst diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst index 6316e086ef43b..b4dd9206c9d52 100644 --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -1078,20 +1078,22 @@ need more precise control over what logging information is collected. Here's a list of things you can do to avoid processing during logging which you don't need: -+-----------------------------------------------+----------------------------------------+ -| What you don't want to collect | How to avoid collecting it | -+===============================================+========================================+ -| Information about where calls were made from. | Set ``logging._srcfile`` to ``None``. | -| | This avoids calling | -| | :func:`sys._getframe`, which may help | -| | to speed up your code in environments | -| | like PyPy (which can't speed up code | -| | that uses :func:`sys._getframe`). | -+-----------------------------------------------+----------------------------------------+ -| Threading information. | Set ``logging.logThreads`` to ``0``. | -+-----------------------------------------------+----------------------------------------+ -| Process information. | Set ``logging.logProcesses`` to ``0``. | -+-----------------------------------------------+----------------------------------------+ ++-----------------------------------------------------+---------------------------------------------------+ +| What you don't want to collect | How to avoid collecting it | ++=====================================================+===================================================+ +| Information about where calls were made from. | Set ``logging._srcfile`` to ``None``. | +| | This avoids calling :func:`sys._getframe`, which | +| | may help to speed up your code in environments | +| | like PyPy (which can't speed up code that uses | +| | :func:`sys._getframe`). | ++-----------------------------------------------------+---------------------------------------------------+ +| Threading information. | Set ``logging.logThreads`` to ``False``. | ++-----------------------------------------------------+---------------------------------------------------+ +| Current process ID (:func:`os.getpid`) | Set ``logging.logProcesses`` to ``False``. | ++-----------------------------------------------------+---------------------------------------------------+ +| Current process name when using ``multiprocessing`` | Set ``logging.logMultiprocessing`` to ``False``. | +| to manage multiple processes. | | ++-----------------------------------------------------+---------------------------------------------------+ Also note that the core logging module only includes the basic handlers. If you don't import :mod:`logging.handlers` and :mod:`logging.config`, they won't

1 0

bpo-39883: Use BSD0 license for code in docs (GH-17635) (GH-22074)
by Miss Islington (bot) 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/9fef7c54a0adfade5ec94259d97f22e05f… commit: 9fef7c54a0adfade5ec94259d97f22e05fe9e3e3 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T07:41:09-07:00 summary: bpo-39883: Use BSD0 license for code in docs (GH-17635) (GH-22074) The PSF board approved this use. (cherry picked from commit e223d06a8b2067c68e9f97c4628c2b0f056dcae6) Co-authored-by: Todd <toddrjen(a)gmail.com> files: A Misc/NEWS.d/next/Documentation/2020-03-07-03-53-39.bpo-39883.1tnb4-.rst M Doc/license.rst M LICENSE diff --git a/Doc/license.rst b/Doc/license.rst index 472a5cf3d88b3..f70c34d3fde60 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -72,6 +72,19 @@ make these releases possible. Terms and conditions for accessing or otherwise using Python ============================================================ +Python software and documentation are licensed under the +:ref:`PSF License Agreement <PSF-license>`. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Agreement +and the :ref:`Zero-Clause BSD license <BSD0>`. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. +See :ref:`OtherLicenses` for an incomplete list of these licenses. + + +.. _PSF-license: PSF LICENSE AGREEMENT FOR PYTHON |release| ------------------------------------------ @@ -258,6 +271,27 @@ CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 SOFTWARE. +.. _BSD0: + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON |release| DOCUMENTATION +---------------------------------------------------------------------- + +.. parsed-literal:: + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH + REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY + AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, + INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM + LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR + OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR + PERFORMANCE OF THIS SOFTWARE. + + +.. _OtherLicenses: + Licenses and Acknowledgements for Incorporated Software ======================================================= diff --git a/LICENSE b/LICENSE index 66a3ac80d729a..f42f8adbed845 100644 --- a/LICENSE +++ b/LICENSE @@ -59,6 +59,17 @@ direction to make these releases possible. B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON =============================================================== +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 -------------------------------------------- @@ -252,3 +263,17 @@ FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. diff --git a/Misc/NEWS.d/next/Documentation/2020-03-07-03-53-39.bpo-39883.1tnb4-.rst b/Misc/NEWS.d/next/Documentation/2020-03-07-03-53-39.bpo-39883.1tnb4-.rst new file mode 100644 index 0000000000000..4941d50a560e2 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-03-07-03-53-39.bpo-39883.1tnb4-.rst @@ -0,0 +1 @@ +Make code, examples, and recipes in the Python documentation be licensed under the more permissive BSD0 license in addition to the existing Python 2.0 license. \ No newline at end of file

1 0

Fix 'gather' rules in the python parser generator (GH-22021)
by Pablo Galindo 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/e55a0e971b4bf9e6473bf0ca6bebdff76c… commit: e55a0e971b4bf9e6473bf0ca6bebdff76c075ef6 branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T15:29:55+01:00 summary: Fix 'gather' rules in the python parser generator (GH-22021) Currently, empty sequences in gather rules make the conditional for gather rules fail as empty sequences evaluate as "False". We need to explicitly check for "None" (the failure condition) to avoid false negatives. files: M Lib/test/test_peg_generator/test_pegen.py M Tools/peg_generator/pegen/python_generator.py diff --git a/Lib/test/test_peg_generator/test_pegen.py b/Lib/test/test_peg_generator/test_pegen.py index 5b4e964d698ad..bcfee3f2c5f8c 100644 --- a/Lib/test/test_peg_generator/test_pegen.py +++ b/Lib/test/test_peg_generator/test_pegen.py @@ -74,7 +74,7 @@ def test_typed_rules(self) -> None: "Rule('term', 'int', Rhs([Alt([NamedItem(None, NameLeaf('NUMBER'))])]))" ) - def test_repeat_with_separator_rules(self) -> None: + def test_gather(self) -> None: grammar = """ start: ','.thing+ NEWLINE thing: NUMBER @@ -85,6 +85,20 @@ def test_repeat_with_separator_rules(self) -> None: "Rule('start', None, Rhs([Alt([NamedItem(None, Gather(StringLeaf(\"','\"), NameLeaf('thing'" )) self.assertEqual(str(rules["thing"]), "thing: NUMBER") + parser_class = make_parser(grammar) + node = parse_string("42\n", parser_class) + assert node == [ + [[TokenInfo(NUMBER, string="42", start=(1, 0), end=(1, 2), line="42\n")]], + TokenInfo(NEWLINE, string="\n", start=(1, 2), end=(1, 3), line="42\n"), + ] + node = parse_string("1, 2\n", parser_class) + assert node == [ + [ + [TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1, 2\n")], + [TokenInfo(NUMBER, string="2", start=(1, 3), end=(1, 4), line="1, 2\n")], + ], + TokenInfo(NEWLINE, string="\n", start=(1, 4), end=(1, 5), line="1, 2\n"), + ] def test_expr_grammar(self) -> None: grammar = """ diff --git a/Tools/peg_generator/pegen/python_generator.py b/Tools/peg_generator/pegen/python_generator.py index 45a75975dbf5e..b786de7fee5b4 100644 --- a/Tools/peg_generator/pegen/python_generator.py +++ b/Tools/peg_generator/pegen/python_generator.py @@ -217,6 +217,9 @@ def visit_Alt(self, node: Alt, is_loop: bool, is_gather: bool) -> None: else: self.print("and") self.visit(item) + if is_gather: + self.print("is not None") + self.print("):") with self.indent(): action = node.action

1 0

bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in the parser (GH-22077)
by Pablo Galindo 03 Sep '20

03 Sep '20

https://github.com/python/cpython/commit/315a61f7a9418d904e0eea14b1f054fac3… commit: 315a61f7a9418d904e0eea14b1f054fac3a90e9f branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T15:29:32+01:00 summary: bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in the parser (GH-22077) files: M Grammar/python.gram M Parser/parser.c M Parser/pegen.c M Parser/pegen.h diff --git a/Grammar/python.gram b/Grammar/python.gram index 84835b731c540..524e88eb38996 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -535,7 +535,7 @@ arguments[expr_ty] (memo): | a=args [','] &')' { a } | incorrect_arguments args[expr_ty]: - | a=','.(starred_expression | named_expression !'=')+ b=[',' k=kwargs {k}] { _PyPegen_collect_call_seqs(p, a, b) } + | a=','.(starred_expression | named_expression !'=')+ b=[',' k=kwargs {k}] { _PyPegen_collect_call_seqs(p, a, b, EXTRA) } | a=kwargs { _Py_Call(_PyPegen_dummy_name(p), CHECK_NULL_ALLOWED(_PyPegen_seq_extract_starred_exprs(p, a)), CHECK_NULL_ALLOWED(_PyPegen_seq_delete_starred_exprs(p, a)), diff --git a/Parser/parser.c b/Parser/parser.c index 3e724a260d90a..8a7cb62fd7cf8 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -12237,7 +12237,16 @@ args_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ args[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.(starred_expression | named_expression !'=')+ [',' kwargs]")); - _res = _PyPegen_collect_call_seqs ( p , a , b ); + Token *_token = _PyPegen_get_last_nonnwhitespace_token(p); + if (_token == NULL) { + D(p->level--); + return NULL; + } + int _end_lineno = _token->end_lineno; + UNUSED(_end_lineno); // Only used by EXTRA macro + int _end_col_offset = _token->end_col_offset; + UNUSED(_end_col_offset); // Only used by EXTRA macro + _res = _PyPegen_collect_call_seqs ( p , a , b , EXTRA ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; D(p->level--); diff --git a/Parser/pegen.c b/Parser/pegen.c index 2507bc4b38280..4beb2abdd296a 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -2219,14 +2219,15 @@ _PyPegen_nonparen_genexp_in_call(Parser *p, expr_ty args) } -expr_ty _PyPegen_collect_call_seqs(Parser *p, asdl_seq *a, asdl_seq *b) { +expr_ty _PyPegen_collect_call_seqs(Parser *p, asdl_seq *a, asdl_seq *b, + int lineno, int col_offset, int end_lineno, + int end_col_offset, PyArena *arena) { Py_ssize_t args_len = asdl_seq_LEN(a); Py_ssize_t total_len = args_len; if (b == NULL) { - expr_ty first = asdl_seq_GET(a, 0); - expr_ty last = asdl_seq_GET(a, args_len - 1); - return _Py_Call(_PyPegen_dummy_name(p), a, NULL, EXTRA_EXPR(first, last)); + return _Py_Call(_PyPegen_dummy_name(p), a, NULL, lineno, col_offset, + end_lineno, end_col_offset, arena); } @@ -2237,7 +2238,7 @@ expr_ty _PyPegen_collect_call_seqs(Parser *p, asdl_seq *a, asdl_seq *b) { total_len += asdl_seq_LEN(starreds); } - asdl_seq *args = _Py_asdl_seq_new(total_len, p->arena); + asdl_seq *args = _Py_asdl_seq_new(total_len, arena); Py_ssize_t i = 0; for (i = 0; i < args_len; i++) { @@ -2247,8 +2248,8 @@ expr_ty _PyPegen_collect_call_seqs(Parser *p, asdl_seq *a, asdl_seq *b) { asdl_seq_SET(args, i, asdl_seq_GET(starreds, i - args_len)); } - expr_ty first = asdl_seq_GET(args, 0); - expr_ty last = asdl_seq_GET(b, asdl_seq_LEN(b)-1); + return _Py_Call(_PyPegen_dummy_name(p), args, keywords, lineno, + col_offset, end_lineno, end_col_offset, arena); + - return _Py_Call(_PyPegen_dummy_name(p), args, keywords, EXTRA_EXPR(first, last)); } diff --git a/Parser/pegen.h b/Parser/pegen.h index 3e74e3ac73e0a..c81681efad208 100644 --- a/Parser/pegen.h +++ b/Parser/pegen.h @@ -257,7 +257,9 @@ stmt_ty _PyPegen_class_def_decorators(Parser *, asdl_seq *, stmt_ty); KeywordOrStarred *_PyPegen_keyword_or_starred(Parser *, void *, int); asdl_seq *_PyPegen_seq_extract_starred_exprs(Parser *, asdl_seq *); asdl_seq *_PyPegen_seq_delete_starred_exprs(Parser *, asdl_seq *); -expr_ty _PyPegen_collect_call_seqs(Parser *, asdl_seq *, asdl_seq *); +expr_ty _PyPegen_collect_call_seqs(Parser *, asdl_seq *, asdl_seq *, + int lineno, int col_offset, int end_lineno, + int end_col_offset, PyArena *arena); expr_ty _PyPegen_concatenate_strings(Parser *p, asdl_seq *); asdl_seq *_PyPegen_join_sequences(Parser *, asdl_seq *, asdl_seq *); int _PyPegen_check_barry_as_flufl(Parser *);

1 0

(no subject)
by Mario Šaško 03 Sep '20

03 Sep '20

To: python-checkins(a)python.org Subject: [doc] Fix a typo in the graphlib docs (#22030) Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 https://github.com/python/cpython/commit/851606007665dc30089e5e1953208c5428… 55b1 commit: 851606007665dc30089e5e1953208c5428e455b1 branch: master author: Mario =C5=A0a=C5=A1ko <mariosasko777(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-03T11:00:10+01:00 summary: [doc] Fix a typo in the graphlib docs (#22030) files: M Doc/library/graphlib.rst diff --git a/Doc/library/graphlib.rst b/Doc/library/graphlib.rst index 820615e723015..0faca2186b268 100644 --- a/Doc/library/graphlib.rst +++ b/Doc/library/graphlib.rst @@ -121,7 +121,7 @@ if ts.is_active(): ... =20 - if possible to simply do:: + it is possible to simply do:: =20 if ts: ...

1 0