Mailman 3 February 2020 - Python-checkins

fixes typos in http.client documentation (GH-18300)
by Miss Islington (bot) Feb. 3, 2020

Feb. 3, 2020

https://github.com/python/cpython/commit/db2f3114b2be4f3ee81a3258a979327d53… commit: db2f3114b2be4f3ee81a3258a979327d539ab51a branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-02-03T04:07:19-08:00 summary: fixes typos in http.client documentation (GH-18300) (cherry picked from commit b94737a4af96b29bd4c025724f671e7bc0f6b6f1) Co-authored-by: James Corbett <james.h.corbett(a)gmail.com> files: M Doc/library/http.client.rst diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 9cdabc2a03a75..be31c3c07154f 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -563,8 +563,8 @@ Here is an example session that shows how to ``POST`` requests:: Client side ``HTTP PUT`` requests are very similar to ``POST`` requests. The difference lies only the server side where HTTP server will allow resources to be created via ``PUT`` request. It should be noted that custom HTTP methods -+are also handled in :class:`urllib.request.Request` by sending the appropriate -+method attribute.Here is an example session that shows how to do ``PUT`` +are also handled in :class:`urllib.request.Request` by setting the appropriate +method attribute. Here is an example session that shows how to send a ``PUT`` request using http.client:: >>> # This creates an HTTP message

1 0

bpo-36051: Fix compiler warning. (GH-18325)
by Inada Naoki Feb. 3, 2020

Feb. 3, 2020

https://github.com/python/cpython/commit/869c0c99b94ff9527acc1ca060164ab3d1… commit: 869c0c99b94ff9527acc1ca060164ab3d1bdcc53 branch: master author: Inada Naoki <songofacandy(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-02-03T19:03:34+09:00 summary: bpo-36051: Fix compiler warning. (GH-18325) files: M Objects/stringlib/join.h diff --git a/Objects/stringlib/join.h b/Objects/stringlib/join.h index 4d023ed1a851e..8ad598ad5c9fd 100644 --- a/Objects/stringlib/join.h +++ b/Objects/stringlib/join.h @@ -20,7 +20,7 @@ STRINGLIB(bytes_join)(PyObject *sep, PyObject *iterable) Py_buffer static_buffers[NB_STATIC_BUFFERS]; #define GIL_THRESHOLD 1048576 int drop_gil = 1; - PyThreadState *save; + PyThreadState *save = NULL; seq = PySequence_Fast(iterable, "can only join an iterable"); if (seq == NULL) {

1 0

bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18175) (#18323)
by Miss Islington (bot) Feb. 3, 2020

Feb. 3, 2020

https://github.com/python/cpython/commit/02395fad8e3a35ef00fa31c30869384401… commit: 02395fad8e3a35ef00fa31c308693844013a1dd4 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-02-03T08:20:41Z summary: bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18175) (#18323) (cherry picked from commit 032de7324e30c6b44ef272cea3be205a3d768759) Co-authored-by: Steve Cirelli <scirelli+git(a)gmail.com> files: A Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst M Lib/unittest/case.py M Lib/unittest/test/test_case.py diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index b639c64d02a7a..e5734b6b7a298 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -529,7 +529,7 @@ def shortDescription(self): the specified test method's docstring. """ doc = self._testMethodDoc - return doc and doc.split("\n")[0].strip() or None + return doc.strip().split("\n")[0].strip() if doc else None def id(self): diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py index c2401c39b917e..f855c4dc00b31 100644 --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -610,6 +610,15 @@ def testShortDescriptionWithMultiLineDocstring(self): 'Tests shortDescription() for a method with a longer ' 'docstring.') + def testShortDescriptionWhitespaceTrimming(self): + """ + Tests shortDescription() whitespace is trimmed, so that the first + line of nonwhite-space text becomes the docstring. + """ + self.assertEqual( + self.shortDescription(), + 'Tests shortDescription() whitespace is trimmed, so that the first') + def testAddTypeEqualityFunc(self): class SadSnake(object): """Dummy class for test_addTypeEqualityFunc.""" diff --git a/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst new file mode 100644 index 0000000000000..55fed519a2d80 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst @@ -0,0 +1,2 @@ +Striped whitespace from docstring before returning it from +:func:`unittest.case.shortDescription`.

1 0

bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18321)
by Miss Islington (bot) Feb. 3, 2020

Feb. 3, 2020

https://github.com/python/cpython/commit/7561e7a83ccccf5118fda6c62fe9c8c345… commit: 7561e7a83ccccf5118fda6c62fe9c8c3458f8cfd branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-02-03T07:25:17Z summary: bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18321) (cherry picked from commit 032de7324e30c6b44ef272cea3be205a3d768759) Co-authored-by: Steve Cirelli <scirelli+git(a)gmail.com> files: A Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst M Lib/unittest/case.py M Lib/unittest/test/test_case.py diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index 811f5df23dd14..24af29057646a 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -493,7 +493,7 @@ def shortDescription(self): the specified test method's docstring. """ doc = self._testMethodDoc - return doc and doc.split("\n")[0].strip() or None + return doc.strip().split("\n")[0].strip() if doc else None def id(self): diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py index 6d58201ea8143..4fac8d5974528 100644 --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -610,6 +610,15 @@ def testShortDescriptionWithMultiLineDocstring(self): 'Tests shortDescription() for a method with a longer ' 'docstring.') + def testShortDescriptionWhitespaceTrimming(self): + """ + Tests shortDescription() whitespace is trimmed, so that the first + line of nonwhite-space text becomes the docstring. + """ + self.assertEqual( + self.shortDescription(), + 'Tests shortDescription() whitespace is trimmed, so that the first') + def testAddTypeEqualityFunc(self): class SadSnake(object): """Dummy class for test_addTypeEqualityFunc.""" diff --git a/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst new file mode 100644 index 0000000000000..55fed519a2d80 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst @@ -0,0 +1,2 @@ +Striped whitespace from docstring before returning it from +:func:`unittest.case.shortDescription`.

1 0

bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18175)
by Steve Cirelli Feb. 3, 2020

Feb. 3, 2020

https://github.com/python/cpython/commit/032de7324e30c6b44ef272cea3be205a3d… commit: 032de7324e30c6b44ef272cea3be205a3d768759 branch: master author: Steve Cirelli <scirelli+git(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-02-03T07:06:50Z summary: bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18175) files: A Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst M Lib/unittest/case.py M Lib/unittest/test/test_case.py diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index fa64a6ea2378c..5e5d535dc6938 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -512,7 +512,7 @@ def shortDescription(self): the specified test method's docstring. """ doc = self._testMethodDoc - return doc and doc.split("\n")[0].strip() or None + return doc.strip().split("\n")[0].strip() if doc else None def id(self): diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py index c2401c39b917e..f855c4dc00b31 100644 --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -610,6 +610,15 @@ def testShortDescriptionWithMultiLineDocstring(self): 'Tests shortDescription() for a method with a longer ' 'docstring.') + def testShortDescriptionWhitespaceTrimming(self): + """ + Tests shortDescription() whitespace is trimmed, so that the first + line of nonwhite-space text becomes the docstring. + """ + self.assertEqual( + self.shortDescription(), + 'Tests shortDescription() whitespace is trimmed, so that the first') + def testAddTypeEqualityFunc(self): class SadSnake(object): """Dummy class for test_addTypeEqualityFunc.""" diff --git a/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst new file mode 100644 index 0000000000000..55fed519a2d80 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst @@ -0,0 +1,2 @@ +Striped whitespace from docstring before returning it from +:func:`unittest.case.shortDescription`.

1 0

[3.8] bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) (#18316)
by Antoine Pitrou Feb. 2, 2020

Feb. 2, 2020

https://github.com/python/cpython/commit/17236873392533ce0c5d7bbf52cbd61bca… commit: 17236873392533ce0c5d7bbf52cbd61bca171c59 branch: 3.8 author: Antoine Pitrou <antoine(a)python.org> committer: GitHub <noreply(a)github.com> date: 2020-02-02T21:22:57+01:00 summary: [3.8] bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) (#18316) https://bugs.python.org/issue39492 Automerge-Triggered-By: @pitrou (cherry picked from commit 0f2f35e) Co-authored-by: Pierre Glaser <pierreglaser(a)msn.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst M Lib/test/pickletester.py M Modules/_pickle.c diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index c9f374678ae35..3a8aee4320c65 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3497,6 +3497,30 @@ class MyClass: ValueError, 'The reducer just failed'): p.dump(h) + @support.cpython_only + def test_reducer_override_no_reference_cycle(self): + # bpo-39492: reducer_override used to induce a spurious reference cycle + # inside the Pickler object, that could prevent all serialized objects + # from being garbage-collected without explicity invoking gc.collect. + + for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + def f(): + pass + + wr = weakref.ref(f) + + bio = io.BytesIO() + p = self.pickler_class(bio, proto) + p.dump(f) + new_f = pickle.loads(bio.getvalue()) + assert new_f == 5 + + del p + del f + + self.assertIsNone(wr()) + class AbstractDispatchTableTests(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst new file mode 100644 index 0000000000000..6e8b715c46365 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst @@ -0,0 +1 @@ +Fix a reference cycle in the C Pickler that was preventing the garbage collection of deleted, pickled objects. \ No newline at end of file diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 8150bf3b33dc3..9f6e66f70a546 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4457,12 +4457,13 @@ static int dump(PicklerObject *self, PyObject *obj) { const char stop_op = STOP; + int status = -1; PyObject *tmp; _Py_IDENTIFIER(reducer_override); if (_PyObject_LookupAttrId((PyObject *)self, &PyId_reducer_override, &tmp) < 0) { - return -1; + goto error; } /* Cache the reducer_override method, if it exists. */ if (tmp != NULL) { @@ -4479,7 +4480,7 @@ dump(PicklerObject *self, PyObject *obj) assert(self->proto >= 0 && self->proto < 256); header[1] = (unsigned char)self->proto; if (_Pickler_Write(self, header, 2) < 0) - return -1; + goto error; if (self->proto >= 4) self->framing = 1; } @@ -4487,9 +4488,22 @@ dump(PicklerObject *self, PyObject *obj) if (save(self, obj, 0) < 0 || _Pickler_Write(self, &stop_op, 1) < 0 || _Pickler_CommitFrame(self) < 0) - return -1; + goto error; + + // Success + status = 0; + + error: self->framing = 0; - return 0; + + /* Break the reference cycle we generated at the beginning this function + * call when setting the reducer_override attribute of the Pickler instance + * to a bound method of the same instance. This is important as the Pickler + * instance holds a reference to each object it has pickled (through its + * memo): thus, these objects wont be garbage-collected as long as the + * Pickler itself is not collected. */ + Py_CLEAR(self->reducer_override); + return status; } /*[clinic input]

1 0

bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266)
by Pierre Glaser Feb. 2, 2020

Feb. 2, 2020

https://github.com/python/cpython/commit/0f2f35e15f9fbee44ce042b724348419d8… commit: 0f2f35e15f9fbee44ce042b724348419d8136bc5 branch: master author: Pierre Glaser <pierreglaser(a)msn.com> committer: GitHub <noreply(a)github.com> date: 2020-02-02T10:55:21-08:00 summary: bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) This also needs a backport to 3.8 https://bugs.python.org/issue39492 Automerge-Triggered-By: @pitrou files: A Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst M Lib/test/pickletester.py M Modules/_pickle.c diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 953fd5c5a278b..ba893f39c2f34 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3499,6 +3499,30 @@ class MyClass: ValueError, 'The reducer just failed'): p.dump(h) + @support.cpython_only + def test_reducer_override_no_reference_cycle(self): + # bpo-39492: reducer_override used to induce a spurious reference cycle + # inside the Pickler object, that could prevent all serialized objects + # from being garbage-collected without explicity invoking gc.collect. + + for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + def f(): + pass + + wr = weakref.ref(f) + + bio = io.BytesIO() + p = self.pickler_class(bio, proto) + p.dump(f) + new_f = pickle.loads(bio.getvalue()) + assert new_f == 5 + + del p + del f + + self.assertIsNone(wr()) + class AbstractDispatchTableTests(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst new file mode 100644 index 0000000000000..6e8b715c46365 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst @@ -0,0 +1 @@ +Fix a reference cycle in the C Pickler that was preventing the garbage collection of deleted, pickled objects. \ No newline at end of file diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 25d5c8da92068..fc48d6057866a 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4455,12 +4455,13 @@ static int dump(PicklerObject *self, PyObject *obj) { const char stop_op = STOP; + int status = -1; PyObject *tmp; _Py_IDENTIFIER(reducer_override); if (_PyObject_LookupAttrId((PyObject *)self, &PyId_reducer_override, &tmp) < 0) { - return -1; + goto error; } /* Cache the reducer_override method, if it exists. */ if (tmp != NULL) { @@ -4477,7 +4478,7 @@ dump(PicklerObject *self, PyObject *obj) assert(self->proto >= 0 && self->proto < 256); header[1] = (unsigned char)self->proto; if (_Pickler_Write(self, header, 2) < 0) - return -1; + goto error; if (self->proto >= 4) self->framing = 1; } @@ -4485,9 +4486,22 @@ dump(PicklerObject *self, PyObject *obj) if (save(self, obj, 0) < 0 || _Pickler_Write(self, &stop_op, 1) < 0 || _Pickler_CommitFrame(self) < 0) - return -1; + goto error; + + // Success + status = 0; + + error: self->framing = 0; - return 0; + + /* Break the reference cycle we generated at the beginning this function + * call when setting the reducer_override attribute of the Pickler instance + * to a bound method of the same instance. This is important as the Pickler + * instance holds a reference to each object it has pickled (through its + * memo): thus, these objects wont be garbage-collected as long as the + * Pickler itself is not collected. */ + Py_CLEAR(self->reducer_override); + return status; } /*[clinic input]

1 0

bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057)
by Kyle Stanley Feb. 2, 2020

Feb. 2, 2020

https://github.com/python/cpython/commit/339fd46cb764277cbbdc3e78dcc5b45b15… commit: 339fd46cb764277cbbdc3e78dcc5b45b156bb6ae branch: master author: Kyle Stanley <aeros167(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-02-02T13:49:00+01:00 summary: bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057) files: A Misc/NEWS.d/next/Library/2020-01-19-04-12-34.bpo-39349.7CV-LC.rst M Doc/library/concurrent.futures.rst M Doc/whatsnew/3.9.rst M Lib/concurrent/futures/process.py M Lib/concurrent/futures/thread.py M Lib/test/test_concurrent_futures.py diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index d71f2d80c9e2d..b21d5594c84fa 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -67,7 +67,7 @@ Executor Objects .. versionchanged:: 3.5 Added the *chunksize* argument. - .. method:: shutdown(wait=True) + .. method:: shutdown(wait=True, \*, cancel_futures=False) Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to @@ -82,6 +82,15 @@ Executor Objects value of *wait*, the entire Python program will not exit until all pending futures are done executing. + If *cancel_futures* is ``True``, this method will cancel all pending + futures that the executor has not started running. Any futures that + are completed or running won't be cancelled, regardless of the value + of *cancel_futures*. + + If both *cancel_futures* and *wait* are ``True``, all futures that the + executor has started running will be completed prior to this method + returning. The remaining futures are cancelled. + You can avoid having to call this method explicitly if you use the :keyword:`with` statement, which will shutdown the :class:`Executor` (waiting as if :meth:`Executor.shutdown` were called with *wait* set to @@ -94,6 +103,9 @@ Executor Objects e.submit(shutil.copy, 'src3.txt', 'dest3.txt') e.submit(shutil.copy, 'src4.txt', 'dest4.txt') + .. versionchanged:: 3.9 + Added *cancel_futures*. + ThreadPoolExecutor ------------------ diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst index ee37b5a5b7265..931f8bf13fbde 100644 --- a/Doc/whatsnew/3.9.rst +++ b/Doc/whatsnew/3.9.rst @@ -146,6 +146,15 @@ that schedules a shutdown for the default executor that waits on the Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher implementation that polls process file descriptors. (:issue:`38692`) +concurrent.futures +------------------ + +Added a new *cancel_futures* parameter to +:meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures +which have not started running, instead of waiting for them to complete before +shutting down the executor. +(Contributed by Kyle Stanley in :issue:`39349`.) + curses ------ diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py index 9e2ab9db64f66..fd9f572b6c711 100644 --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -435,6 +435,24 @@ def shutdown_worker(): # is not gc-ed yet. if executor is not None: executor._shutdown_thread = True + # Unless there are pending work items, we have nothing to cancel. + if pending_work_items and executor._cancel_pending_futures: + # Cancel all pending futures and update pending_work_items + # to only have futures that are currently running. + new_pending_work_items = {} + for work_id, work_item in pending_work_items.items(): + if not work_item.future.cancel(): + new_pending_work_items[work_id] = work_item + + pending_work_items = new_pending_work_items + # Drain work_ids_queue since we no longer need to + # add items to the call queue. + while True: + try: + work_ids_queue.get_nowait() + except queue.Empty: + break + # Since no new work items can be added, it is safe to shutdown # this thread if there are no pending work items. if not pending_work_items: @@ -546,6 +564,7 @@ def __init__(self, max_workers=None, mp_context=None, self._broken = False self._queue_count = 0 self._pending_work_items = {} + self._cancel_pending_futures = False # Create communication channels for the executor # Make the call queue slightly larger than the number of processes to @@ -660,9 +679,11 @@ def map(self, fn, *iterables, timeout=None, chunksize=1): timeout=timeout) return _chain_from_iterable_of_lists(results) - def shutdown(self, wait=True): + def shutdown(self, wait=True, *, cancel_futures=False): with self._shutdown_lock: + self._cancel_pending_futures = cancel_futures self._shutdown_thread = True + if self._queue_management_thread: # Wake up queue management thread self._queue_management_thread_wakeup.wakeup() diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py index b89f8f24d4d67..be79161bf8561 100644 --- a/Lib/concurrent/futures/thread.py +++ b/Lib/concurrent/futures/thread.py @@ -215,9 +215,22 @@ def _initializer_failed(self): if work_item is not None: work_item.future.set_exception(BrokenThreadPool(self._broken)) - def shutdown(self, wait=True): + def shutdown(self, wait=True, *, cancel_futures=False): with self._shutdown_lock: self._shutdown = True + if cancel_futures: + # Drain all work items from the queue, and then cancel their + # associated futures. + while True: + try: + work_item = self._work_queue.get_nowait() + except queue.Empty: + break + if work_item is not None: + work_item.future.cancel() + + # Send a wake-up to prevent threads calling + # _work_queue.get(block=True) from permanently blocking. self._work_queue.put(None) if wait: for t in self._threads: diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index c8fa35e9eeafa..af77f81341910 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -342,6 +342,29 @@ def test_hang_issue12364(self): for f in fs: f.result() + def test_cancel_futures(self): + executor = self.executor_type(max_workers=3) + fs = [executor.submit(time.sleep, .1) for _ in range(50)] + executor.shutdown(cancel_futures=True) + # We can't guarantee the exact number of cancellations, but we can + # guarantee that *some* were cancelled. With setting max_workers to 3, + # most of the submitted futures should have been cancelled. + cancelled = [fut for fut in fs if fut.cancelled()] + self.assertTrue(len(cancelled) >= 35, msg=f"{len(cancelled)=}") + + # Ensure the other futures were able to finish. + # Use "not fut.cancelled()" instead of "fut.done()" to include futures + # that may have been left in a pending state. + others = [fut for fut in fs if not fut.cancelled()] + for fut in others: + self.assertTrue(fut.done(), msg=f"{fut._state=}") + self.assertIsNone(fut.exception()) + + # Similar to the number of cancelled futures, we can't guarantee the + # exact number that completed. But, we can guarantee that at least + # one finished. + self.assertTrue(len(others) > 0, msg=f"{len(others)=}") + def test_hang_issue39205(self): """shutdown(wait=False) doesn't hang at exit with running futures. @@ -422,6 +445,22 @@ def test_thread_names_default(self): self.assertRegex(t.name, r'ThreadPoolExecutor-\d+_[0-4]$') t.join() + def test_cancel_futures_wait_false(self): + # Can only be reliably tested for TPE, since PPE often hangs with + # `wait=False` (even without *cancel_futures*). + rc, out, err = assert_python_ok('-c', """if True: + from concurrent.futures import ThreadPoolExecutor + from test.test_concurrent_futures import sleep_and_print + if __name__ == "__main__": + t = ThreadPoolExecutor() + t.submit(sleep_and_print, .1, "apple") + t.shutdown(wait=False, cancel_futures=True) + """.format(executor_type=self.executor_type.__name__)) + # Errors in atexit hooks don't change the process exit code, check + # stderr manually. + self.assertFalse(err) + self.assertEqual(out.strip(), b"apple") + class ProcessPoolShutdownTest(ExecutorShutdownTest): def _prime_executor(self): diff --git a/Misc/NEWS.d/next/Library/2020-01-19-04-12-34.bpo-39349.7CV-LC.rst b/Misc/NEWS.d/next/Library/2020-01-19-04-12-34.bpo-39349.7CV-LC.rst new file mode 100644 index 0000000000000..cc52700f67031 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-19-04-12-34.bpo-39349.7CV-LC.rst @@ -0,0 +1,4 @@ +Added a new *cancel_futures* parameter to +:meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures +which have not started running, instead of waiting for them to complete before +shutting down the executor. \ No newline at end of file

1 0

Fix 5-space indentation and trailing whitespace (GH-18311)
by Mark Dickinson Feb. 2, 2020

Feb. 2, 2020

https://github.com/python/cpython/commit/be8147bdc6111a225ec284a45142773047… commit: be8147bdc6111a225ec284a4514277304726c3d0 branch: master author: Mark Dickinson <mdickinson(a)enthought.com> committer: GitHub <noreply(a)github.com> date: 2020-02-02T11:37:02Z summary: Fix 5-space indentation and trailing whitespace (GH-18311) files: M Objects/floatobject.c diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 89f60b65cd55c..ab486d4ee3d2e 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -641,7 +641,7 @@ _float_div_mod(double vx, double wx, double *floordiv, double *mod) *floordiv = floor(div); if (div - *floordiv > 0.5) { *floordiv += 1.0; - } + } } else { /* div is zero - get the same sign as the true quotient */ @@ -652,16 +652,16 @@ _float_div_mod(double vx, double wx, double *floordiv, double *mod) static PyObject * float_divmod(PyObject *v, PyObject *w) { - double vx, wx; - double mod, floordiv; - CONVERT_TO_DOUBLE(v, vx); - CONVERT_TO_DOUBLE(w, wx); - if (wx == 0.0) { - PyErr_SetString(PyExc_ZeroDivisionError, "float divmod()"); - return NULL; - } - _float_div_mod(vx, wx, &floordiv, &mod); - return Py_BuildValue("(dd)", floordiv, mod); + double vx, wx; + double mod, floordiv; + CONVERT_TO_DOUBLE(v, vx); + CONVERT_TO_DOUBLE(w, wx); + if (wx == 0.0) { + PyErr_SetString(PyExc_ZeroDivisionError, "float divmod()"); + return NULL; + } + _float_div_mod(vx, wx, &floordiv, &mod); + return Py_BuildValue("(dd)", floordiv, mod); } static PyObject *

1 0

bpo-39496: Remove redundant checks from _sqlite/cursor.c (GH-18270)
by Alex Henrie Feb. 1, 2020

Feb. 1, 2020

https://github.com/python/cpython/commit/78c7183f470b60a39ac2dd0ad1a94d49d1… commit: 78c7183f470b60a39ac2dd0ad1a94d49d1e0b062 branch: master author: Alex Henrie <alexhenrie24(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-02-01T23:45:34+03:00 summary: bpo-39496: Remove redundant checks from _sqlite/cursor.c (GH-18270) files: M Modules/_sqlite/cursor.c diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c index 2302ca9edac2d..06275ecb26849 100644 --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -786,17 +786,9 @@ PyObject* pysqlite_cursor_fetchmany(pysqlite_Cursor* self, PyObject* args, PyObj return NULL; } - /* just make sure we enter the loop */ - row = Py_None; - - while (row) { - row = pysqlite_cursor_iternext(self); - if (row) { - PyList_Append(list, row); - Py_DECREF(row); - } else { - break; - } + while ((row = pysqlite_cursor_iternext(self))) { + PyList_Append(list, row); + Py_XDECREF(row); if (++counter == maxrows) { break; @@ -821,15 +813,9 @@ PyObject* pysqlite_cursor_fetchall(pysqlite_Cursor* self, PyObject* args) return NULL; } - /* just make sure we enter the loop */ - row = (PyObject*)Py_None; - - while (row) { - row = pysqlite_cursor_iternext(self); - if (row) { - PyList_Append(list, row); - Py_DECREF(row); - } + while ((row = pysqlite_cursor_iternext(self))) { + PyList_Append(list, row); + Py_XDECREF(row); } if (PyErr_Occurred()) {

1 0