Mailman 3 August 2020 - Python-checkins

bpo-33660: Fix PosixPath to resolve a relative path on root (GH-21975)
by Miss Islington (bot) Aug. 27, 2020

Aug. 27, 2020

https://github.com/python/cpython/commit/7475aa2c590e33a47f5e79e4079bca0645… commit: 7475aa2c590e33a47f5e79e4079bca0645e93f2f branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-08-27T02:47:10+02:00 summary: bpo-33660: Fix PosixPath to resolve a relative path on root (GH-21975) (cherry picked from commit 94ad6c674f7687ef22853cb8d42b440d6b42ddc8) Co-authored-by: Dong-hee Na <donghee.… [View More]

1 0

bpo-41624: fix documentation of typing.Coroutine (GH-21952)
by MingZhe Hu Aug. 27, 2020

Aug. 27, 2020

https://github.com/python/cpython/commit/8c58d2a216ca2b5965361df9b8d8944bc7… commit: 8c58d2a216ca2b5965361df9b8d8944bc7d4854d branch: master author: MingZhe Hu <humingzhework(a)163.com> committer: GitHub <noreply(a)github.com> date: 2020-08-27T02:42:37+02:00 summary: bpo-41624: fix documentation of typing.Coroutine (GH-21952) files: A Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/… [View More]

1 0

(no subject)
by Łukasz Langa Aug. 27, 2020

Aug. 27, 2020

To: python-checkins(a)python.org Subject: bpo-33660: Fix PosixPath to resolve a relative path on root Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 https://github.com/python/cpython/commit/94ad6c674f7687ef22853cb8d42b440d6b… ddc8 commit: 94ad6c674f7687ef22853cb8d42b440d6b42ddc8 branch: master author: Dong-hee Na <donghee.na92(a)gmail.com> committer: =C5=81ukasz Langa <lukasz(a)langa.pl> date: 2020-08-27T02:24:38+02:00 … [View More]

1 0

[3.8] bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (GH-21895) (#21967)
by Elvis Pranskevichus Aug. 26, 2020

Aug. 26, 2020

https://github.com/python/cpython/commit/57b698886b47bb81c782c2ba80a8a72fe6… commit: 57b698886b47bb81c782c2ba80a8a72fe66c7aad branch: 3.8 author: Elvis Pranskevichus <elvis(a)magic.io> committer: GitHub <noreply(a)github.com> date: 2020-08-26T13:59:17-07:00 summary: [3.8] bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (GH-21895) (#21967) When I was fixing bpo-32751 back in GH-7216 I missed the case when *timeout* is zero or negative. This takes care of … [View More]that. Props to @aaliddell for noticing the inconsistency.. (cherry picked from commit c517fc712105c8e5930cb42baaebdbe37fc3e15f) files: A Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 373be37fb4bdd..9ca9fa0a94a93 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -460,8 +460,13 @@ async def wait_for(fut, timeout, *, loop=None): if fut.done(): return fut.result() - fut.cancel() - raise exceptions.TimeoutError() + await _cancel_and_wait(fut, loop=loop) + try: + fut.result() + except exceptions.CancelledError as exc: + raise exceptions.TimeoutError() from exc + else: + raise exceptions.TimeoutError() waiter = loop.create_future() timeout_handle = loop.call_later(timeout, _release_waiter, waiter) diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index a70dc0a87e1c4..c402f8f021f3a 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -888,6 +888,9 @@ async def inner(): nonlocal task_done try: await asyncio.sleep(0.2) + except asyncio.CancelledError: + await asyncio.sleep(0.1) + raise finally: task_done = True @@ -900,6 +903,34 @@ async def inner(): loop.run_until_complete(foo()) + def test_wait_for_waits_for_task_cancellation_w_timeout_0(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + task_done = False + + async def foo(): + async def inner(): + nonlocal task_done + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(0.1) + raise + finally: + task_done = True + + inner_task = self.new_task(loop, inner()) + await asyncio.sleep(0.1) + await asyncio.wait_for(inner_task, timeout=0) + + with self.assertRaises(asyncio.TimeoutError) as cm: + loop.run_until_complete(foo()) + + self.assertTrue(task_done) + chained = cm.exception.__context__ + self.assertEqual(type(chained), asyncio.CancelledError) + def test_wait_for_self_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst new file mode 100644 index 0000000000000..c172ce5d9e948 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst @@ -0,0 +1,3 @@ +When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now +wait until the cancellation is complete also in the case when *timeout* is +<= 0, like it does with positive timeouts. [View Less]

1 0

Fix typos in comment (GH-21966)
by Raymond Hettinger Aug. 26, 2020

Aug. 26, 2020

https://github.com/python/cpython/commit/82e79480d6e61940d7007d9026fbff0b1a… commit: 82e79480d6e61940d7007d9026fbff0b1a11ad9a branch: master author: Raymond Hettinger <rhettinger(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-08-26T13:09:40-07:00 summary: Fix typos in comment (GH-21966) files: M Modules/mathmodule.c diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 1d6174132814b..1704d8efd31c3 100644 --- a/Modules/mathmodule.c +++ b/… [View More]

1 0

bpo-37658: Fix asyncio.wait_for() to respect waited task status (GH-21894) (#21965)
by Miss Islington (bot) Aug. 26, 2020

Aug. 26, 2020

https://github.com/python/cpython/commit/6e1954cd8286e083e7f8d09516d91b6b15… commit: 6e1954cd8286e083e7f8d09516d91b6b15769a4e branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-08-26T11:26:28-07:00 summary: bpo-37658: Fix asyncio.wait_for() to respect waited task status (GH-21894) (#21965) Currently, if `asyncio.wait_for()` itself is cancelled it will always raise `CancelledError` … [View More]regardless if the underlying task is still running. This is similar to a race with the timeout, which is handled already. (cherry picked from commit a2118a14627256197bddcf4fcecad4c264c1e39d) Co-authored-by: Elvis Pranskevichus <elvis(a)magic.io> files: A Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 66e81f9200a36..373be37fb4bdd 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -475,9 +475,12 @@ async def wait_for(fut, timeout, *, loop=None): try: await waiter except exceptions.CancelledError: - fut.remove_done_callback(cb) - fut.cancel() - raise + if fut.done(): + return fut.result() + else: + fut.remove_done_callback(cb) + fut.cancel() + raise if fut.done(): return fut.result() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index c21b06938fe3f..a70dc0a87e1c4 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -861,6 +861,22 @@ def gen(): res = loop.run_until_complete(task) self.assertEqual(res, "ok") + def test_wait_for_cancellation_race_condition(self): + def gen(): + yield 0.1 + yield 0.1 + yield 0.1 + yield 0.1 + + loop = self.new_test_loop(gen) + + fut = self.new_future(loop) + loop.call_later(0.1, fut.set_result, "ok") + task = loop.create_task(asyncio.wait_for(fut, timeout=1)) + loop.call_later(0.1, task.cancel) + res = loop.run_until_complete(task) + self.assertEqual(res, "ok") + def test_wait_for_waits_for_task_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst new file mode 100644 index 0000000000000..694fbbbf346dc --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst @@ -0,0 +1,2 @@ +:meth:`asyncio.wait_for` now properly handles races between cancellation of +itself and the completion of the wrapped awaitable. [View Less]

1 0

bpo-40077: Convert _operator to use PyType_FromSpec (GH-21954)
by Dong-hee Na Aug. 26, 2020

Aug. 26, 2020

https://github.com/python/cpython/commit/31967fd8d0220af84e2698172d1378bffc… commit: 31967fd8d0220af84e2698172d1378bffc8cd851 branch: master author: Dong-hee Na <donghee.na92(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-08-27T02:22:27+09:00 summary: bpo-40077: Convert _operator to use PyType_FromSpec (GH-21954) files: A Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst M Modules/_operator.c diff --git a/Misc/NEWS.d/next/Core and … [View More]Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst new file mode 100644 index 0000000000000..ee950010e6d13 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst @@ -0,0 +1 @@ +Convert :mod:`_operator` to use :c:func:`PyType_FromSpec`. diff --git a/Modules/_operator.c b/Modules/_operator.c index 8a54829e5bbcc..c9d38aa342366 100644 --- a/Modules/_operator.c +++ b/Modules/_operator.c @@ -3,6 +3,20 @@ #include "clinic/_operator.c.h" +typedef struct { + PyObject *itemgetter_type; + PyObject *attrgetter_type; + PyObject *methodcaller_type; +} _operator_state; + +static inline _operator_state* +get_operator_state(PyObject *module) +{ + void *state = PyModule_GetState(module); + assert(state != NULL); + return (_operator_state *)state; +} + /*[clinic input] module _operator [clinic start generated code]*/ @@ -942,8 +956,6 @@ typedef struct { Py_ssize_t index; // -1 unless *item* is a single non-negative integer index } itemgetterobject; -static PyTypeObject itemgetter_type; - /* AC 3.5: treats first argument as an iterable, otherwise uses *args */ static PyObject * itemgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -960,13 +972,15 @@ itemgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) if (nitems <= 1) { if (!PyArg_UnpackTuple(args, "itemgetter", 1, 1, &item)) return NULL; - } else + } else { item = args; - + } + _operator_state *state = PyType_GetModuleState(type); /* create itemgetterobject structure */ - ig = PyObject_GC_New(itemgetterobject, &itemgetter_type); - if (ig == NULL) + ig = PyObject_GC_New(itemgetterobject, (PyTypeObject *) state->itemgetter_type); + if (ig == NULL) { return NULL; + } Py_INCREF(item); ig->item = item; @@ -994,9 +1008,11 @@ itemgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void itemgetter_dealloc(itemgetterobject *ig) { + PyTypeObject *tp = Py_TYPE(ig); PyObject_GC_UnTrack(ig); Py_XDECREF(ig->item); - PyObject_GC_Del(ig); + tp->tp_free(ig); + Py_DECREF(tp); } static int @@ -1093,49 +1109,25 @@ Return a callable object that fetches the given item(s) from its operand.\n\ After f = itemgetter(2), the call f(r) returns r[2].\n\ After g = itemgetter(2, 5, 3), the call g(r) returns (r[2], r[5], r[3])"); -static PyTypeObject itemgetter_type = { - PyVarObject_HEAD_INIT(NULL, 0) - "operator.itemgetter", /* tp_name */ - sizeof(itemgetterobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)itemgetter_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)itemgetter_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - (ternaryfunc)itemgetter_call, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - itemgetter_doc, /* tp_doc */ - (traverseproc)itemgetter_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - itemgetter_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - itemgetter_new, /* tp_new */ - 0, /* tp_free */ +static PyType_Slot itemgetter_type_slots[] = { + {Py_tp_doc, (void *)itemgetter_doc}, + {Py_tp_dealloc, itemgetter_dealloc}, + {Py_tp_call, itemgetter_call}, + {Py_tp_traverse, itemgetter_traverse}, + {Py_tp_methods, itemgetter_methods}, + {Py_tp_new, itemgetter_new}, + {Py_tp_getattro, PyObject_GenericGetAttr}, + {Py_tp_repr, itemgetter_repr}, + {0, 0} }; +static PyType_Spec itemgetter_type_spec = { + .name = "operator.itemgetter", + .basicsize = sizeof(itemgetterobject), + .itemsize = 0, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .slots = itemgetter_type_slots, +}; /* attrgetter object **********************************************************/ @@ -1145,8 +1137,6 @@ typedef struct { PyObject *attr; } attrgetterobject; -static PyTypeObject attrgetter_type; - /* AC 3.5: treats first argument as an iterable, otherwise uses *args */ static PyObject * attrgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -1246,8 +1236,9 @@ attrgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } } + _operator_state *state = PyType_GetModuleState(type); /* create attrgetterobject structure */ - ag = PyObject_GC_New(attrgetterobject, &attrgetter_type); + ag = PyObject_GC_New(attrgetterobject, (PyTypeObject *)state->attrgetter_type); if (ag == NULL) { Py_DECREF(attr); return NULL; @@ -1263,9 +1254,11 @@ attrgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void attrgetter_dealloc(attrgetterobject *ag) { + PyTypeObject *tp = Py_TYPE(ag); PyObject_GC_UnTrack(ag); Py_XDECREF(ag->attr); - PyObject_GC_Del(ag); + tp->tp_free(ag); + Py_DECREF(tp); } static int @@ -1438,47 +1431,24 @@ After g = attrgetter('name', 'date'), the call g(r) returns (r.name, r.date).\n\ After h = attrgetter('name.first', 'name.last'), the call h(r) returns\n\ (r.name.first, r.name.last)."); -static PyTypeObject attrgetter_type = { - PyVarObject_HEAD_INIT(NULL, 0) - "operator.attrgetter", /* tp_name */ - sizeof(attrgetterobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)attrgetter_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)attrgetter_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - (ternaryfunc)attrgetter_call, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - attrgetter_doc, /* tp_doc */ - (traverseproc)attrgetter_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - attrgetter_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - attrgetter_new, /* tp_new */ - 0, /* tp_free */ +static PyType_Slot attrgetter_type_slots[] = { + {Py_tp_doc, (void *)attrgetter_doc}, + {Py_tp_dealloc, attrgetter_dealloc}, + {Py_tp_call, attrgetter_call}, + {Py_tp_traverse, attrgetter_traverse}, + {Py_tp_methods, attrgetter_methods}, + {Py_tp_new, attrgetter_new}, + {Py_tp_getattro, PyObject_GenericGetAttr}, + {Py_tp_repr, attrgetter_repr}, + {0, 0} +}; + +static PyType_Spec attrgetter_type_spec = { + .name = "operator.attrgetter", + .basicsize = sizeof(attrgetterobject), + .itemsize = 0, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .slots = attrgetter_type_slots, }; @@ -1491,8 +1461,6 @@ typedef struct { PyObject *kwds; } methodcallerobject; -static PyTypeObject methodcaller_type; - /* AC 3.5: variable number of arguments, not currently support by AC */ static PyObject * methodcaller_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -1513,10 +1481,12 @@ methodcaller_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return NULL; } + _operator_state *state = PyType_GetModuleState(type); /* create methodcallerobject structure */ - mc = PyObject_GC_New(methodcallerobject, &methodcaller_type); - if (mc == NULL) + mc = PyObject_GC_New(methodcallerobject, (PyTypeObject *)state->methodcaller_type); + if (mc == NULL) { return NULL; + } name = PyTuple_GET_ITEM(args, 0); Py_INCREF(name); @@ -1539,11 +1509,13 @@ methodcaller_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void methodcaller_dealloc(methodcallerobject *mc) { + PyTypeObject *tp = Py_TYPE(mc); PyObject_GC_UnTrack(mc); Py_XDECREF(mc->name); Py_XDECREF(mc->args); Py_XDECREF(mc->kwds); - PyObject_GC_Del(mc); + tp->tp_free(mc); + Py_DECREF(tp); } static int @@ -1704,63 +1676,52 @@ After f = methodcaller('name'), the call f(r) returns r.name().\n\ After g = methodcaller('name', 'date', foo=1), the call g(r) returns\n\ r.name('date', foo=1)."); -static PyTypeObject methodcaller_type = { - PyVarObject_HEAD_INIT(NULL, 0) - "operator.methodcaller", /* tp_name */ - sizeof(methodcallerobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)methodcaller_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)methodcaller_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - (ternaryfunc)methodcaller_call, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ - methodcaller_doc, /* tp_doc */ - (traverseproc)methodcaller_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - methodcaller_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - methodcaller_new, /* tp_new */ - 0, /* tp_free */ +static PyType_Slot methodcaller_type_slots[] = { + {Py_tp_doc, (void *)methodcaller_doc}, + {Py_tp_dealloc, methodcaller_dealloc}, + {Py_tp_call, methodcaller_call}, + {Py_tp_traverse, methodcaller_traverse}, + {Py_tp_methods, methodcaller_methods}, + {Py_tp_new, methodcaller_new}, + {Py_tp_getattro, PyObject_GenericGetAttr}, + {Py_tp_repr, methodcaller_repr}, + {0, 0} }; +static PyType_Spec methodcaller_type_spec = { + .name = "operator.methodcaller", + .basicsize = sizeof(methodcallerobject), + .itemsize = 0, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .slots = methodcaller_type_slots, +}; static int operator_exec(PyObject *module) { - PyTypeObject *types[] = { - &itemgetter_type, - &attrgetter_type, - &methodcaller_type - }; + _operator_state *state = get_operator_state(module); + state->attrgetter_type = PyType_FromModuleAndSpec(module, &attrgetter_type_spec, NULL); + if (state->attrgetter_type == NULL) { + return -1; + } + if (PyModule_AddType(module, (PyTypeObject *)state->attrgetter_type) < 0) { + return -1; + } - for (size_t i = 0; i < Py_ARRAY_LENGTH(types); i++) { - if (PyModule_AddType(module, types[i]) < 0) { - return -1; - } + state->itemgetter_type = PyType_FromModuleAndSpec(module, &itemgetter_type_spec, NULL); + if (state->itemgetter_type == NULL) { + return -1; + } + if (PyModule_AddType(module, (PyTypeObject *)state->itemgetter_type) < 0) { + return -1; + } + + state->methodcaller_type = PyType_FromModuleAndSpec(module, &methodcaller_type_spec, NULL); + if (state->methodcaller_type == NULL) { + return -1; + } + if (PyModule_AddType(module, (PyTypeObject *)state->methodcaller_type) < 0) { + return -1; } return 0; @@ -1772,17 +1733,42 @@ static struct PyModuleDef_Slot operator_slots[] = { {0, NULL} }; +static int +operator_traverse(PyObject *module, visitproc visit, void *arg) +{ + _operator_state *state = get_operator_state(module); + Py_VISIT(state->attrgetter_type); + Py_VISIT(state->itemgetter_type); + Py_VISIT(state->methodcaller_type); + return 0; +} + +static int +operator_clear(PyObject *module) +{ + _operator_state *state = get_operator_state(module); + Py_CLEAR(state->attrgetter_type); + Py_CLEAR(state->itemgetter_type); + Py_CLEAR(state->methodcaller_type); + return 0; +} + +static void +operator_free(void *module) +{ + operator_clear((PyObject *)module); +} static struct PyModuleDef operatormodule = { PyModuleDef_HEAD_INIT, - "_operator", - operator_doc, - 0, - operator_methods, - operator_slots, - NULL, - NULL, - NULL + .m_name = "_operator", + .m_doc = operator_doc, + .m_size = sizeof(_operator_state), + .m_methods = operator_methods, + .m_slots = operator_slots, + .m_traverse = operator_traverse, + .m_clear = operator_clear, + .m_free = operator_free, }; PyMODINIT_FUNC [View Less]

1 0

bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (#21895)
by Elvis Pranskevichus Aug. 26, 2020

Aug. 26, 2020

https://github.com/python/cpython/commit/c517fc712105c8e5930cb42baaebdbe37f… commit: c517fc712105c8e5930cb42baaebdbe37fc3e15f branch: master author: Elvis Pranskevichus <elvis(a)magic.io> committer: GitHub <noreply(a)github.com> date: 2020-08-26T09:42:22-07:00 summary: bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (#21895) When I was fixing bpo-32751 back in GH-7216 I missed the case when *timeout* is zero or negative. This takes care of that. Props … [View More]to @aaliddell for noticing the inconsistency. files: A Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index c37f0e1387980..7ecec9638489d 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -445,8 +445,13 @@ async def wait_for(fut, timeout, *, loop=None): if fut.done(): return fut.result() - fut.cancel() - raise exceptions.TimeoutError() + await _cancel_and_wait(fut, loop=loop) + try: + fut.result() + except exceptions.CancelledError as exc: + raise exceptions.TimeoutError() from exc + else: + raise exceptions.TimeoutError() waiter = loop.create_future() timeout_handle = loop.call_later(timeout, _release_waiter, waiter) diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index f9db066ce8983..511961c32005a 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1131,6 +1131,9 @@ async def inner(): nonlocal task_done try: await asyncio.sleep(0.2) + except asyncio.CancelledError: + await asyncio.sleep(_EPSILON) + raise finally: task_done = True @@ -1145,6 +1148,34 @@ async def inner(): chained = cm.exception.__context__ self.assertEqual(type(chained), asyncio.CancelledError) + def test_wait_for_waits_for_task_cancellation_w_timeout_0(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + task_done = False + + async def foo(): + async def inner(): + nonlocal task_done + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(_EPSILON) + raise + finally: + task_done = True + + inner_task = self.new_task(loop, inner()) + await asyncio.sleep(_EPSILON) + await asyncio.wait_for(inner_task, timeout=0) + + with self.assertRaises(asyncio.TimeoutError) as cm: + loop.run_until_complete(foo()) + + self.assertTrue(task_done) + chained = cm.exception.__context__ + self.assertEqual(type(chained), asyncio.CancelledError) + def test_wait_for_reraises_exception_during_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst new file mode 100644 index 0000000000000..c172ce5d9e948 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst @@ -0,0 +1,3 @@ +When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now +wait until the cancellation is complete also in the case when *timeout* is +<= 0, like it does with positive timeouts. [View Less]

1 0

bpo-37658: Fix asyncio.wait_for() to respect waited task status (#21894)
by Elvis Pranskevichus Aug. 26, 2020

Aug. 26, 2020

https://github.com/python/cpython/commit/a2118a14627256197bddcf4fcecad4c264… commit: a2118a14627256197bddcf4fcecad4c264c1e39d branch: master author: Elvis Pranskevichus <elvis(a)magic.io> committer: GitHub <noreply(a)github.com> date: 2020-08-26T09:42:45-07:00 summary: bpo-37658: Fix asyncio.wait_for() to respect waited task status (#21894) Currently, if `asyncio.wait_for()` itself is cancelled it will always raise `CancelledError` regardless if the underlying task is still … [View More]running. This is similar to a race with the timeout, which is handled already. files: A Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 7ecec9638489d..8b05434f273b5 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -465,9 +465,12 @@ async def wait_for(fut, timeout, *, loop=None): try: await waiter except exceptions.CancelledError: - fut.remove_done_callback(cb) - fut.cancel() - raise + if fut.done(): + return fut.result() + else: + fut.remove_done_callback(cb) + fut.cancel() + raise if fut.done(): return fut.result() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index 511961c32005a..74fc1e4a42133 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1120,6 +1120,22 @@ def gen(): res = loop.run_until_complete(task) self.assertEqual(res, "ok") + def test_wait_for_cancellation_race_condition(self): + def gen(): + yield 0.1 + yield 0.1 + yield 0.1 + yield 0.1 + + loop = self.new_test_loop(gen) + + fut = self.new_future(loop) + loop.call_later(0.1, fut.set_result, "ok") + task = loop.create_task(asyncio.wait_for(fut, timeout=1)) + loop.call_later(0.1, task.cancel) + res = loop.run_until_complete(task) + self.assertEqual(res, "ok") + def test_wait_for_waits_for_task_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst new file mode 100644 index 0000000000000..694fbbbf346dc --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst @@ -0,0 +1,2 @@ +:meth:`asyncio.wait_for` now properly handles races between cancellation of +itself and the completion of the wrapped awaitable. [View Less]

1 0

bpo-41513: More accurate hypot() (GH-21916)
by Raymond Hettinger Aug. 25, 2020

Aug. 25, 2020

https://github.com/python/cpython/commit/8e19c8be87017f6bef8e4c936b1e6ddacb… commit: 8e19c8be87017f6bef8e4c936b1e6ddacb558ad2 branch: master author: Raymond Hettinger <rhettinger(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-08-24T17:40:08-07:00 summary: bpo-41513: More accurate hypot() (GH-21916) files: A Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst M Modules/mathmodule.c diff --git a/Misc/NEWS.d/next/Library/2020-08-23-14-… [View More]23-18.bpo-41513.DGqc_I.rst b/Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst new file mode 100644 index 0000000000000..b4d0d9b63cf87 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst @@ -0,0 +1,3 @@ +Improved the accuracy of math.hypot(). Internally, each step is computed +with extra precision so that the result is now almost always correctly +rounded. diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 489802cc36745..1d6174132814b 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2404,52 +2404,79 @@ math_fmod_impl(PyObject *module, double x, double y) } /* -Given an *n* length *vec* of values and a value *max*, compute: +Given a *vec* of values, compute the vector norm: - sqrt(sum((x * scale) ** 2 for x in vec)) / scale + sqrt(sum(x ** 2 for x in vec)) - where scale is the first power of two - greater than max. - -or compute: - - max * sqrt(sum((x / max) ** 2 for x in vec)) - -The value of the *max* variable must be non-negative and -equal to the absolute value of the largest magnitude -entry in the vector. If n==0, then *max* should be 0.0. +The *max* variable should be equal to the largest fabs(x). +The *n* variable is the length of *vec*. +If n==0, then *max* should be 0.0. If an infinity is present in the vec, *max* should be INF. - The *found_nan* variable indicates whether some member of the *vec* is a NaN. -To improve accuracy and to increase the number of cases where -vector_norm() is commutative, we use a variant of Neumaier -summation specialized to exploit that we always know that -|csum| >= |x|. - -The *csum* variable tracks the cumulative sum and *frac* tracks -the cumulative fractional errors at each step. Since this -variant assumes that |csum| >= |x| at each step, we establish -the precondition by starting the accumulation from 1.0 which -represents the largest possible value of (x*scale)**2 or (x/max)**2. - -After the loop is finished, the initial 1.0 is subtracted out -for a net zero effect on the final sum. Since *csum* will be -greater than 1.0, the subtraction of 1.0 will not cause -fractional digits to be dropped from *csum*. - -To get the full benefit from compensated summation, the -largest addend should be in the range: 0.5 <= x <= 1.0. -Accordingly, scaling or division by *max* should not be skipped -even if not otherwise needed to prevent overflow or loss of precision. +To avoid overflow/underflow and to achieve high accuracy giving results +that are almost always correctly rounded, four techniques are used: + +* lossless scaling using a power-of-two scaling factor +* accurate squaring using Veltkamp-Dekker splitting +* compensated summation using a variant of the Neumaier algorithm +* differential correction of the square root + +The usual presentation of the Neumaier summation algorithm has an +expensive branch depending on which operand has the larger +magnitude. We avoid this cost by arranging the calculation so that +fabs(csum) is always as large as fabs(x). + +To establish the invariant, *csum* is initialized to 1.0 which is +always larger than x**2 after scaling or division by *max*. +After the loop is finished, the initial 1.0 is subtracted out for a +net zero effect on the final sum. Since *csum* will be greater than +1.0, the subtraction of 1.0 will not cause fractional digits to be +dropped from *csum*. + +To get the full benefit from compensated summation, the largest +addend should be in the range: 0.5 <= |x| <= 1.0. Accordingly, +scaling or division by *max* should not be skipped even if not +otherwise needed to prevent overflow or loss of precision. + +The assertion that hi*hi >= 1.0 is a bit subtle. Each vector element +gets scaled to a magnitude below 1.0. The Veltkamp-Dekker splitting +algorithm gives a *hi* value that is correctly rounded to half +precision. When a value at or below 1.0 is correctly rounded, it +never goes above 1.0. And when values at or below 1.0 are squared, +they remain at or below 1.0, thus preserving the summation invariant. + +The square root differential correction is needed because a +correctly rounded square root of a correctly rounded sum of +squares can still be off by as much as one ulp. + +The differential correction starts with a value *x* that is +the difference between the square of *h*, the possibly inaccurately +rounded square root, and the accurately computed sum of squares. +The correction is the first order term of the Maclaurin series +expansion of sqrt(h**2 + x) == h + x/(2*h) + O(x**2). + +Essentially, this differential correction is equivalent to one +refinement step in the Newton divide-and-average square root +algorithm, effectively doubling the number of accurate bits. +This technique is used in Dekker's SQRT2 algorithm and again in +Borges' ALGORITHM 4 and 5. + +References: + +1. Veltkamp-Dekker splitting: http://csclub.uwaterloo.ca/~pbarfuss/dekker1971.pdf +2. Compensated summation: http://www.ti3.tu-harburg.de/paper/rump/Ru08b.pdf +3. Square root diffential correction: https://arxiv.org/pdf/1904.09481.pdf */ static inline double vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) { + const double T27 = 134217729.0; /* ldexp(1.0, 27)+1.0) */ double x, csum = 1.0, oldcsum, frac = 0.0, scale; + double t, hi, lo, h; int max_e; Py_ssize_t i; @@ -2470,15 +2497,62 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) for (i=0 ; i < n ; i++) { x = vec[i]; assert(Py_IS_FINITE(x) && fabs(x) <= max); + x *= scale; - x = x*x; + assert(fabs(x) < 1.0); + + t = x * T27; + hi = t - (t - x); + lo = x - hi; + assert(hi + lo == x); + + x = hi * hi; assert(x <= 1.0); - assert(csum >= x); + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = 2.0 * hi * lo; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = lo * lo; + assert(fabs(csum) >= fabs(x)); oldcsum = csum; csum += x; frac += (oldcsum - csum) + x; } - return sqrt(csum - 1.0 + frac) / scale; + h = sqrt(csum - 1.0 + frac); + + x = h; + t = x * T27; + hi = t - (t - x); + lo = x - hi; + assert (hi + lo == x); + + x = -hi * hi; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = -2.0 * hi * lo; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = -lo * lo; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = csum - 1.0 + frac; + return (h + x / (2.0 * h)) / scale; } /* When max_e < -1023, ldexp(1.0, -max_e) overflows. So instead of multiplying by a scale, we just divide by *max*. @@ -2489,7 +2563,7 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) x /= max; x = x*x; assert(x <= 1.0); - assert(csum >= x); + assert(fabs(csum) >= fabs(x)); oldcsum = csum; csum += x; frac += (oldcsum - csum) + x; [View Less]

1 0