Mailman 3 March 2016 - Python-checkins

cpython (2.7): Issue #26492: Added additional tests for exhausted iterators of mutable
by serhiy.storchaka March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/249ef9d02aa6 changeset: 100805:249ef9d02aa6 branch: 2.7 parent: 100802:cff06d875678 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Wed Mar 30 21:02:00 2016 +0300 summary: Issue #26492: Added additional tests for exhausted iterators of mutable sequences. files: Lib/test/list_tests.py | 11 +++++++++++ Lib/test/test_bytes.py | 10 ++++++++++ Lib/test/test_iter.py | 11 +++++++++++ 3 files changed, 32 insertions(+), 0 deletions(-) diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py --- a/Lib/test/list_tests.py +++ b/Lib/test/list_tests.py @@ -532,3 +532,14 @@ def __iter__(self): raise KeyboardInterrupt self.assertRaises(KeyboardInterrupt, list, F()) + + def test_exhausted_iterator(self): + a = self.type2test([1, 2, 3]) + exhit = iter(a) + empit = iter(a) + for x in exhit: # exhaust the iterator + next(empit) # not exhausted + a.append(9) + self.assertEqual(list(exhit), []) + self.assertEqual(list(empit), [9]) + self.assertEqual(a, self.type2test([1, 2, 3, 9])) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -900,6 +900,16 @@ # PyByteArray_AS_STRING() C macro. self.assertRaises(ValueError, int, bytearray(b'')) + def test_exhausted_iterator(self): + a = self.type2test([1, 2, 3]) + exhit = iter(a) + empit = iter(a) + for x in exhit: # exhaust the iterator + next(empit) # not exhausted + a.append(9) + self.assertEqual(list(exhit), []) + self.assertEqual(list(empit), [9]) + self.assertEqual(a, self.type2test([1, 2, 3, 9])) class AssortedBytesTest(unittest.TestCase): # diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -122,6 +122,17 @@ def test_seq_class_iter(self): self.check_iterator(iter(SequenceClass(10)), range(10)) + def test_mutating_seq_class_exhausted_iter(self): + a = SequenceClass(5) + exhit = iter(a) + empit = iter(a) + for x in exhit: # exhaust the iterator + next(empit) # not exhausted + a.n = 7 + self.assertEqual(list(exhit), []) + self.assertEqual(list(empit), [5, 6]) + self.assertEqual(list(a), [0, 1, 2, 3, 4, 5, 6]) + # Test a new_style class with __iter__ but no next() method def test_new_style_iter_class(self): class IterClass(object): -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.5): Issue #26492: Added additional tests for exhausted iterators of mutable
by serhiy.storchaka March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/89ba67ee83d6 changeset: 100803:89ba67ee83d6 branch: 3.5 parent: 100800:905b5944119c user: Serhiy Storchaka <storchaka(a)gmail.com> date: Wed Mar 30 21:01:26 2016 +0300 summary: Issue #26492: Added additional tests for exhausted iterators of mutable sequences. files: Lib/test/list_tests.py | 11 +++++++++++ Lib/test/test_bytes.py | 2 ++ Lib/test/test_iter.py | 11 +++++++++++ 3 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py --- a/Lib/test/list_tests.py +++ b/Lib/test/list_tests.py @@ -593,3 +593,14 @@ def __iter__(self): raise KeyboardInterrupt self.assertRaises(KeyboardInterrupt, list, F()) + + def test_exhausted_iterator(self): + a = self.type2test([1, 2, 3]) + exhit = iter(a) + empit = iter(a) + for x in exhit: # exhaust the iterator + next(empit) # not exhausted + a.append(9) + self.assertEqual(list(exhit), []) + self.assertEqual(list(empit), [9]) + self.assertEqual(a, self.type2test([1, 2, 3, 9])) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -17,6 +17,7 @@ import test.support import test.string_tests import test.buffer_tests +import test.list_tests from test.support import bigaddrspacetest, MAX_Py_ssize_t @@ -1323,6 +1324,7 @@ b[:] = data self.assertEqual(list(it), []) + test_exhausted_iterator = test.list_tests.CommonTest.test_exhausted_iterator class AssortedBytesTest(unittest.TestCase): # diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -190,6 +190,17 @@ self.assertTrue(isinstance(it, collections.abc.Iterator)) self.assertEqual(list(it), []) + def test_mutating_seq_class_exhausted_iter(self): + a = SequenceClass(5) + exhit = iter(a) + empit = iter(a) + for x in exhit: # exhaust the iterator + next(empit) # not exhausted + a.n = 7 + self.assertEqual(list(exhit), []) + self.assertEqual(list(empit), [5, 6]) + self.assertEqual(list(a), [0, 1, 2, 3, 4, 5, 6]) + # Test a new_style class with __iter__ but no next() method def test_new_style_iter_class(self): class IterClass(object): -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.5 -> default): Issue #26494: Fixed crash on iterating exhausting iterators.
by serhiy.storchaka March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/73ce47d4a7b2 changeset: 100801:73ce47d4a7b2 parent: 100799:854018bf929f parent: 100800:905b5944119c user: Serhiy Storchaka <storchaka(a)gmail.com> date: Wed Mar 30 20:41:15 2016 +0300 summary: Issue #26494: Fixed crash on iterating exhausting iterators. Affected classes are generic sequence iterators, iterators of str, bytes, bytearray, list, tuple, set, frozenset, dict, OrderedDict, corresponding views and os.scandir() iterator. files: Lib/test/seq_tests.py | 5 +++++ Lib/test/support/__init__.py | 19 +++++++++++++++++++ Lib/test/test_bytes.py | 4 ++++ Lib/test/test_deque.py | 4 ++++ Lib/test/test_dict.py | 6 ++++++ Lib/test/test_iter.py | 4 ++++ Lib/test/test_ordered_dict.py | 6 ++++++ Lib/test/test_set.py | 3 +++ Lib/test/test_unicode.py | 4 ++++ Misc/NEWS | 5 +++++ Modules/posixmodule.c | 16 ++++++++++------ Objects/bytearrayobject.c | 2 +- Objects/bytesobject.c | 2 +- Objects/dictobject.c | 6 +++--- Objects/iterobject.c | 2 +- Objects/listobject.c | 20 +++++++++++++------- Objects/setobject.c | 2 +- Objects/tupleobject.c | 2 +- Objects/unicodeobject.c | 2 +- 19 files changed, 92 insertions(+), 22 deletions(-) diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py --- a/Lib/test/seq_tests.py +++ b/Lib/test/seq_tests.py @@ -5,6 +5,7 @@ import unittest import sys import pickle +from test import support # Various iterables # This is used for checking the constructor (here and in test_deque.py) @@ -408,3 +409,7 @@ lst2 = pickle.loads(pickle.dumps(lst, proto)) self.assertEqual(lst2, lst) self.assertNotEqual(id(lst2), id(lst)) + + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.type2test) + support.check_free_after_iterating(self, reversed, self.type2test) diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -2432,3 +2432,22 @@ "memory allocations") import _testcapi return _testcapi.run_in_subinterp(code) + + +def check_free_after_iterating(test, iter, cls, args=()): + class A(cls): + def __del__(self): + nonlocal done + done = True + try: + next(it) + except StopIteration: + pass + + done = False + it = iter(A(*args)) + # Issue 26494: Shouldn't crash + test.assertRaises(StopIteration, next, it) + # The sequence should be deallocated just after the end of iterating + gc_collect() + test.assertTrue(done) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -761,6 +761,10 @@ self.assertRaisesRegex(TypeError, r'\bendswith\b', b.endswith, x, None, None, None) + def test_free_after_iterating(self): + test.support.check_free_after_iterating(self, iter, self.type2test) + test.support.check_free_after_iterating(self, reversed, self.type2test) + class BytesTest(BaseBytesTest, unittest.TestCase): type2test = bytes diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -918,6 +918,10 @@ # For now, bypass tests that require slicing pass + def test_free_after_iterating(self): + # For now, bypass tests that require slicing + self.skipTest("Exhausted deque iterator doesn't free a deque") + #============================================================================== libreftest = """ diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -954,6 +954,12 @@ d = {X(): 0, 1: 1} self.assertRaises(RuntimeError, d.update, other) + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, dict) + support.check_free_after_iterating(self, lambda d: iter(d.keys()), dict) + support.check_free_after_iterating(self, lambda d: iter(d.values()), dict) + support.check_free_after_iterating(self, lambda d: iter(d.items()), dict) + from test import mapping_tests class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol): diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -3,6 +3,7 @@ import sys import unittest from test.support import run_unittest, TESTFN, unlink, cpython_only +from test.support import check_free_after_iterating import pickle import collections.abc @@ -980,6 +981,9 @@ self.assertEqual(next(it), 0) self.assertEqual(next(it), 1) + def test_free_after_iterating(self): + check_free_after_iterating(self, iter, SequenceClass, (0,)) + def test_main(): run_unittest(TestCase) diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -608,6 +608,12 @@ gc.collect() self.assertIsNone(r()) + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.OrderedDict) + support.check_free_after_iterating(self, lambda d: iter(d.keys()), self.OrderedDict) + support.check_free_after_iterating(self, lambda d: iter(d.values()), self.OrderedDict) + support.check_free_after_iterating(self, lambda d: iter(d.items()), self.OrderedDict) + class PurePythonOrderedDictTests(OrderedDictTests, unittest.TestCase): diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py --- a/Lib/test/test_set.py +++ b/Lib/test/test_set.py @@ -364,6 +364,9 @@ gc.collect() self.assertTrue(ref() is None, "Cycle was not collected") + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.thetype) + class TestSet(TestJointOps, unittest.TestCase): thetype = set basetype = set diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2729,6 +2729,10 @@ # Check that the second call returns the same result self.assertEqual(getargs_s_hash(s), chr(k).encode() * (i + 1)) + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, str) + support.check_free_after_iterating(self, reversed, str) + class StringModuleTest(unittest.TestCase): def test_formatter_parser(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,11 @@ Core and Builtins ----------------- +- Issue #26494: Fixed crash on iterating exhausting iterators. + Affected classes are generic sequence iterators, iterators of str, bytes, + bytearray, list, tuple, set, frozenset, dict, OrderedDict, corresponding + views and os.scandir() iterator. + - Issue #26574: Optimize ``bytes.replace(b'', b'.')`` and ``bytearray.replace(b'', b'.')``. Patch written by Josh Snider. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -11956,13 +11956,15 @@ static void ScandirIterator_closedir(ScandirIterator *iterator) { - if (iterator->handle == INVALID_HANDLE_VALUE) + HANDLE handle = iterator->handle; + + if (handle == INVALID_HANDLE_VALUE) return; + iterator->handle = INVALID_HANDLE_VALUE; Py_BEGIN_ALLOW_THREADS - FindClose(iterator->handle); + FindClose(handle); Py_END_ALLOW_THREADS - iterator->handle = INVALID_HANDLE_VALUE; } static PyObject * @@ -12018,13 +12020,15 @@ static void ScandirIterator_closedir(ScandirIterator *iterator) { - if (!iterator->dirp) + DIR *dirp = iterator->dirp; + + if (!dirp) return; + iterator->dirp = NULL; Py_BEGIN_ALLOW_THREADS - closedir(iterator->dirp); + closedir(dirp); Py_END_ALLOW_THREADS - iterator->dirp = NULL; return; } diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -3126,8 +3126,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -3806,8 +3806,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/dictobject.c b/Objects/dictobject.c --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -2988,8 +2988,8 @@ return key; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } @@ -3069,8 +3069,8 @@ return value; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } @@ -3164,8 +3164,8 @@ return result; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } diff --git a/Objects/iterobject.c b/Objects/iterobject.c --- a/Objects/iterobject.c +++ b/Objects/iterobject.c @@ -69,8 +69,8 @@ PyErr_ExceptionMatches(PyExc_StopIteration)) { PyErr_Clear(); + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; } return NULL; } diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2776,8 +2776,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } @@ -2906,9 +2906,17 @@ listreviter_next(listreviterobject *it) { PyObject *item; - Py_ssize_t index = it->it_index; - PyListObject *seq = it->it_seq; + Py_ssize_t index; + PyListObject *seq; + assert(it != NULL); + seq = it->it_seq; + if (seq == NULL) { + return NULL; + } + assert(PyList_Check(seq)); + + index = it->it_index; if (index>=0 && index < PyList_GET_SIZE(seq)) { item = PyList_GET_ITEM(seq, index); it->it_index--; @@ -2916,10 +2924,8 @@ return item; } it->it_index = -1; - if (seq != NULL) { - it->it_seq = NULL; - Py_DECREF(seq); - } + it->it_seq = NULL; + Py_DECREF(seq); return NULL; } diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -916,8 +916,8 @@ return key; fail: + si->si_set = NULL; Py_DECREF(so); - si->si_set = NULL; return NULL; } diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -961,8 +961,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -15401,8 +15401,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } -- Repository URL: https://hg.python.org/cpython

1 0

cpython (2.7): Issue #26494: Fixed crash on iterating exhausting iterators.
by serhiy.storchaka March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/cff06d875678 changeset: 100802:cff06d875678 branch: 2.7 parent: 100778:f4cff2bf9903 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Wed Mar 30 20:43:06 2016 +0300 summary: Issue #26494: Fixed crash on iterating exhausting iterators. Affected classes are generic sequence iterators, iterators of bytearray, list, tuple, set, frozenset, dict, OrderedDict and corresponding views. files: Lib/test/seq_tests.py | 5 +++++ Lib/test/test_bytes.py | 4 ++++ Lib/test/test_deque.py | 4 ++++ Lib/test/test_dict.py | 9 +++++++++ Lib/test/test_iter.py | 6 +++++- Lib/test/test_ordered_dict.py | 9 +++++++++ Lib/test/test_set.py | 3 +++ Lib/test/test_support.py | 18 ++++++++++++++++++ Lib/test/test_unicode.py | 5 +++++ Misc/NEWS | 4 ++++ Objects/bytearrayobject.c | 2 +- Objects/dictobject.c | 6 +++--- Objects/iterobject.c | 2 +- Objects/listobject.c | 20 +++++++++++++------- Objects/setobject.c | 2 +- Objects/tupleobject.c | 2 +- 16 files changed, 86 insertions(+), 15 deletions(-) diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py --- a/Lib/test/seq_tests.py +++ b/Lib/test/seq_tests.py @@ -4,6 +4,7 @@ import unittest import sys +from test import test_support as support # Various iterables # This is used for checking the constructor (here and in test_deque.py) @@ -402,3 +403,7 @@ self.assertEqual(a.index(0, -4*sys.maxint, 4*sys.maxint), 2) self.assertRaises(ValueError, a.index, 0, 4*sys.maxint,-4*sys.maxint) self.assertRaises(ValueError, a.index, 2, 0, -10) + + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.type2test) + support.check_free_after_iterating(self, reversed, self.type2test) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -518,6 +518,10 @@ self.assertRaisesRegexp(TypeError, r'\bendswith\b', b.endswith, x, None, None, None) + def test_free_after_iterating(self): + test.test_support.check_free_after_iterating(self, iter, self.type2test) + test.test_support.check_free_after_iterating(self, reversed, self.type2test) + class ByteArrayTest(BaseBytesTest): type2test = bytearray diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -669,6 +669,10 @@ # SF bug #1486663 -- this used to erroneously raise a TypeError SubclassWithKwargs(newarg=1) + def test_free_after_iterating(self): + # For now, bypass tests that require slicing + self.skipTest("Exhausted deque iterator doesn't free a deque") + #============================================================================== libreftest = """ diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -681,6 +681,15 @@ self._tracked(MyDict()) + def test_free_after_iterating(self): + test_support.check_free_after_iterating(self, iter, dict) + test_support.check_free_after_iterating(self, lambda d: d.iterkeys(), dict) + test_support.check_free_after_iterating(self, lambda d: d.itervalues(), dict) + test_support.check_free_after_iterating(self, lambda d: d.iteritems(), dict) + test_support.check_free_after_iterating(self, lambda d: iter(d.viewkeys()), dict) + test_support.check_free_after_iterating(self, lambda d: iter(d.viewvalues()), dict) + test_support.check_free_after_iterating(self, lambda d: iter(d.viewitems()), dict) + from test import mapping_tests class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol): diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -2,7 +2,8 @@ import unittest from test.test_support import run_unittest, TESTFN, unlink, have_unicode, \ - check_py3k_warnings, cpython_only + check_py3k_warnings, cpython_only, \ + check_free_after_iterating # Test result of triple loop (too big to inline) TRIPLETS = [(0, 0, 0), (0, 0, 1), (0, 0, 2), @@ -921,6 +922,9 @@ lst.extend(gen()) self.assertEqual(len(lst), 760) + def test_free_after_iterating(self): + check_free_after_iterating(self, iter, SequenceClass, (0,)) + def test_main(): run_unittest(TestCase) diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -267,6 +267,15 @@ items = [('a', 1), ('c', 3), ('b', 2)] self.assertEqual(list(MyOD(items).items()), items) + def test_free_after_iterating(self): + test_support.check_free_after_iterating(self, iter, OrderedDict) + test_support.check_free_after_iterating(self, lambda d: d.iterkeys(), OrderedDict) + test_support.check_free_after_iterating(self, lambda d: d.itervalues(), OrderedDict) + test_support.check_free_after_iterating(self, lambda d: d.iteritems(), OrderedDict) + test_support.check_free_after_iterating(self, lambda d: iter(d.viewkeys()), OrderedDict) + test_support.check_free_after_iterating(self, lambda d: iter(d.viewvalues()), OrderedDict) + test_support.check_free_after_iterating(self, lambda d: iter(d.viewitems()), OrderedDict) + class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol): type2test = OrderedDict diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py --- a/Lib/test/test_set.py +++ b/Lib/test/test_set.py @@ -340,6 +340,9 @@ gc.collect() self.assertTrue(ref() is None, "Cycle was not collected") + def test_free_after_iterating(self): + test_support.check_free_after_iterating(self, iter, self.thetype) + class TestSet(TestJointOps): thetype = set diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -1659,3 +1659,21 @@ """ stderr = re.sub(br"\[\d+ refs\]\r?\n?$", b"", stderr).strip() return stderr + + +def check_free_after_iterating(test, iter, cls, args=()): + class A(cls): + def __del__(self): + done[0] = True + try: + next(it) + except StopIteration: + pass + + done = [False] + it = iter(A(*args)) + # Issue 26494: Shouldn't crash + test.assertRaises(StopIteration, next, it) + # The sequence should be deallocated just after the end of iterating + gc_collect() + test.assertTrue(done[0]) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1857,6 +1857,11 @@ unicode_encodedecimal(u"123" + s, "xmlcharrefreplace"), '123' + exp) + def test_free_after_iterating(self): + test_support.check_free_after_iterating(self, iter, unicode) + test_support.check_free_after_iterating(self, reversed, unicode) + + def test_main(): test_support.run_unittest(__name__) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #26494: Fixed crash on iterating exhausting iterators. + Affected classes are generic sequence iterators, iterators of bytearray, + list, tuple, set, frozenset, dict, OrderedDict and corresponding views. + - Issue #26581: If coding cookie is specified multiple times on a line in Python source code file, only the first one is taken to account. diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -2982,8 +2982,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/dictobject.c b/Objects/dictobject.c --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -2586,8 +2586,8 @@ return key; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } @@ -2658,8 +2658,8 @@ return value; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } @@ -2744,8 +2744,8 @@ return result; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } diff --git a/Objects/iterobject.c b/Objects/iterobject.c --- a/Objects/iterobject.c +++ b/Objects/iterobject.c @@ -69,8 +69,8 @@ PyErr_ExceptionMatches(PyExc_StopIteration)) { PyErr_Clear(); + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; } return NULL; } diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2915,8 +2915,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } @@ -3018,9 +3018,17 @@ listreviter_next(listreviterobject *it) { PyObject *item; - Py_ssize_t index = it->it_index; - PyListObject *seq = it->it_seq; + Py_ssize_t index; + PyListObject *seq; + assert(it != NULL); + seq = it->it_seq; + if (seq == NULL) { + return NULL; + } + assert(PyList_Check(seq)); + + index = it->it_index; if (index>=0 && index < PyList_GET_SIZE(seq)) { item = PyList_GET_ITEM(seq, index); it->it_index--; @@ -3028,10 +3036,8 @@ return item; } it->it_index = -1; - if (seq != NULL) { - it->it_seq = NULL; - Py_DECREF(seq); - } + it->it_seq = NULL; + Py_DECREF(seq); return NULL; } diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -871,8 +871,8 @@ return key; fail: + si->si_set = NULL; Py_DECREF(so); - si->si_set = NULL; return NULL; } diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -966,8 +966,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.5): Issue #26494: Fixed crash on iterating exhausting iterators.
by serhiy.storchaka March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/905b5944119c changeset: 100800:905b5944119c branch: 3.5 parent: 100798:a38c6e6084cb user: Serhiy Storchaka <storchaka(a)gmail.com> date: Wed Mar 30 20:40:02 2016 +0300 summary: Issue #26494: Fixed crash on iterating exhausting iterators. Affected classes are generic sequence iterators, iterators of str, bytes, bytearray, list, tuple, set, frozenset, dict, OrderedDict, corresponding views and os.scandir() iterator. files: Lib/test/seq_tests.py | 5 +++++ Lib/test/support/__init__.py | 19 +++++++++++++++++++ Lib/test/test_bytes.py | 4 ++++ Lib/test/test_deque.py | 4 ++++ Lib/test/test_dict.py | 6 ++++++ Lib/test/test_iter.py | 4 ++++ Lib/test/test_ordered_dict.py | 6 ++++++ Lib/test/test_set.py | 3 +++ Lib/test/test_unicode.py | 4 ++++ Misc/NEWS | 5 +++++ Modules/posixmodule.c | 16 ++++++++++------ Objects/bytearrayobject.c | 2 +- Objects/bytesobject.c | 2 +- Objects/dictobject.c | 6 +++--- Objects/iterobject.c | 2 +- Objects/listobject.c | 20 +++++++++++++------- Objects/setobject.c | 2 +- Objects/tupleobject.c | 2 +- Objects/unicodeobject.c | 2 +- 19 files changed, 92 insertions(+), 22 deletions(-) diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py --- a/Lib/test/seq_tests.py +++ b/Lib/test/seq_tests.py @@ -5,6 +5,7 @@ import unittest import sys import pickle +from test import support # Various iterables # This is used for checking the constructor (here and in test_deque.py) @@ -408,3 +409,7 @@ lst2 = pickle.loads(pickle.dumps(lst, proto)) self.assertEqual(lst2, lst) self.assertNotEqual(id(lst2), id(lst)) + + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.type2test) + support.check_free_after_iterating(self, reversed, self.type2test) diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -2366,3 +2366,22 @@ "memory allocations") import _testcapi return _testcapi.run_in_subinterp(code) + + +def check_free_after_iterating(test, iter, cls, args=()): + class A(cls): + def __del__(self): + nonlocal done + done = True + try: + next(it) + except StopIteration: + pass + + done = False + it = iter(A(*args)) + # Issue 26494: Shouldn't crash + test.assertRaises(StopIteration, next, it) + # The sequence should be deallocated just after the end of iterating + gc_collect() + test.assertTrue(done) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -747,6 +747,10 @@ self.assertRaisesRegex(TypeError, r'\bendswith\b', b.endswith, x, None, None, None) + def test_free_after_iterating(self): + test.support.check_free_after_iterating(self, iter, self.type2test) + test.support.check_free_after_iterating(self, reversed, self.type2test) + class BytesTest(BaseBytesTest, unittest.TestCase): type2test = bytes diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -905,6 +905,10 @@ # For now, bypass tests that require slicing pass + def test_free_after_iterating(self): + # For now, bypass tests that require slicing + self.skipTest("Exhausted deque iterator doesn't free a deque") + #============================================================================== libreftest = """ diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -952,6 +952,12 @@ d = {X(): 0, 1: 1} self.assertRaises(RuntimeError, d.update, other) + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, dict) + support.check_free_after_iterating(self, lambda d: iter(d.keys()), dict) + support.check_free_after_iterating(self, lambda d: iter(d.values()), dict) + support.check_free_after_iterating(self, lambda d: iter(d.items()), dict) + from test import mapping_tests class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol): diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -3,6 +3,7 @@ import sys import unittest from test.support import run_unittest, TESTFN, unlink, cpython_only +from test.support import check_free_after_iterating import pickle import collections.abc @@ -980,6 +981,9 @@ self.assertEqual(next(it), 0) self.assertEqual(next(it), 1) + def test_free_after_iterating(self): + check_free_after_iterating(self, iter, SequenceClass, (0,)) + def test_main(): run_unittest(TestCase) diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -598,6 +598,12 @@ gc.collect() self.assertIsNone(r()) + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.OrderedDict) + support.check_free_after_iterating(self, lambda d: iter(d.keys()), self.OrderedDict) + support.check_free_after_iterating(self, lambda d: iter(d.values()), self.OrderedDict) + support.check_free_after_iterating(self, lambda d: iter(d.items()), self.OrderedDict) + class PurePythonOrderedDictTests(OrderedDictTests, unittest.TestCase): diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py --- a/Lib/test/test_set.py +++ b/Lib/test/test_set.py @@ -362,6 +362,9 @@ gc.collect() self.assertTrue(ref() is None, "Cycle was not collected") + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, self.thetype) + class TestSet(TestJointOps, unittest.TestCase): thetype = set basetype = set diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2729,6 +2729,10 @@ # Check that the second call returns the same result self.assertEqual(getargs_s_hash(s), chr(k).encode() * (i + 1)) + def test_free_after_iterating(self): + support.check_free_after_iterating(self, iter, str) + support.check_free_after_iterating(self, reversed, str) + class StringModuleTest(unittest.TestCase): def test_formatter_parser(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,11 @@ Core and Builtins ----------------- +- Issue #26494: Fixed crash on iterating exhausting iterators. + Affected classes are generic sequence iterators, iterators of str, bytes, + bytearray, list, tuple, set, frozenset, dict, OrderedDict, corresponding + views and os.scandir() iterator. + - Issue #26581: If coding cookie is specified multiple times on a line in Python source code file, only the first one is taken to account. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -11928,13 +11928,15 @@ static void ScandirIterator_close(ScandirIterator *iterator) { - if (iterator->handle == INVALID_HANDLE_VALUE) + HANDLE handle = iterator->handle; + + if (handle == INVALID_HANDLE_VALUE) return; + iterator->handle = INVALID_HANDLE_VALUE; Py_BEGIN_ALLOW_THREADS - FindClose(iterator->handle); + FindClose(handle); Py_END_ALLOW_THREADS - iterator->handle = INVALID_HANDLE_VALUE; } static PyObject * @@ -11984,13 +11986,15 @@ static void ScandirIterator_close(ScandirIterator *iterator) { - if (!iterator->dirp) + DIR *dirp = iterator->dirp; + + if (!dirp) return; + iterator->dirp = NULL; Py_BEGIN_ALLOW_THREADS - closedir(iterator->dirp); + closedir(dirp); Py_END_ALLOW_THREADS - iterator->dirp = NULL; return; } diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -3186,8 +3186,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -3628,8 +3628,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/dictobject.c b/Objects/dictobject.c --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -2985,8 +2985,8 @@ return key; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } @@ -3066,8 +3066,8 @@ return value; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } @@ -3161,8 +3161,8 @@ return result; fail: + di->di_dict = NULL; Py_DECREF(d); - di->di_dict = NULL; return NULL; } diff --git a/Objects/iterobject.c b/Objects/iterobject.c --- a/Objects/iterobject.c +++ b/Objects/iterobject.c @@ -69,8 +69,8 @@ PyErr_ExceptionMatches(PyExc_StopIteration)) { PyErr_Clear(); + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; } return NULL; } diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2782,8 +2782,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } @@ -2912,9 +2912,17 @@ listreviter_next(listreviterobject *it) { PyObject *item; - Py_ssize_t index = it->it_index; - PyListObject *seq = it->it_seq; + Py_ssize_t index; + PyListObject *seq; + assert(it != NULL); + seq = it->it_seq; + if (seq == NULL) { + return NULL; + } + assert(PyList_Check(seq)); + + index = it->it_index; if (index>=0 && index < PyList_GET_SIZE(seq)) { item = PyList_GET_ITEM(seq, index); it->it_index--; @@ -2922,10 +2930,8 @@ return item; } it->it_index = -1; - if (seq != NULL) { - it->it_seq = NULL; - Py_DECREF(seq); - } + it->it_seq = NULL; + Py_DECREF(seq); return NULL; } diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -839,8 +839,8 @@ return key; fail: + si->si_set = NULL; Py_DECREF(so); - si->si_set = NULL; return NULL; } diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -964,8 +964,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -15149,8 +15149,8 @@ return item; } + it->it_seq = NULL; Py_DECREF(seq); - it->it_seq = NULL; return NULL; } -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.5 -> default): Fix typo in xml.dom.pulldom.rst
by berker.peksag March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/854018bf929f changeset: 100799:854018bf929f parent: 100797:e47e00723e5d parent: 100798:a38c6e6084cb user: Berker Peksag <berker.peksag(a)gmail.com> date: Wed Mar 30 16:29:01 2016 +0300 summary: Fix typo in xml.dom.pulldom.rst Reported by Matthew Cole on docs(a)p.o. files: Doc/library/xml.dom.pulldom.rst | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -114,13 +114,15 @@ Expands all children of *node* into *node*. Example:: + from xml.dom import pulldom + xml = '<html><title>Foo</title> <p>Some text <div>and more</div></p> </html>' doc = pulldom.parseString(xml) for event, node in doc: if event == pulldom.START_ELEMENT and node.tagName == 'p': # Following statement only prints '<p/>' print(node.toxml()) - doc.exandNode(node) + doc.expandNode(node) # Following statement prints node with all its children '<p>Some text <div>and more</div></p>' print(node.toxml()) -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.5): Fix typo in xml.dom.pulldom.rst
by berker.peksag March 30, 2016

March 30, 2016

https://hg.python.org/cpython/rev/a38c6e6084cb changeset: 100798:a38c6e6084cb branch: 3.5 parent: 100792:7bd4736195ce user: Berker Peksag <berker.peksag(a)gmail.com> date: Wed Mar 30 16:28:43 2016 +0300 summary: Fix typo in xml.dom.pulldom.rst Reported by Matthew Cole on docs(a)p.o. files: Doc/library/xml.dom.pulldom.rst | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -114,13 +114,15 @@ Expands all children of *node* into *node*. Example:: + from xml.dom import pulldom + xml = '<html><title>Foo</title> <p>Some text <div>and more</div></p> </html>' doc = pulldom.parseString(xml) for event, node in doc: if event == pulldom.START_ELEMENT and node.tagName == 'p': # Following statement only prints '<p/>' print(node.toxml()) - doc.exandNode(node) + doc.expandNode(node) # Following statement prints node with all its children '<p>Some text <div>and more</div></p>' print(node.toxml()) -- Repository URL: https://hg.python.org/cpython

1 0

BAD Benchmark Results for Python 2.7 2016-03-30
by lp_benchmark_robot＠intel.com March 30, 2016

March 30, 2016

No new revisions. Here are the previous results: Results for project Python 2.7, build date 2016-03-30 02:58:15 +0000 commit: f4cff2bf9903 previous commit: b84d136e0028 revision date: 2016-03-28 00:22:09 +0000 environment: Haswell-EP cpu: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, stepping 2, LLC 45 MB mem: 128 GB os: CentOS 7.1 kernel: Linux 3.10.0-229.4.2.el7.x86_64 Baseline results were generated using release v2.7.10, with hash 15c95b7d81dc from 2015-05-23 16:02:14+00:00 ---------------------------------------------------------------------------------- benchmark relative change since change since current rev run std_dev* last run baseline with PGO ---------------------------------------------------------------------------------- :-) django_v2 0.13% -1.10% 4.22% 7.37% :-) pybench 0.09% -0.03% 6.43% 3.87% :-( regex_v8 0.78% -0.23% -2.87% 11.65% :-) nbody 0.09% -0.71% 7.90% 3.70% :-) json_dump_v2 0.29% -0.18% 4.41% 11.11% :-( normal_startup 1.71% -0.18% -6.07% 2.88% :-) ssbench 0.13% -0.06% 2.07% 1.43% ---------------------------------------------------------------------------------- * Relative Standard Deviation (Standard Deviation/Average) If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/bad-benchmark-results-for-python-2-7-… Note: Benchmark results for ssbench are measured in requests/second while all other are measured in seconds. Subject Label Legend: Attributes are determined based on the performance evolution of the workloads compared to the previous measurement iteration. NEUTRAL: performance did not change by more than 1% for any workload GOOD: performance improved by more than 1% for at least one workload and there is no regression greater than 1% BAD: performance dropped by more than 1% for at least one workload and there is no improvement greater than 1% UGLY: performance improved by more than 1% for at least one workload and also dropped by more than 1% for at least one workload Our lab does a nightly source pull and build of the Python project and measures performance changes against the previous stable version and the previous nightly measurement. This is provided as a service to the community so that quality issues with current hardware can be identified quickly. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.

1 0

UGLY Benchmark Results for Python Default 2016-03-30
by lp_benchmark_robot＠intel.com March 30, 2016

March 30, 2016

Results for project Python default, build date 2016-03-30 02:06:53 +0000 commit: c83349c129c9 previous commit: 9fdeca5fdbf0 revision date: 2016-03-30 00:33:52 +0000 environment: Haswell-EP cpu: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, stepping 2, LLC 45 MB mem: 128 GB os: CentOS 7.1 kernel: Linux 3.10.0-229.4.2.el7.x86_64 Baseline results were generated using release v3.4.3, with hash b4cbecbc0781 from 2015-02-25 12:15:33+00:00 ---------------------------------------------------------------------------------- benchmark relative change since change since current rev run std_dev* last run baseline with PGO ---------------------------------------------------------------------------------- :-) django_v2 0.26% -2.31% 7.49% 17.26% :-| pybench 0.20% -0.02% 0.73% 5.69% :-( regex_v8 2.68% -0.54% -3.95% 4.65% :-| nbody 0.06% 0.45% -0.18% 8.05% :-| json_dump_v2 0.22% 1.32% -0.53% 10.33% :-| normal_startup 0.94% -0.12% 0.19% 5.29% ---------------------------------------------------------------------------------- * Relative Standard Deviation (Standard Deviation/Average) If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/ugly-benchmark-results-for-python-def… Note: Benchmark results are measured in seconds. Subject Label Legend: Attributes are determined based on the performance evolution of the workloads compared to the previous measurement iteration. NEUTRAL: performance did not change by more than 1% for any workload GOOD: performance improved by more than 1% for at least one workload and there is no regression greater than 1% BAD: performance dropped by more than 1% for at least one workload and there is no improvement greater than 1% UGLY: performance improved by more than 1% for at least one workload and also dropped by more than 1% for at least one workload Our lab does a nightly source pull and build of the Python project and measures performance changes against the previous stable version and the previous nightly measurement. This is provided as a service to the community so that quality issues with current hardware can be identified quickly. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.

1 0

Daily reference leaks (c83349c129c9): sum=4
by solipsis＠pitrou.net March 30, 2016

March 30, 2016

results for c83349c129c9 on branch "default" -------------------------------------------- test_functools leaked [0, 2, 2] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogBu6UQ4', '--timeout', '7200']

1 0