Mailman 3 November 2013 - Python-checkins

cpython (merge 3.3 -> default): Use @bigmemtest more accurately.
by mark.dickinson Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/08bcbc210a0f changeset: 87592:08bcbc210a0f parent: 87590:835007ccf2b0 parent: 87591:8662734296a0 user: Mark Dickinson <dickinsm(a)gmail.com> date: Tue Nov 26 20:29:06 2013 +0000 summary: Use @bigmemtest more accurately. files: Lib/test/test_strtod.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_strtod.py b/Lib/test/test_strtod.py --- a/Lib/test/test_strtod.py +++ b/Lib/test/test_strtod.py @@ -248,15 +248,15 @@ else: assert False, "expected ValueError" - @test.support.bigmemtest(size=5 * test.support._1G, memuse=1, dry_run=False) + @test.support.bigmemtest(size=test.support._2G+10, memuse=3, dry_run=False) def test_oversized_digit_strings(self, maxsize): # Input string whose length doesn't fit in an INT. - s = "1." + "1" * int(2.2e9) + s = "1." + "1" * maxsize with self.assertRaises(ValueError): float(s) del s - s = "0." + "0" * int(2.2e9) + "1" + s = "0." + "0" * maxsize + "1" with self.assertRaises(ValueError): float(s) del s -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Use @bigmemtest more accurately.
by mark.dickinson Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/8662734296a0 changeset: 87591:8662734296a0 branch: 3.3 parent: 87588:8abbdbe86c01 user: Mark Dickinson <dickinsm(a)gmail.com> date: Tue Nov 26 20:28:29 2013 +0000 summary: Use @bigmemtest more accurately. files: Lib/test/test_strtod.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_strtod.py b/Lib/test/test_strtod.py --- a/Lib/test/test_strtod.py +++ b/Lib/test/test_strtod.py @@ -248,15 +248,15 @@ else: assert False, "expected ValueError" - @test.support.bigmemtest(size=5 * test.support._1G, memuse=1, dry_run=False) + @test.support.bigmemtest(size=test.support._2G+10, memuse=4, dry_run=False) def test_oversized_digit_strings(self, maxsize): # Input string whose length doesn't fit in an INT. - s = "1." + "1" * int(2.2e9) + s = "1." + "1" * maxsize with self.assertRaises(ValueError): float(s) del s - s = "0." + "0" * int(2.2e9) + "1" + s = "0." + "0" * maxsize + "1" with self.assertRaises(ValueError): float(s) del s -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #19793: Improved markup for True/False constants in pathlib documentation.
by serhiy.storchaka Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/835007ccf2b0 changeset: 87590:835007ccf2b0 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Tue Nov 26 21:37:12 2013 +0200 summary: Issue #19793: Improved markup for True/False constants in pathlib documentation. files: Doc/library/pathlib.rst | 56 ++++++++++++++-------------- 1 files changed, 28 insertions(+), 28 deletions(-) diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -439,9 +439,9 @@ .. method:: PurePath.is_reserved() - With :class:`PureWindowsPath`, return True if the path is considered - reserved under Windows, False otherwise. With :class:`PurePosixPath`, - False is always returned. + With :class:`PureWindowsPath`, return ``True`` if the path is considered + reserved under Windows, ``False`` otherwise. With :class:`PurePosixPath`, + ``False`` is always returned. >>> PureWindowsPath('nul').is_reserved() True @@ -469,8 +469,8 @@ .. method:: PurePath.match(pattern) - Match this path against the provided glob-style pattern. Return True - if matching is successful, False otherwise. + Match this path against the provided glob-style pattern. Return ``True`` + if matching is successful, ``False`` otherwise. If *pattern* is relative, the path can be either relative or absolute, and matching is done from the right:: @@ -661,63 +661,63 @@ .. method:: Path.is_dir() - Return True if the path points to a directory (or a symbolic link - pointing to a directory), False if it points to another kind of file. + Return ``True`` if the path points to a directory (or a symbolic link + pointing to a directory), ``False`` if it points to another kind of file. - False is also returned if the path doesn't exist or is a broken symlink; + ``False`` is also returned if the path doesn't exist or is a broken symlink; other errors (such as permission errors) are propagated. .. method:: Path.is_file() - Return True if the path points to a regular file (or a symbolic link - pointing to a regular file), False if it points to another kind of file. + Return ``True`` if the path points to a regular file (or a symbolic link + pointing to a regular file), ``False`` if it points to another kind of file. - False is also returned if the path doesn't exist or is a broken symlink; + ``False`` is also returned if the path doesn't exist or is a broken symlink; other errors (such as permission errors) are propagated. .. method:: Path.is_symlink() - Return True if the path points to a symbolic link, False otherwise. + Return ``True`` if the path points to a symbolic link, ``False`` otherwise. - False is also returned if the path doesn't exist; other errors (such + ``False`` is also returned if the path doesn't exist; other errors (such as permission errors) are propagated. .. method:: Path.is_socket() - Return True if the path points to a Unix socket (or a symbolic link - pointing to a Unix socket), False if it points to another kind of file. + Return ``True`` if the path points to a Unix socket (or a symbolic link + pointing to a Unix socket), ``False`` if it points to another kind of file. - False is also returned if the path doesn't exist or is a broken symlink; + ``False`` is also returned if the path doesn't exist or is a broken symlink; other errors (such as permission errors) are propagated. .. method:: Path.is_fifo() - Return True if the path points to a FIFO (or a symbolic link - pointing to a FIFO), False if it points to another kind of file. + Return ``True`` if the path points to a FIFO (or a symbolic link + pointing to a FIFO), ``False`` if it points to another kind of file. - False is also returned if the path doesn't exist or is a broken symlink; + ``False`` is also returned if the path doesn't exist or is a broken symlink; other errors (such as permission errors) are propagated. .. method:: Path.is_block_device() - Return True if the path points to a block device (or a symbolic link - pointing to a block device), False if it points to another kind of file. + Return ``True`` if the path points to a block device (or a symbolic link + pointing to a block device), ``False`` if it points to another kind of file. - False is also returned if the path doesn't exist or is a broken symlink; + ``False`` is also returned if the path doesn't exist or is a broken symlink; other errors (such as permission errors) are propagated. .. method:: Path.is_char_device() - Return True if the path points to a character device (or a symbolic link - pointing to a character device), False if it points to another kind of file. + Return ``True`` if the path points to a character device (or a symbolic link + pointing to a character device), ``False`` if it points to another kind of file. - False is also returned if the path doesn't exist or is a broken symlink; + ``False`` is also returned if the path doesn't exist or is a broken symlink; other errors (such as permission errors) are propagated. @@ -755,8 +755,8 @@ combined with the process' ``umask`` value to determine the file mode and access flags. If the path already exists, :exc:`OSError` is raised. - If *parents* is True, any missing parents of this path are created - as needed. If *parents* is False (the default), a missing parent raises + If *parents* is true, any missing parents of this path are created + as needed. If *parents* is false (the default), a missing parent raises :exc:`OSError`. @@ -841,7 +841,7 @@ .. method:: Path.symlink_to(target, target_is_directory=False) Make this path a symbolic link to *target*. Under Windows, - *target_is_directory* must be True (default False) if the link's target + *target_is_directory* must be true (default ``False``) if the link's target is a directory. Under POSIX, *target_is_directory*'s value is ignored. >>> p = Path('mylink') -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Issue #11489: JSON decoder now accepts lone surrogates.
by serhiy.storchaka Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/5f7326ed850f changeset: 87589:5f7326ed850f parent: 87583:9c7ab3e68243 parent: 87588:8abbdbe86c01 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Tue Nov 26 21:27:11 2013 +0200 summary: Issue #11489: JSON decoder now accepts lone surrogates. files: Lib/json/decoder.py | 35 +++++----- Lib/test/test_json/test_scanstring.py | 51 +++++++++++++- Misc/NEWS | 2 + Modules/_json.c | 26 ++----- 4 files changed, 73 insertions(+), 41 deletions(-) diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py --- a/Lib/json/decoder.py +++ b/Lib/json/decoder.py @@ -58,6 +58,16 @@ 'b': '\b', 'f': '\f', 'n': '\n', 'r': '\r', 't': '\t', } +def _decode_uXXXX(s, pos): + esc = s[pos + 1:pos + 5] + if len(esc) == 4 and esc[1] not in 'xX': + try: + return int(esc, 16) + except ValueError: + pass + msg = "Invalid \\uXXXX escape" + raise ValueError(errmsg(msg, s, pos)) + def py_scanstring(s, end, strict=True, _b=BACKSLASH, _m=STRINGCHUNK.match): """Scan the string s for a JSON string. End is the index of the @@ -107,25 +117,14 @@ raise ValueError(errmsg(msg, s, end)) end += 1 else: - esc = s[end + 1:end + 5] - next_end = end + 5 - if len(esc) != 4: - msg = "Invalid \\uXXXX escape" - raise ValueError(errmsg(msg, s, end)) - uni = int(esc, 16) - if 0xd800 <= uni <= 0xdbff: - msg = "Invalid \\uXXXX\\uXXXX surrogate pair" - if not s[end + 5:end + 7] == '\\u': - raise ValueError(errmsg(msg, s, end)) - esc2 = s[end + 7:end + 11] - if len(esc2) != 4: - raise ValueError(errmsg(msg, s, end)) - uni2 = int(esc2, 16) - uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00)) - next_end += 6 + uni = _decode_uXXXX(s, end) + end += 5 + if 0xd800 <= uni <= 0xdbff and s[end:end + 2] == '\\u': + uni2 = _decode_uXXXX(s, end + 1) + if 0xdc00 <= uni2 <= 0xdfff: + uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00)) + end += 6 char = chr(uni) - - end = next_end _append(char) return ''.join(chunks), end diff --git a/Lib/test/test_json/test_scanstring.py b/Lib/test/test_json/test_scanstring.py --- a/Lib/test/test_json/test_scanstring.py +++ b/Lib/test/test_json/test_scanstring.py @@ -6,10 +6,6 @@ def test_scanstring(self): scanstring = self.json.decoder.scanstring self.assertEqual( - scanstring('"z\\ud834\\udd20x"', 1, True), - ('z\U0001d120x', 16)) - - self.assertEqual( scanstring('"z\U0001d120x"', 1, True), ('z\U0001d120x', 5)) @@ -89,6 +85,53 @@ scanstring('["Bad value", truth]', 2, True), ('Bad value', 12)) + def test_surrogates(self): + scanstring = self.json.decoder.scanstring + def assertScan(given, expect): + self.assertEqual(scanstring(given, 1, True), + (expect, len(given))) + + assertScan('"z\\ud834\\u0079x"', 'z\ud834yx') + assertScan('"z\\ud834\\udd20x"', 'z\U0001d120x') + assertScan('"z\\ud834\\ud834\\udd20x"', 'z\ud834\U0001d120x') + assertScan('"z\\ud834x"', 'z\ud834x') + assertScan('"z\\ud834\udd20x12345"', 'z\ud834\udd20x12345') + assertScan('"z\\udd20x"', 'z\udd20x') + assertScan('"z\ud834\udd20x"', 'z\ud834\udd20x') + assertScan('"z\ud834\\udd20x"', 'z\ud834\udd20x') + assertScan('"z\ud834x"', 'z\ud834x') + + def test_bad_escapes(self): + scanstring = self.json.decoder.scanstring + bad_escapes = [ + '"\\"', + '"\\x"', + '"\\u"', + '"\\u0"', + '"\\u01"', + '"\\u012"', + '"\\uz012"', + '"\\u0z12"', + '"\\u01z2"', + '"\\u012z"', + '"\\u0x12"', + '"\\u0X12"', + '"\\ud834\\"', + '"\\ud834\\u"', + '"\\ud834\\ud"', + '"\\ud834\\udd"', + '"\\ud834\\udd2"', + '"\\ud834\\uzdd2"', + '"\\ud834\\udzd2"', + '"\\ud834\\uddz2"', + '"\\ud834\\udd2z"', + '"\\ud834\\u0x20"', + '"\\ud834\\u0X20"', + ] + for s in bad_escapes: + with self.assertRaises(ValueError, msg=s): + scanstring(s, 1, True) + def test_overflow(self): with self.assertRaises(OverflowError): self.json.decoder.scanstring(b"xxx", sys.maxsize+1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -16,6 +16,8 @@ Library ------- +- Issue #11489: JSON decoder now accepts lone surrogates. + - Issue #19545: Avoid chained exceptions while passing stray % to time.strptime(). Initial patch by Claudiu Popa. diff --git a/Modules/_json.c b/Modules/_json.c --- a/Modules/_json.c +++ b/Modules/_json.c @@ -409,17 +409,10 @@ } } /* Surrogate pair */ - if (Py_UNICODE_IS_HIGH_SURROGATE(c)) { + if (Py_UNICODE_IS_HIGH_SURROGATE(c) && end + 6 < len && + PyUnicode_READ(kind, buf, next++) == '\\' && + PyUnicode_READ(kind, buf, next++) == 'u') { Py_UCS4 c2 = 0; - if (end + 6 >= len) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - if (PyUnicode_READ(kind, buf, next++) != '\\' || - PyUnicode_READ(kind, buf, next++) != 'u') { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } end += 6; /* Decode 4 hex digits */ for (; next < end; next++) { @@ -440,15 +433,10 @@ goto bail; } } - if (!Py_UNICODE_IS_LOW_SURROGATE(c2)) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - c = Py_UNICODE_JOIN_SURROGATES(c, c2); - } - else if (Py_UNICODE_IS_LOW_SURROGATE(c)) { - raise_errmsg("Unpaired low surrogate", pystr, end - 5); - goto bail; + if (Py_UNICODE_IS_LOW_SURROGATE(c2)) + c = Py_UNICODE_JOIN_SURROGATES(c, c2); + else + end -= 6; } } APPEND_OLD_CHUNK -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Issue #11489: JSON decoder now accepts lone surrogates.
by serhiy.storchaka Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/8abbdbe86c01 changeset: 87588:8abbdbe86c01 branch: 3.3 parent: 87582:eac133e13bb5 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Tue Nov 26 21:25:28 2013 +0200 summary: Issue #11489: JSON decoder now accepts lone surrogates. files: Lib/json/decoder.py | 35 +++++----- Lib/test/test_json/test_scanstring.py | 51 +++++++++++++- Misc/NEWS | 2 + Modules/_json.c | 26 ++----- 4 files changed, 73 insertions(+), 41 deletions(-) diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py --- a/Lib/json/decoder.py +++ b/Lib/json/decoder.py @@ -66,6 +66,16 @@ 'b': '\b', 'f': '\f', 'n': '\n', 'r': '\r', 't': '\t', } +def _decode_uXXXX(s, pos): + esc = s[pos + 1:pos + 5] + if len(esc) == 4 and esc[1] not in 'xX': + try: + return int(esc, 16) + except ValueError: + pass + msg = "Invalid \\uXXXX escape" + raise ValueError(errmsg(msg, s, pos)) + def py_scanstring(s, end, strict=True, _b=BACKSLASH, _m=STRINGCHUNK.match): """Scan the string s for a JSON string. End is the index of the @@ -115,25 +125,14 @@ raise ValueError(errmsg(msg, s, end)) end += 1 else: - esc = s[end + 1:end + 5] - next_end = end + 5 - if len(esc) != 4: - msg = "Invalid \\uXXXX escape" - raise ValueError(errmsg(msg, s, end)) - uni = int(esc, 16) - if 0xd800 <= uni <= 0xdbff: - msg = "Invalid \\uXXXX\\uXXXX surrogate pair" - if not s[end + 5:end + 7] == '\\u': - raise ValueError(errmsg(msg, s, end)) - esc2 = s[end + 7:end + 11] - if len(esc2) != 4: - raise ValueError(errmsg(msg, s, end)) - uni2 = int(esc2, 16) - uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00)) - next_end += 6 + uni = _decode_uXXXX(s, end) + end += 5 + if 0xd800 <= uni <= 0xdbff and s[end:end + 2] == '\\u': + uni2 = _decode_uXXXX(s, end + 1) + if 0xdc00 <= uni2 <= 0xdfff: + uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00)) + end += 6 char = chr(uni) - - end = next_end _append(char) return ''.join(chunks), end diff --git a/Lib/test/test_json/test_scanstring.py b/Lib/test/test_json/test_scanstring.py --- a/Lib/test/test_json/test_scanstring.py +++ b/Lib/test/test_json/test_scanstring.py @@ -6,10 +6,6 @@ def test_scanstring(self): scanstring = self.json.decoder.scanstring self.assertEqual( - scanstring('"z\\ud834\\udd20x"', 1, True), - ('z\U0001d120x', 16)) - - self.assertEqual( scanstring('"z\U0001d120x"', 1, True), ('z\U0001d120x', 5)) @@ -89,6 +85,53 @@ scanstring('["Bad value", truth]', 2, True), ('Bad value', 12)) + def test_surrogates(self): + scanstring = self.json.decoder.scanstring + def assertScan(given, expect): + self.assertEqual(scanstring(given, 1, True), + (expect, len(given))) + + assertScan('"z\\ud834\\u0079x"', 'z\ud834yx') + assertScan('"z\\ud834\\udd20x"', 'z\U0001d120x') + assertScan('"z\\ud834\\ud834\\udd20x"', 'z\ud834\U0001d120x') + assertScan('"z\\ud834x"', 'z\ud834x') + assertScan('"z\\ud834\udd20x12345"', 'z\ud834\udd20x12345') + assertScan('"z\\udd20x"', 'z\udd20x') + assertScan('"z\ud834\udd20x"', 'z\ud834\udd20x') + assertScan('"z\ud834\\udd20x"', 'z\ud834\udd20x') + assertScan('"z\ud834x"', 'z\ud834x') + + def test_bad_escapes(self): + scanstring = self.json.decoder.scanstring + bad_escapes = [ + '"\\"', + '"\\x"', + '"\\u"', + '"\\u0"', + '"\\u01"', + '"\\u012"', + '"\\uz012"', + '"\\u0z12"', + '"\\u01z2"', + '"\\u012z"', + '"\\u0x12"', + '"\\u0X12"', + '"\\ud834\\"', + '"\\ud834\\u"', + '"\\ud834\\ud"', + '"\\ud834\\udd"', + '"\\ud834\\udd2"', + '"\\ud834\\uzdd2"', + '"\\ud834\\udzd2"', + '"\\ud834\\uddz2"', + '"\\ud834\\udd2z"', + '"\\ud834\\u0x20"', + '"\\ud834\\u0X20"', + ] + for s in bad_escapes: + with self.assertRaises(ValueError, msg=s): + scanstring(s, 1, True) + def test_overflow(self): with self.assertRaises(OverflowError): self.json.decoder.scanstring(b"xxx", sys.maxsize+1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -16,6 +16,8 @@ Library ------- +- Issue #11489: JSON decoder now accepts lone surrogates. + - Issue #19545: Avoid chained exceptions while passing stray % to time.strptime(). Initial patch by Claudiu Popa. diff --git a/Modules/_json.c b/Modules/_json.c --- a/Modules/_json.c +++ b/Modules/_json.c @@ -433,17 +433,10 @@ } } /* Surrogate pair */ - if ((c & 0xfc00) == 0xd800) { + if (Py_UNICODE_IS_HIGH_SURROGATE(c) && end + 6 < len && + PyUnicode_READ(kind, buf, next++) == '\\' && + PyUnicode_READ(kind, buf, next++) == 'u') { Py_UCS4 c2 = 0; - if (end + 6 >= len) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - if (PyUnicode_READ(kind, buf, next++) != '\\' || - PyUnicode_READ(kind, buf, next++) != 'u') { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } end += 6; /* Decode 4 hex digits */ for (; next < end; next++) { @@ -464,15 +457,10 @@ goto bail; } } - if ((c2 & 0xfc00) != 0xdc00) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); - } - else if ((c & 0xfc00) == 0xdc00) { - raise_errmsg("Unpaired low surrogate", pystr, end - 5); - goto bail; + if (Py_UNICODE_IS_LOW_SURROGATE(c2)) + c = Py_UNICODE_JOIN_SURROGATES(c, c2); + else + end -= 6; } } APPEND_OLD_CHUNK -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #11489: JSON decoder now accepts lone surrogates.
by serhiy.storchaka Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/c85305a54e6d changeset: 87587:c85305a54e6d branch: 2.7 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Tue Nov 26 21:25:15 2013 +0200 summary: Issue #11489: JSON decoder now accepts lone surrogates. files: Lib/json/decoder.py | 35 ++++++----- Lib/json/tests/test_scanstring.py | 55 +++++++++++++++++- Misc/NEWS | 2 + Modules/_json.c | 49 ++++------------ 4 files changed, 84 insertions(+), 57 deletions(-) diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py --- a/Lib/json/decoder.py +++ b/Lib/json/decoder.py @@ -62,6 +62,16 @@ DEFAULT_ENCODING = "utf-8" +def _decode_uXXXX(s, pos): + esc = s[pos + 1:pos + 5] + if len(esc) == 4 and esc[1] not in 'xX': + try: + return int(esc, 16) + except ValueError: + pass + msg = "Invalid \\uXXXX escape" + raise ValueError(errmsg(msg, s, pos)) + def py_scanstring(s, end, encoding=None, strict=True, _b=BACKSLASH, _m=STRINGCHUNK.match): """Scan the string s for a JSON string. End is the index of the @@ -116,25 +126,16 @@ end += 1 else: # Unicode escape sequence - esc = s[end + 1:end + 5] - next_end = end + 5 - if len(esc) != 4: - msg = "Invalid \\uXXXX escape" - raise ValueError(errmsg(msg, s, end)) - uni = int(esc, 16) + uni = _decode_uXXXX(s, end) + end += 5 # Check for surrogate pair on UCS-4 systems - if 0xd800 <= uni <= 0xdbff and sys.maxunicode > 65535: - msg = "Invalid \\uXXXX\\uXXXX surrogate pair" - if not s[end + 5:end + 7] == '\\u': - raise ValueError(errmsg(msg, s, end)) - esc2 = s[end + 7:end + 11] - if len(esc2) != 4: - raise ValueError(errmsg(msg, s, end)) - uni2 = int(esc2, 16) - uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00)) - next_end += 6 + if sys.maxunicode > 65535 and \ + 0xd800 <= uni <= 0xdbff and s[end:end + 2] == '\\u': + uni2 = _decode_uXXXX(s, end + 1) + if 0xdc00 <= uni2 <= 0xdfff: + uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00)) + end += 6 char = unichr(uni) - end = next_end # Append the unescaped character _append(char) return u''.join(chunks), end diff --git a/Lib/json/tests/test_scanstring.py b/Lib/json/tests/test_scanstring.py --- a/Lib/json/tests/test_scanstring.py +++ b/Lib/json/tests/test_scanstring.py @@ -5,10 +5,6 @@ class TestScanstring(object): def test_scanstring(self): scanstring = self.json.decoder.scanstring - self.assertEqual( - scanstring('"z\\ud834\\udd20x"', 1, None, True), - (u'z\U0001d120x', 16)) - if sys.maxunicode == 65535: self.assertEqual( scanstring(u'"z\U0001d120x"', 1, None, True), @@ -94,6 +90,57 @@ scanstring('["Bad value", truth]', 2, None, True), (u'Bad value', 12)) + def test_surrogates(self): + scanstring = self.json.decoder.scanstring + def assertScan(given, expect): + self.assertEqual(scanstring(given, 1, None, True), + (expect, len(given))) + if not isinstance(given, unicode): + given = unicode(given) + self.assertEqual(scanstring(given, 1, None, True), + (expect, len(given))) + + assertScan('"z\\ud834\\u0079x"', u'z\ud834yx') + assertScan('"z\\ud834\\udd20x"', u'z\U0001d120x') + assertScan('"z\\ud834\\ud834\\udd20x"', u'z\ud834\U0001d120x') + assertScan('"z\\ud834x"', u'z\ud834x') + assertScan(u'"z\\ud834\udd20x12345"', u'z\ud834\udd20x12345') + assertScan('"z\\udd20x"', u'z\udd20x') + assertScan(u'"z\ud834\udd20x"', u'z\ud834\udd20x') + assertScan(u'"z\ud834\\udd20x"', u'z\ud834\udd20x') + assertScan(u'"z\ud834x"', u'z\ud834x') + + def test_bad_escapes(self): + scanstring = self.json.decoder.scanstring + bad_escapes = [ + '"\\"', + '"\\x"', + '"\\u"', + '"\\u0"', + '"\\u01"', + '"\\u012"', + '"\\uz012"', + '"\\u0z12"', + '"\\u01z2"', + '"\\u012z"', + '"\\u0x12"', + '"\\u0X12"', + '"\\ud834\\"', + '"\\ud834\\u"', + '"\\ud834\\ud"', + '"\\ud834\\udd"', + '"\\ud834\\udd2"', + '"\\ud834\\uzdd2"', + '"\\ud834\\udzd2"', + '"\\ud834\\uddz2"', + '"\\ud834\\udd2z"', + '"\\ud834\\u0x20"', + '"\\ud834\\u0X20"', + ] + for s in bad_escapes: + with self.assertRaises(ValueError): + scanstring(s, 1, None, True) + def test_issue3623(self): self.assertRaises(ValueError, self.json.decoder.scanstring, b"xxx", 1, "xxx") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #11489: JSON decoder now accepts lone surrogates. + - Fix test.test_support.bind_port() to not cause an error when Python was compiled on a system with SO_REUSEPORT defined in the headers but run on a system with an OS kernel that does not support that new socket option. diff --git a/Modules/_json.c b/Modules/_json.c --- a/Modules/_json.c +++ b/Modules/_json.c @@ -524,16 +524,10 @@ } #ifdef Py_UNICODE_WIDE /* Surrogate pair */ - if ((c & 0xfc00) == 0xd800) { + if ((c & 0xfc00) == 0xd800 && end + 6 < len && + buf[next++] == '\\' && + buf[next++] == 'u') { Py_UNICODE c2 = 0; - if (end + 6 >= len) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - if (buf[next++] != '\\' || buf[next++] != 'u') { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } end += 6; /* Decode 4 hex digits */ for (; next < end; next++) { @@ -554,15 +548,10 @@ goto bail; } } - if ((c2 & 0xfc00) != 0xdc00) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); - } - else if ((c & 0xfc00) == 0xdc00) { - raise_errmsg("Unpaired low surrogate", pystr, end - 5); - goto bail; + if ((c2 & 0xfc00) == 0xdc00) + c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); + else + end -= 6; } #endif } @@ -703,16 +692,9 @@ } #ifdef Py_UNICODE_WIDE /* Surrogate pair */ - if ((c & 0xfc00) == 0xd800) { + if ((c & 0xfc00) == 0xd800 && end + 6 < len && + buf[next++] == '\\' && buf[next++] == 'u') { Py_UNICODE c2 = 0; - if (end + 6 >= len) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - if (buf[next++] != '\\' || buf[next++] != 'u') { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } end += 6; /* Decode 4 hex digits */ for (; next < end; next++) { @@ -733,15 +715,10 @@ goto bail; } } - if ((c2 & 0xfc00) != 0xdc00) { - raise_errmsg("Unpaired high surrogate", pystr, end - 5); - goto bail; - } - c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); - } - else if ((c & 0xfc00) == 0xdc00) { - raise_errmsg("Unpaired low surrogate", pystr, end - 5); - goto bail; + if ((c2 & 0xfc00) == 0xdc00) + c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); + else + end -= 6; } #endif } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Fix misuse of the bigmemtest decorator
by antoine.pitrou Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/461a4d10753a changeset: 87586:461a4d10753a branch: 2.7 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Tue Nov 26 20:00:34 2013 +0100 summary: Fix misuse of the bigmemtest decorator files: Lib/test/test_strtod.py | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_strtod.py b/Lib/test/test_strtod.py --- a/Lib/test/test_strtod.py +++ b/Lib/test/test_strtod.py @@ -249,11 +249,9 @@ else: assert False, "expected ValueError" - @test_support.bigmemtest(minsize=5 * test_support._1G, memuse=1) + @test_support.precisionbigmemtest(size=test_support._2G, memuse=3, + dry_run=False) def test_oversized_digit_strings(self, maxsize): - if sys.maxsize <= 0x7fffffff: - self.skipTest("Only applies on 64-bit systems.") - # Input string whose length doesn't fit in an INT. s = "1." + "1" * int(2.2e9) with self.assertRaises(ValueError): -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #19638: Skip large digit string tests on 32-bit platforms.
by mark.dickinson Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/4c523ea2b429 changeset: 87585:4c523ea2b429 branch: 2.7 user: Mark Dickinson <dickinsm(a)gmail.com> date: Tue Nov 26 17:02:46 2013 +0000 summary: Issue #19638: Skip large digit string tests on 32-bit platforms. files: Lib/test/test_strtod.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_strtod.py b/Lib/test/test_strtod.py --- a/Lib/test/test_strtod.py +++ b/Lib/test/test_strtod.py @@ -251,6 +251,9 @@ @test_support.bigmemtest(minsize=5 * test_support._1G, memuse=1) def test_oversized_digit_strings(self, maxsize): + if sys.maxsize <= 0x7fffffff: + self.skipTest("Only applies on 64-bit systems.") + # Input string whose length doesn't fit in an INT. s = "1." + "1" * int(2.2e9) with self.assertRaises(ValueError): -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #19638: Raise ValueError instead of crashing when converting billion
by mark.dickinson Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/6d6a018a3bb0 changeset: 87584:6d6a018a3bb0 branch: 2.7 parent: 87579:04c4f141874b user: Mark Dickinson <dickinsm(a)gmail.com> date: Tue Nov 26 16:38:25 2013 +0000 summary: Issue #19638: Raise ValueError instead of crashing when converting billion character strings to float. files: Lib/test/test_strtod.py | 31 +++++++++++++ Misc/NEWS | 3 + Python/dtoa.c | 65 +++++++++++++++++++++------- 3 files changed, 82 insertions(+), 17 deletions(-) diff --git a/Lib/test/test_strtod.py b/Lib/test/test_strtod.py --- a/Lib/test/test_strtod.py +++ b/Lib/test/test_strtod.py @@ -249,6 +249,37 @@ else: assert False, "expected ValueError" + @test_support.bigmemtest(minsize=5 * test_support._1G, memuse=1) + def test_oversized_digit_strings(self, maxsize): + # Input string whose length doesn't fit in an INT. + s = "1." + "1" * int(2.2e9) + with self.assertRaises(ValueError): + float(s) + del s + + s = "0." + "0" * int(2.2e9) + "1" + with self.assertRaises(ValueError): + float(s) + del s + + def test_large_exponents(self): + # Verify that the clipping of the exponent in strtod doesn't affect the + # output values. + def positive_exp(n): + """ Long string with value 1.0 and exponent n""" + return '0.{}1e+{}'.format('0'*(n-1), n) + + def negative_exp(n): + """ Long string with value 1.0 and exponent -n""" + return '1{}e-{}'.format('0'*n, n) + + self.assertEqual(float(positive_exp(10000)), 1.0) + self.assertEqual(float(positive_exp(20000)), 1.0) + self.assertEqual(float(positive_exp(30000)), 1.0) + self.assertEqual(float(negative_exp(10000)), 1.0) + self.assertEqual(float(negative_exp(20000)), 1.0) + self.assertEqual(float(negative_exp(30000)), 1.0) + def test_particular(self): # inputs that produced crashes or incorrectly rounded results with # previous versions of dtoa.c, for various reasons diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,9 @@ Core and Builtins ----------------- +- Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 + billion characters) input strings in _Py_dg_strtod. + Library ------- diff --git a/Python/dtoa.c b/Python/dtoa.c --- a/Python/dtoa.c +++ b/Python/dtoa.c @@ -204,7 +204,24 @@ MAX_ABS_EXP in absolute value get truncated to +-MAX_ABS_EXP. MAX_ABS_EXP should fit into an int. */ #ifndef MAX_ABS_EXP -#define MAX_ABS_EXP 19999U +#define MAX_ABS_EXP 1100000000U +#endif +/* Bound on length of pieces of input strings in _Py_dg_strtod; specifically, + this is used to bound the total number of digits ignoring leading zeros and + the number of digits that follow the decimal point. Ideally, MAX_DIGITS + should satisfy MAX_DIGITS + 400 < MAX_ABS_EXP; that ensures that the + exponent clipping in _Py_dg_strtod can't affect the value of the output. */ +#ifndef MAX_DIGITS +#define MAX_DIGITS 1000000000U +#endif + +/* Guard against trying to use the above values on unusual platforms with ints + * of width less than 32 bits. */ +#if MAX_ABS_EXP > INT_MAX +#error "MAX_ABS_EXP should fit in an int" +#endif +#if MAX_DIGITS > INT_MAX +#error "MAX_DIGITS should fit in an int" #endif /* The following definition of Storeinc is appropriate for MIPS processors. @@ -1498,6 +1515,7 @@ Long L; BCinfo bc; Bigint *bb, *bb1, *bd, *bd0, *bs, *delta; + size_t ndigits, fraclen; dval(&rv) = 0.; @@ -1520,40 +1538,53 @@ c = *++s; lz = s != s1; - /* Point s0 at the first nonzero digit (if any). nd0 will be the position - of the point relative to s0. nd will be the total number of digits - ignoring leading zeros. */ + /* Point s0 at the first nonzero digit (if any). fraclen will be the + number of digits between the decimal point and the end of the + digit string. ndigits will be the total number of digits ignoring + leading zeros. */ s0 = s1 = s; while ('0' <= c && c <= '9') c = *++s; - nd0 = nd = s - s1; + ndigits = s - s1; + fraclen = 0; /* Parse decimal point and following digits. */ if (c == '.') { c = *++s; - if (!nd) { + if (!ndigits) { s1 = s; while (c == '0') c = *++s; lz = lz || s != s1; - nd0 -= s - s1; + fraclen += (s - s1); s0 = s; } s1 = s; while ('0' <= c && c <= '9') c = *++s; - nd += s - s1; + ndigits += s - s1; + fraclen += s - s1; } - /* Now lz is true if and only if there were leading zero digits, and nd - gives the total number of digits ignoring leading zeros. A valid input - must have at least one digit. */ - if (!nd && !lz) { + /* Now lz is true if and only if there were leading zero digits, and + ndigits gives the total number of digits ignoring leading zeros. A + valid input must have at least one digit. */ + if (!ndigits && !lz) { if (se) *se = (char *)s00; goto parse_error; } + /* Range check ndigits and fraclen to make sure that they, and values + computed with them, can safely fit in an int. */ + if (ndigits > MAX_DIGITS || fraclen > MAX_DIGITS) { + if (se) + *se = (char *)s00; + goto parse_error; + } + nd = (int)ndigits; + nd0 = (int)ndigits - (int)fraclen; + /* Parse exponent. */ e = 0; if (c == 'e' || c == 'E') { @@ -1886,20 +1917,20 @@ bd2++; /* At this stage bd5 - bb5 == e == bd2 - bb2 + bbe, bb2 - bs2 == 1, - and bs == 1, so: + and bs == 1, so: tdv == bd * 10**e = bd * 2**(bbe - bb2 + bd2) * 5**(bd5 - bb5) srv == bb * 2**bbe = bb * 2**(bbe - bb2 + bb2) - 0.5 ulp(srv) == 2**(bbe-1) = bs * 2**(bbe - bb2 + bs2) + 0.5 ulp(srv) == 2**(bbe-1) = bs * 2**(bbe - bb2 + bs2) - It follows that: + It follows that: M * tdv = bd * 2**bd2 * 5**bd5 M * srv = bb * 2**bb2 * 5**bb5 M * 0.5 ulp(srv) = bs * 2**bs2 * 5**bb5 - for some constant M. (Actually, M == 2**(bb2 - bbe) * 5**bb5, but - this fact is not needed below.) + for some constant M. (Actually, M == 2**(bb2 - bbe) * 5**bb5, but + this fact is not needed below.) */ /* Remove factor of 2**i, where i = min(bb2, bd2, bs2). */ -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Issue #19638: Merge from 3.3
by mark.dickinson Nov. 26, 2013

Nov. 26, 2013

http://hg.python.org/cpython/rev/9c7ab3e68243 changeset: 87583:9c7ab3e68243 parent: 87581:67e642f5ab5d parent: 87582:eac133e13bb5 user: Mark Dickinson <dickinsm(a)gmail.com> date: Tue Nov 26 16:19:38 2013 +0000 summary: Issue #19638: Merge from 3.3 files: Lib/test/test_strtod.py | 31 ++++++++++++++++ Misc/NEWS | 3 + Python/dtoa.c | 55 ++++++++++++++++++++++------ 3 files changed, 77 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_strtod.py b/Lib/test/test_strtod.py --- a/Lib/test/test_strtod.py +++ b/Lib/test/test_strtod.py @@ -248,6 +248,37 @@ else: assert False, "expected ValueError" + @test.support.bigmemtest(size=5 * test.support._1G, memuse=1, dry_run=False) + def test_oversized_digit_strings(self, maxsize): + # Input string whose length doesn't fit in an INT. + s = "1." + "1" * int(2.2e9) + with self.assertRaises(ValueError): + float(s) + del s + + s = "0." + "0" * int(2.2e9) + "1" + with self.assertRaises(ValueError): + float(s) + del s + + def test_large_exponents(self): + # Verify that the clipping of the exponent in strtod doesn't affect the + # output values. + def positive_exp(n): + """ Long string with value 1.0 and exponent n""" + return '0.{}1e+{}'.format('0'*(n-1), n) + + def negative_exp(n): + """ Long string with value 1.0 and exponent -n""" + return '1{}e-{}'.format('0'*n, n) + + self.assertEqual(float(positive_exp(10000)), 1.0) + self.assertEqual(float(positive_exp(20000)), 1.0) + self.assertEqual(float(positive_exp(30000)), 1.0) + self.assertEqual(float(negative_exp(10000)), 1.0) + self.assertEqual(float(negative_exp(20000)), 1.0) + self.assertEqual(float(negative_exp(30000)), 1.0) + def test_particular(self): # inputs that produced crashes or incorrectly rounded results with # previous versions of dtoa.c, for various reasons diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 + billion characters) input strings in _Py_dg_strtod. + Library ------- diff --git a/Python/dtoa.c b/Python/dtoa.c --- a/Python/dtoa.c +++ b/Python/dtoa.c @@ -204,7 +204,24 @@ MAX_ABS_EXP in absolute value get truncated to +-MAX_ABS_EXP. MAX_ABS_EXP should fit into an int. */ #ifndef MAX_ABS_EXP -#define MAX_ABS_EXP 19999U +#define MAX_ABS_EXP 1100000000U +#endif +/* Bound on length of pieces of input strings in _Py_dg_strtod; specifically, + this is used to bound the total number of digits ignoring leading zeros and + the number of digits that follow the decimal point. Ideally, MAX_DIGITS + should satisfy MAX_DIGITS + 400 < MAX_ABS_EXP; that ensures that the + exponent clipping in _Py_dg_strtod can't affect the value of the output. */ +#ifndef MAX_DIGITS +#define MAX_DIGITS 1000000000U +#endif + +/* Guard against trying to use the above values on unusual platforms with ints + * of width less than 32 bits. */ +#if MAX_ABS_EXP > INT_MAX +#error "MAX_ABS_EXP should fit in an int" +#endif +#if MAX_DIGITS > INT_MAX +#error "MAX_DIGITS should fit in an int" #endif /* The following definition of Storeinc is appropriate for MIPS processors. @@ -1538,6 +1555,7 @@ Long L; BCinfo bc; Bigint *bb, *bb1, *bd, *bd0, *bs, *delta; + size_t ndigits, fraclen; dval(&rv) = 0.; @@ -1560,40 +1578,53 @@ c = *++s; lz = s != s1; - /* Point s0 at the first nonzero digit (if any). nd0 will be the position - of the point relative to s0. nd will be the total number of digits - ignoring leading zeros. */ + /* Point s0 at the first nonzero digit (if any). fraclen will be the + number of digits between the decimal point and the end of the + digit string. ndigits will be the total number of digits ignoring + leading zeros. */ s0 = s1 = s; while ('0' <= c && c <= '9') c = *++s; - nd0 = nd = s - s1; + ndigits = s - s1; + fraclen = 0; /* Parse decimal point and following digits. */ if (c == '.') { c = *++s; - if (!nd) { + if (!ndigits) { s1 = s; while (c == '0') c = *++s; lz = lz || s != s1; - nd0 -= s - s1; + fraclen += (s - s1); s0 = s; } s1 = s; while ('0' <= c && c <= '9') c = *++s; - nd += s - s1; + ndigits += s - s1; + fraclen += s - s1; } - /* Now lz is true if and only if there were leading zero digits, and nd - gives the total number of digits ignoring leading zeros. A valid input - must have at least one digit. */ - if (!nd && !lz) { + /* Now lz is true if and only if there were leading zero digits, and + ndigits gives the total number of digits ignoring leading zeros. A + valid input must have at least one digit. */ + if (!ndigits && !lz) { if (se) *se = (char *)s00; goto parse_error; } + /* Range check ndigits and fraclen to make sure that they, and values + computed with them, can safely fit in an int. */ + if (ndigits > MAX_DIGITS || fraclen > MAX_DIGITS) { + if (se) + *se = (char *)s00; + goto parse_error; + } + nd = (int)ndigits; + nd0 = (int)ndigits - (int)fraclen; + /* Parse exponent. */ e = 0; if (c == 'e' || c == 'E') { -- Repository URL: http://hg.python.org/cpython

1 0