Mailman 3 October 2016 - Python-checkins

BAD Benchmark Results for Python Default 2016-10-31
by lp_benchmark_robot＠intel.com Oct. 31, 2016

Oct. 31, 2016

Results for project Python default, build date 2016-10-31 03:01:27 +0000 commit: e9169a8c0692 previous commit: 72e64fc8746b revision date: 2016-10-30 21:00:20 +0000 environment: Haswell-EP cpu: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, stepping 2, LLC 45 MB mem: 128 GB os: CentOS 7.1 kernel: Linux 3.10.0-229.4.2.el7.x86_64 Baseline results were generated using release v3.4.3, with hash b4cbecbc0781 from 2015-02-25 12:15:33+00:00 ---------------------------------------------------------------------------------- benchmark relative change since change since current rev run std_dev* last run baseline with PGO ---------------------------------------------------------------------------------- :-) django_v2 0.29% -0.28% 4.10% 16.79% :-) pybench 0.06% -0.34% 5.60% 4.78% :-( regex_v8 3.84% -2.75% -3.22% 3.50% :-) nbody 0.27% -1.17% 2.44% 4.84% :-( json_dump_v2 0.25% -0.93% -12.79% 18.06% :-| normal_startup 0.72% -0.51% 0.56% 6.64% ---------------------------------------------------------------------------------- * Relative Standard Deviation (Standard Deviation/Average) If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/bad-benchmark-results-for-python-defa… Note: Benchmark results are measured in seconds. Subject Label Legend: Attributes are determined based on the performance evolution of the workloads compared to the previous measurement iteration. NEUTRAL: performance did not change by more than 1% for any workload GOOD: performance improved by more than 1% for at least one workload and there is no regression greater than 1% BAD: performance dropped by more than 1% for at least one workload and there is no improvement greater than 1% UGLY: performance improved by more than 1% for at least one workload and also dropped by more than 1% for at least one workload Our lab does a nightly source pull and build of the Python project and measures performance changes against the previous stable version and the previous nightly measurement. This is provided as a service to the community so that quality issues with current hardware can be identified quickly. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.

1 0

NEUTRAL Benchmark Results for Python 2.7 2016-10-31
by lp_benchmark_robot＠intel.com Oct. 31, 2016

Oct. 31, 2016

Results for project Python 2.7, build date 2016-10-31 03:47:42 +0000 commit: e2dd0f48e643 previous commit: 8a564ab1d208 revision date: 2016-10-30 18:52:55 +0000 environment: Haswell-EP cpu: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, stepping 2, LLC 45 MB mem: 128 GB os: CentOS 7.1 kernel: Linux 3.10.0-229.4.2.el7.x86_64 Baseline results were generated using release v2.7.10, with hash 15c95b7d81dc from 2015-05-23 16:02:14+00:00 ---------------------------------------------------------------------------------- benchmark relative change since change since current rev run std_dev* last run baseline with PGO ---------------------------------------------------------------------------------- :-) django_v2 0.16% -0.27% 4.30% 7.77% :-) pybench 0.14% -0.09% 5.74% 4.21% :-| regex_v8 0.66% 0.21% -1.99% 10.68% :-) nbody 0.07% 0.01% 7.96% 4.02% :-) json_dump_v2 0.29% 0.05% 2.69% 10.74% :-| normal_startup 0.77% -0.53% -0.30% 2.15% :-) ssbench 0.23% -0.24% 2.14% 2.43% ---------------------------------------------------------------------------------- * Relative Standard Deviation (Standard Deviation/Average) If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/neutral-benchmark-results-for-python-… Note: Benchmark results for ssbench are measured in requests/second while all other are measured in seconds. Subject Label Legend: Attributes are determined based on the performance evolution of the workloads compared to the previous measurement iteration. NEUTRAL: performance did not change by more than 1% for any workload GOOD: performance improved by more than 1% for at least one workload and there is no regression greater than 1% BAD: performance dropped by more than 1% for at least one workload and there is no improvement greater than 1% UGLY: performance improved by more than 1% for at least one workload and also dropped by more than 1% for at least one workload Our lab does a nightly source pull and build of the Python project and measures performance changes against the previous stable version and the previous nightly measurement. This is provided as a service to the community so that quality issues with current hardware can be identified quickly. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.

1 0

cpython: Issue 28128: Print out better error/warning messages for invalid string escapes.
by eric.smith Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/259745f9a1e4 changeset: 104848:259745f9a1e4 user: Eric V. Smith <eric(a)trueblade.com> date: Mon Oct 31 09:22:08 2016 -0400 summary: Issue 28128: Print out better error/warning messages for invalid string escapes. files: Include/bytesobject.h | 5 + Include/unicodeobject.h | 11 +++ Lib/test/test_string_literals.py | 27 ++++++++ Lib/test/test_unicode.py | 7 -- Misc/NEWS | 4 + Objects/bytesobject.c | 37 ++++++++++- Objects/unicodeobject.c | 38 +++++++++- Python/ast.c | 66 ++++++++++++++++++- 8 files changed, 173 insertions(+), 22 deletions(-) diff --git a/Include/bytesobject.h b/Include/bytesobject.h --- a/Include/bytesobject.h +++ b/Include/bytesobject.h @@ -74,6 +74,11 @@ PyAPI_FUNC(PyObject *) PyBytes_DecodeEscape(const char *, Py_ssize_t, const char *, Py_ssize_t, const char *); +/* Helper for PyBytes_DecodeEscape that detects invalid escape chars. */ +PyAPI_FUNC(PyObject *) _PyBytes_DecodeEscape(const char *, Py_ssize_t, + const char *, Py_ssize_t, + const char *, + const char **); /* Macro, trading safety for speed */ #ifndef Py_LIMITED_API diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -1486,6 +1486,17 @@ const char *errors /* error handling */ ); +/* Helper for PyUnicode_DecodeUnicodeEscape that detects invalid escape + chars. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscape( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + const char **first_invalid_escape /* on return, points to first + invalid escaped char in + string. */ +); + PyAPI_FUNC(PyObject*) PyUnicode_AsUnicodeEscapeString( PyObject *unicode /* Unicode object */ ); diff --git a/Lib/test/test_string_literals.py b/Lib/test/test_string_literals.py --- a/Lib/test/test_string_literals.py +++ b/Lib/test/test_string_literals.py @@ -31,6 +31,7 @@ import sys import shutil import tempfile +import warnings import unittest @@ -104,6 +105,19 @@ self.assertRaises(SyntaxError, eval, r""" '\U000000' """) self.assertRaises(SyntaxError, eval, r""" '\U0000000' """) + def test_eval_str_invalid_escape(self): + for b in range(1, 128): + if b in b"""\n\r"'01234567NU\\abfnrtuvx""": + continue + with self.assertWarns(DeprecationWarning): + self.assertEqual(eval(r"'\%c'" % b), '\\' + chr(b)) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=DeprecationWarning) + eval("'''\n\\z'''") + self.assertEqual(len(w), 1) + self.assertEqual(w[0].filename, '<string>') + self.assertEqual(w[0].lineno, 2) + def test_eval_str_raw(self): self.assertEqual(eval(""" r'x' """), 'x') self.assertEqual(eval(r""" r'\x01' """), '\\' + 'x01') @@ -130,6 +144,19 @@ self.assertRaises(SyntaxError, eval, r""" b'\x' """) self.assertRaises(SyntaxError, eval, r""" b'\x0' """) + def test_eval_bytes_invalid_escape(self): + for b in range(1, 128): + if b in b"""\n\r"'01234567\\abfnrtvx""": + continue + with self.assertWarns(DeprecationWarning): + self.assertEqual(eval(r"b'\%c'" % b), b'\\' + bytes([b])) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=DeprecationWarning) + eval("b'''\n\\z'''") + self.assertEqual(len(w), 1) + self.assertEqual(w[0].filename, '<string>') + self.assertEqual(w[0].lineno, 2) + def test_eval_bytes_raw(self): self.assertEqual(eval(""" br'x' """), b'x') self.assertEqual(eval(""" rb'x' """), b'x') diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2413,13 +2413,6 @@ support.check_free_after_iterating(self, iter, str) support.check_free_after_iterating(self, reversed, str) - def test_invalid_sequences(self): - for letter in string.ascii_letters + "89": # 0-7 are octal escapes - if letter in "abfnrtuvxNU": - continue - with self.assertWarns(DeprecationWarning): - eval(r"'\%s'" % letter) - class CAPITest(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #28128: Deprecation warning for invalid str and byte escape + sequences now prints better information about where the error + occurs. Patch by Serhiy Storchaka and Eric Smith. + - Issue #28509: dict.update() no longer allocate unnecessary large memory. - Issue #28426: Fixed potential crash in PyUnicode_AsDecodedObject() in debug diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -1105,11 +1105,12 @@ return p; } -PyObject *PyBytes_DecodeEscape(const char *s, +PyObject *_PyBytes_DecodeEscape(const char *s, Py_ssize_t len, const char *errors, Py_ssize_t unicode, - const char *recode_encoding) + const char *recode_encoding, + const char **first_invalid_escape) { int c; char *p; @@ -1123,6 +1124,8 @@ return NULL; writer.overallocate = 1; + *first_invalid_escape = NULL; + end = s + len; while (s < end) { if (*s != '\\') { @@ -1207,9 +1210,12 @@ break; default: - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, "invalid escape sequence '\\%c'", *(--s)) < 0) - goto failed; + if (*first_invalid_escape == NULL) { + *first_invalid_escape = s-1; /* Back up one char, since we've + already incremented s. */ + } *p++ = '\\'; + s--; goto non_esc; /* an arbitrary number of unescaped UTF-8 bytes may follow. */ } @@ -1222,6 +1228,29 @@ return NULL; } +PyObject *PyBytes_DecodeEscape(const char *s, + Py_ssize_t len, + const char *errors, + Py_ssize_t unicode, + const char *recode_encoding) +{ + const char* first_invalid_escape; + PyObject *result = _PyBytes_DecodeEscape(s, len, errors, unicode, + recode_encoding, + &first_invalid_escape); + if (result == NULL) + return NULL; + if (first_invalid_escape != NULL) { + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "invalid escape sequence '\\%c'", + *first_invalid_escape) < 0) { + Py_DECREF(result); + return NULL; + } + } + return result; + +} /* -------------------------------------------------------------------- */ /* object api */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5877,9 +5877,10 @@ static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL; PyObject * -PyUnicode_DecodeUnicodeEscape(const char *s, - Py_ssize_t size, - const char *errors) +_PyUnicode_DecodeUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors, + const char **first_invalid_escape) { const char *starts = s; _PyUnicodeWriter writer; @@ -5887,6 +5888,9 @@ PyObject *errorHandler = NULL; PyObject *exc = NULL; + // so we can remember if we've seen an invalid escape char or not + *first_invalid_escape = NULL; + if (size == 0) { _Py_RETURN_UNICODE_EMPTY(); } @@ -6061,9 +6065,10 @@ goto error; default: - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "invalid escape sequence '\\%c'", c) < 0) - goto onError; + if (*first_invalid_escape == NULL) { + *first_invalid_escape = s-1; /* Back up one char, since we've + already incremented s. */ + } WRITE_ASCII_CHAR('\\'); WRITE_CHAR(c); continue; @@ -6098,6 +6103,27 @@ return NULL; } +PyObject * +PyUnicode_DecodeUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + const char *first_invalid_escape; + PyObject *result = _PyUnicode_DecodeUnicodeEscape(s, size, errors, + &first_invalid_escape); + if (result == NULL) + return NULL; + if (first_invalid_escape != NULL) { + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "invalid escape sequence '\\%c'", + *first_invalid_escape) < 0) { + Py_DECREF(result); + return NULL; + } + } + return result; +} + /* Return a Unicode-Escape string version of the Unicode object. If quotes is true, the string is enclosed in u"" or u'' quotes as diff --git a/Python/ast.c b/Python/ast.c --- a/Python/ast.c +++ b/Python/ast.c @@ -4113,8 +4113,34 @@ return PyUnicode_DecodeUTF8(t, s - t, NULL); } +static int +warn_invalid_escape_sequence(struct compiling *c, const node *n, + char first_invalid_escape_char) +{ + PyObject *msg = PyUnicode_FromFormat("invalid escape sequence \\%c", + first_invalid_escape_char); + if (msg == NULL) { + return -1; + } + if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, msg, + c->c_filename, LINENO(n), + NULL, NULL) < 0 && + PyErr_ExceptionMatches(PyExc_DeprecationWarning)) + { + const char *s = PyUnicode_AsUTF8(msg); + if (s != NULL) { + ast_error(c, n, s); + } + Py_DECREF(msg); + return -1; + } + Py_DECREF(msg); + return 0; +} + static PyObject * -decode_unicode_with_escapes(struct compiling *c, const char *s, size_t len) +decode_unicode_with_escapes(struct compiling *c, const node *n, const char *s, + size_t len) { PyObject *v, *u; char *buf; @@ -4167,11 +4193,41 @@ len = p - buf; s = buf; - v = PyUnicode_DecodeUnicodeEscape(s, len, NULL); + const char *first_invalid_escape; + v = _PyUnicode_DecodeUnicodeEscape(s, len, NULL, &first_invalid_escape); + + if (v != NULL && first_invalid_escape != NULL) { + if (warn_invalid_escape_sequence(c, n, *first_invalid_escape) < 0) { + /* We have not decref u before because first_invalid_escape points + inside u. */ + Py_XDECREF(u); + Py_DECREF(v); + return NULL; + } + } Py_XDECREF(u); return v; } +static PyObject * +decode_bytes_with_escapes(struct compiling *c, const node *n, const char *s, + size_t len) +{ + const char *first_invalid_escape; + PyObject *result = _PyBytes_DecodeEscape(s, len, NULL, 0, NULL, + &first_invalid_escape); + if (result == NULL) + return NULL; + + if (first_invalid_escape != NULL) { + if (warn_invalid_escape_sequence(c, n, *first_invalid_escape) < 0) { + Py_DECREF(result); + return NULL; + } + } + return result; +} + /* Compile this expression in to an expr_ty. Add parens around the expression, in order to allow leading spaces in the expression. */ static expr_ty @@ -4310,7 +4366,7 @@ literal_end-literal_start, NULL, NULL); else - *literal = decode_unicode_with_escapes(c, literal_start, + *literal = decode_unicode_with_escapes(c, n, literal_start, literal_end-literal_start); if (!*literal) return -1; @@ -5048,12 +5104,12 @@ if (*rawmode) *result = PyBytes_FromStringAndSize(s, len); else - *result = PyBytes_DecodeEscape(s, len, NULL, /* ignored */ 0, NULL); + *result = decode_bytes_with_escapes(c, n, s, len); } else { if (*rawmode) *result = PyUnicode_DecodeUTF8Stateful(s, len, NULL, NULL); else - *result = decode_unicode_with_escapes(c, s, len); + *result = decode_unicode_with_escapes(c, n, s, len); } return *result == NULL ? -1 : 0; } -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.6 -> default): Issue #28553: Fix logic error in example code of int.to_bytes doc.
by inada.naoki Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/66f255754ce9 changeset: 104847:66f255754ce9 parent: 104844:98c078fca8e0 parent: 104846:2ae3f1599c34 user: INADA Naoki <songofacandy(a)gmail.com> date: Mon Oct 31 17:42:25 2016 +0900 summary: Issue #28553: Fix logic error in example code of int.to_bytes doc. files: Doc/library/stdtypes.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -485,7 +485,7 @@ >>> (-1024).to_bytes(10, byteorder='big', signed=True) b'\xff\xff\xff\xff\xff\xff\xff\xff\xfc\x00' >>> x = 1000 - >>> x.to_bytes((x.bit_length() // 8) + 1, byteorder='little') + >>> x.to_bytes((x.bit_length() + 7) // 8, byteorder='little') b'\xe8\x03' The integer is represented using *length* bytes. An :exc:`OverflowError` -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.5 -> 3.6): Issue #28553: Fix logic error in example code of int.to_bytes doc.
by inada.naoki Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/2ae3f1599c34 changeset: 104846:2ae3f1599c34 branch: 3.6 parent: 104843:fb64c7a81010 parent: 104845:32d8c89e90d1 user: INADA Naoki <songofacandy(a)gmail.com> date: Mon Oct 31 17:42:10 2016 +0900 summary: Issue #28553: Fix logic error in example code of int.to_bytes doc. files: Doc/library/stdtypes.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -485,7 +485,7 @@ >>> (-1024).to_bytes(10, byteorder='big', signed=True) b'\xff\xff\xff\xff\xff\xff\xff\xff\xfc\x00' >>> x = 1000 - >>> x.to_bytes((x.bit_length() // 8) + 1, byteorder='little') + >>> x.to_bytes((x.bit_length() + 7) // 8, byteorder='little') b'\xe8\x03' The integer is represented using *length* bytes. An :exc:`OverflowError` -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.5): Issue #28553: Fix logic error in example code of int.to_bytes doc.
by inada.naoki Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/32d8c89e90d1 changeset: 104845:32d8c89e90d1 branch: 3.5 parent: 104842:b06f15507978 user: INADA Naoki <songofacandy(a)gmail.com> date: Mon Oct 31 17:41:47 2016 +0900 summary: Issue #28553: Fix logic error in example code of int.to_bytes doc. files: Doc/library/stdtypes.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -485,7 +485,7 @@ >>> (-1024).to_bytes(10, byteorder='big', signed=True) b'\xff\xff\xff\xff\xff\xff\xff\xff\xfc\x00' >>> x = 1000 - >>> x.to_bytes((x.bit_length() // 8) + 1, byteorder='little') + >>> x.to_bytes((x.bit_length() + 7) // 8, byteorder='little') b'\xe8\x03' The integer is represented using *length* bytes. An :exc:`OverflowError` -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.5 -> 3.6): Issue #27517: LZMA compressor and decompressor no longer raise exceptions if
by serhiy.storchaka Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/fb64c7a81010 changeset: 104843:fb64c7a81010 branch: 3.6 parent: 104840:f43078ec598c parent: 104842:b06f15507978 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Mon Oct 31 08:31:13 2016 +0200 summary: Issue #27517: LZMA compressor and decompressor no longer raise exceptions if given empty data twice. Patch by Benjamin Fogle. files: Lib/test/test_lzma.py | 38 +++++++++++++++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 ++ Modules/_lzmamodule.c | 5 ++++ 4 files changed, 47 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -137,6 +137,21 @@ self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_empty(self): + lzd = LZMADecompressor() + out = [] + for i in range(0, len(COMPRESSED_XZ), 10): + self.assertFalse(lzd.eof) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(COMPRESSED_XZ[i:i+10])) + out = b"".join(out) + self.assertEqual(out, INPUT) + self.assertEqual(lzd.check, lzma.CHECK_CRC64) + self.assertTrue(lzd.eof) + self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_maxsize(self): lzd = LZMADecompressor() max_length = 100 @@ -274,6 +289,16 @@ lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + def test_roundtrip_raw_empty(self): + lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + cdata = lzc.compress(INPUT) + cdata += lzc.compress(b'') + cdata += lzc.compress(b'') + cdata += lzc.compress(b'') + cdata += lzc.flush() + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + def test_roundtrip_chunks(self): lzc = LZMACompressor() cdata = [] @@ -284,6 +309,19 @@ lzd = LZMADecompressor() self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + def test_roundtrip_empty_chunks(self): + lzc = LZMACompressor() + cdata = [] + for i in range(0, len(INPUT), 10): + cdata.append(lzc.compress(INPUT[i:i+10])) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.flush()) + cdata = b"".join(cdata) + lzd = LZMADecompressor() + self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + # LZMADecompressor intentionally does not handle concatenated streams. def test_decompressor_multistream(self): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -452,6 +452,7 @@ Tom Flanagan Matt Fleming Hernán Martínez Foffani +Benjamin Fogle Artem Fokin Arnaud Fontaine Michael Foord diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -30,6 +30,9 @@ Library ------- +- Issue #27517: LZMA compressor and decompressor no longer raise exceptions if + given empty data twice. Patch by Benjamin Fogle. + - Issue #28549: Fixed segfault in curses's addch() with ncurses6. - Issue #28449: tarfile.open() with mode "r" or "r:" now tries to open a tar diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -521,6 +521,8 @@ Py_BEGIN_ALLOW_THREADS lzret = lzma_code(&c->lzs, action); data_size = (char *)c->lzs.next_out - PyBytes_AS_STRING(result); + if (lzret == LZMA_BUF_ERROR && len == 0 && c->lzs.avail_out > 0) + lzret = LZMA_OK; /* That wasn't a real error */ Py_END_ALLOW_THREADS if (catch_lzma_error(lzret)) goto error; @@ -896,6 +898,9 @@ PyObject *result; lzma_stream *lzs = &d->lzs; + if (lzs->avail_in == 0) + return PyBytes_FromStringAndSize(NULL, 0); + if (max_length < 0 || max_length >= INITIAL_BUFFER_SIZE) result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); else -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.5): Issue #27517: LZMA compressor and decompressor no longer raise exceptions if
by serhiy.storchaka Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/b06f15507978 changeset: 104842:b06f15507978 branch: 3.5 parent: 104835:d06bf822585c user: Serhiy Storchaka <storchaka(a)gmail.com> date: Mon Oct 31 08:30:09 2016 +0200 summary: Issue #27517: LZMA compressor and decompressor no longer raise exceptions if given empty data twice. Patch by Benjamin Fogle. files: Lib/test/test_lzma.py | 38 +++++++++++++++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 ++ Modules/_lzmamodule.c | 5 ++++ 4 files changed, 47 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -136,6 +136,21 @@ self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_empty(self): + lzd = LZMADecompressor() + out = [] + for i in range(0, len(COMPRESSED_XZ), 10): + self.assertFalse(lzd.eof) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(COMPRESSED_XZ[i:i+10])) + out = b"".join(out) + self.assertEqual(out, INPUT) + self.assertEqual(lzd.check, lzma.CHECK_CRC64) + self.assertTrue(lzd.eof) + self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_maxsize(self): lzd = LZMADecompressor() max_length = 100 @@ -273,6 +288,16 @@ lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + def test_roundtrip_raw_empty(self): + lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + cdata = lzc.compress(INPUT) + cdata += lzc.compress(b'') + cdata += lzc.compress(b'') + cdata += lzc.compress(b'') + cdata += lzc.flush() + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + def test_roundtrip_chunks(self): lzc = LZMACompressor() cdata = [] @@ -283,6 +308,19 @@ lzd = LZMADecompressor() self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + def test_roundtrip_empty_chunks(self): + lzc = LZMACompressor() + cdata = [] + for i in range(0, len(INPUT), 10): + cdata.append(lzc.compress(INPUT[i:i+10])) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.flush()) + cdata = b"".join(cdata) + lzd = LZMADecompressor() + self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + # LZMADecompressor intentionally does not handle concatenated streams. def test_decompressor_multistream(self): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -446,6 +446,7 @@ Tom Flanagan Matt Fleming Hernán Martínez Foffani +Benjamin Fogle Artem Fokin Arnaud Fontaine Michael Foord diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -113,6 +113,9 @@ Library ------- +- Issue #27517: LZMA compressor and decompressor no longer raise exceptions if + given empty data twice. Patch by Benjamin Fogle. + - Issue #28549: Fixed segfault in curses's addch() with ncurses6. - Issue #28449: tarfile.open() with mode "r" or "r:" now tries to open a tar diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -527,6 +527,8 @@ Py_BEGIN_ALLOW_THREADS lzret = lzma_code(&c->lzs, action); data_size = (char *)c->lzs.next_out - PyBytes_AS_STRING(result); + if (lzret == LZMA_BUF_ERROR && len == 0 && c->lzs.avail_out > 0) + lzret = LZMA_OK; /* That wasn't a real error */ Py_END_ALLOW_THREADS if (catch_lzma_error(lzret)) goto error; @@ -906,6 +908,9 @@ PyObject *result; lzma_stream *lzs = &d->lzs; + if (lzs->avail_in == 0) + return PyBytes_FromStringAndSize(NULL, 0); + if (max_length < 0 || max_length >= INITIAL_BUFFER_SIZE) result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); else -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.6 -> default): Issue #27517: LZMA compressor and decompressor no longer raise exceptions if
by serhiy.storchaka Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/98c078fca8e0 changeset: 104844:98c078fca8e0 parent: 104841:931410a04240 parent: 104843:fb64c7a81010 user: Serhiy Storchaka <storchaka(a)gmail.com> date: Mon Oct 31 08:31:41 2016 +0200 summary: Issue #27517: LZMA compressor and decompressor no longer raise exceptions if given empty data twice. Patch by Benjamin Fogle. files: Lib/test/test_lzma.py | 38 +++++++++++++++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 ++ Modules/_lzmamodule.c | 5 ++++ 4 files changed, 47 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -137,6 +137,21 @@ self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_empty(self): + lzd = LZMADecompressor() + out = [] + for i in range(0, len(COMPRESSED_XZ), 10): + self.assertFalse(lzd.eof) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(b'')) + out.append(lzd.decompress(COMPRESSED_XZ[i:i+10])) + out = b"".join(out) + self.assertEqual(out, INPUT) + self.assertEqual(lzd.check, lzma.CHECK_CRC64) + self.assertTrue(lzd.eof) + self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_maxsize(self): lzd = LZMADecompressor() max_length = 100 @@ -274,6 +289,16 @@ lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + def test_roundtrip_raw_empty(self): + lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + cdata = lzc.compress(INPUT) + cdata += lzc.compress(b'') + cdata += lzc.compress(b'') + cdata += lzc.compress(b'') + cdata += lzc.flush() + lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4) + self._test_decompressor(lzd, cdata, lzma.CHECK_NONE) + def test_roundtrip_chunks(self): lzc = LZMACompressor() cdata = [] @@ -284,6 +309,19 @@ lzd = LZMADecompressor() self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + def test_roundtrip_empty_chunks(self): + lzc = LZMACompressor() + cdata = [] + for i in range(0, len(INPUT), 10): + cdata.append(lzc.compress(INPUT[i:i+10])) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.compress(b'')) + cdata.append(lzc.flush()) + cdata = b"".join(cdata) + lzd = LZMADecompressor() + self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64) + # LZMADecompressor intentionally does not handle concatenated streams. def test_decompressor_multistream(self): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -453,6 +453,7 @@ Tom Flanagan Matt Fleming Hernán Martínez Foffani +Benjamin Fogle Artem Fokin Arnaud Fontaine Michael Foord diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -99,6 +99,9 @@ Library ------- +- Issue #27517: LZMA compressor and decompressor no longer raise exceptions if + given empty data twice. Patch by Benjamin Fogle. + - Issue #28549: Fixed segfault in curses's addch() with ncurses6. - Issue #28449: tarfile.open() with mode "r" or "r:" now tries to open a tar diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -521,6 +521,8 @@ Py_BEGIN_ALLOW_THREADS lzret = lzma_code(&c->lzs, action); data_size = (char *)c->lzs.next_out - PyBytes_AS_STRING(result); + if (lzret == LZMA_BUF_ERROR && len == 0 && c->lzs.avail_out > 0) + lzret = LZMA_OK; /* That wasn't a real error */ Py_END_ALLOW_THREADS if (catch_lzma_error(lzret)) goto error; @@ -896,6 +898,9 @@ PyObject *result; lzma_stream *lzs = &d->lzs; + if (lzs->avail_in == 0) + return PyBytes_FromStringAndSize(NULL, 0); + if (max_length < 0 || max_length >= INITIAL_BUFFER_SIZE) result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); else -- Repository URL: https://hg.python.org/cpython

1 0

cpython (merge 3.6 -> default): Update the f-string test broken in issue #28385.
by serhiy.storchaka Oct. 31, 2016

Oct. 31, 2016

https://hg.python.org/cpython/rev/931410a04240 changeset: 104841:931410a04240 parent: 104839:e9169a8c0692 parent: 104840:f43078ec598c user: Serhiy Storchaka <storchaka(a)gmail.com> date: Mon Oct 31 08:13:30 2016 +0200 summary: Update the f-string test broken in issue #28385. files: Lib/test/test_fstring.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_fstring.py b/Lib/test/test_fstring.py --- a/Lib/test/test_fstring.py +++ b/Lib/test/test_fstring.py @@ -720,7 +720,7 @@ def test_errors(self): # see issue 26287 - self.assertAllRaise(TypeError, 'non-empty', + self.assertAllRaise(TypeError, 'unsupported', [r"f'{(lambda: 0):x}'", r"f'{(0,):x}'", ]) -- Repository URL: https://hg.python.org/cpython

1 0