Mailman 3 December 2011 - Python-checkins

cpython (2.7): Issue #13522: Fix _Py_co_pow() documentation
by antoine.pitrou Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/2863470caebb changeset: 74030:2863470caebb branch: 2.7 parent: 74025:059e4d752fbe user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 18 02:56:18 2011 +0100 summary: Issue #13522: Fix _Py_co_pow() documentation Patch written by Arnaud Calmettes. files: Doc/c-api/complex.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -72,7 +72,7 @@ Return the exponentiation of *num* by *exp*, using the C :ctype:`Py_complex` representation. - If :attr:`exp.imag` is not null, or :attr:`exp.real` is negative, + If *num* is null and *exp* is not a positive real number, this method returns zero and sets :cdata:`errno` to :cdata:`EDOM`. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13522: Fix _Py_co_pow() documentation
by antoine.pitrou Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/edc981ce8748 changeset: 74029:edc981ce8748 branch: 3.2 parent: 74023:68cbf6551710 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 18 02:56:18 2011 +0100 summary: Issue #13522: Fix _Py_co_pow() documentation Patch written by Arnaud Calmettes. files: Doc/c-api/complex.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -72,7 +72,7 @@ Return the exponentiation of *num* by *exp*, using the C :c:type:`Py_complex` representation. - If :attr:`exp.imag` is not null, or :attr:`exp.real` is negative, + If *num* is null and *exp* is not a positive real number, this method returns zero and sets :c:data:`errno` to :c:data:`EDOM`. -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (20f2f5193b04): sum=0
by solipsis＠pitrou.net Dec. 18, 2011

Dec. 18, 2011

results for 20f2f5193b04 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogMs7WDq', '-x']

1 0

cpython: Optimize str * n for len(str)==1 and UCS-2 or UCS-4
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/20f2f5193b04 changeset: 74028:20f2f5193b04 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 18 03:26:31 2011 +0100 summary: Optimize str * n for len(str)==1 and UCS-2 or UCS-4 files: Objects/unicodeobject.c | 15 +++++++++++---- 1 files changed, 11 insertions(+), 4 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -12028,12 +12028,19 @@ if (PyUnicode_GET_LENGTH(str) == 1) { const int kind = PyUnicode_KIND(str); const Py_UCS4 fill_char = PyUnicode_READ(kind, PyUnicode_DATA(str), 0); - void *to = PyUnicode_DATA(u); - if (kind == PyUnicode_1BYTE_KIND) + if (kind == PyUnicode_1BYTE_KIND) { + void *to = PyUnicode_DATA(u); memset(to, (unsigned char)fill_char, len); - else { + } + else if (kind == PyUnicode_2BYTE_KIND) { + Py_UCS2 *ucs2 = PyUnicode_2BYTE_DATA(u); for (n = 0; n < len; ++n) - PyUnicode_WRITE(kind, to, n, fill_char); + ucs2[n] = fill_char; + } else { + Py_UCS4 *ucs4 = PyUnicode_4BYTE_DATA(u); + assert(kind == PyUnicode_4BYTE_KIND); + for (n = 0; n < len; ++n) + ucs4[n] = fill_char; } } else { -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13522: Fix _Py_co_pow() documentation
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/67a4e8fe650e changeset: 74027:67a4e8fe650e user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 18 02:56:18 2011 +0100 summary: Issue #13522: Fix _Py_co_pow() documentation Patch written by Arnaud Calmettes. files: Doc/c-api/complex.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -72,7 +72,7 @@ Return the exponentiation of *num* by *exp*, using the C :c:type:`Py_complex` representation. - If :attr:`exp.imag` is not null, or :attr:`exp.real` is negative, + If *num* is null and *exp* is not a positive real number, this method returns zero and sets :c:data:`errno` to :c:data:`EDOM`. -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13621: Optimize str.replace(char1, char2)
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/c802bfc8acfc changeset: 74026:c802bfc8acfc parent: 74024:1f096611baf4 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 18 02:43:08 2011 +0100 summary: Issue #13621: Optimize str.replace(char1, char2) Use findchar() which is more optimized than a dummy loop using PyUnicode_READ(). PyUnicode_READ() is a complex and slow macro. files: Objects/unicodeobject.c | 30 ++++++++++++++++++++-------- 1 files changed, 21 insertions(+), 9 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -10285,7 +10285,6 @@ maxchar = Py_MAX(maxchar, maxchar_str2); if (len1 == len2) { - Py_ssize_t i; /* same length */ if (len1 == 0) goto nothing; @@ -10293,9 +10292,12 @@ /* replace characters */ Py_UCS4 u1, u2; int rkind; + Py_ssize_t index, pos; + char *src; + u1 = PyUnicode_READ_CHAR(str1, 0); - if (findchar(sbuf, PyUnicode_KIND(self), - slen, u1, 1) < 0) + pos = findchar(sbuf, PyUnicode_KIND(self), slen, u1, 1); + if (pos < 0) goto nothing; u2 = PyUnicode_READ_CHAR(str2, 0); u = PyUnicode_New(slen, maxchar); @@ -10303,16 +10305,26 @@ goto error; copy_characters(u, 0, self, 0, slen); rkind = PyUnicode_KIND(u); - for (i = 0; i < PyUnicode_GET_LENGTH(u); i++) - if (PyUnicode_READ(rkind, PyUnicode_DATA(u), i) == u1) { - if (--maxcount < 0) - break; - PyUnicode_WRITE(rkind, PyUnicode_DATA(u), i, u2); - } + + PyUnicode_WRITE(rkind, PyUnicode_DATA(u), pos, u2); + index = 0; + src = sbuf; + while (--maxcount) + { + pos++; + src += pos * PyUnicode_KIND(self); + slen -= pos; + index += pos; + pos = findchar(src, PyUnicode_KIND(self), slen, u1, 1); + if (pos < 0) + break; + PyUnicode_WRITE(rkind, PyUnicode_DATA(u), index + pos, u2); + } } else { int rkind = skind; char *res; + Py_ssize_t i; if (kind1 < rkind) { /* widen substring */ -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13522: document error return values of some float and complex C API
by antoine.pitrou Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/059e4d752fbe changeset: 74025:059e4d752fbe branch: 2.7 parent: 74003:766a21ebf82e user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sun Dec 18 01:25:27 2011 +0100 summary: Issue #13522: document error return values of some float and complex C API functions. files: Doc/c-api/complex.rst | 7 +++++++ Doc/c-api/float.rst | 2 ++ Misc/ACKS | 1 + 3 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -63,12 +63,18 @@ Return the quotient of two complex numbers, using the C :ctype:`Py_complex` representation. + If *divisor* is null, this method returns zero and sets + :cdata:`errno` to :cdata:`EDOM`. + .. cfunction:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp) Return the exponentiation of *num* by *exp*, using the C :ctype:`Py_complex` representation. + If :attr:`exp.imag` is not null, or :attr:`exp.real` is negative, + this method returns zero and sets :cdata:`errno` to :cdata:`EDOM`. + Complex Numbers as Python Objects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -125,6 +131,7 @@ .. cfunction:: Py_complex PyComplex_AsCComplex(PyObject *op) Return the :ctype:`Py_complex` value of the complex number *op*. + Upon failure, this method returns ``-1.0`` as a real value. .. versionchanged:: 2.6 If *op* is not a Python complex number object but has a :meth:`__complex__` diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst --- a/Doc/c-api/float.rst +++ b/Doc/c-api/float.rst @@ -55,6 +55,8 @@ Return a C :ctype:`double` representation of the contents of *pyfloat*. If *pyfloat* is not a Python floating point object but has a :meth:`__float__` method, this method will first be called to convert *pyfloat* into a float. + This method returns ``-1.0`` upon failure, so one should call + :cfunc:`PyErr_Occurred` to check for errors. .. cfunction:: double PyFloat_AS_DOUBLE(PyObject *pyfloat) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -120,6 +120,7 @@ Lee Busby Ralph Butler Jp Calderone +Arnaud Calmettes Daniel Calvelo Tony Campbell Brett Cannon -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13522: document error return values of some float and complex C API
by antoine.pitrou Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/1f096611baf4 changeset: 74024:1f096611baf4 parent: 74022:75648db1b3f3 parent: 74023:68cbf6551710 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sun Dec 18 01:26:05 2011 +0100 summary: Issue #13522: document error return values of some float and complex C API functions. files: Doc/c-api/complex.rst | 8 +++++++- Doc/c-api/float.rst | 2 ++ Misc/ACKS | 1 + 3 files changed, 10 insertions(+), 1 deletions(-) diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -63,12 +63,18 @@ Return the quotient of two complex numbers, using the C :c:type:`Py_complex` representation. + If *divisor* is null, this method returns zero and sets + :c:data:`errno` to :c:data:`EDOM`. + .. c:function:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp) Return the exponentiation of *num* by *exp*, using the C :c:type:`Py_complex` representation. + If :attr:`exp.imag` is not null, or :attr:`exp.real` is negative, + this method returns zero and sets :c:data:`errno` to :c:data:`EDOM`. + Complex Numbers as Python Objects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -123,4 +129,4 @@ If *op* is not a Python complex number object but has a :meth:`__complex__` method, this method will first be called to convert *op* to a Python complex - number object. + number object. Upon failure, this method returns ``-1.0`` as a real value. diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst --- a/Doc/c-api/float.rst +++ b/Doc/c-api/float.rst @@ -47,6 +47,8 @@ Return a C :c:type:`double` representation of the contents of *pyfloat*. If *pyfloat* is not a Python floating point object but has a :meth:`__float__` method, this method will first be called to convert *pyfloat* into a float. + This method returns ``-1.0`` upon failure, so one should call + :c:func:`PyErr_Occurred` to check for errors. .. c:function:: double PyFloat_AS_DOUBLE(PyObject *pyfloat) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -149,6 +149,7 @@ Ralph Butler Nicolas Cadou Jp Calderone +Arnaud Calmettes Daniel Calvelo Tony Campbell Brett Cannon -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13522: document error return values of some float and complex C API
by antoine.pitrou Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/68cbf6551710 changeset: 74023:68cbf6551710 branch: 3.2 parent: 74020:beac7d6c7be4 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sun Dec 18 01:25:27 2011 +0100 summary: Issue #13522: document error return values of some float and complex C API functions. files: Doc/c-api/complex.rst | 8 +++++++- Doc/c-api/float.rst | 2 ++ Misc/ACKS | 1 + 3 files changed, 10 insertions(+), 1 deletions(-) diff --git a/Doc/c-api/complex.rst b/Doc/c-api/complex.rst --- a/Doc/c-api/complex.rst +++ b/Doc/c-api/complex.rst @@ -63,12 +63,18 @@ Return the quotient of two complex numbers, using the C :c:type:`Py_complex` representation. + If *divisor* is null, this method returns zero and sets + :c:data:`errno` to :c:data:`EDOM`. + .. c:function:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp) Return the exponentiation of *num* by *exp*, using the C :c:type:`Py_complex` representation. + If :attr:`exp.imag` is not null, or :attr:`exp.real` is negative, + this method returns zero and sets :c:data:`errno` to :c:data:`EDOM`. + Complex Numbers as Python Objects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -123,4 +129,4 @@ If *op* is not a Python complex number object but has a :meth:`__complex__` method, this method will first be called to convert *op* to a Python complex - number object. + number object. Upon failure, this method returns ``-1.0`` as a real value. diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst --- a/Doc/c-api/float.rst +++ b/Doc/c-api/float.rst @@ -47,6 +47,8 @@ Return a C :c:type:`double` representation of the contents of *pyfloat*. If *pyfloat* is not a Python floating point object but has a :meth:`__float__` method, this method will first be called to convert *pyfloat* into a float. + This method returns ``-1.0`` upon failure, so one should call + :c:func:`PyErr_Occurred` to check for errors. .. c:function:: double PyFloat_AS_DOUBLE(PyObject *pyfloat) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -139,6 +139,7 @@ Lee Busby Ralph Butler Jp Calderone +Arnaud Calmettes Daniel Calvelo Tony Campbell Brett Cannon -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13623: Fix a performance regression introduced by issue #12170 in
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/75648db1b3f3 changeset: 74022:75648db1b3f3 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 18 01:17:41 2011 +0100 summary: Issue #13623: Fix a performance regression introduced by issue #12170 in bytes.find() and handle correctly OverflowError (raise the same ValueError than the error for -1). files: Lib/test/test_bytes.py | 5 ++++ Objects/stringlib/find.h | 33 ++++++++++++++++----------- 2 files changed, 25 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -362,6 +362,11 @@ self.assertEqual(b.find(i, 1, 3), 1) self.assertEqual(b.find(w, 1, 3), -1) + for index in (-1, 256, sys.maxsize + 1): + self.assertRaisesRegex( + ValueError, r'byte must be in range$0, 256$', + b.find, index) + def test_rfind(self): b = self.type2test(b'mississippi') i = 105 diff --git a/Objects/stringlib/find.h b/Objects/stringlib/find.h --- a/Objects/stringlib/find.h +++ b/Objects/stringlib/find.h @@ -186,27 +186,34 @@ { PyObject *tmp_subobj; Py_ssize_t ival; + PyObject *err; if(!STRINGLIB(parse_args_finds)(function_name, args, &tmp_subobj, start, end)) return 0; - ival = PyNumber_AsSsize_t(tmp_subobj, PyExc_ValueError); - if (ival == -1 && PyErr_Occurred()) { - PyErr_Clear(); + if (!PyNumber_Check(tmp_subobj)) { *subobj = tmp_subobj; - } - else { - /* The first argument was an integer */ - if(ival < 0 || ival > 255) { - PyErr_SetString(PyExc_ValueError, "byte must be in range(0, 256)"); - return 0; - } - - *subobj = NULL; - *byte = (char)ival; + return 1; } + ival = PyNumber_AsSsize_t(tmp_subobj, PyExc_OverflowError); + if (ival == -1) { + err = PyErr_Occurred(); + if (err && !PyErr_GivenExceptionMatches(err, PyExc_OverflowError)) { + PyErr_Clear(); + *subobj = tmp_subobj; + return 1; + } + } + + if (ival < 0 || ival > 255) { + PyErr_SetString(PyExc_ValueError, "byte must be in range(0, 256)"); + return 0; + } + + *subobj = NULL; + *byte = (char)ival; return 1; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #11231: Fix bytes and bytearray docstrings
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/beac7d6c7be4 changeset: 74020:beac7d6c7be4 branch: 3.2 parent: 74018:46c418d8a480 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 23:18:07 2011 +0100 summary: Issue #11231: Fix bytes and bytearray docstrings Patch written by Brice Berna. files: Misc/ACKS | 1 + Objects/bytearrayobject.c | 12 +++++------- Objects/bytesobject.c | 7 ++++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -80,6 +80,7 @@ Andy Bensky Michel Van den Bergh Julian Berman +Brice Berna Eric Beser Steven Bethard Stephen Bevan diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -2797,18 +2797,16 @@ PyDoc_STRVAR(bytearray_doc, "bytearray(iterable_of_ints) -> bytearray\n\ bytearray(string, encoding[, errors]) -> bytearray\n\ -bytearray(bytes_or_bytearray) -> mutable copy of bytes_or_bytearray\n\ -bytearray(memory_view) -> bytearray\n\ +bytearray(bytes_or_buffer) -> mutable copy of bytes_or_buffer\n\ +bytearray(int) -> bytes array of size given by the parameter initialized with null bytes\n\ +bytearray() -> empty bytes array\n\ \n\ Construct an mutable bytearray object from:\n\ - an iterable yielding integers in range(256)\n\ - a text string encoded using the specified encoding\n\ - - a bytes or a bytearray object\n\ + - a bytes or a buffer object\n\ - any object implementing the buffer API.\n\ -\n\ -bytearray(int) -> bytearray\n\ -\n\ -Construct a zero-initialized bytearray of the given length."); + - an integer"); static PyObject *bytearray_iter(PyObject *seq); diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2721,13 +2721,14 @@ "bytes(iterable_of_ints) -> bytes\n\ bytes(string, encoding[, errors]) -> bytes\n\ bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer\n\ -bytes(memory_view) -> bytes\n\ +bytes(int) -> bytes object of size given by the parameter initialized with null bytes\n\ +bytes() -> empty bytes object\n\ \n\ Construct an immutable array of bytes from:\n\ - an iterable yielding integers in range(256)\n\ - a text string encoded using the specified encoding\n\ - - a bytes or a buffer object\n\ - - any object implementing the buffer API."); + - any object implementing the buffer API.\n\ + - an integer"); static PyObject *bytes_iter(PyObject *seq); -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #11231: Fix bytes and bytearray docstrings
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/dc670add1e28 changeset: 74021:dc670add1e28 parent: 74019:b05caa600c40 parent: 74020:beac7d6c7be4 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 23:18:43 2011 +0100 summary: Issue #11231: Fix bytes and bytearray docstrings Patch written by Brice Berna. files: Misc/ACKS | 1 + Objects/bytearrayobject.c | 12 +++++------- Objects/bytesobject.c | 7 ++++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -84,6 +84,7 @@ Andy Bensky Michel Van den Bergh Julian Berman +Brice Berna Eric Beser Steven Bethard Stephen Bevan diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -2890,18 +2890,16 @@ PyDoc_STRVAR(bytearray_doc, "bytearray(iterable_of_ints) -> bytearray\n\ bytearray(string, encoding[, errors]) -> bytearray\n\ -bytearray(bytes_or_bytearray) -> mutable copy of bytes_or_bytearray\n\ -bytearray(memory_view) -> bytearray\n\ +bytearray(bytes_or_buffer) -> mutable copy of bytes_or_buffer\n\ +bytearray(int) -> bytes array of size given by the parameter initialized with null bytes\n\ +bytearray() -> empty bytes array\n\ \n\ Construct an mutable bytearray object from:\n\ - an iterable yielding integers in range(256)\n\ - a text string encoded using the specified encoding\n\ - - a bytes or a bytearray object\n\ + - a bytes or a buffer object\n\ - any object implementing the buffer API.\n\ -\n\ -bytearray(int) -> bytearray\n\ -\n\ -Construct a zero-initialized bytearray of the given length."); + - an integer"); static PyObject *bytearray_iter(PyObject *seq); diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2732,13 +2732,14 @@ "bytes(iterable_of_ints) -> bytes\n\ bytes(string, encoding[, errors]) -> bytes\n\ bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer\n\ -bytes(memory_view) -> bytes\n\ +bytes(int) -> bytes object of size given by the parameter initialized with null bytes\n\ +bytes() -> empty bytes object\n\ \n\ Construct an immutable array of bytes from:\n\ - an iterable yielding integers in range(256)\n\ - a text string encoded using the specified encoding\n\ - - a bytes or a buffer object\n\ - - any object implementing the buffer API."); + - any object implementing the buffer API.\n\ + - an integer"); static PyObject *bytes_iter(PyObject *seq); -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13530: Document os.lseek() result
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/b05caa600c40 changeset: 74019:b05caa600c40 parent: 74017:cc2af102c73b parent: 74018:46c418d8a480 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 23:15:22 2011 +0100 summary: Issue #13530: Document os.lseek() result Patch written by Jérémy Anger. files: Doc/library/os.rst | 2 +- Misc/ACKS | 1 + Modules/posixmodule.c | 3 ++- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -977,7 +977,7 @@ by *how*: :const:`SEEK_SET` or ``0`` to set the position relative to the beginning of the file; :const:`SEEK_CUR` or ``1`` to set it relative to the current position; :const:`os.SEEK_END` or ``2`` to set it relative to the end of - the file. + the file. Return the new cursor position in bytes, starting from the beginning. Availability: Unix, Windows. diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -32,6 +32,7 @@ Erik Andersén Oliver Andrich Ross Andrus +Jérémy Anger Jon Anglin Éric Araujo Alicia Arlen diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -6946,7 +6946,8 @@ PyDoc_STRVAR(posix_lseek__doc__, "lseek(fd, pos, how) -> newpos\n\n\ -Set the current position of a file descriptor."); +Set the current position of a file descriptor.\n\ +Return the new cursor position in bytes, starting from the beginning."); static PyObject * posix_lseek(PyObject *self, PyObject *args) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13530: Document os.lseek() result
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/46c418d8a480 changeset: 74018:46c418d8a480 branch: 3.2 parent: 74015:d7862e855274 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 23:15:09 2011 +0100 summary: Issue #13530: Document os.lseek() result Patch written by Jérémy Anger. files: Doc/library/os.rst | 2 +- Misc/ACKS | 1 + Modules/posixmodule.c | 3 ++- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -706,7 +706,7 @@ by *how*: :const:`SEEK_SET` or ``0`` to set the position relative to the beginning of the file; :const:`SEEK_CUR` or ``1`` to set it relative to the current position; :const:`os.SEEK_END` or ``2`` to set it relative to the end of - the file. + the file. Return the new cursor position in bytes, starting from the beginning. Availability: Unix, Windows. diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -30,6 +30,7 @@ Erik Andersén Oliver Andrich Ross Andrus +Jérémy Anger Jon Anglin Éric Araujo Alicia Arlen diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -5687,7 +5687,8 @@ PyDoc_STRVAR(posix_lseek__doc__, "lseek(fd, pos, how) -> newpos\n\n\ -Set the current position of a file descriptor."); +Set the current position of a file descriptor.\n\ +Return the new cursor position in bytes, starting from the beginning."); static PyObject * posix_lseek(PyObject *self, PyObject *args) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge default -> default): Merge
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/cc2af102c73b changeset: 74017:cc2af102c73b parent: 74016:49b85dba251d parent: 74013:652605cdd3d6 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 22:40:19 2011 +0100 summary: Merge files: Lib/logging/__init__.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -789,7 +789,7 @@ def setLevel(self, level): """ - Set the logging level of this handler. + Set the logging level of this handler. level must be an int or a str. """ self.level = _checkLevel(level) @@ -1194,7 +1194,7 @@ def setLevel(self, level): """ - Set the logging level of this logger. + Set the logging level of this logger. level must be an int or a str. """ self.level = _checkLevel(level) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #10951: Fix compiler warnings in timemodule.c and unicodeobject.c
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/49b85dba251d changeset: 74016:49b85dba251d parent: 74014:7876cd49300d parent: 74015:d7862e855274 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 22:39:43 2011 +0100 summary: Issue #10951: Fix compiler warnings in timemodule.c and unicodeobject.c Thanks Jérémy Anger for the fix. files: Modules/timemodule.c | 4 ++++ Objects/unicodeobject.c | 2 +- 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -515,14 +515,18 @@ * will be ahead of time... */ for (i = 1024; ; i += i) { +#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__) int err; +#endif outbuf = (time_char *)PyMem_Malloc(i*sizeof(time_char)); if (outbuf == NULL) { PyErr_NoMemory(); break; } buflen = format_time(outbuf, i, fmt, &buf); +#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__) err = errno; +#endif if (buflen > 0 || i >= 256 * fmtlen) { /* If the buffer is 256 times as long as the format, it's probably not failing for lack of room! diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4832,7 +4832,7 @@ if (maxchar < 128 && size == unicode_size) { if (consumed) *consumed = size; - return unicode_fromascii(s, size); + return unicode_fromascii((const unsigned char *)s, size); } unicode = PyUnicode_New(unicode_size, maxchar); -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #10951: Fix a compiler warning in timemodule.c
by victor.stinner Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/d7862e855274 changeset: 74015:d7862e855274 branch: 3.2 parent: 74004:0b86da9d6964 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 22:37:18 2011 +0100 summary: Issue #10951: Fix a compiler warning in timemodule.c files: Modules/timemodule.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -527,14 +527,18 @@ * will be ahead of time... */ for (i = 1024; ; i += i) { +#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__) int err; +#endif outbuf = (time_char *)PyMem_Malloc(i*sizeof(time_char)); if (outbuf == NULL) { PyErr_NoMemory(); break; } buflen = format_time(outbuf, i, fmt, &buf); +#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__) err = errno; +#endif if (buflen > 0 || i >= 256 * fmtlen) { /* If the buffer is 256 times as long as the format, it's probably not failing for lack of room! -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Mention that level can be an int or str in the setLevel docstring.
by gregory.p.smith Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/652605cdd3d6 changeset: 74013:652605cdd3d6 user: Gregory P. Smith <greg(a)krypto.org> date: Sat Dec 17 12:36:34 2011 -0800 summary: Mention that level can be an int or str in the setLevel docstring. files: Lib/logging/__init__.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -789,7 +789,7 @@ def setLevel(self, level): """ - Set the logging level of this handler. + Set the logging level of this handler. level must be an int or a str. """ self.level = _checkLevel(level) @@ -1194,7 +1194,7 @@ def setLevel(self, level): """ - Set the logging level of this logger. + Set the logging level of this logger. level must be an int or a str. """ self.level = _checkLevel(level) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #12809: Expose IP_TRANSPARENT in the socket module. Patch by Michael
by charles-francois.natali Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/59ea1d1a4137 changeset: 74012:59ea1d1a4137 user: Charles-François Natali <neologix(a)free.fr> date: Sat Dec 17 14:59:56 2011 +0100 summary: Issue #12809: Expose IP_TRANSPARENT in the socket module. Patch by Michael Farrell. files: Misc/ACKS | 1 + Modules/socketmodule.c | 3 +++ 2 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -304,6 +304,7 @@ Clovis Fabricio Andreas Faerber Bill Fancher +Michael Farrell Troy J. Farrell Mark Favas Boris Feld diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -6192,6 +6192,9 @@ #ifdef IP_MAX_MEMBERSHIPS PyModule_AddIntConstant(m, "IP_MAX_MEMBERSHIPS", IP_MAX_MEMBERSHIPS); #endif +#ifdef IP_TRANSPARENT + PyModule_AddIntConstant(m, "IP_TRANSPARENT", IP_TRANSPARENT); +#endif /* IPv6 [gs]etsockopt options, defined in RFC2553 */ #ifdef IPV6_JOIN_GROUP -- Repository URL: http://hg.python.org/cpython

1 0

cpython: fix possible NULL dereference
by benjamin.peterson Dec. 18, 2011

Dec. 18, 2011

http://hg.python.org/cpython/rev/8c355edc5b1d changeset: 74011:8c355edc5b1d user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 17 08:02:20 2011 -0500 summary: fix possible NULL dereference files: Objects/descrobject.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -628,7 +628,9 @@ Py_DECREF(descr); descr = NULL; } - descr->d_qualname = NULL; + else { + descr->d_qualname = NULL; + } } return descr; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython: The locale decoder raises a UnicodeDecodeError instead of an OSError
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/ea421c534305 changeset: 74010:ea421c534305 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 07:08:30 2011 +0100 summary: The locale decoder raises a UnicodeDecodeError instead of an OSError Search the invalid character using mbrtowc(). files: Objects/unicodeobject.c | 105 +++++++++++++++++++++++---- 1 files changed, 87 insertions(+), 18 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3084,9 +3084,7 @@ #endif char outbuf[MB_LEN_MAX]; const wchar_t *start, *previous; - int save_errno; - - save_errno = errno; + #if SIZEOF_WCHAR_T == 2 buf[2] = 0; #else @@ -3114,14 +3112,11 @@ wstr++; #endif len = wcstombs(outbuf, buf, sizeof(outbuf)); - if (len == (size_t)-1) { - errno = save_errno; + if (len == (size_t)-1) return previous - start; - } } /* failed to find the unencodable character */ - errno = save_errno; return 0; } @@ -3199,7 +3194,7 @@ len = wcstombs(NULL, wstr, 0); if (len == (size_t)-1) { - error_pos = wcstombs_errorpos(wstr); + error_pos = (size_t)-1; goto encode_error; } @@ -3211,7 +3206,7 @@ len2 = wcstombs(PyBytes_AS_STRING(bytes), wstr, len+1); if (len2 == (size_t)-1 || len2 > len) { - error_pos = wcstombs_errorpos(wstr); + error_pos = (size_t)-1; goto encode_error; } PyMem_Free(wstr); @@ -3221,12 +3216,23 @@ encode_error: errmsg = strerror(errno); assert(errmsg != NULL); + + if (error_pos == (size_t)-1) + error_pos = wcstombs_errorpos(wstr); + PyMem_Free(wstr); Py_XDECREF(bytes); - if (errmsg != NULL) - reason = PyUnicode_DecodeLocale(errmsg, "surrogateescape"); - else + if (errmsg != NULL) { + size_t errlen; + wstr = _Py_char2wchar(errmsg, &errlen); + if (wstr != NULL) { + reason = PyUnicode_FromWideChar(wstr, errlen); + PyMem_Free(wstr); + } else + errmsg = NULL; + } + if (errmsg == NULL) reason = PyUnicode_FromString( "wcstombs() encountered an unencodable " "wide character"); @@ -3376,6 +3382,37 @@ return NULL; } +static size_t +mbstowcs_errorpos(const char *str, size_t len) +{ +#ifdef HAVE_MBRTOWC + const char *start = str; + mbstate_t mbs; + size_t converted; + wchar_t ch; + + memset(&mbs, 0, sizeof mbs); + while (len) + { + converted = mbrtowc(&ch, (char*)str, len, &mbs); + if (converted == 0) + /* Reached end of string */ + break; + if (converted == (size_t)-1 || converted == (size_t)-2) { + /* Conversion error or incomplete character */ + return str - start; + } + else { + str += converted; + len -= converted; + } + } + /* failed to find the undecodable byte sequence */ + return 0; +#endif + return 0; +} + PyObject* PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len, const char *errors) @@ -3386,6 +3423,9 @@ size_t wlen, wlen2; PyObject *unicode; int surrogateescape; + size_t error_pos; + char *errmsg; + PyObject *reason, *exc; if (locale_error_handler(errors, &surrogateescape) < 0) return NULL; @@ -3415,10 +3455,8 @@ #else wlen = len; #endif - if (wlen == (size_t)-1) { - PyErr_SetFromErrno(PyExc_OSError); - return NULL; - } + if (wlen == (size_t)-1) + goto decode_error; if (wlen+1 <= smallbuf_len) { wstr = smallbuf; } @@ -3436,8 +3474,7 @@ if (wlen2 == (size_t)-1) { if (wstr != smallbuf) PyMem_Free(wstr); - PyErr_SetFromErrno(PyExc_OSError); - return NULL; + goto decode_error; } #ifdef HAVE_BROKEN_MBSTOWCS assert(wlen2 == wlen); @@ -3447,6 +3484,38 @@ PyMem_Free(wstr); } return unicode; + +decode_error: + errmsg = strerror(errno); + assert(errmsg != NULL); + + error_pos = mbstowcs_errorpos(str, len); + if (errmsg != NULL) { + size_t errlen; + wstr = _Py_char2wchar(errmsg, &errlen); + if (wstr != NULL) { + reason = PyUnicode_FromWideChar(wstr, errlen); + PyMem_Free(wstr); + } else + errmsg = NULL; + } + if (errmsg == NULL) + reason = PyUnicode_FromString( + "mbstowcs() encountered an invalid multibyte sequence"); + if (reason == NULL) + return NULL; + + exc = PyObject_CallFunction(PyExc_UnicodeDecodeError, "sy#nnO", + "locale", str, len, + (Py_ssize_t)error_pos, + (Py_ssize_t)(error_pos+1), + reason); + Py_DECREF(reason); + if (exc != NULL) { + PyCodec_StrictErrors(exc); + Py_XDECREF(exc); + } + return NULL; } PyObject* -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13560: Locale codec functions use the classic "errors" parameter,
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/07802351ccad changeset: 74009:07802351ccad user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 05:47:23 2011 +0100 summary: Issue #13560: Locale codec functions use the classic "errors" parameter, instead of surrogateescape So it would be possible to support more error handlers later. files: Include/unicodeobject.h | 6 +- Modules/main.c | 2 +- Modules/posixmodule.c | 2 +- Modules/timemodule.c | 9 +++-- Objects/unicodeobject.c | 47 ++++++++++++++++++++++++---- Python/errors.c | 2 +- 6 files changed, 50 insertions(+), 18 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -1608,14 +1608,14 @@ PyAPI_FUNC(PyObject*) PyUnicode_DecodeLocaleAndSize( const char *str, Py_ssize_t len, - int surrogateescape); + const char *errors); /* Similar to PyUnicode_DecodeLocaleAndSize(), but compute the string length using strlen(). */ PyAPI_FUNC(PyObject*) PyUnicode_DecodeLocale( const char *str, - int surrogateescape); + const char *errors); /* Encode a Unicode object to the current locale encoding. The encoder is strict is *surrogateescape* is equal to zero, otherwise the @@ -1624,7 +1624,7 @@ PyAPI_FUNC(PyObject*) PyUnicode_EncodeLocale( PyObject *unicode, - int surrogateescape + const char *errors ); /* --- File system encoding ---------------------------------------------- */ diff --git a/Modules/main.c b/Modules/main.c --- a/Modules/main.c +++ b/Modules/main.c @@ -495,7 +495,7 @@ /* Use utf-8 on Mac OS X */ unicode = PyUnicode_FromString(p); #else - unicode = PyUnicode_DecodeLocale(p, 1); + unicode = PyUnicode_DecodeLocale(p, "surrogateescape"); #endif if (unicode == NULL) { /* ignore errors */ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -7891,7 +7891,7 @@ "strerror() argument out of range"); return NULL; } - return PyUnicode_DecodeLocale(message, 1); + return PyUnicode_DecodeLocale(message, "surrogateescape"); } diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -486,7 +486,7 @@ fmt = format; #else /* Convert the unicode string to an ascii one */ - format = PyUnicode_EncodeLocale(format_arg, 1); + format = PyUnicode_EncodeLocale(format_arg, "surrogateescape"); if (format == NULL) return NULL; fmt = PyBytes_AS_STRING(format); @@ -532,7 +532,8 @@ #ifdef HAVE_WCSFTIME ret = PyUnicode_FromWideChar(outbuf, buflen); #else - ret = PyUnicode_DecodeLocaleAndSize(outbuf, buflen, 1); + ret = PyUnicode_DecodeLocaleAndSize(outbuf, buflen, + "surrogateescape"); #endif PyMem_Free(outbuf); break; @@ -764,8 +765,8 @@ #endif /* PYOS_OS2 */ #endif PyModule_AddIntConstant(m, "daylight", daylight); - otz0 = PyUnicode_DecodeLocale(tzname[0], 1); - otz1 = PyUnicode_DecodeLocale(tzname[1], 1); + otz0 = PyUnicode_DecodeLocale(tzname[0], "surrogateescape"); + otz1 = PyUnicode_DecodeLocale(tzname[1], "surrogateescape"); PyModule_AddObject(m, "tzname", Py_BuildValue("(NN)", otz0, otz1)); #else /* !HAVE_TZNAME || __GLIBC__ || __CYGWIN__*/ #ifdef HAVE_STRUCT_TM_TM_ZONE diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3125,8 +3125,31 @@ return 0; } -PyObject * -PyUnicode_EncodeLocale(PyObject *unicode, int surrogateescape) +static int +locale_error_handler(const char *errors, int *surrogateescape) +{ + if (errors == NULL) { + *surrogateescape = 0; + return 0; + } + + if (strcmp(errors, "strict") == 0) { + *surrogateescape = 0; + return 0; + } + if (strcmp(errors, "surrogateescape") == 0) { + *surrogateescape = 1; + return 0; + } + PyErr_Format(PyExc_ValueError, + "only 'strict' and 'surrogateescape' error handlers " + "are supported, not '%s'", + errors); + return -1; +} + +PyObject * +PyUnicode_EncodeLocale(PyObject *unicode, const char *errors) { Py_ssize_t wlen, wlen2; wchar_t *wstr; @@ -3135,6 +3158,10 @@ PyObject *reason; PyObject *exc; size_t error_pos; + int surrogateescape; + + if (locale_error_handler(errors, &surrogateescape) < 0) + return NULL; wstr = PyUnicode_AsWideCharString(unicode, &wlen); if (wstr == NULL) @@ -3198,7 +3225,7 @@ Py_XDECREF(bytes); if (errmsg != NULL) - reason = PyUnicode_DecodeLocale(errmsg, 1); + reason = PyUnicode_DecodeLocale(errmsg, "surrogateescape"); else reason = PyUnicode_FromString( "wcstombs() encountered an unencodable " @@ -3243,7 +3270,7 @@ "surrogateescape"); } else { - return PyUnicode_EncodeLocale(unicode, 1); + return PyUnicode_EncodeLocale(unicode, "surrogateescape"); } #endif } @@ -3351,13 +3378,17 @@ PyObject* PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len, - int surrogateescape) + const char *errors) { wchar_t smallbuf[256]; size_t smallbuf_len = Py_ARRAY_LENGTH(smallbuf); wchar_t *wstr; size_t wlen, wlen2; PyObject *unicode; + int surrogateescape; + + if (locale_error_handler(errors, &surrogateescape) < 0) + return NULL; if (str[len] != '\0' || len != strlen(str)) { PyErr_SetString(PyExc_TypeError, "embedded null character"); @@ -3419,10 +3450,10 @@ } PyObject* -PyUnicode_DecodeLocale(const char *str, int surrogateescape) +PyUnicode_DecodeLocale(const char *str, const char *errors) { Py_ssize_t size = (Py_ssize_t)strlen(str); - return PyUnicode_DecodeLocaleAndSize(str, size, surrogateescape); + return PyUnicode_DecodeLocaleAndSize(str, size, errors); } @@ -3456,7 +3487,7 @@ "surrogateescape"); } else { - return PyUnicode_DecodeLocaleAndSize(s, size, 1); + return PyUnicode_DecodeLocaleAndSize(s, size, "surrogateescape"); } #endif } diff --git a/Python/errors.c b/Python/errors.c --- a/Python/errors.c +++ b/Python/errors.c @@ -355,7 +355,7 @@ #ifndef MS_WINDOWS if (i != 0) { char *s = strerror(i); - message = PyUnicode_DecodeLocale(s, 1); + message = PyUnicode_DecodeLocale(s, "surrogateescape"); } else { /* Sometimes errno didn't get set */ -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (f2e1867f33b8): sum=0
by solipsis＠pitrou.net Dec. 17, 2011

Dec. 17, 2011

results for f2e1867f33b8 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogB9PJEQ', '-x']

1 0

cpython: What's New in Python 3.3: complete the deprecation list
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/880c7cffd7eb changeset: 74008:880c7cffd7eb user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 04:59:06 2011 +0100 summary: What's New in Python 3.3: complete the deprecation list Add also FIXMEs in unicodeobject.c files: Doc/whatsnew/3.3.rst | 3 +++ Objects/unicodeobject.c | 2 ++ 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -798,6 +798,8 @@ strings) * :c:func:`PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy` or :c:func:`PyUnicode_AsWideCharString` + * :c:func:`PyUnicode_GetMax` + Functions and macros manipulating Py_UNICODE* strings: @@ -813,6 +815,7 @@ * :c:macro:`Py_UNICODE_strchr`, :c:macro:`Py_UNICODE_strrchr`: use :c:func:`PyUnicode_FindChar` * :c:macro:`Py_UNICODE_FILL` + * :c:macro:`Py_UNICODE_MATCH` Encoders: diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7042,6 +7042,7 @@ if (*v == NULL) { /* Create unicode object */ + /* FIXME: don't use _PyUnicode_New(), but allocate a wchar_t* buffer */ *v = (PyObject*)_PyUnicode_New(outsize); if (*v == NULL) return -1; @@ -7122,6 +7123,7 @@ PyErr_NoMemory(); goto error; } + /* FIXME: don't use _PyUnicode_New(), but allocate a wchar_t* buffer */ *v = (PyObject*)_PyUnicode_New(size * Py_ARRAY_LENGTH(buffer)); if (*v == NULL) goto error; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13560: os.strerror() now uses the current locale encoding instead of
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/51412b4b81ae changeset: 74007:51412b4b81ae user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 04:45:09 2011 +0100 summary: Issue #13560: os.strerror() now uses the current locale encoding instead of UTF-8 files: Misc/NEWS | 3 +++ Modules/posixmodule.c | 2 +- Modules/socketmodule.c | 5 ++--- Objects/unicodeobject.c | 28 ++++++++++++++++++++-------- Python/errors.c | 17 +++++++++-------- 5 files changed, 35 insertions(+), 20 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -419,6 +419,9 @@ Library ------- +- Issue #13560: os.strerror() now uses the current locale encoding instead of + UTF-8. + - Issue #13560: Add PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() and PyUnicode_EncodeLocale() functions to the C API to decode/encode from/to the current locale encoding. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -7891,7 +7891,7 @@ "strerror() argument out of range"); return NULL; } - return PyUnicode_FromString(message); + return PyUnicode_DecodeLocale(message, 1); } diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -4032,9 +4032,8 @@ if (h->h_addrtype != af) { /* Let's get real error message to return */ - PyErr_SetString(PyExc_OSError, - (char *)strerror(EAFNOSUPPORT)); - + errno = EAFNOSUPPORT; + PyErr_SetFromErrno(PyExc_OSError); return NULL; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3132,6 +3132,7 @@ wchar_t *wstr; PyObject *bytes = NULL; char *errmsg; + PyObject *reason; PyObject *exc; size_t error_pos; @@ -3193,17 +3194,28 @@ encode_error: errmsg = strerror(errno); assert(errmsg != NULL); - if (errmsg == NULL) - errmsg = "wcstombs() encountered an unencodable wide character"; PyMem_Free(wstr); Py_XDECREF(bytes); - exc = NULL; - raise_encode_exception(&exc, - "locale", unicode, - error_pos, error_pos+1, - errmsg); - Py_XDECREF(exc); + if (errmsg != NULL) + reason = PyUnicode_DecodeLocale(errmsg, 1); + else + reason = PyUnicode_FromString( + "wcstombs() encountered an unencodable " + "wide character"); + if (reason == NULL) + return NULL; + + exc = PyObject_CallFunction(PyExc_UnicodeEncodeError, "sOnnO", + "locale", unicode, + (Py_ssize_t)error_pos, + (Py_ssize_t)(error_pos+1), + reason); + Py_DECREF(reason); + if (exc != NULL) { + PyCodec_StrictErrors(exc); + Py_XDECREF(exc); + } return NULL; } diff --git a/Python/errors.c b/Python/errors.c --- a/Python/errors.c +++ b/Python/errors.c @@ -343,9 +343,7 @@ PyObject *message; PyObject *v, *args; int i = errno; -#ifndef MS_WINDOWS - char *s; -#else +#ifdef MS_WINDOWS WCHAR *s_buf = NULL; #endif /* Unix/Windows */ @@ -355,11 +353,14 @@ #endif #ifndef MS_WINDOWS - if (i == 0) - s = "Error"; /* Sometimes errno didn't get set */ - else - s = strerror(i); - message = PyUnicode_DecodeUTF8(s, strlen(s), "ignore"); + if (i != 0) { + char *s = strerror(i); + message = PyUnicode_DecodeLocale(s, 1); + } + else { + /* Sometimes errno didn't get set */ + message = PyUnicode_FromString("Error"); + } #else if (i == 0) message = PyUnicode_FromString("Error"); /* Sometimes errno didn't get set */ -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13560: Add PyUnicode_EncodeLocale()
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/88198b93ff2f changeset: 74006:88198b93ff2f user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 17 04:13:41 2011 +0100 summary: Issue #13560: Add PyUnicode_EncodeLocale() * Use PyUnicode_EncodeLocale() in time.strftime() if wcsftime() is not available * Document my last changes in Misc/NEWS files: Doc/c-api/unicode.rst | 25 ++++- Include/unicodeobject.h | 12 +- Misc/NEWS | 8 +- Modules/timemodule.c | 2 +- Objects/unicodeobject.c | 167 ++++++++++++++++++++++----- 5 files changed, 177 insertions(+), 37 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -713,7 +713,7 @@ bytes. If a byte sequence can be decoded as a surrogate character and *surrogateescape* is not equal to zero, the byte sequence is escaped using the ``'surrogateescape'`` error handler instead of being decoded. *str* - must end with a null character but cannot contain embedded null character. + must end with a null character but cannot contain embedded null characters. .. seealso:: @@ -732,6 +732,22 @@ .. versionadded:: 3.3 +.. c:function:: PyObject* PyUnicode_EncodeLocale(PyObject *unicode, int surrogateescape) + + Encode a Unicode object to the current locale encoding. The encoder is + strict if *surrogateescape* is equal to zero, otherwise it uses the + ``'surrogateescape'`` error handler (:pep:`383`). Return a :class:`bytes` + object. *str* cannot contain embedded null characters. + + .. seealso:: + + Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + + .. versionadded:: 3.3 + + File System Encoding """""""""""""""""""" @@ -806,6 +822,13 @@ If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + .. seealso:: + + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to encode a + string to the current locale encoding, use + :c:func:`PyUnicode_EncodeLocale`. + .. versionadded:: 3.2 diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -1603,7 +1603,7 @@ be decoded as a surrogate character and *surrogateescape* is not equal to zero, the byte sequence is escaped using the 'surrogateescape' error handler instead of being decoded. *str* must end with a null character but cannot - contain embedded null character. */ + contain embedded null characters. */ PyAPI_FUNC(PyObject*) PyUnicode_DecodeLocaleAndSize( const char *str, @@ -1617,6 +1617,16 @@ const char *str, int surrogateescape); +/* Encode a Unicode object to the current locale encoding. The encoder is + strict is *surrogateescape* is equal to zero, otherwise the + "surrogateescape" error handler is used. Return a bytes object. The string + cannot contain embedded null characters.. */ + +PyAPI_FUNC(PyObject*) PyUnicode_EncodeLocale( + PyObject *unicode, + int surrogateescape + ); + /* --- File system encoding ---------------------------------------------- */ /* ParseTuple converter: encode str objects to bytes using diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -419,6 +419,10 @@ Library ------- +- Issue #13560: Add PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() + and PyUnicode_EncodeLocale() functions to the C API to decode/encode from/to + the current locale encoding. + - Issue #8373: The filesystem path of AF_UNIX sockets now uses the filesystem encoding and the surrogateescape error handler, rather than UTF-8. Patch by David Watson. @@ -451,8 +455,8 @@ 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', and private attributes of 'smtpd.SMTPChannel'. -- Issue #5905: time.strftime() is now using the locale encoding, instead of - UTF-8, if the wcsftime() function is not available. +- Issue #5905, #13560: time.strftime() is now using the current locale + encoding, instead of UTF-8, if the wcsftime() function is not available. - Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". Patch by Tal Einat. diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -486,7 +486,7 @@ fmt = format; #else /* Convert the unicode string to an ascii one */ - format = PyUnicode_EncodeFSDefault(format_arg); + format = PyUnicode_EncodeLocale(format_arg, 1); if (format == NULL) return NULL; fmt = PyBytes_AS_STRING(format); diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3073,6 +3073,140 @@ return NULL; } +static size_t +wcstombs_errorpos(const wchar_t *wstr) +{ + size_t len; +#if SIZEOF_WCHAR_T == 2 + wchar_t buf[3]; +#else + wchar_t buf[2]; +#endif + char outbuf[MB_LEN_MAX]; + const wchar_t *start, *previous; + int save_errno; + + save_errno = errno; +#if SIZEOF_WCHAR_T == 2 + buf[2] = 0; +#else + buf[1] = 0; +#endif + start = wstr; + while (*wstr != L'\0') + { + previous = wstr; +#if SIZEOF_WCHAR_T == 2 + if (Py_UNICODE_IS_HIGH_SURROGATE(wstr[0]) + && Py_UNICODE_IS_LOW_SURROGATE(wstr[1])) + { + buf[0] = wstr[0]; + buf[1] = wstr[1]; + wstr += 2; + } + else { + buf[0] = *wstr; + buf[1] = 0; + wstr++; + } +#else + buf[0] = *wstr; + wstr++; +#endif + len = wcstombs(outbuf, buf, sizeof(outbuf)); + if (len == (size_t)-1) { + errno = save_errno; + return previous - start; + } + } + + /* failed to find the unencodable character */ + errno = save_errno; + return 0; +} + +PyObject * +PyUnicode_EncodeLocale(PyObject *unicode, int surrogateescape) +{ + Py_ssize_t wlen, wlen2; + wchar_t *wstr; + PyObject *bytes = NULL; + char *errmsg; + PyObject *exc; + size_t error_pos; + + wstr = PyUnicode_AsWideCharString(unicode, &wlen); + if (wstr == NULL) + return NULL; + + wlen2 = wcslen(wstr); + if (wlen2 != wlen) { + PyMem_Free(wstr); + PyErr_SetString(PyExc_TypeError, "embedded null character"); + return NULL; + } + + if (surrogateescape) { + /* locale encoding with surrogateescape */ + char *str; + + str = _Py_wchar2char(wstr, &error_pos); + if (str == NULL) { + if (error_pos == (size_t)-1) { + PyErr_NoMemory(); + PyMem_Free(wstr); + return NULL; + } + else { + goto encode_error; + } + } + PyMem_Free(wstr); + + bytes = PyBytes_FromString(str); + PyMem_Free(str); + } + else { + size_t len, len2; + + len = wcstombs(NULL, wstr, 0); + if (len == (size_t)-1) { + error_pos = wcstombs_errorpos(wstr); + goto encode_error; + } + + bytes = PyBytes_FromStringAndSize(NULL, len); + if (bytes == NULL) { + PyMem_Free(wstr); + return NULL; + } + + len2 = wcstombs(PyBytes_AS_STRING(bytes), wstr, len+1); + if (len2 == (size_t)-1 || len2 > len) { + error_pos = wcstombs_errorpos(wstr); + goto encode_error; + } + PyMem_Free(wstr); + } + return bytes; + +encode_error: + errmsg = strerror(errno); + assert(errmsg != NULL); + if (errmsg == NULL) + errmsg = "wcstombs() encountered an unencodable wide character"; + PyMem_Free(wstr); + Py_XDECREF(bytes); + + exc = NULL; + raise_encode_exception(&exc, + "locale", unicode, + error_pos, error_pos+1, + errmsg); + Py_XDECREF(exc); + return NULL; +} + PyObject * PyUnicode_EncodeFSDefault(PyObject *unicode) { @@ -3097,38 +3231,7 @@ "surrogateescape"); } else { - /* locale encoding with surrogateescape */ - wchar_t *wchar; - char *bytes; - PyObject *bytes_obj; - size_t error_pos; - - wchar = PyUnicode_AsWideCharString(unicode, NULL); - if (wchar == NULL) - return NULL; - bytes = _Py_wchar2char(wchar, &error_pos); - if (bytes == NULL) { - if (error_pos != (size_t)-1) { - char *errmsg = strerror(errno); - PyObject *exc = NULL; - if (errmsg == NULL) - errmsg = "Py_wchar2char() failed"; - raise_encode_exception(&exc, - "filesystemencoding", unicode, - error_pos, error_pos+1, - errmsg); - Py_XDECREF(exc); - } - else - PyErr_NoMemory(); - PyMem_Free(wchar); - return NULL; - } - PyMem_Free(wchar); - - bytes_obj = PyBytes_FromString(bytes); - PyMem_Free(bytes); - return bytes_obj; + return PyUnicode_EncodeLocale(unicode, 1); } #endif } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): #13613: merge with 3.2.
by ezio.melotti Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/f2e1867f33b8 changeset: 74005:f2e1867f33b8 parent: 74002:279b0aee0cfb parent: 74004:0b86da9d6964 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Sat Dec 17 01:20:02 2011 +0200 summary: #13613: merge with 3.2. files: Doc/library/re.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1021,16 +1021,16 @@ Suppose you are writing a poker program where a player's hand is represented as a 5-character string with each character representing a card, "a" for ace, "k" -for king, "q" for queen, j for jack, "0" for 10, and "1" through "9" +for king, "q" for queen, "j" for jack, "t" for 10, and "2" through "9" representing the card with that value. To see if a given string is a valid hand, one could do the following: - >>> valid = re.compile(r"[0-9akqj]{5}$") - >>> displaymatch(valid.match("ak05q")) # Valid. - "<Match: 'ak05q', groups=()>" - >>> displaymatch(valid.match("ak05e")) # Invalid. - >>> displaymatch(valid.match("ak0")) # Invalid. + >>> valid = re.compile(r"^[a2-9tjqk]{5}$") + >>> displaymatch(valid.match("akt5q")) # Valid. + "<Match: 'akt5q', groups=()>" + >>> displaymatch(valid.match("akt5e")) # Invalid. + >>> displaymatch(valid.match("akt")) # Invalid. >>> displaymatch(valid.match("727ak")) # Valid. "<Match: '727ak', groups=()>" -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): #13613: fix example in re doc.
by ezio.melotti Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/0b86da9d6964 changeset: 74004:0b86da9d6964 branch: 3.2 parent: 74000:3b17210f34d9 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Sat Dec 17 01:17:17 2011 +0200 summary: #13613: fix example in re doc. files: Doc/library/re.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1018,16 +1018,16 @@ Suppose you are writing a poker program where a player's hand is represented as a 5-character string with each character representing a card, "a" for ace, "k" -for king, "q" for queen, j for jack, "0" for 10, and "1" through "9" +for king, "q" for queen, "j" for jack, "t" for 10, and "2" through "9" representing the card with that value. To see if a given string is a valid hand, one could do the following: - >>> valid = re.compile(r"[0-9akqj]{5}$") - >>> displaymatch(valid.match("ak05q")) # Valid. - "<Match: 'ak05q', groups=()>" - >>> displaymatch(valid.match("ak05e")) # Invalid. - >>> displaymatch(valid.match("ak0")) # Invalid. + >>> valid = re.compile(r"^[a2-9tjqk]{5}$") + >>> displaymatch(valid.match("akt5q")) # Valid. + "<Match: 'akt5q', groups=()>" + >>> displaymatch(valid.match("akt5e")) # Invalid. + >>> displaymatch(valid.match("akt")) # Invalid. >>> displaymatch(valid.match("727ak")) # Valid. "<Match: '727ak', groups=()>" -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): #13613: fix example in re doc.
by ezio.melotti Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/766a21ebf82e changeset: 74003:766a21ebf82e branch: 2.7 parent: 73987:f2a5dcced66d user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Sat Dec 17 01:17:17 2011 +0200 summary: #13613: fix example in re doc. files: Doc/library/re.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1000,16 +1000,16 @@ Suppose you are writing a poker program where a player's hand is represented as a 5-character string with each character representing a card, "a" for ace, "k" -for king, "q" for queen, j for jack, "0" for 10, and "1" through "9" +for king, "q" for queen, "j" for jack, "t" for 10, and "2" through "9" representing the card with that value. To see if a given string is a valid hand, one could do the following: - >>> valid = re.compile(r"[0-9akqj]{5}$") - >>> displaymatch(valid.match("ak05q")) # Valid. - "<Match: 'ak05q', groups=()>" - >>> displaymatch(valid.match("ak05e")) # Invalid. - >>> displaymatch(valid.match("ak0")) # Invalid. + >>> valid = re.compile(r"^[a2-9tjqk]{5}$") + >>> displaymatch(valid.match("akt5q")) # Valid. + "<Match: 'akt5q', groups=()>" + >>> displaymatch(valid.match("akt5e")) # Invalid. + >>> displaymatch(valid.match("akt")) # Invalid. >>> displaymatch(valid.match("727ak")) # Valid. "<Match: '727ak', groups=()>" -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Add PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale()
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/279b0aee0cfb changeset: 74002:279b0aee0cfb user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 16 23:56:01 2011 +0100 summary: Add PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale() * PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale() decode a string from the current locale encoding * _Py_char2wchar() writes an "error code" in the size argument to indicate if the function failed because of memory allocation failure or because of a decoding error. The function doesn't write the error message directly to stderr. * Fix time.strftime() (if wcsftime() is missing): decode strftime() result from the current locale encoding, not from the filesystem encoding. files: Doc/c-api/unicode.rst | 40 ++++++++++++ Include/unicodeobject.h | 22 ++++++ Modules/_localemodule.c | 57 +++-------------- Modules/main.c | 15 +-- Modules/timemodule.c | 6 +- Objects/unicodeobject.c | 95 +++++++++++++++++++++++----- Python/fileutils.c | 25 ++++-- 7 files changed, 175 insertions(+), 85 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -699,6 +699,39 @@ throughout the interpreter whenever coercion to Unicode is needed. +Locale Encoding +""""""""""""""" + +The current locale encoding can be used to decode text from the operating +system. + +.. c:function:: PyObject* PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len, int surrogateescape) + + Decode a string from the current locale encoding. The decoder is strict if + *surrogateescape* is equal to zero, otherwise it uses the + ``'surrogateescape'`` error handler (:pep:`383`) to escape undecodable + bytes. If a byte sequence can be decoded as a surrogate character and + *surrogateescape* is not equal to zero, the byte sequence is escaped using + the ``'surrogateescape'`` error handler instead of being decoded. *str* + must end with a null character but cannot contain embedded null character. + + .. seealso:: + + Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + + .. versionadded:: 3.3 + + +.. c:function:: PyObject* PyUnicode_DecodeLocale(const char *str, int surrogateescape) + + Similar to :c:func:`PyUnicode_DecodeLocaleAndSize`, but compute the string + length using :c:func:`strlen`. + + .. versionadded:: 3.3 + + File System Encoding """""""""""""""""""" @@ -739,6 +772,13 @@ If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + .. seealso:: + + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to decode a + string from the current locale encoding, use + :c:func:`PyUnicode_DecodeLocaleAndSize`. + .. versionchanged:: 3.2 Use ``'strict'`` error handler on Windows. diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -1595,6 +1595,28 @@ ); #endif +/* --- Locale encoding --------------------------------------------------- */ + +/* Decode a string from the current locale encoding. The decoder is strict if + *surrogateescape* is equal to zero, otherwise it uses the 'surrogateescape' + error handler (PEP 383) to escape undecodable bytes. If a byte sequence can + be decoded as a surrogate character and *surrogateescape* is not equal to + zero, the byte sequence is escaped using the 'surrogateescape' error handler + instead of being decoded. *str* must end with a null character but cannot + contain embedded null character. */ + +PyAPI_FUNC(PyObject*) PyUnicode_DecodeLocaleAndSize( + const char *str, + Py_ssize_t len, + int surrogateescape); + +/* Similar to PyUnicode_DecodeLocaleAndSize(), but compute the string + length using strlen(). */ + +PyAPI_FUNC(PyObject*) PyUnicode_DecodeLocale( + const char *str, + int surrogateescape); + /* --- File system encoding ---------------------------------------------- */ /* ParseTuple converter: encode str objects to bytes using diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c --- a/Modules/_localemodule.c +++ b/Modules/_localemodule.c @@ -42,43 +42,6 @@ static PyObject *Error; -/* Convert a char* to a Unicode object according to the current locale */ -static PyObject* -str2uni(const char* s) -{ -#ifdef HAVE_BROKEN_MBSTOWCS - size_t needed = strlen(s); -#else - size_t needed = mbstowcs(NULL, s, 0); -#endif - size_t res1; - wchar_t smallbuf[30]; - wchar_t *dest; - PyObject *res2; - if (needed == (size_t)-1) { - PyErr_SetString(PyExc_ValueError, "Cannot convert byte to string"); - return NULL; - } - if (needed*sizeof(wchar_t) < sizeof(smallbuf)) - dest = smallbuf; - else { - dest = PyMem_Malloc((needed+1)*sizeof(wchar_t)); - if (!dest) - return PyErr_NoMemory(); - } - /* This shouldn't fail now */ - res1 = mbstowcs(dest, s, needed+1); -#ifdef HAVE_BROKEN_MBSTOWCS - assert(res1 != (size_t)-1); -#else - assert(res1 == needed); -#endif - res2 = PyUnicode_FromWideChar(dest, res1); - if (dest != smallbuf) - PyMem_Free(dest); - return res2; -} - /* support functions for formatting floating point numbers */ PyDoc_STRVAR(setlocale__doc__, @@ -149,7 +112,7 @@ PyErr_SetString(Error, "unsupported locale setting"); return NULL; } - result_object = str2uni(result); + result_object = PyUnicode_DecodeLocale(result, 0); if (!result_object) return NULL; } else { @@ -159,7 +122,7 @@ PyErr_SetString(Error, "locale query failed"); return NULL; } - result_object = str2uni(result); + result_object = PyUnicode_DecodeLocale(result, 0); } return result_object; } @@ -185,7 +148,7 @@ involved herein */ #define RESULT_STRING(s)\ - x = str2uni(l->s); \ + x = PyUnicode_DecodeLocale(l->s, 0); \ if (!x) goto failed;\ PyDict_SetItemString(result, #s, x);\ Py_XDECREF(x) @@ -476,7 +439,7 @@ instead of an empty string for nl_langinfo(ERA). */ const char *result = nl_langinfo(item); result = result != NULL ? result : ""; - return str2uni(result); + return PyUnicode_DecodeLocale(result, 0); } PyErr_SetString(PyExc_ValueError, "unsupported langinfo constant"); return NULL; @@ -495,7 +458,7 @@ char *in; if (!PyArg_ParseTuple(args, "s", &in)) return 0; - return str2uni(gettext(in)); + return PyUnicode_DecodeLocale(gettext(in), 0); } PyDoc_STRVAR(dgettext__doc__, @@ -508,7 +471,7 @@ char *domain, *in; if (!PyArg_ParseTuple(args, "zs", &domain, &in)) return 0; - return str2uni(dgettext(domain, in)); + return PyUnicode_DecodeLocale(dgettext(domain, in), 0); } PyDoc_STRVAR(dcgettext__doc__, @@ -522,7 +485,7 @@ int category; if (!PyArg_ParseTuple(args, "zsi", &domain, &msgid, &category)) return 0; - return str2uni(dcgettext(domain,msgid,category)); + return PyUnicode_DecodeLocale(dcgettext(domain,msgid,category), 0); } PyDoc_STRVAR(textdomain__doc__, @@ -540,7 +503,7 @@ PyErr_SetFromErrno(PyExc_OSError); return NULL; } - return str2uni(domain); + return PyUnicode_DecodeLocale(domain, 0); } PyDoc_STRVAR(bindtextdomain__doc__, @@ -572,7 +535,7 @@ PyErr_SetFromErrno(PyExc_OSError); return NULL; } - result = str2uni(current_dirname); + result = PyUnicode_DecodeLocale(current_dirname, 0); Py_XDECREF(dirname_bytes); return result; } @@ -590,7 +553,7 @@ return NULL; codeset = bind_textdomain_codeset(domain, codeset); if (codeset) - return str2uni(codeset); + return PyUnicode_DecodeLocale(codeset, 0); Py_RETURN_NONE; } #endif diff --git a/Modules/main.c b/Modules/main.c --- a/Modules/main.c +++ b/Modules/main.c @@ -495,16 +495,13 @@ /* Use utf-8 on Mac OS X */ unicode = PyUnicode_FromString(p); #else - wchar_t *wchar; - size_t len; - wchar = _Py_char2wchar(p, &len); - if (wchar == NULL) + unicode = PyUnicode_DecodeLocale(p, 1); +#endif + if (unicode == NULL) { + /* ignore errors */ + PyErr_Clear(); continue; - unicode = PyUnicode_FromWideChar(wchar, len); - PyMem_Free(wchar); -#endif - if (unicode == NULL) - continue; + } PySys_AddWarnOptionUnicode(unicode); Py_DECREF(unicode); } diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -532,7 +532,7 @@ #ifdef HAVE_WCSFTIME ret = PyUnicode_FromWideChar(outbuf, buflen); #else - ret = PyUnicode_DecodeFSDefaultAndSize(outbuf, buflen); + ret = PyUnicode_DecodeLocaleAndSize(outbuf, buflen, 1); #endif PyMem_Free(outbuf); break; @@ -764,8 +764,8 @@ #endif /* PYOS_OS2 */ #endif PyModule_AddIntConstant(m, "daylight", daylight); - otz0 = PyUnicode_DecodeFSDefaultAndSize(tzname[0], strlen(tzname[0])); - otz1 = PyUnicode_DecodeFSDefaultAndSize(tzname[1], strlen(tzname[1])); + otz0 = PyUnicode_DecodeLocale(tzname[0], 1); + otz1 = PyUnicode_DecodeLocale(tzname[1], 1); PyModule_AddObject(m, "tzname", Py_BuildValue("(NN)", otz0, otz1)); #else /* !HAVE_TZNAME || __GLIBC__ || __CYGWIN__*/ #ifdef HAVE_STRUCT_TM_TM_ZONE diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3235,6 +3235,83 @@ } PyObject* +PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t len, + int surrogateescape) +{ + wchar_t smallbuf[256]; + size_t smallbuf_len = Py_ARRAY_LENGTH(smallbuf); + wchar_t *wstr; + size_t wlen, wlen2; + PyObject *unicode; + + if (str[len] != '\0' || len != strlen(str)) { + PyErr_SetString(PyExc_TypeError, "embedded null character"); + return NULL; + } + + if (surrogateescape) + { + wstr = _Py_char2wchar(str, &wlen); + if (wstr == NULL) { + if (wlen == (size_t)-1) + PyErr_NoMemory(); + else + PyErr_SetFromErrno(PyExc_OSError); + return NULL; + } + + unicode = PyUnicode_FromWideChar(wstr, wlen); + PyMem_Free(wstr); + } + else { +#ifndef HAVE_BROKEN_MBSTOWCS + wlen = mbstowcs(NULL, str, 0); +#else + wlen = len; +#endif + if (wlen == (size_t)-1) { + PyErr_SetFromErrno(PyExc_OSError); + return NULL; + } + if (wlen+1 <= smallbuf_len) { + wstr = smallbuf; + } + else { + if (wlen > PY_SSIZE_T_MAX / sizeof(wchar_t) - 1) + return PyErr_NoMemory(); + + wstr = PyMem_Malloc((wlen+1) * sizeof(wchar_t)); + if (!wstr) + return PyErr_NoMemory(); + } + + /* This shouldn't fail now */ + wlen2 = mbstowcs(wstr, str, wlen+1); + if (wlen2 == (size_t)-1) { + if (wstr != smallbuf) + PyMem_Free(wstr); + PyErr_SetFromErrno(PyExc_OSError); + return NULL; + } +#ifdef HAVE_BROKEN_MBSTOWCS + assert(wlen2 == wlen); +#endif + unicode = PyUnicode_FromWideChar(wstr, wlen2); + if (wstr != smallbuf) + PyMem_Free(wstr); + } + return unicode; +} + +PyObject* +PyUnicode_DecodeLocale(const char *str, int surrogateescape) +{ + Py_ssize_t size = (Py_ssize_t)strlen(str); + return PyUnicode_DecodeLocaleAndSize(str, size, surrogateescape); +} + + +PyObject* PyUnicode_DecodeFSDefault(const char *s) { Py_ssize_t size = (Py_ssize_t)strlen(s); return PyUnicode_DecodeFSDefaultAndSize(s, size); @@ -3264,23 +3341,7 @@ "surrogateescape"); } else { - /* locale encoding with surrogateescape */ - wchar_t *wchar; - PyObject *unicode; - size_t len; - - if (s[size] != '\0' || size != strlen(s)) { - PyErr_SetString(PyExc_TypeError, "embedded NUL character"); - return NULL; - } - - wchar = _Py_char2wchar(s, &len); - if (wchar == NULL) - return PyErr_NoMemory(); - - unicode = PyUnicode_FromWideChar(wchar, len); - PyMem_Free(wchar); - return unicode; + return PyUnicode_DecodeLocaleAndSize(s, size, 1); } #endif } diff --git a/Python/fileutils.c b/Python/fileutils.c --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -16,7 +16,9 @@ Return a pointer to a newly allocated wide character string (use PyMem_Free() to free the memory) and write the number of written wide characters excluding the null character into *size if size is not NULL, or - NULL on error (conversion or memory allocation error). + NULL on error (decoding or memory allocation error). If size is not NULL, + *size is set to (size_t)-1 on memory error and (size_t)-2 on decoding + error. Conversion errors should never happen, unless there is a bug in the C library. */ @@ -82,8 +84,9 @@ since we provide everything that we have - unless there is a bug in the C library, or I misunderstood how mbrtowc works. */ - fprintf(stderr, "unexpected mbrtowc result -2\n"); PyMem_Free(res); + if (size != NULL) + *size = (size_t)-2; return NULL; } if (converted == (size_t)-1) { @@ -112,7 +115,8 @@ is ASCII (i.e. escape all bytes > 128. This will still roundtrip correctly in the locale's charset, which must be an ASCII superset. */ res = PyMem_Malloc((strlen(arg)+1)*sizeof(wchar_t)); - if (!res) goto oom; + if (!res) + goto oom; in = (unsigned char*)arg; out = res; while(*in) @@ -126,7 +130,8 @@ *size = out - res; return res; oom: - fprintf(stderr, "out of memory\n"); + if (size != NULL) + *size = (size_t)-1; return NULL; } @@ -137,10 +142,10 @@ This function is the reverse of _Py_char2wchar(). Return a pointer to a newly allocated byte string (use PyMem_Free() to free - the memory), or NULL on conversion or memory allocation error. + the memory), or NULL on encoding or memory allocation error. If error_pos is not NULL: *error_pos is the index of the invalid character - on conversion error, or (size_t)-1 otherwise. */ + on encoding error, or (size_t)-1 otherwise. */ char* _Py_wchar2char(const wchar_t *text, size_t *error_pos) { @@ -328,7 +333,7 @@ #ifdef HAVE_READLINK /* Read value of symbolic link. Encode the path to the locale encoding, decode - the result from the locale encoding. */ + the result from the locale encoding. Return -1 on error. */ int _Py_wreadlink(const wchar_t *path, wchar_t *buf, size_t bufsiz) @@ -372,7 +377,8 @@ #ifdef HAVE_REALPATH /* Return the canonicalized absolute pathname. Encode path to the locale - encoding, decode the result from the locale encoding. */ + encoding, decode the result from the locale encoding. + Return NULL on error. */ wchar_t* _Py_wrealpath(const wchar_t *path, @@ -410,7 +416,8 @@ #endif /* Get the current directory. size is the buffer size in wide characters - including the null character. Decode the path from the locale encoding. */ + including the null character. Decode the path from the locale encoding. + Return NULL on error. */ wchar_t* _Py_wgetcwd(wchar_t *buf, size_t size) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): main() now displays an error message before exiting if a command line argument
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/3b17210f34d9 changeset: 74000:3b17210f34d9 branch: 3.2 parent: 73996:6a966179c73a user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 16 23:48:31 2011 +0100 summary: main() now displays an error message before exiting if a command line argument cannot be decoded files: Modules/python.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Modules/python.c b/Modules/python.c --- a/Modules/python.c +++ b/Modules/python.c @@ -50,8 +50,12 @@ #else argv_copy[i] = _Py_char2wchar(argv[i], NULL); #endif - if (!argv_copy[i]) + if (!argv_copy[i]) { + fprintf(stderr, "Fatal Python error: " + "unable to decode the command line argument #%i\n", + i + 1); return 1; + } argv_copy2[i] = argv_copy[i]; } setlocale(LC_ALL, oldloc); -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): (Merge 3.2) main() now displays an error message before exiting if a command
by victor.stinner Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/5d305270e58f changeset: 74001:5d305270e58f parent: 73999:fb887a9ce2c9 parent: 74000:3b17210f34d9 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 16 23:48:55 2011 +0100 summary: (Merge 3.2) main() now displays an error message before exiting if a command line argument cannot be decoded files: Modules/python.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Modules/python.c b/Modules/python.c --- a/Modules/python.c +++ b/Modules/python.c @@ -50,8 +50,12 @@ #else argv_copy[i] = _Py_char2wchar(argv[i], NULL); #endif - if (!argv_copy[i]) + if (!argv_copy[i]) { + fprintf(stderr, "Fatal Python error: " + "unable to decode the command line argument #%i\n", + i + 1); return 1; + } argv_copy2[i] = argv_copy[i]; } setlocale(LC_ALL, oldloc); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Skip test if the path is too long for a AF_UNIX socket
by antoine.pitrou Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/fb887a9ce2c9 changeset: 73999:fb887a9ce2c9 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 15:04:12 2011 +0100 summary: Skip test if the path is too long for a AF_UNIX socket files: Lib/test/test_socket.py | 20 ++++++++++++++++---- 1 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -4268,17 +4268,29 @@ "system encoding {1!r}".format( path, sys.getfilesystemencoding())) + def bind(self, sock, path): + # Bind the socket + try: + sock.bind(path) + except OSError as e: + if str(e) == "AF_UNIX path too long": + self.skipTest( + "Pathname {0!a} is too long to serve as a AF_UNIX path" + .format(path)) + else: + raise + def testStrAddr(self): # Test binding to and retrieving a normal string pathname. path = os.path.abspath(support.TESTFN) - self.sock.bind(path) + self.bind(self.sock, path) self.addCleanup(support.unlink, path) self.assertEqual(self.sock.getsockname(), path) def testBytesAddr(self): # Test binding to a bytes pathname. path = os.path.abspath(support.TESTFN) - self.sock.bind(self.encoded(path)) + self.bind(self.sock, self.encoded(path)) self.addCleanup(support.unlink, path) self.assertEqual(self.sock.getsockname(), path) @@ -4287,7 +4299,7 @@ # non-ASCII bytes supplied using surrogateescape encoding. path = os.path.abspath(support.TESTFN_UNICODE) b = self.encoded(path) - self.sock.bind(b.decode("ascii", "surrogateescape")) + self.bind(self.sock, b.decode("ascii", "surrogateescape")) self.addCleanup(support.unlink, path) self.assertEqual(self.sock.getsockname(), path) @@ -4297,7 +4309,7 @@ if support.TESTFN_UNENCODABLE is None: self.skipTest("No unencodable filename available") path = os.path.abspath(support.TESTFN_UNENCODABLE) - self.sock.bind(path) + self.bind(self.sock, path) self.addCleanup(support.unlink, path) self.assertEqual(self.sock.getsockname(), path) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #8373: The filesystem path of AF_UNIX sockets now uses the filesystem
by antoine.pitrou Dec. 17, 2011

Dec. 17, 2011

http://hg.python.org/cpython/rev/1f23bb74f4bc changeset: 73998:1f23bb74f4bc user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 14:46:36 2011 +0100 summary: Issue #8373: The filesystem path of AF_UNIX sockets now uses the filesystem encoding and the surrogateescape error handler, rather than UTF-8. Patch by David Watson. files: Doc/library/socket.rst | 18 +++++++- Lib/test/test_socket.py | 63 ++++++++++++++++++++++++++++- Misc/NEWS | 4 + Modules/socketmodule.c | 25 ++++++++-- 4 files changed, 101 insertions(+), 9 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -40,9 +40,23 @@ Depending on the system and the build options, various socket families are supported by this module. -Socket addresses are represented as follows: +The address format required by a particular socket object is automatically +selected based on the address family specified when the socket object was +created. Socket addresses are represented as follows: -- A single string is used for the :const:`AF_UNIX` address family. +- The address of an :const:`AF_UNIX` socket bound to a file system node + is represented as a string, using the file system encoding and the + ``'surrogateescape'`` error handler (see :pep:`383`). An address in + Linux's abstract namespace is returned as a :class:`bytes` object with + an initial null byte; note that sockets in this namespace can + communicate with normal file system sockets, so programs intended to + run on Linux may need to deal with both types of address. A string or + :class:`bytes` object can be used for either type of address when + passing it as an argument. + + .. versionchanged:: 3.3 + Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 + encoding. - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -1538,7 +1538,6 @@ def _testRecvFromNegative(self): self.cli.sendto(MSG, 0, (HOST, self.port)) - # Tests for the sendmsg()/recvmsg() interface. Where possible, the # same test code is used with different families and types of socket # (e.g. stream, datagram), and tests using recvmsg() are repeated @@ -4241,6 +4240,66 @@ with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s: self.assertRaises(socket.error, s.bind, address) + def testStrName(self): + # Check that an abstract name can be passed as a string. + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + try: + s.bind("\x00python\x00test\x00") + self.assertEqual(s.getsockname(), b"\x00python\x00test\x00") + finally: + s.close() + +class TestUnixDomain(unittest.TestCase): + + def setUp(self): + self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + + def tearDown(self): + self.sock.close() + + def encoded(self, path): + # Return the given path encoded in the file system encoding, + # or skip the test if this is not possible. + try: + return os.fsencode(path) + except UnicodeEncodeError: + self.skipTest( + "Pathname {0!a} cannot be represented in file " + "system encoding {1!r}".format( + path, sys.getfilesystemencoding())) + + def testStrAddr(self): + # Test binding to and retrieving a normal string pathname. + path = os.path.abspath(support.TESTFN) + self.sock.bind(path) + self.addCleanup(support.unlink, path) + self.assertEqual(self.sock.getsockname(), path) + + def testBytesAddr(self): + # Test binding to a bytes pathname. + path = os.path.abspath(support.TESTFN) + self.sock.bind(self.encoded(path)) + self.addCleanup(support.unlink, path) + self.assertEqual(self.sock.getsockname(), path) + + def testSurrogateescapeBind(self): + # Test binding to a valid non-ASCII pathname, with the + # non-ASCII bytes supplied using surrogateescape encoding. + path = os.path.abspath(support.TESTFN_UNICODE) + b = self.encoded(path) + self.sock.bind(b.decode("ascii", "surrogateescape")) + self.addCleanup(support.unlink, path) + self.assertEqual(self.sock.getsockname(), path) + + def testUnencodableAddr(self): + # Test binding to a pathname that cannot be encoded in the + # file system encoding. + if support.TESTFN_UNENCODABLE is None: + self.skipTest("No unencodable filename available") + path = os.path.abspath(support.TESTFN_UNENCODABLE) + self.sock.bind(path) + self.addCleanup(support.unlink, path) + self.assertEqual(self.sock.getsockname(), path) @unittest.skipUnless(thread, 'Threading required for this test.') class BufferIOTest(SocketConnectedTest): @@ -4517,6 +4576,8 @@ ]) if hasattr(socket, "socketpair"): tests.append(BasicSocketPairTest) + if hasattr(socket, "AF_UNIX"): + tests.append(TestUnixDomain) if sys.platform == 'linux': tests.append(TestLinuxAbstractNamespace) if isTipcAvailable(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -419,6 +419,10 @@ Library ------- +- Issue #8373: The filesystem path of AF_UNIX sockets now uses the filesystem + encoding and the surrogateescape error handler, rather than UTF-8. Patch + by David Watson. + - Issue #10350: Read and save errno before calling a function which might overwrite it. Original patch by Hallvard B Furuseth. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1073,7 +1073,7 @@ #endif /* linux */ { /* regular NULL-terminated string */ - return PyUnicode_FromString(a->sun_path); + return PyUnicode_DecodeFSDefault(a->sun_path); } } #endif /* AF_UNIX */ @@ -1269,8 +1269,18 @@ struct sockaddr_un* addr; char *path; int len; - if (!PyArg_Parse(args, "s#", &path, &len)) - return 0; + int retval = 0; + + /* PEP 383. Not using PyUnicode_FSConverter since we need to + allow embedded nulls on Linux. */ + if (PyUnicode_Check(args)) { + if ((args = PyUnicode_EncodeFSDefault(args)) == NULL) + return 0; + } + else + Py_INCREF(args); + if (!PyArg_Parse(args, "y#", &path, &len)) + goto unix_out; addr = (struct sockaddr_un*)addr_ret; #ifdef linux @@ -1279,7 +1289,7 @@ if (len > sizeof addr->sun_path) { PyErr_SetString(PyExc_OSError, "AF_UNIX path too long"); - return 0; + goto unix_out; } } else @@ -1289,7 +1299,7 @@ if (len >= sizeof addr->sun_path) { PyErr_SetString(PyExc_OSError, "AF_UNIX path too long"); - return 0; + goto unix_out; } addr->sun_path[len] = 0; } @@ -1300,7 +1310,10 @@ #else *len_ret = len + offsetof(struct sockaddr_un, sun_path); #endif - return 1; + retval = 1; + unix_out: + Py_DECREF(args); + return retval; } #endif /* AF_UNIX */ -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #10350: Read and save errno before calling a function which might
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/8e0b2e75ca7a changeset: 73997:8e0b2e75ca7a parent: 73995:ffe981aeb06f parent: 73996:6a966179c73a user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 12:29:37 2011 +0100 summary: Issue #10350: Read and save errno before calling a function which might overwrite it. Original patch by Hallvard B Furuseth. files: Misc/NEWS | 3 +++ Modules/_io/fileio.c | 14 +++++++++++--- Modules/_multiprocessing/semaphore.c | 4 +++- Modules/main.c | 3 ++- Modules/readline.c | 13 ++++++++----- Modules/timemodule.c | 4 +++- Parser/myreadline.c | 4 +++- 7 files changed, 33 insertions(+), 12 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -419,6 +419,9 @@ Library ------- +- Issue #10350: Read and save errno before calling a function which might + overwrite it. Original patch by Hallvard B Furuseth. + - Issue #11610: Introduce a more general way to declare abstract properties. - Issue #13591: A bug in importlib has been fixed that caused import_module diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -530,6 +530,7 @@ { Py_buffer pbuf; Py_ssize_t n, len; + int err; if (self->fd < 0) return err_closed(); @@ -553,10 +554,12 @@ Py_END_ALLOW_THREADS } else n = -1; + err = errno; PyBuffer_Release(&pbuf); if (n < 0) { - if (errno == EAGAIN) + if (err == EAGAIN) Py_RETURN_NONE; + errno = err; PyErr_SetFromErrno(PyExc_IOError); return NULL; } @@ -726,9 +729,11 @@ n = -1; if (n < 0) { + int err = errno; Py_DECREF(bytes); - if (errno == EAGAIN) + if (err == EAGAIN) Py_RETURN_NONE; + errno = err; PyErr_SetFromErrno(PyExc_IOError); return NULL; } @@ -748,6 +753,7 @@ { Py_buffer pbuf; Py_ssize_t n, len; + int err; if (self->fd < 0) return err_closed(); @@ -778,12 +784,14 @@ Py_END_ALLOW_THREADS } else n = -1; + err = errno; PyBuffer_Release(&pbuf); if (n < 0) { - if (errno == EAGAIN) + if (err == EAGAIN) Py_RETURN_NONE; + errno = err; PyErr_SetFromErrno(PyExc_IOError); return NULL; } diff --git a/Modules/_multiprocessing/semaphore.c b/Modules/_multiprocessing/semaphore.c --- a/Modules/_multiprocessing/semaphore.c +++ b/Modules/_multiprocessing/semaphore.c @@ -255,7 +255,7 @@ static PyObject * semlock_acquire(SemLockObject *self, PyObject *args, PyObject *kwds) { - int blocking = 1, res; + int blocking = 1, res, err = 0; double timeout; PyObject *timeout_obj = Py_None; struct timespec deadline = {0}; @@ -301,11 +301,13 @@ else res = sem_timedwait(self->handle, &deadline); Py_END_ALLOW_THREADS + err = errno; if (res == MP_EXCEPTION_HAS_BEEN_SET) break; } while (res < 0 && errno == EINTR && !PyErr_CheckSignals()); if (res < 0) { + errno = err; if (errno == EAGAIN || errno == ETIMEDOUT) Py_RETURN_FALSE; else if (errno == EINTR) diff --git a/Modules/main.c b/Modules/main.c --- a/Modules/main.c +++ b/Modules/main.c @@ -655,13 +655,14 @@ if (fp == NULL) { char *cfilename_buffer; const char *cfilename; + int err = errno; cfilename_buffer = _Py_wchar2char(filename, NULL); if (cfilename_buffer != NULL) cfilename = cfilename_buffer; else cfilename = "<unprintable file name>"; fprintf(stderr, "%ls: can't open file '%s': [Errno %d] %s\n", - argv[0], cfilename, errno, strerror(errno)); + argv[0], cfilename, err, strerror(err)); if (cfilename_buffer) PyMem_Free(cfilename_buffer); return 2; diff --git a/Modules/readline.c b/Modules/readline.c --- a/Modules/readline.c +++ b/Modules/readline.c @@ -154,6 +154,7 @@ { PyObject *filename_obj = Py_None, *filename_bytes; char *filename; + int err; if (!PyArg_ParseTuple(args, "|O:write_history_file", &filename_obj)) return NULL; if (filename_obj != Py_None) { @@ -164,10 +165,11 @@ filename_bytes = NULL; filename = NULL; } - errno = write_history(filename); - if (!errno && _history_length >= 0) + errno = err = write_history(filename); + if (!err && _history_length >= 0) history_truncate_file(filename, _history_length); Py_XDECREF(filename_bytes); + errno = err; if (errno) return PyErr_SetFromErrno(PyExc_IOError); Py_RETURN_NONE; @@ -969,7 +971,7 @@ completed_input_string = not_done_reading; while (completed_input_string == not_done_reading) { - int has_input = 0; + int has_input = 0, err = 0; while (!has_input) { struct timeval timeout = {0, 100000}; /* 0.1 seconds */ @@ -983,13 +985,14 @@ /* select resets selectset if no input was available */ has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, timeoutp); + err = errno; if(PyOS_InputHook) PyOS_InputHook(); } - if(has_input > 0) { + if (has_input > 0) { rl_callback_read_char(); } - else if (errno == EINTR) { + else if (err == EINTR) { int s; #ifdef WITH_THREAD PyEval_RestoreThread(_PyOS_ReadlineTState); diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -515,12 +515,14 @@ * will be ahead of time... */ for (i = 1024; ; i += i) { + int err; outbuf = (time_char *)PyMem_Malloc(i*sizeof(time_char)); if (outbuf == NULL) { PyErr_NoMemory(); break; } buflen = format_time(outbuf, i, fmt, &buf); + err = errno; if (buflen > 0 || i >= 256 * fmtlen) { /* If the buffer is 256 times as long as the format, it's probably not failing for lack of room! @@ -538,7 +540,7 @@ PyMem_Free(outbuf); #if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__) /* VisualStudio .NET 2005 does this properly */ - if (buflen == 0 && errno == EINVAL) { + if (buflen == 0 && err == EINVAL) { PyErr_SetString(PyExc_ValueError, "Invalid format string"); break; } diff --git a/Parser/myreadline.c b/Parser/myreadline.c --- a/Parser/myreadline.c +++ b/Parser/myreadline.c @@ -36,6 +36,7 @@ my_fgets(char *buf, int len, FILE *fp) { char *p; + int err; while (1) { if (PyOS_InputHook != NULL) (void)(PyOS_InputHook)(); @@ -44,6 +45,7 @@ p = fgets(buf, len, fp); if (p != NULL) return 0; /* No error */ + err = errno; #ifdef MS_WINDOWS /* In the case of a Ctrl+C or some other external event interrupting the operation: @@ -78,7 +80,7 @@ return -1; /* EOF */ } #ifdef EINTR - if (errno == EINTR) { + if (err == EINTR) { int s; #ifdef WITH_THREAD PyEval_RestoreThread(_PyOS_ReadlineTState); -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #10350: Read and save errno before calling a function which might
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/6a966179c73a changeset: 73996:6a966179c73a branch: 3.2 parent: 73985:5ec7ecf62c1d user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 12:28:32 2011 +0100 summary: Issue #10350: Read and save errno before calling a function which might overwrite it. Original patch by Hallvard B Furuseth. files: Misc/NEWS | 3 +++ Modules/_io/fileio.c | 14 +++++++++++--- Modules/_multiprocessing/semaphore.c | 4 +++- Modules/main.c | 3 ++- Modules/readline.c | 13 ++++++++----- Modules/timemodule.c | 4 +++- Parser/myreadline.c | 4 +++- 7 files changed, 33 insertions(+), 12 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -97,6 +97,9 @@ Library ------- +- Issue #10350: Read and save errno before calling a function which might + overwrite it. Original patch by Hallvard B Furuseth. + - Issue #13591: A bug in importlib has been fixed that caused import_module to load a module twice. diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -506,6 +506,7 @@ { Py_buffer pbuf; Py_ssize_t n, len; + int err; if (self->fd < 0) return err_closed(); @@ -529,10 +530,12 @@ Py_END_ALLOW_THREADS } else n = -1; + err = errno; PyBuffer_Release(&pbuf); if (n < 0) { - if (errno == EAGAIN) + if (err == EAGAIN) Py_RETURN_NONE; + errno = err; PyErr_SetFromErrno(PyExc_IOError); return NULL; } @@ -675,9 +678,11 @@ n = -1; if (n < 0) { + int err = errno; Py_DECREF(bytes); - if (errno == EAGAIN) + if (err == EAGAIN) Py_RETURN_NONE; + errno = err; PyErr_SetFromErrno(PyExc_IOError); return NULL; } @@ -697,6 +702,7 @@ { Py_buffer pbuf; Py_ssize_t n, len; + int err; if (self->fd < 0) return err_closed(); @@ -727,12 +733,14 @@ Py_END_ALLOW_THREADS } else n = -1; + err = errno; PyBuffer_Release(&pbuf); if (n < 0) { - if (errno == EAGAIN) + if (err == EAGAIN) Py_RETURN_NONE; + errno = err; PyErr_SetFromErrno(PyExc_IOError); return NULL; } diff --git a/Modules/_multiprocessing/semaphore.c b/Modules/_multiprocessing/semaphore.c --- a/Modules/_multiprocessing/semaphore.c +++ b/Modules/_multiprocessing/semaphore.c @@ -267,7 +267,7 @@ static PyObject * semlock_acquire(SemLockObject *self, PyObject *args, PyObject *kwds) { - int blocking = 1, res; + int blocking = 1, res, err = 0; double timeout; PyObject *timeout_obj = Py_None; struct timespec deadline = {0}; @@ -313,11 +313,13 @@ else res = sem_timedwait(self->handle, &deadline); Py_END_ALLOW_THREADS + err = errno; if (res == MP_EXCEPTION_HAS_BEEN_SET) break; } while (res < 0 && errno == EINTR && !PyErr_CheckSignals()); if (res < 0) { + errno = err; if (errno == EAGAIN || errno == ETIMEDOUT) Py_RETURN_FALSE; else if (errno == EINTR) diff --git a/Modules/main.c b/Modules/main.c --- a/Modules/main.c +++ b/Modules/main.c @@ -654,13 +654,14 @@ if (fp == NULL) { char *cfilename_buffer; const char *cfilename; + int err = errno; cfilename_buffer = _Py_wchar2char(filename, NULL); if (cfilename_buffer != NULL) cfilename = cfilename_buffer; else cfilename = "<unprintable file name>"; fprintf(stderr, "%ls: can't open file '%s': [Errno %d] %s\n", - argv[0], cfilename, errno, strerror(errno)); + argv[0], cfilename, err, strerror(err)); if (cfilename_buffer) PyMem_Free(cfilename_buffer); return 2; diff --git a/Modules/readline.c b/Modules/readline.c --- a/Modules/readline.c +++ b/Modules/readline.c @@ -154,6 +154,7 @@ { PyObject *filename_obj = Py_None, *filename_bytes; char *filename; + int err; if (!PyArg_ParseTuple(args, "|O:write_history_file", &filename_obj)) return NULL; if (filename_obj != Py_None) { @@ -164,10 +165,11 @@ filename_bytes = NULL; filename = NULL; } - errno = write_history(filename); - if (!errno && _history_length >= 0) + errno = err = write_history(filename); + if (!err && _history_length >= 0) history_truncate_file(filename, _history_length); Py_XDECREF(filename_bytes); + errno = err; if (errno) return PyErr_SetFromErrno(PyExc_IOError); Py_RETURN_NONE; @@ -970,7 +972,7 @@ completed_input_string = not_done_reading; while (completed_input_string == not_done_reading) { - int has_input = 0; + int has_input = 0, err = 0; while (!has_input) { struct timeval timeout = {0, 100000}; /* 0.1 seconds */ @@ -984,13 +986,14 @@ /* select resets selectset if no input was available */ has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, timeoutp); + err = errno; if(PyOS_InputHook) PyOS_InputHook(); } - if(has_input > 0) { + if (has_input > 0) { rl_callback_read_char(); } - else if (errno == EINTR) { + else if (err == EINTR) { int s; #ifdef WITH_THREAD PyEval_RestoreThread(_PyOS_ReadlineTState); diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -527,12 +527,14 @@ * will be ahead of time... */ for (i = 1024; ; i += i) { + int err; outbuf = (time_char *)PyMem_Malloc(i*sizeof(time_char)); if (outbuf == NULL) { PyErr_NoMemory(); break; } buflen = format_time(outbuf, i, fmt, &buf); + err = errno; if (buflen > 0 || i >= 256 * fmtlen) { /* If the buffer is 256 times as long as the format, it's probably not failing for lack of room! @@ -550,7 +552,7 @@ PyMem_Free(outbuf); #if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__) /* VisualStudio .NET 2005 does this properly */ - if (buflen == 0 && errno == EINVAL) { + if (buflen == 0 && err == EINVAL) { PyErr_SetString(PyExc_ValueError, "Invalid format string"); break; } diff --git a/Parser/myreadline.c b/Parser/myreadline.c --- a/Parser/myreadline.c +++ b/Parser/myreadline.c @@ -36,6 +36,7 @@ my_fgets(char *buf, int len, FILE *fp) { char *p; + int err; while (1) { if (PyOS_InputHook != NULL) (void)(PyOS_InputHook)(); @@ -44,6 +45,7 @@ p = fgets(buf, len, fp); if (p != NULL) return 0; /* No error */ + err = errno; #ifdef MS_WINDOWS /* In the case of a Ctrl+C or some other external event interrupting the operation: @@ -78,7 +80,7 @@ return -1; /* EOF */ } #ifdef EINTR - if (errno == EINTR) { + if (err == EINTR) { int s; #ifdef WITH_THREAD PyEval_RestoreThread(_PyOS_ReadlineTState); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #6695: Full garbage collection runs now clear the freelist of set objects.
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/57f0af61da53 changeset: 73994:57f0af61da53 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 11:24:27 2011 +0100 summary: Issue #6695: Full garbage collection runs now clear the freelist of set objects. Initial patch by Matthias Troffaes. files: Doc/c-api/set.rst | 7 +++++++ Include/setobject.h | 2 ++ Misc/NEWS | 3 +++ Modules/gcmodule.c | 1 + Objects/setobject.c | 12 ++++++++++-- 5 files changed, 23 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst --- a/Doc/c-api/set.rst +++ b/Doc/c-api/set.rst @@ -157,3 +157,10 @@ .. c:function:: int PySet_Clear(PyObject *set) Empty an existing set of all elements. + + +.. c:function:: int PySet_ClearFreeList() + + Clear the free list. Return the total number of freed items. + + .. versionadded:: 3.3 diff --git a/Include/setobject.h b/Include/setobject.h --- a/Include/setobject.h +++ b/Include/setobject.h @@ -99,6 +99,8 @@ PyAPI_FUNC(PyObject *) PySet_Pop(PyObject *set); #ifndef Py_LIMITED_API PyAPI_FUNC(int) _PySet_Update(PyObject *set, PyObject *iterable); + +PyAPI_FUNC(int) PySet_ClearFreeList(void); #endif #ifdef __cplusplus diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #6695: Full garbage collection runs now clear the freelist of set + objects. Initial patch by Matthias Troffaes. + - Fix OSError.__init__ and OSError.__new__ so that each of them can be overriden and take additional arguments (followup to issue #12555). diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -764,6 +764,7 @@ (void)PyFloat_ClearFreeList(); (void)PyList_ClearFreeList(); (void)PyDict_ClearFreeList(); + (void)PySet_ClearFreeList(); } static double diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1068,9 +1068,10 @@ return emptyfrozenset; } -void -PySet_Fini(void) +int +PySet_ClearFreeList(void) { + int freelist_size = numfree; PySetObject *so; while (numfree) { @@ -1078,6 +1079,13 @@ so = free_list[numfree]; PyObject_GC_Del(so); } + return freelist_size; +} + +void +PySet_Fini(void) +{ + PySet_ClearFreeList(); Py_CLEAR(dummy); Py_CLEAR(emptyfrozenset); } -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Add ACKS entry for 57f0af61da53.
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/ffe981aeb06f changeset: 73995:ffe981aeb06f user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 11:25:15 2011 +0100 summary: Add ACKS entry for 57f0af61da53. files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -988,6 +988,7 @@ Sandro Tosi Richard Townsend Laurence Tratt +Matthias Troffaes John Tromp Jason Trowbridge Anthony Tuininga -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (8d6d068e9966): sum=0
by solipsis＠pitrou.net Dec. 16, 2011

Dec. 16, 2011

results for 8d6d068e9966 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog09suNZ', '-x']

1 0

benchmarks: Fix gcbench for Python 3
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/benchmarks/rev/3f1cef15cf22 changeset: 154:3f1cef15cf22 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 16 00:21:23 2011 +0100 summary: Fix gcbench for Python 3 files: performance/gcbench.py | 31 ++++++++++++++++------------- 1 files changed, 17 insertions(+), 14 deletions(-) diff --git a/performance/gcbench.py b/performance/gcbench.py --- a/performance/gcbench.py +++ b/performance/gcbench.py @@ -43,6 +43,9 @@ # - No attempt to measure variation with object size # - Results are sensitive to locking cost, but we dont # check for proper locking + +from __future__ import print_function + import os, time USAGE = """gcbench [num_repetitions] [--depths=N,N,N..] [--threads=N]""" @@ -67,7 +70,7 @@ def num_iters(i): "Number of iterations to use for a given tree depth" - return 2 * tree_size(kStretchTreeDepth) / tree_size(i); + return 2 * tree_size(kStretchTreeDepth) // tree_size(i); def populate(depth, node): "Build tree top down, assigning to older objects." @@ -93,20 +96,20 @@ def time_construction(depth): niters = num_iters(depth) - print "Creating %d trees of depth %d" % (niters, depth) + print("Creating %d trees of depth %d" % (niters, depth)) t_start = time.time() for i in range(niters): temp_tree = Node() populate(depth, temp_tree) temp_tree = None t_finish = time.time() - print "\tTop down constrution took %f ms" % ((t_finish-t_start)*1000.) + print("\tTop down constrution took %f ms" % ((t_finish-t_start)*1000.)) t_start = time.time() for i in range(niters): temp_tree = make_tree(depth) temp_tree = None t_finish = time.time() - print "\tBottom up constrution took %f ms" % ((t_finish-t_start)*1000.) + print("\tBottom up constrution took %f ms" % ((t_finish-t_start)*1000.)) DEFAULT_DEPTHS = range(kMinTreeDepth, kMaxTreeDepth+1, 2) @@ -117,30 +120,30 @@ def time_parallel_constructions(depths, nthreads): import threading threadlist = [] - print "Starting %d parallel threads..." % (nthreads,) + print("Starting %d parallel threads..." % (nthreads,)) for n in range(nthreads): t = threading.Thread(target=time_constructions, args=(depths,)) t.start() threadlist.append(t) for t in threadlist: t.join() - print "All %d threads finished" % (nthreads,) + print("All %d threads finished" % (nthreads,)) def main(depths=DEFAULT_DEPTHS, threads=0): - print "Garbage Collector Test" - print " Stretching memory with a binary tree of depth %d" % kStretchTreeDepth + print("Garbage Collector Test") + print(" Stretching memory with a binary tree of depth %d" % kStretchTreeDepth) print_diagnostics() t_start = time.time() temp_tree = make_tree(kStretchTreeDepth) temp_tree = None # Create a long lived object - print " Creating a long-lived binary tree of depth %d" % kLongLivedTreeDepth + print(" Creating a long-lived binary tree of depth %d" % kLongLivedTreeDepth) long_lived_tree = Node() populate(kLongLivedTreeDepth, long_lived_tree) # Create long-lived array, filling half of it - print " Creating a long-lived array of %d doubles" % kArraySize + print(" Creating a long-lived array of %d doubles" % kArraySize) array = [0.0] * kArraySize i = 1 while i < kArraySize/2: @@ -158,15 +161,15 @@ t_finish = time.time() print_diagnostics() - print "Completed in %f ms." % ((t_finish-t_start)*1000.) + print("Completed in %f ms." % ((t_finish-t_start)*1000.)) class Failed(Exception): pass def argerror(): - print "Usage:" - print " ", USAGE + print("Usage:") + print(" ", USAGE) return 2 def entry_point(argv): @@ -177,7 +180,7 @@ if arg.startswith('--threads='): arg = arg[len('--threads='):] if not ENABLE_THREADS: - print "threads disabled (they cannot be translated)" + print("threads disabled (they cannot be translated)") return 1 try: threads = int(arg) -- Repository URL: http://hg.python.org/benchmarks

2 1

cpython: property -> staticmethod
by benjamin.peterson Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/8d6d068e9966 changeset: 73993:8d6d068e9966 user: Benjamin Peterson <benjamin(a)python.org> date: Thu Dec 15 19:24:49 2011 -0500 summary: property -> staticmethod files: Doc/whatsnew/3.3.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -365,7 +365,7 @@ * :class:`abc.abstractclassmethod` has been deprecated, use :class:`classmethod` with :func:`abc.abstractmethod` instead. * :class:`abc.abstractstaticmethod` has been deprecated, use - :class:`property` with :func:`abc.abstractmethod` instead. + :class:`staticmethod` with :func:`abc.abstractmethod` instead. (Contributed by Darren Dale in :issue:`11610`) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: improve abstract property support (closes #11610)
by benjamin.peterson Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/e979b26a9172 changeset: 73984:e979b26a9172 user: Benjamin Peterson <benjamin(a)python.org> date: Thu Dec 15 15:34:02 2011 -0500 summary: improve abstract property support (closes #11610) Thanks to Darren Dale for patch. files: Doc/library/abc.rst | 75 +++++++++-- Doc/whatsnew/3.3.rst | 17 ++ Include/object.h | 1 + Lib/abc.py | 19 ++- Lib/numbers.py | 14 +- Lib/test/test_abc.py | 172 ++++++++++++++++++++++++- Lib/test/test_property.py | 23 +++ Misc/ACKS | 1 + Misc/NEWS | 2 + Objects/descrobject.c | 39 +++++- Objects/funcobject.c | 46 ++++++- Objects/object.c | 23 +++ 12 files changed, 396 insertions(+), 36 deletions(-) diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -127,19 +127,18 @@ available as a method of ``Foo``, so it is provided separately. -It also provides the following decorators: +The :mod:`abc` module also provides the following decorators: .. decorator:: abstractmethod(function) A decorator indicating abstract methods. - Using this decorator requires that the class's metaclass is :class:`ABCMeta` or - is derived from it. - A class that has a metaclass derived from :class:`ABCMeta` - cannot be instantiated unless all of its abstract methods and - properties are overridden. - The abstract methods can be called using any of the normal 'super' call - mechanisms. + Using this decorator requires that the class's metaclass is :class:`ABCMeta` + or is derived from it. A class that has a metaclass derived from + :class:`ABCMeta` cannot be instantiated unless all of its abstract methods + and properties are overridden. The abstract methods can be called using any + of the normal 'super' call mechanisms. :func:`abstractmethod` may be used + to declare abstract methods for properties and descriptors. Dynamically adding abstract methods to a class, or attempting to modify the abstraction status of a method or class once it is created, are not @@ -147,12 +146,52 @@ regular inheritance; "virtual subclasses" registered with the ABC's :meth:`register` method are not affected. - Usage:: + When :func:`abstractmethod` is applied in combination with other method + descriptors, it should be applied as the innermost decorator, as shown in + the following usage examples:: class C(metaclass=ABCMeta): @abstractmethod def my_abstract_method(self, ...): ... + @classmethod + @abstractmethod + def my_abstract_classmethod(cls, ...): + ... + @staticmethod + @abstractmethod + def my_abstract_staticmethod(...): + ... + + @property + @abstractmethod + def my_abstract_property(self): + ... + @my_abstract_property.setter + @abstractmethod + def my_abstract_property(self, val): + ... + + @abstractmethod + def _get_x(self): + ... + @abstractmethod + def _set_x(self, val): + ... + x = property(_get_x, _set_x) + + In order to correctly interoperate with the abstract base class machinery, + the descriptor must identify itself as abstract using + :attr:`__isabstractmethod__`. In general, this attribute should be ``True`` + if any of the methods used to compose the descriptor are abstract. For + example, Python's built-in property does the equivalent of:: + + class Descriptor: + ... + @property + def __isabstractmethod__(self): + return any(getattr(f, '__isabstractmethod__', False) for + f in (self._fget, self._fset, self._fdel)) .. note:: @@ -177,6 +216,8 @@ ... .. versionadded:: 3.2 + .. deprecated:: 3.3 + Use :class:`classmethod` with :func:`abstractmethod` instead .. decorator:: abstractstaticmethod(function) @@ -192,18 +233,19 @@ ... .. versionadded:: 3.2 + .. deprecated:: 3.3 + Use :class:`staticmethod` with :func:`abstractmethod` instead .. decorator:: abstractproperty(fget=None, fset=None, fdel=None, doc=None) A subclass of the built-in :func:`property`, indicating an abstract property. - Using this function requires that the class's metaclass is :class:`ABCMeta` or - is derived from it. - A class that has a metaclass derived from :class:`ABCMeta` cannot be - instantiated unless all of its abstract methods and properties are overridden. - The abstract properties can be called using any of the normal - 'super' call mechanisms. + Using this function requires that the class's metaclass is :class:`ABCMeta` + or is derived from it. A class that has a metaclass derived from + :class:`ABCMeta` cannot be instantiated unless all of its abstract methods + and properties are overridden. The abstract properties can be called using + any of the normal 'super' call mechanisms. Usage:: @@ -220,6 +262,9 @@ def setx(self, value): ... x = abstractproperty(getx, setx) + .. deprecated:: 3.3 + Use :class:`property` with :func:`abstractmethod` instead + .. rubric:: Footnotes diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -352,6 +352,23 @@ (Contributed by Iñigo Serna in :issue:`6755`) +abc +--- + +Improved support for abstract base classes containing descriptors composed with +abstract methods. The recommended approach to declaring abstract descriptors is +now to provide :attr:`__isabstractmethod__` as a dynamically updated +property. The built-in descriptors have been updated accordingly. + + * :class:`abc.abstractproperty` has been deprecated, use :class:`property` + with :func:`abc.abstractmethod` instead. + * :class:`abc.abstractclassmethod` has been deprecated, use + :class:`classmethod` with :func:`abc.abstractmethod` instead. + * :class:`abc.abstractstaticmethod` has been deprecated, use + :class:`property` with :func:`abc.abstractmethod` instead. + +(Contributed by Darren Dale in :issue:`11610`) + faulthandler ------------ diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -473,6 +473,7 @@ PyAPI_FUNC(PyObject *) PyObject_GetAttr(PyObject *, PyObject *); PyAPI_FUNC(int) PyObject_SetAttr(PyObject *, PyObject *, PyObject *); PyAPI_FUNC(int) PyObject_HasAttr(PyObject *, PyObject *); +PyAPI_FUNC(int) _PyObject_IsAbstract(PyObject *); PyAPI_FUNC(PyObject *) _PyObject_GetAttrId(PyObject *, struct _Py_Identifier *); PyAPI_FUNC(int) _PyObject_SetAttrId(PyObject *, struct _Py_Identifier *, PyObject *); PyAPI_FUNC(int) _PyObject_HasAttrId(PyObject *, struct _Py_Identifier *); diff --git a/Lib/abc.py b/Lib/abc.py --- a/Lib/abc.py +++ b/Lib/abc.py @@ -26,7 +26,8 @@ class abstractclassmethod(classmethod): - """A decorator indicating abstract classmethods. + """ + A decorator indicating abstract classmethods. Similar to abstractmethod. @@ -36,6 +37,9 @@ @abstractclassmethod def my_abstract_classmethod(cls, ...): ... + + 'abstractclassmethod' is deprecated. Use 'classmethod' with + 'abstractmethod' instead. """ __isabstractmethod__ = True @@ -46,7 +50,8 @@ class abstractstaticmethod(staticmethod): - """A decorator indicating abstract staticmethods. + """ + A decorator indicating abstract staticmethods. Similar to abstractmethod. @@ -56,6 +61,9 @@ @abstractstaticmethod def my_abstract_staticmethod(...): ... + + 'abstractstaticmethod' is deprecated. Use 'staticmethod' with + 'abstractmethod' instead. """ __isabstractmethod__ = True @@ -66,7 +74,8 @@ class abstractproperty(property): - """A decorator indicating abstract properties. + """ + A decorator indicating abstract properties. Requires that the metaclass is ABCMeta or derived from it. A class that has a metaclass derived from ABCMeta cannot be @@ -88,7 +97,11 @@ def getx(self): ... def setx(self, value): ... x = abstractproperty(getx, setx) + + 'abstractproperty' is deprecated. Use 'property' with 'abstractmethod' + instead. """ + __isabstractmethod__ = True diff --git a/Lib/numbers.py b/Lib/numbers.py --- a/Lib/numbers.py +++ b/Lib/numbers.py @@ -5,7 +5,7 @@ TODO: Fill out more detailed documentation on the operators.""" -from abc import ABCMeta, abstractmethod, abstractproperty +from abc import ABCMeta, abstractmethod __all__ = ["Number", "Complex", "Real", "Rational", "Integral"] @@ -50,7 +50,8 @@ """True if self != 0. Called for bool(self).""" return self != 0 - @abstractproperty + @property + @abstractmethod def real(self): """Retrieve the real component of this number. @@ -58,7 +59,8 @@ """ raise NotImplementedError - @abstractproperty + @property + @abstractmethod def imag(self): """Retrieve the imaginary component of this number. @@ -272,11 +274,13 @@ __slots__ = () - @abstractproperty + @property + @abstractmethod def numerator(self): raise NotImplementedError - @abstractproperty + @property + @abstractmethod def denominator(self): raise NotImplementedError diff --git a/Lib/test/test_abc.py b/Lib/test/test_abc.py --- a/Lib/test/test_abc.py +++ b/Lib/test/test_abc.py @@ -10,14 +10,7 @@ from inspect import isabstract -class TestABC(unittest.TestCase): - - def test_abstractmethod_basics(self): - @abc.abstractmethod - def foo(self): pass - self.assertTrue(foo.__isabstractmethod__) - def bar(self): pass - self.assertFalse(hasattr(bar, "__isabstractmethod__")) +class TestLegacyAPI(unittest.TestCase): def test_abstractproperty_basics(self): @abc.abstractproperty @@ -29,10 +22,12 @@ class C(metaclass=abc.ABCMeta): @abc.abstractproperty def foo(self): return 3 + self.assertRaises(TypeError, C) class D(C): @property def foo(self): return super().foo self.assertEqual(D().foo, 3) + self.assertFalse(getattr(D.foo, "__isabstractmethod__", False)) def test_abstractclassmethod_basics(self): @abc.abstractclassmethod @@ -40,7 +35,7 @@ self.assertTrue(foo.__isabstractmethod__) @classmethod def bar(cls): pass - self.assertFalse(hasattr(bar, "__isabstractmethod__")) + self.assertFalse(getattr(bar, "__isabstractmethod__", False)) class C(metaclass=abc.ABCMeta): @abc.abstractclassmethod @@ -58,7 +53,7 @@ self.assertTrue(foo.__isabstractmethod__) @staticmethod def bar(): pass - self.assertFalse(hasattr(bar, "__isabstractmethod__")) + self.assertFalse(getattr(bar, "__isabstractmethod__", False)) class C(metaclass=abc.ABCMeta): @abc.abstractstaticmethod @@ -98,6 +93,163 @@ self.assertRaises(TypeError, F) # because bar is abstract now self.assertTrue(isabstract(F)) + +class TestABC(unittest.TestCase): + + def test_abstractmethod_basics(self): + @abc.abstractmethod + def foo(self): pass + self.assertTrue(foo.__isabstractmethod__) + def bar(self): pass + self.assertFalse(hasattr(bar, "__isabstractmethod__")) + + def test_abstractproperty_basics(self): + @property + @abc.abstractmethod + def foo(self): pass + self.assertTrue(foo.__isabstractmethod__) + def bar(self): pass + self.assertFalse(getattr(bar, "__isabstractmethod__", False)) + + class C(metaclass=abc.ABCMeta): + @property + @abc.abstractmethod + def foo(self): return 3 + self.assertRaises(TypeError, C) + class D(C): + @C.foo.getter + def foo(self): return super().foo + self.assertEqual(D().foo, 3) + + def test_abstractclassmethod_basics(self): + @classmethod + @abc.abstractmethod + def foo(cls): pass + self.assertTrue(foo.__isabstractmethod__) + @classmethod + def bar(cls): pass + self.assertFalse(getattr(bar, "__isabstractmethod__", False)) + + class C(metaclass=abc.ABCMeta): + @classmethod + @abc.abstractmethod + def foo(cls): return cls.__name__ + self.assertRaises(TypeError, C) + class D(C): + @classmethod + def foo(cls): return super().foo() + self.assertEqual(D.foo(), 'D') + self.assertEqual(D().foo(), 'D') + + def test_abstractstaticmethod_basics(self): + @staticmethod + @abc.abstractmethod + def foo(): pass + self.assertTrue(foo.__isabstractmethod__) + @staticmethod + def bar(): pass + self.assertFalse(getattr(bar, "__isabstractmethod__", False)) + + class C(metaclass=abc.ABCMeta): + @staticmethod + @abc.abstractmethod + def foo(): return 3 + self.assertRaises(TypeError, C) + class D(C): + @staticmethod + def foo(): return 4 + self.assertEqual(D.foo(), 4) + self.assertEqual(D().foo(), 4) + + def test_abstractmethod_integration(self): + for abstractthing in [abc.abstractmethod, abc.abstractproperty, + abc.abstractclassmethod, + abc.abstractstaticmethod]: + class C(metaclass=abc.ABCMeta): + @abstractthing + def foo(self): pass # abstract + def bar(self): pass # concrete + self.assertEqual(C.__abstractmethods__, {"foo"}) + self.assertRaises(TypeError, C) # because foo is abstract + self.assertTrue(isabstract(C)) + class D(C): + def bar(self): pass # concrete override of concrete + self.assertEqual(D.__abstractmethods__, {"foo"}) + self.assertRaises(TypeError, D) # because foo is still abstract + self.assertTrue(isabstract(D)) + class E(D): + def foo(self): pass + self.assertEqual(E.__abstractmethods__, set()) + E() # now foo is concrete, too + self.assertFalse(isabstract(E)) + class F(E): + @abstractthing + def bar(self): pass # abstract override of concrete + self.assertEqual(F.__abstractmethods__, {"bar"}) + self.assertRaises(TypeError, F) # because bar is abstract now + self.assertTrue(isabstract(F)) + + def test_descriptors_with_abstractmethod(self): + class C(metaclass=abc.ABCMeta): + @property + @abc.abstractmethod + def foo(self): return 3 + @foo.setter + @abc.abstractmethod + def foo(self, val): pass + self.assertRaises(TypeError, C) + class D(C): + @C.foo.getter + def foo(self): return super().foo + self.assertRaises(TypeError, D) + class E(D): + @D.foo.setter + def foo(self, val): pass + self.assertEqual(E().foo, 3) + # check that the property's __isabstractmethod__ descriptor does the + # right thing when presented with a value that fails truth testing: + class NotBool(object): + def __nonzero__(self): + raise ValueError() + __len__ = __nonzero__ + with self.assertRaises(ValueError): + class F(C): + def bar(self): + pass + bar.__isabstractmethod__ = NotBool() + foo = property(bar) + + + def test_customdescriptors_with_abstractmethod(self): + class Descriptor: + def __init__(self, fget, fset=None): + self._fget = fget + self._fset = fset + def getter(self, callable): + return Descriptor(callable, self._fget) + def setter(self, callable): + return Descriptor(self._fget, callable) + @property + def __isabstractmethod__(self): + return (getattr(self._fget, '__isabstractmethod__', False) + or getattr(self._fset, '__isabstractmethod__', False)) + class C(metaclass=abc.ABCMeta): + @Descriptor + @abc.abstractmethod + def foo(self): return 3 + @foo.setter + @abc.abstractmethod + def foo(self, val): pass + self.assertRaises(TypeError, C) + class D(C): + @C.foo.getter + def foo(self): return super().foo + self.assertRaises(TypeError, D) + class E(D): + @D.foo.setter + def foo(self, val): pass + self.assertFalse(E.foo.__isabstractmethod__) + def test_metaclass_abc(self): # Metaclasses can be ABCs, too. class A(metaclass=abc.ABCMeta): diff --git a/Lib/test/test_property.py b/Lib/test/test_property.py --- a/Lib/test/test_property.py +++ b/Lib/test/test_property.py @@ -128,6 +128,29 @@ self.assertEqual(newgetter.spam, 8) self.assertEqual(newgetter.__class__.spam.__doc__, "new docstring") + def test_property___isabstractmethod__descriptor(self): + for val in (True, False, [], [1], '', '1'): + class C(object): + def foo(self): + pass + foo.__isabstractmethod__ = val + foo = property(foo) + self.assertIs(C.foo.__isabstractmethod__, bool(val)) + + # check that the property's __isabstractmethod__ descriptor does the + # right thing when presented with a value that fails truth testing: + class NotBool(object): + def __nonzero__(self): + raise ValueError() + __len__ = __nonzero__ + with self.assertRaises(ValueError): + class C(object): + def foo(self): + pass + foo.__isabstractmethod__ = NotBool() + foo = property(foo) + C.foo.__isabstractmethod__ + # Issue 5890: subclasses of property do not preserve method __doc__ strings class PropertySub(property): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -220,6 +220,7 @@ Antonio Cuni Brian Curtin Lisandro Dalcin +Darren Dale Andrew Dalke Lars Damerow Evan Dandrea diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -416,6 +416,8 @@ Library ------- +- Issue #11610: Introduce a more general way to declare abstract properties. + - Issue #13591: A bug in importlib has been fixed that caused import_module to load a module twice. diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -1380,6 +1380,43 @@ return 0; } +static PyObject * +property_get___isabstractmethod__(propertyobject *prop, void *closure) +{ + int res = _PyObject_IsAbstract(prop->prop_get); + if (res == -1) { + return NULL; + } + else if (res) { + Py_RETURN_TRUE; + } + + res = _PyObject_IsAbstract(prop->prop_set); + if (res == -1) { + return NULL; + } + else if (res) { + Py_RETURN_TRUE; + } + + res = _PyObject_IsAbstract(prop->prop_del); + if (res == -1) { + return NULL; + } + else if (res) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; +} + +static PyGetSetDef property_getsetlist[] = { + {"__isabstractmethod__", + (getter)property_get___isabstractmethod__, NULL, + NULL, + NULL}, + {NULL} /* Sentinel */ +}; + PyDoc_STRVAR(property_doc, "property(fget=None, fset=None, fdel=None, doc=None) -> property attribute\n" "\n" @@ -1445,7 +1482,7 @@ 0, /* tp_iternext */ property_methods, /* tp_methods */ property_members, /* tp_members */ - 0, /* tp_getset */ + property_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ property_descr_get, /* tp_descr_get */ diff --git a/Objects/funcobject.c b/Objects/funcobject.c --- a/Objects/funcobject.c +++ b/Objects/funcobject.c @@ -814,6 +814,27 @@ {NULL} /* Sentinel */ }; +static PyObject * +cm_get___isabstractmethod__(classmethod *cm, void *closure) +{ + int res = _PyObject_IsAbstract(cm->cm_callable); + if (res == -1) { + return NULL; + } + else if (res) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; +} + +static PyGetSetDef cm_getsetlist[] = { + {"__isabstractmethod__", + (getter)cm_get___isabstractmethod__, NULL, + NULL, + NULL}, + {NULL} /* Sentinel */ +}; + PyDoc_STRVAR(classmethod_doc, "classmethod(function) -> method\n\ \n\ @@ -865,7 +886,7 @@ 0, /* tp_iternext */ 0, /* tp_methods */ cm_memberlist, /* tp_members */ - 0, /* tp_getset */ + cm_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ cm_descr_get, /* tp_descr_get */ @@ -969,6 +990,27 @@ {NULL} /* Sentinel */ }; +static PyObject * +sm_get___isabstractmethod__(staticmethod *sm, void *closure) +{ + int res = _PyObject_IsAbstract(sm->sm_callable); + if (res == -1) { + return NULL; + } + else if (res) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; +} + +static PyGetSetDef sm_getsetlist[] = { + {"__isabstractmethod__", + (getter)sm_get___isabstractmethod__, NULL, + NULL, + NULL}, + {NULL} /* Sentinel */ +}; + PyDoc_STRVAR(staticmethod_doc, "staticmethod(function) -> method\n\ \n\ @@ -1017,7 +1059,7 @@ 0, /* tp_iternext */ 0, /* tp_methods */ sm_memberlist, /* tp_members */ - 0, /* tp_getset */ + sm_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ sm_descr_get, /* tp_descr_get */ diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -840,6 +840,29 @@ return res; } +int +_PyObject_IsAbstract(PyObject *obj) +{ + int res; + PyObject* isabstract; + _Py_IDENTIFIER(__isabstractmethod__); + + if (obj == NULL) + return 0; + + isabstract = _PyObject_GetAttrId(obj, &PyId___isabstractmethod__); + if (isabstract == NULL) { + if (PyErr_ExceptionMatches(PyExc_AttributeError)) { + PyErr_Clear(); + return 0; + } + return -1; + } + res = PyObject_IsTrue(isabstract); + Py_DECREF(isabstract); + return res; +} + PyObject * _PyObject_GetAttrId(PyObject *v, _Py_Identifier *name) { -- Repository URL: http://hg.python.org/cpython

2 1

cpython (merge default -> default): merge heads
by benjamin.peterson Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/df57bea858f3 changeset: 73992:df57bea858f3 parent: 73991:c9ae0eb66959 parent: 73990:06d83098d9a9 user: Benjamin Peterson <benjamin(a)python.org> date: Thu Dec 15 15:57:47 2011 -0500 summary: merge heads files: Makefile.pre.in | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -379,6 +379,7 @@ Objects/unicodectype.o \ Objects/weakrefobject.o +SYSCONFIGDATA=$(srcdir)/Lib/_sysconfigdata.py ########################################################################## # objects that get linked into the Python library @@ -396,7 +397,7 @@ # Default target all: build_all -build_all: $(BUILDPYTHON) sysconfig oldsharedmods sharedmods gdbhooks Modules/_testembed +build_all: $(BUILDPYTHON) $(SYSCONFIGDATA) oldsharedmods sharedmods gdbhooks Modules/_testembed # Compile a binary with gcc profile guided optimization. profile-opt: @@ -429,15 +430,15 @@ $(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(LINKCC) $(PY_LDFLAGS) $(LINKFORSHARED) -o $@ Modules/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST) -platform: $(BUILDPYTHON) sysconfig +platform: $(BUILDPYTHON) $(SYSCONFIGDATA) $(RUNSHARED) ./$(BUILDPYTHON) -E -c 'import sys ; from sysconfig import get_platform ; print(get_platform()+"-"+sys.version[0:3])' >platform # Generate the sysconfig build-time data -sysconfig: $(BUILDPYTHON) +$(SYSCONFIGDATA): $(BUILDPYTHON) $(RUNSHARED) ./$(BUILDPYTHON) -SE -m sysconfig --generate-posix-vars # Build the shared modules -sharedmods: $(BUILDPYTHON) sysconfig +sharedmods: $(BUILDPYTHON) $(SYSCONFIGDATA) @case $$MAKEFLAGS in \ *s*) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' ./$(BUILDPYTHON) -E $(srcdir)/setup.py -q build;; \ *) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' ./$(BUILDPYTHON) -E $(srcdir)/setup.py build;; \ @@ -1315,7 +1316,7 @@ find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name 'fficonfig.py' -exec rm -f {} ';' || true -rm -f Lib/lib2to3/*Grammar*.pickle - -rm -f Lib/_sysconfigdata.py + -rm -f $(SYSCONFIGDATA) -rm -f Modules/_testembed profile-removal: @@ -1393,7 +1394,7 @@ Python/thread.o: @THREADHEADERS@ # Declare targets that aren't real files -.PHONY: all build_all sysconfig sharedmods oldsharedmods test quicktest +.PHONY: all build_all sharedmods oldsharedmods test quicktest .PHONY: install altinstall oldsharedinstall bininstall altbininstall .PHONY: maninstall libinstall inclinstall libainstall sharedinstall .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure -- Repository URL: http://hg.python.org/cpython

1 0

cpython: fix this test to actually test something (closes #13606)
by benjamin.peterson Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/c9ae0eb66959 changeset: 73991:c9ae0eb66959 parent: 73989:df5c28c01deb user: Benjamin Peterson <benjamin(a)python.org> date: Thu Dec 15 15:57:15 2011 -0500 summary: fix this test to actually test something (closes #13606) Thanks Mark Shannon. files: Lib/test/test_module.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_module.py b/Lib/test/test_module.py --- a/Lib/test/test_module.py +++ b/Lib/test/test_module.py @@ -70,9 +70,11 @@ m = ModuleType("foo") m.destroyed = destroyed s = """class A: + def __init__(self, l): + self.l = l def __del__(self): - destroyed.append(1) -a = A()""" + self.l.append(1) +a = A(destroyed)""" exec(s, m.__dict__) del m gc_collect() -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Close #13596: Only recompile Lib/_sysconfigdata.py when needed
by victor.stinner Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/06d83098d9a9 changeset: 73990:06d83098d9a9 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 15 21:48:39 2011 +0100 summary: Close #13596: Only recompile Lib/_sysconfigdata.py when needed files: Makefile.pre.in | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -379,6 +379,7 @@ Objects/unicodectype.o \ Objects/weakrefobject.o +SYSCONFIGDATA=$(srcdir)/Lib/_sysconfigdata.py ########################################################################## # objects that get linked into the Python library @@ -396,7 +397,7 @@ # Default target all: build_all -build_all: $(BUILDPYTHON) sysconfig oldsharedmods sharedmods gdbhooks Modules/_testembed +build_all: $(BUILDPYTHON) $(SYSCONFIGDATA) oldsharedmods sharedmods gdbhooks Modules/_testembed # Compile a binary with gcc profile guided optimization. profile-opt: @@ -429,15 +430,15 @@ $(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(LINKCC) $(PY_LDFLAGS) $(LINKFORSHARED) -o $@ Modules/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST) -platform: $(BUILDPYTHON) sysconfig +platform: $(BUILDPYTHON) $(SYSCONFIGDATA) $(RUNSHARED) ./$(BUILDPYTHON) -E -c 'import sys ; from sysconfig import get_platform ; print(get_platform()+"-"+sys.version[0:3])' >platform # Generate the sysconfig build-time data -sysconfig: $(BUILDPYTHON) +$(SYSCONFIGDATA): $(BUILDPYTHON) $(RUNSHARED) ./$(BUILDPYTHON) -SE -m sysconfig --generate-posix-vars # Build the shared modules -sharedmods: $(BUILDPYTHON) sysconfig +sharedmods: $(BUILDPYTHON) $(SYSCONFIGDATA) @case $$MAKEFLAGS in \ *s*) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' ./$(BUILDPYTHON) -E $(srcdir)/setup.py -q build;; \ *) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' ./$(BUILDPYTHON) -E $(srcdir)/setup.py build;; \ @@ -1315,7 +1316,7 @@ find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name 'fficonfig.py' -exec rm -f {} ';' || true -rm -f Lib/lib2to3/*Grammar*.pickle - -rm -f Lib/_sysconfigdata.py + -rm -f $(SYSCONFIGDATA) -rm -f Modules/_testembed profile-removal: @@ -1393,7 +1394,7 @@ Python/thread.o: @THREADHEADERS@ # Declare targets that aren't real files -.PHONY: all build_all sysconfig sharedmods oldsharedmods test quicktest +.PHONY: all build_all sharedmods oldsharedmods test quicktest .PHONY: install altinstall oldsharedinstall bininstall altbininstall .PHONY: maninstall libinstall inclinstall libainstall sharedinstall .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge default -> default): merge heads
by benjamin.peterson Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/df5c28c01deb changeset: 73989:df5c28c01deb parent: 73988:4de895bb25f6 parent: 73986:a9ee21ac0879 user: Benjamin Peterson <benjamin(a)python.org> date: Thu Dec 15 15:44:16 2011 -0500 summary: merge heads files: Lib/platform.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -186,7 +186,7 @@ elif so: if lib != 'glibc': lib = 'libc' - if soversion > version: + if soversion and soversion > version: version = soversion if threads and version[-len(threads):] != threads: version = version + threads -- Repository URL: http://hg.python.org/cpython

1 0

cpython: input() in this sense is gone
by benjamin.peterson Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/4de895bb25f6 changeset: 73988:4de895bb25f6 parent: 73984:e979b26a9172 user: Benjamin Peterson <benjamin(a)python.org> date: Thu Dec 15 15:43:56 2011 -0500 summary: input() in this sense is gone files: Grammar/Grammar | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Grammar/Grammar b/Grammar/Grammar --- a/Grammar/Grammar +++ b/Grammar/Grammar @@ -13,7 +13,7 @@ # Start symbols for the grammar: # single_input is a single interactive statement; # file_input is a module or sequence of commands read from an input file; -# eval_input is the input for the eval() and input() functions. +# eval_input is the input for the eval() functions. # NB: compound_stmt in single_input is followed by extra NEWLINE! single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE file_input: (NEWLINE | stmt)* ENDMARKER -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13545: Fix platform.libc_version() is the SO version is missing
by victor.stinner Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/f2a5dcced66d changeset: 73987:f2a5dcced66d branch: 2.7 parent: 73980:da1d7f8cd487 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 15 21:42:03 2011 +0100 summary: Issue #13545: Fix platform.libc_version() is the SO version is missing files: Lib/platform.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -183,7 +183,7 @@ elif so: if lib != 'glibc': lib = 'libc' - if soversion > version: + if soversion and soversion > version: version = soversion if threads and version[-len(threads):] != threads: version = version + threads -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): (Merge 3.2) Issue #13545: Fix platform.libc_version() is the SO version is
by victor.stinner Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/a9ee21ac0879 changeset: 73986:a9ee21ac0879 parent: 73984:e979b26a9172 parent: 73985:5ec7ecf62c1d user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 15 21:42:28 2011 +0100 summary: (Merge 3.2) Issue #13545: Fix platform.libc_version() is the SO version is missing files: Lib/platform.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -186,7 +186,7 @@ elif so: if lib != 'glibc': lib = 'libc' - if soversion > version: + if soversion and soversion > version: version = soversion if threads and version[-len(threads):] != threads: version = version + threads -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13545: Fix platform.libc_version() is the SO version is missing
by victor.stinner Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/5ec7ecf62c1d changeset: 73985:5ec7ecf62c1d branch: 3.2 parent: 73982:313fa7ea6c79 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 15 21:42:03 2011 +0100 summary: Issue #13545: Fix platform.libc_version() is the SO version is missing files: Lib/platform.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -181,7 +181,7 @@ elif so: if lib != 'glibc': lib = 'libc' - if soversion > version: + if soversion and soversion > version: version = soversion if threads and version[-len(threads):] != threads: version = version + threads -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13597: Improve documentation of standard streams.
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/313fa7ea6c79 changeset: 73982:313fa7ea6c79 branch: 3.2 parent: 73978:b196bcd7c34f user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 16:25:34 2011 +0100 summary: Issue #13597: Improve documentation of standard streams. files: Doc/library/sys.rst | 47 ++++++++++++++++++++------------ 1 files changed, 29 insertions(+), 18 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -934,31 +934,42 @@ stdout stderr - :term:`File objects <file object>` corresponding to the interpreter's standard - input, output and error streams. ``stdin`` is used for all interpreter input - except for scripts but including calls to :func:`input`. ``stdout`` is used - for the output of :func:`print` and :term:`expression` statements and for the - prompts of :func:`input`. The interpreter's own prompts - and (almost all of) its error messages go to ``stderr``. ``stdout`` and - ``stderr`` needn't be built-in file objects: any object is acceptable as long - as it has a :meth:`write` method that takes a string argument. (Changing these - objects doesn't affect the standard I/O streams of processes executed by - :func:`os.popen`, :func:`os.system` or the :func:`exec\*` family of functions in - the :mod:`os` module.) + :term:`File objects <file object>` used by the interpreter for standard + input, output and errors: - The standard streams are in text mode by default. To write or read binary - data to these, use the underlying binary buffer. For example, to write bytes - to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``. Using - :meth:`io.TextIOBase.detach` streams can be made binary by default. This + * ``stdin`` is used for all interactive input (including calls to + :func:`input`); + * ``stdout`` is used for the output of :func:`print` and :term:`expression` + statements and for the prompts of :func:`input`; + * The interpreter's own prompts and its error messages go to ``stderr``. + + By default, these streams are regular text streams as returned by the + :func:`open` function. Their parameters are chosen as follows: + + * The character encoding is platform-dependent. Under Windows, if the stream + is interactive (that is, if its :meth:`isatty` method returns True), the + console codepage is used, otherwise the ANSI code page. Under other + platforms, the locale encoding is used (see :meth:`locale.getpreferredencoding`). + + Under all platforms though, you can override this value by setting the + :envvar:`PYTHONIOENCODING` environment variable. + + * When interactive, standard streams are line-buffered. Otherwise, they + are block-buffered like regular text files. You can override this + value with the :option:`-u` command-line option. + + To write or read binary data from/to the standard streams, use the + underlying binary :data:`~io.TextIOBase.buffer`. For example, to write + bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``. Using + :meth:`io.TextIOBase.detach`, streams can be made binary by default. This function sets :data:`stdin` and :data:`stdout` to binary:: def make_streams_binary(): sys.stdin = sys.stdin.detach() sys.stdout = sys.stdout.detach() - Note that the streams can be replaced with objects (like - :class:`io.StringIO`) that do not support the - :attr:`~io.BufferedIOBase.buffer` attribute or the + Note that the streams may be replaced with objects (like :class:`io.StringIO`) + that do not support the :attr:`~io.BufferedIOBase.buffer` attribute or the :meth:`~io.BufferedIOBase.detach` method and can raise :exc:`AttributeError` or :exc:`io.UnsupportedOperation`. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13597: Improve documentation of standard streams.
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/7343730185a3 changeset: 73983:7343730185a3 parent: 73981:d22c99e77768 parent: 73982:313fa7ea6c79 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 16:26:03 2011 +0100 summary: Issue #13597: Improve documentation of standard streams. files: Doc/library/sys.rst | 47 ++++++++++++++++++++------------ 1 files changed, 29 insertions(+), 18 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -941,31 +941,42 @@ stdout stderr - :term:`File objects <file object>` corresponding to the interpreter's standard - input, output and error streams. ``stdin`` is used for all interpreter input - except for scripts but including calls to :func:`input`. ``stdout`` is used - for the output of :func:`print` and :term:`expression` statements and for the - prompts of :func:`input`. The interpreter's own prompts - and (almost all of) its error messages go to ``stderr``. ``stdout`` and - ``stderr`` needn't be built-in file objects: any object is acceptable as long - as it has a :meth:`write` method that takes a string argument. (Changing these - objects doesn't affect the standard I/O streams of processes executed by - :func:`os.popen`, :func:`os.system` or the :func:`exec\*` family of functions in - the :mod:`os` module.) + :term:`File objects <file object>` used by the interpreter for standard + input, output and errors: - The standard streams are in text mode by default. To write or read binary - data to these, use the underlying binary buffer. For example, to write bytes - to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``. Using - :meth:`io.TextIOBase.detach` streams can be made binary by default. This + * ``stdin`` is used for all interactive input (including calls to + :func:`input`); + * ``stdout`` is used for the output of :func:`print` and :term:`expression` + statements and for the prompts of :func:`input`; + * The interpreter's own prompts and its error messages go to ``stderr``. + + By default, these streams are regular text streams as returned by the + :func:`open` function. Their parameters are chosen as follows: + + * The character encoding is platform-dependent. Under Windows, if the stream + is interactive (that is, if its :meth:`isatty` method returns True), the + console codepage is used, otherwise the ANSI code page. Under other + platforms, the locale encoding is used (see :meth:`locale.getpreferredencoding`). + + Under all platforms though, you can override this value by setting the + :envvar:`PYTHONIOENCODING` environment variable. + + * When interactive, standard streams are line-buffered. Otherwise, they + are block-buffered like regular text files. You can override this + value with the :option:`-u` command-line option. + + To write or read binary data from/to the standard streams, use the + underlying binary :data:`~io.TextIOBase.buffer`. For example, to write + bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``. Using + :meth:`io.TextIOBase.detach`, streams can be made binary by default. This function sets :data:`stdin` and :data:`stdout` to binary:: def make_streams_binary(): sys.stdin = sys.stdin.detach() sys.stdout = sys.stdout.detach() - Note that the streams can be replaced with objects (like - :class:`io.StringIO`) that do not support the - :attr:`~io.BufferedIOBase.buffer` attribute or the + Note that the streams may be replaced with objects (like :class:`io.StringIO`) + that do not support the :attr:`~io.BufferedIOBase.buffer` attribute or the :meth:`~io.BufferedIOBase.detach` method and can raise :exc:`AttributeError` or :exc:`io.UnsupportedOperation`. -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix OSError.__init__ and OSError.__new__ so that each of them can be
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/d22c99e77768 changeset: 73981:d22c99e77768 parent: 73979:195bfd4c3ea1 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 14:31:28 2011 +0100 summary: Fix OSError.__init__ and OSError.__new__ so that each of them can be overriden and take additional arguments (followup to issue #12555). files: Lib/test/test_pep3151.py | 60 ++++++- Misc/NEWS | 3 + Objects/exceptions.c | 216 +++++++++++++++++++------- 3 files changed, 217 insertions(+), 62 deletions(-) diff --git a/Lib/test/test_pep3151.py b/Lib/test/test_pep3151.py --- a/Lib/test/test_pep3151.py +++ b/Lib/test/test_pep3151.py @@ -12,6 +12,23 @@ class SubOSError(OSError): pass +class SubOSErrorWithInit(OSError): + def __init__(self, message, bar): + self.bar = bar + super().__init__(message) + +class SubOSErrorWithNew(OSError): + def __new__(cls, message, baz): + self = super().__new__(cls, message) + self.baz = baz + return self + +class SubOSErrorCombinedInitFirst(SubOSErrorWithInit, SubOSErrorWithNew): + pass + +class SubOSErrorCombinedNewFirst(SubOSErrorWithNew, SubOSErrorWithInit): + pass + class HierarchyTest(unittest.TestCase): @@ -74,11 +91,6 @@ e = OSError(errcode, "Some message") self.assertIs(type(e), OSError) - def test_OSError_subclass_mapping(self): - # When constructing an OSError subclass, errno mapping isn't done - e = SubOSError(EEXIST, "Bad file descriptor") - self.assertIs(type(e), SubOSError) - def test_try_except(self): filename = "some_hopefully_non_existing_file" @@ -144,6 +156,44 @@ # XXX VMSError not tested +class ExplicitSubclassingTest(unittest.TestCase): + + def test_errno_mapping(self): + # When constructing an OSError subclass, errno mapping isn't done + e = SubOSError(EEXIST, "Bad file descriptor") + self.assertIs(type(e), SubOSError) + + def test_init_overriden(self): + e = SubOSErrorWithInit("some message", "baz") + self.assertEqual(e.bar, "baz") + self.assertEqual(e.args, ("some message",)) + + def test_init_kwdargs(self): + e = SubOSErrorWithInit("some message", bar="baz") + self.assertEqual(e.bar, "baz") + self.assertEqual(e.args, ("some message",)) + + def test_new_overriden(self): + e = SubOSErrorWithNew("some message", "baz") + self.assertEqual(e.baz, "baz") + self.assertEqual(e.args, ("some message",)) + + def test_new_kwdargs(self): + e = SubOSErrorWithNew("some message", baz="baz") + self.assertEqual(e.baz, "baz") + self.assertEqual(e.args, ("some message",)) + + def test_init_new_overriden(self): + e = SubOSErrorCombinedInitFirst("some message", "baz") + self.assertEqual(e.bar, "baz") + self.assertEqual(e.baz, "baz") + self.assertEqual(e.args, ("some message",)) + e = SubOSErrorCombinedNewFirst("some message", "baz") + self.assertEqual(e.bar, "baz") + self.assertEqual(e.baz, "baz") + self.assertEqual(e.args, ("some message",)) + + def test_main(): support.run_unittest(__name__) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Fix OSError.__init__ and OSError.__new__ so that each of them can be + overriden and take additional arguments (followup to issue #12555). + - Fix the fix for issue #12149: it was incorrect, although it had the side effect of appearing to resolve the issue. Thanks to Mark Shannon for noticing. diff --git a/Objects/exceptions.c b/Objects/exceptions.c --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -58,7 +58,7 @@ if (!_PyArg_NoKeywords(Py_TYPE(self)->tp_name, kwds)) return -1; - Py_DECREF(self->args); + Py_XDECREF(self->args); self->args = args; Py_INCREF(self->args); @@ -587,37 +587,34 @@ * when it was supplied. */ -static PyObject * -OSError_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +/* This function doesn't cleanup on error, the caller should */ +static int +oserror_parse_args(PyObject **p_args, + PyObject **myerrno, PyObject **strerror, + PyObject **filename +#ifdef MS_WINDOWS + , PyObject **winerror +#endif + ) { - PyOSErrorObject *self = NULL; Py_ssize_t nargs; + PyObject *args = *p_args; - PyObject *myerrno = NULL, *strerror = NULL, *filename = NULL; - PyObject *subslice = NULL; -#ifdef MS_WINDOWS - PyObject *winerror = NULL; - long winerrcode = 0; -#endif - - if (!_PyArg_NoKeywords(type->tp_name, kwds)) - return NULL; - Py_INCREF(args); nargs = PyTuple_GET_SIZE(args); #ifdef MS_WINDOWS if (nargs >= 2 && nargs <= 4) { if (!PyArg_UnpackTuple(args, "OSError", 2, 4, - &myerrno, &strerror, &filename, &winerror)) - goto error; - if (winerror && PyLong_Check(winerror)) { - long errcode; + myerrno, strerror, filename, winerror)) + return -1; + if (*winerror && PyLong_Check(*winerror)) { + long errcode, winerrcode; PyObject *newargs; Py_ssize_t i; - winerrcode = PyLong_AsLong(winerror); + winerrcode = PyLong_AsLong(*winerror); if (winerrcode == -1 && PyErr_Occurred()) - goto error; + return -1; /* Set errno to the corresponding POSIX errno (overriding first argument). Windows Socket error codes (>= 10000) have the same value as their POSIX counterparts. @@ -626,59 +623,55 @@ errcode = winerror_to_errno(winerrcode); else errcode = winerrcode; - myerrno = PyLong_FromLong(errcode); - if (!myerrno) - goto error; + *myerrno = PyLong_FromLong(errcode); + if (!*myerrno) + return -1; newargs = PyTuple_New(nargs); if (!newargs) - goto error; - PyTuple_SET_ITEM(newargs, 0, myerrno); + return -1; + PyTuple_SET_ITEM(newargs, 0, *myerrno); for (i = 1; i < nargs; i++) { PyObject *val = PyTuple_GET_ITEM(args, i); Py_INCREF(val); PyTuple_SET_ITEM(newargs, i, val); } Py_DECREF(args); - args = newargs; + args = *p_args = newargs; } } #else if (nargs >= 2 && nargs <= 3) { if (!PyArg_UnpackTuple(args, "OSError", 2, 3, - &myerrno, &strerror, &filename)) - goto error; + myerrno, strerror, filename)) + return -1; } #endif - if (myerrno && PyLong_Check(myerrno) && - errnomap && (PyObject *) type == PyExc_OSError) { - PyObject *newtype; - newtype = PyDict_GetItem(errnomap, myerrno); - if (newtype) { - assert(PyType_Check(newtype)); - type = (PyTypeObject *) newtype; - } - else if (PyErr_Occurred()) - goto error; - } - self = (PyOSErrorObject *) type->tp_alloc(type, 0); - if (!self) - goto error; + return 0; +} - self->dict = NULL; - self->traceback = self->cause = self->context = NULL; - self->written = -1; +static int +oserror_init(PyOSErrorObject *self, PyObject **p_args, + PyObject *myerrno, PyObject *strerror, + PyObject *filename +#ifdef MS_WINDOWS + , PyObject *winerror +#endif + ) +{ + PyObject *args = *p_args; + Py_ssize_t nargs = PyTuple_GET_SIZE(args); /* self->filename will remain Py_None otherwise */ if (filename && filename != Py_None) { - if ((PyObject *) type == PyExc_BlockingIOError && + if (Py_TYPE(self) == (PyTypeObject *) PyExc_BlockingIOError && PyNumber_Check(filename)) { /* BlockingIOError's 3rd argument can be the number of * characters written. */ self->written = PyNumber_AsSsize_t(filename, PyExc_ValueError); if (self->written == -1 && PyErr_Occurred()) - goto error; + return -1; } else { Py_INCREF(filename); @@ -687,20 +680,15 @@ if (nargs >= 2 && nargs <= 3) { /* filename is removed from the args tuple (for compatibility purposes, see test_exceptions.py) */ - subslice = PyTuple_GetSlice(args, 0, 2); + PyObject *subslice = PyTuple_GetSlice(args, 0, 2); if (!subslice) - goto error; + return -1; Py_DECREF(args); /* replacing args */ - args = subslice; + *p_args = args = subslice; } } } - - /* Steals the reference to args */ - self->args = args; - args = NULL; - Py_XINCREF(myerrno); self->myerrno = myerrno; @@ -712,6 +700,90 @@ self->winerror = winerror; #endif + /* Steals the reference to args */ + self->args = args; + args = NULL; + + return 0; +} + +static PyObject * +OSError_new(PyTypeObject *type, PyObject *args, PyObject *kwds); +static int +OSError_init(PyOSErrorObject *self, PyObject *args, PyObject *kwds); + +static int +oserror_use_init(PyTypeObject *type) +{ + /* When __init__ is defined in a OSError subclass, we want any + extraneous argument to __new__ to be ignored. The only reasonable + solution, given __new__ takes a variable number of arguments, + is to defer arg parsing and initialization to __init__. + + But when __new__ is overriden as well, it should call our __new__ + with the right arguments. + + (see http://bugs.python.org/issue12555#msg148829 ) + */ + if (type->tp_init != (initproc) OSError_init && + type->tp_new == (newfunc) OSError_new) { + assert((PyObject *) type != PyExc_OSError); + return 1; + } + return 0; +} + +static PyObject * +OSError_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +{ + PyOSErrorObject *self = NULL; + PyObject *myerrno = NULL, *strerror = NULL, *filename = NULL; +#ifdef MS_WINDOWS + PyObject *winerror = NULL; +#endif + + if (!oserror_use_init(type)) { + if (!_PyArg_NoKeywords(type->tp_name, kwds)) + return NULL; + + Py_INCREF(args); + if (oserror_parse_args(&args, &myerrno, &strerror, &filename +#ifdef MS_WINDOWS + , &winerror +#endif + )) + goto error; + + if (myerrno && PyLong_Check(myerrno) && + errnomap && (PyObject *) type == PyExc_OSError) { + PyObject *newtype; + newtype = PyDict_GetItem(errnomap, myerrno); + if (newtype) { + assert(PyType_Check(newtype)); + type = (PyTypeObject *) newtype; + } + else if (PyErr_Occurred()) + goto error; + } + } + + self = (PyOSErrorObject *) type->tp_alloc(type, 0); + if (!self) + goto error; + + self->dict = NULL; + self->traceback = self->cause = self->context = NULL; + self->written = -1; + + if (!oserror_use_init(type)) { + if (oserror_init(self, &args, myerrno, strerror, filename +#ifdef MS_WINDOWS + , winerror +#endif + )) + goto error; + } + return (PyObject *) self; error: @@ -721,10 +793,40 @@ } static int -OSError_init(PySyntaxErrorObject *self, PyObject *args, PyObject *kwds) +OSError_init(PyOSErrorObject *self, PyObject *args, PyObject *kwds) { - /* Everything already done in OSError_new */ + PyObject *myerrno = NULL, *strerror = NULL, *filename = NULL; +#ifdef MS_WINDOWS + PyObject *winerror = NULL; +#endif + + if (!oserror_use_init(Py_TYPE(self))) + /* Everything already done in OSError_new */ + return 0; + + if (!_PyArg_NoKeywords(Py_TYPE(self)->tp_name, kwds)) + return -1; + + Py_INCREF(args); + if (oserror_parse_args(&args, &myerrno, &strerror, &filename +#ifdef MS_WINDOWS + , &winerror +#endif + )) + goto error; + + if (oserror_init(self, &args, myerrno, strerror, filename +#ifdef MS_WINDOWS + , winerror +#endif + )) + goto error; + return 0; + +error: + Py_XDECREF(args); + return -1; } static int -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Fix the fix for issue #12149: it was incorrect, although it had the side
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/da1d7f8cd487 changeset: 73980:da1d7f8cd487 branch: 2.7 parent: 73970:f6ca22bbf138 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 14:15:31 2011 +0100 summary: Fix the fix for issue #12149: it was incorrect, although it had the side effect of appearing to resolve the issue. Thanks to Mark Shannon for noticing. files: Misc/NEWS | 4 ++++ Objects/typeobject.c | 18 ++++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,10 @@ Core and Builtins ----------------- +- Fix the fix for issue #12149: it was incorrect, although it had the side + effect of appearing to resolve the issue. Thanks to Mark Shannon for + noticing. + - Issue #13546: Fixed an overflow issue that could crash the intepreter when calling sys.setrecursionlimit((1<<31)-1). diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -1013,8 +1013,6 @@ assert(basedealloc); basedealloc(self); - PyType_Modified(type); - /* Can't reference self beyond this point */ Py_DECREF(type); @@ -2707,15 +2705,16 @@ for heaptypes. */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); - /* The only field we need to clear is tp_mro, which is part of a - hard cycle (its first element is the class itself) that won't - be broken otherwise (it's a tuple and tuples don't have a + /* We need to invalidate the method cache carefully before clearing + the dict, so that other objects caught in a reference cycle + don't start calling destroyed methods. + + Otherwise, the only field we need to clear is tp_mro, which is + part of a hard cycle (its first element is the class itself) that + won't be broken otherwise (it's a tuple and tuples don't have a tp_clear handler). None of the other fields need to be cleared, and here's why: - tp_dict: - It is a dict, so the collector will call its tp_clear. - tp_cache: Not used; if it were, it would be a dict. @@ -2732,6 +2731,9 @@ A tuple of strings can't be part of a cycle. */ + PyType_Modified(type); + if (type->tp_dict) + PyDict_Clear(type->tp_dict); Py_CLEAR(type->tp_mro); return 0; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Fix the fix for issue #12149: it was incorrect, although it had the side
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/195bfd4c3ea1 changeset: 73979:195bfd4c3ea1 parent: 73977:54a77c556d9a parent: 73978:b196bcd7c34f user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 14:17:36 2011 +0100 summary: Fix the fix for issue #12149: it was incorrect, although it had the side effect of appearing to resolve the issue. Thanks to Mark Shannon for noticing. files: Misc/NEWS | 4 ++++ Objects/typeobject.c | 18 ++++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Fix the fix for issue #12149: it was incorrect, although it had the side + effect of appearing to resolve the issue. Thanks to Mark Shannon for + noticing. + - Issue #13505: Pickle bytes objects in a way that is compatible with Python 2 when using protocols <= 2. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -999,8 +999,6 @@ assert(basedealloc); basedealloc(self); - PyType_Modified(type); - /* Can't reference self beyond this point */ Py_DECREF(type); @@ -2776,15 +2774,16 @@ for heaptypes. */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); - /* The only field we need to clear is tp_mro, which is part of a - hard cycle (its first element is the class itself) that won't - be broken otherwise (it's a tuple and tuples don't have a + /* We need to invalidate the method cache carefully before clearing + the dict, so that other objects caught in a reference cycle + don't start calling destroyed methods. + + Otherwise, the only field we need to clear is tp_mro, which is + part of a hard cycle (its first element is the class itself) that + won't be broken otherwise (it's a tuple and tuples don't have a tp_clear handler). None of the other fields need to be cleared, and here's why: - tp_dict: - It is a dict, so the collector will call its tp_clear. - tp_cache: Not used; if it were, it would be a dict. @@ -2801,6 +2800,9 @@ A tuple of strings can't be part of a cycle. */ + PyType_Modified(type); + if (type->tp_dict) + PyDict_Clear(type->tp_dict); Py_CLEAR(type->tp_mro); return 0; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix the fix for issue #12149: it was incorrect, although it had the side
by antoine.pitrou Dec. 16, 2011

Dec. 16, 2011

http://hg.python.org/cpython/rev/b196bcd7c34f changeset: 73978:b196bcd7c34f branch: 3.2 parent: 73975:541f215a31f7 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 14:15:31 2011 +0100 summary: Fix the fix for issue #12149: it was incorrect, although it had the side effect of appearing to resolve the issue. Thanks to Mark Shannon for noticing. files: Misc/NEWS | 4 ++++ Objects/typeobject.c | 18 ++++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Fix the fix for issue #12149: it was incorrect, although it had the side + effect of appearing to resolve the issue. Thanks to Mark Shannon for + noticing. + - Issue #13505: Pickle bytes objects in a way that is compatible with Python 2 when using protocols <= 2. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -967,8 +967,6 @@ assert(basedealloc); basedealloc(self); - PyType_Modified(type); - /* Can't reference self beyond this point */ Py_DECREF(type); @@ -2645,15 +2643,16 @@ for heaptypes. */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); - /* The only field we need to clear is tp_mro, which is part of a - hard cycle (its first element is the class itself) that won't - be broken otherwise (it's a tuple and tuples don't have a + /* We need to invalidate the method cache carefully before clearing + the dict, so that other objects caught in a reference cycle + don't start calling destroyed methods. + + Otherwise, the only field we need to clear is tp_mro, which is + part of a hard cycle (its first element is the class itself) that + won't be broken otherwise (it's a tuple and tuples don't have a tp_clear handler). None of the other fields need to be cleared, and here's why: - tp_dict: - It is a dict, so the collector will call its tp_clear. - tp_cache: Not used; if it were, it would be a dict. @@ -2670,6 +2669,9 @@ A tuple of strings can't be part of a cycle. */ + PyType_Modified(type); + if (type->tp_dict) + PyDict_Clear(type->tp_dict); Py_CLEAR(type->tp_mro); return 0; -- Repository URL: http://hg.python.org/cpython

1 0

benchmarks: Duplicate lines
by antoine.pitrou Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/benchmarks/rev/eec31e939b64 changeset: 153:eec31e939b64 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 13:42:10 2011 +0100 summary: Duplicate lines files: performance/bm_logging.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/performance/bm_logging.py b/performance/bm_logging.py --- a/performance/bm_logging.py +++ b/performance/bm_logging.py @@ -98,8 +98,6 @@ else: raise RuntimeError("Need to specify one of %s" % benchmarks) - logger = logging.getLogger("benchlogger") - logger.propagate = False # NOTE: StringIO performance will impact the results... if sys.version_info >= (3,): sio = io.StringIO() -- Repository URL: http://hg.python.org/benchmarks

1 0

benchmarks: Add a logging benchmark
by antoine.pitrou Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/benchmarks/rev/135d44d0c02a changeset: 152:135d44d0c02a user: Antoine Pitrou <solipsis(a)pitrou.net> date: Thu Dec 15 13:17:14 2011 +0100 summary: Add a logging benchmark files: perf.py | 49 ++++++++++- performance/bm_logging.py | 119 ++++++++++++++++++++++++++ 2 files changed, 167 insertions(+), 1 deletions(-) diff --git a/perf.py b/perf.py --- a/perf.py +++ b/perf.py @@ -1597,6 +1597,52 @@ def BM_NQueens(*args, **kwargs): return SimpleBenchmark(MeasureNQueens, *args, **kwargs) +def MeasureLogging(python, options, extra_args): + """Test the performance of Python's logging module. + + Args: + python: prefix of a command line for the Python binary. + options: optparse.Values instance. + extra_args: list of arguments to append to the command line. + + Returns: + RawData instance. + """ + bm_path = Relative("performance/bm_logging.py") + return MeasureGeneric(python, options, bm_path, extra_args=extra_args) + + +def _LoggingBenchmark(base_python, changed_python, options, extra_args): + """Test the performance of Python's logging module. + + Args: + base_python: prefix of a command line for the reference + Python binary. + changed_python: prefix of a command line for the + experimental Python binary. + options: optparse.Values instance. + extra_args: list of arguments to append to the command line. + + Returns: + Summary of whether the experiemental Python is better/worse than the + baseline. + """ + return SimpleBenchmark(MeasureLogging, + base_python, changed_python, options, extra_args) + + +def BM_Silent_Logging(base_python, changed_python, options): + args = ["no_output"] + return _LoggingBenchmark(base_python, changed_python, options, args) + +def BM_Simple_Logging(base_python, changed_python, options): + args = ["simple_output"] + return _LoggingBenchmark(base_python, changed_python, options, args) + +def BM_Formatted_Logging(base_python, changed_python, options): + args = ["formatted_output"] + return _LoggingBenchmark(base_python, changed_python, options, args) + def _StartupPython(command, mem_usage, track_memory, inherit_env): startup_env = BuildEnv(inherit_env=inherit_env) @@ -1935,11 +1981,12 @@ "call_method_unknown"], "math": ["nbody", "float", "pidigits"], "template" : ["slowspitfire", "django", "mako"], + "logging": ["silent_logging", "simple_logging", "formatted_logging"], # Benchmarks natively 2.x- and 3.x-compatible "2n3": ["calls", "math", "fastpickle", "fastunpickle", "json_dump", "json_load", "regex", "threading", "nqueens", "unpack_sequence", "richards", - "normal_startup", "startup_nosite"], + "logging", "normal_startup", "startup_nosite"], # After 2to3-conversion "py3k": ["2to3", "2n3", "mako"] } diff --git a/performance/bm_logging.py b/performance/bm_logging.py new file mode 100644 --- /dev/null +++ b/performance/bm_logging.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python + +"""Script for testing the performance of logging simple messages. +""" + +# Python imports +import io +import logging +import optparse +import sys +import time + +# Local imports +import util +from compat import xrange + +# A simple format for parametered logging +FORMAT = 'important: %s' +MESSAGE = 'some important information to be logged' + + +def test_no_output(iterations, logger): + times = [] + m = MESSAGE + for _ in xrange(iterations): + t0 = time.time() + for _ in xrange(10000): + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + logger.debug(m) + t1 = time.time() + times.append(t1 - t0) + return times + + +def test_simple_output(iterations, logger): + times = [] + m = MESSAGE + for _ in xrange(iterations): + t0 = time.time() + for _ in xrange(1000): + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + logger.warn(m) + t1 = time.time() + times.append(t1 - t0) + return times + + +def test_formatted_output(iterations, logger): + times = [] + f = FORMAT + m = MESSAGE + for _ in xrange(iterations): + t0 = time.time() + for _ in xrange(1000): + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + logger.warn(f, m) + t1 = time.time() + times.append(t1 - t0) + return times + + +if __name__ == "__main__": + parser = optparse.OptionParser( + usage="%prog [no_output|simple_output|formatted_output] [options]", + description=("Test the performance of logging.")) + util.add_standard_options_to(parser) + options, args = parser.parse_args() + + benchmarks = ["no_output", "simple_output", "formatted_output"] + for bench_name in benchmarks: + if bench_name in args: + benchmark = globals()["test_" + bench_name] + break + else: + raise RuntimeError("Need to specify one of %s" % benchmarks) + + logger = logging.getLogger("benchlogger") + logger.propagate = False + # NOTE: StringIO performance will impact the results... + if sys.version_info >= (3,): + sio = io.StringIO() + else: + sio = io.BytesIO() + handler = logging.StreamHandler(stream=sio) + logger = logging.getLogger("benchlogger") + logger.propagate = False + logger.addHandler(handler) + logger.setLevel(logging.WARNING) + + util.run_benchmark(options, options.num_runs, benchmark, logger) + + if benchmark is not test_no_output: + assert len(sio.getvalue()) > 0 + else: + assert len(sio.getvalue()) == 0 -- Repository URL: http://hg.python.org/benchmarks

1 0

cpython: Issue #13593: updating the importlib utility decorators for __qualname__.
by meador.inge Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/54a77c556d9a changeset: 73977:54a77c556d9a user: Meador Inge <meadori(a)gmail.com> date: Wed Dec 14 22:53:13 2011 -0600 summary: Issue #13593: updating the importlib utility decorators for __qualname__. files: Lib/importlib/_bootstrap.py | 2 +- Lib/importlib/test/test_util.py | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -111,7 +111,7 @@ def _wrap(new, old): """Simple substitute for functools.wraps.""" - for replace in ['__module__', '__name__', '__doc__']: + for replace in ['__module__', '__name__', '__qualname__', '__doc__']: setattr(new, replace, getattr(old, replace)) new.__dict__.update(old.__dict__) diff --git a/Lib/importlib/test/test_util.py b/Lib/importlib/test/test_util.py --- a/Lib/importlib/test/test_util.py +++ b/Lib/importlib/test/test_util.py @@ -59,6 +59,11 @@ self.raise_exception(name) self.assertIs(module, sys.modules[name]) + def test_decorator_attrs(self): + def fxn(self, module): pass + wrapped = util.module_for_loader(fxn) + self.assertEqual(wrapped.__name__, fxn.__name__) + self.assertEqual(wrapped.__qualname__, fxn.__qualname__) class SetPackageTests(unittest.TestCase): @@ -108,6 +113,11 @@ module.__package__ = value self.verify(module, value) + def test_decorator_attrs(self): + def fxn(module): pass + wrapped = util.set_package(fxn) + self.assertEqual(wrapped.__name__, fxn.__name__) + self.assertEqual(wrapped.__qualname__, fxn.__qualname__) def test_main(): from test import support -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13591: Moving the NEWS line to the right release.
by meador.inge Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/92e94fd303d4 changeset: 73976:92e94fd303d4 parent: 73974:e8fb61a0a2d7 parent: 73975:541f215a31f7 user: Meador Inge <meadori(a)gmail.com> date: Wed Dec 14 22:39:28 2011 -0600 summary: Issue #13591: Moving the NEWS line to the right release. files: Misc/NEWS | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -409,6 +409,9 @@ Library ------- +- Issue #13591: A bug in importlib has been fixed that caused import_module + to load a module twice. + - Issue #4625: If IDLE cannot write to its recent file or breakpoint files, display a message popup and continue rather than crash. (original patch by Roger Serwy) @@ -2548,9 +2551,6 @@ Library ------- -- Issue #13591: A bug in importlib has been fixed that caused import_module - to load a module twice. - - logging: added "handler of last resort". See http://bit.ly/last-resort-handler - test.support: Added TestHandler and Matcher classes for better support of -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13591: Moving the NEWS line to the right release.
by meador.inge Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/541f215a31f7 changeset: 73975:541f215a31f7 branch: 3.2 parent: 73973:d2504d30f259 user: Meador Inge <meadori(a)gmail.com> date: Wed Dec 14 22:37:48 2011 -0600 summary: Issue #13591: Moving the NEWS line to the right release. files: Misc/NEWS | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -93,6 +93,9 @@ Library ------- +- Issue #13591: A bug in importlib has been fixed that caused import_module + to load a module twice. + - Issue #4625: If IDLE cannot write to its recent file or breakpoint files, display a message popup and continue rather than crash. (original patch by Roger Serwy) @@ -1868,9 +1871,6 @@ Library ------- -- Issue #13591: A bug in importlib has been fixed that caused import_module - to load a module twice. - - logging: added "handler of last resort". See http://bit.ly/last-resort-handler - test.support: Added TestHandler and Matcher classes for better support of -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (f6510cdf4ada): sum=0
by solipsis＠pitrou.net Dec. 15, 2011

Dec. 15, 2011

results for f6510cdf4ada on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogGobnbL', '-x']

1 0

cpython (merge 3.2 -> default): Issue #13591: import_module potentially imports a module twice.
by meador.inge Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/e8fb61a0a2d7 changeset: 73974:e8fb61a0a2d7 parent: 73972:f6510cdf4ada parent: 73973:d2504d30f259 user: Meador Inge <meadori(a)gmail.com> date: Wed Dec 14 22:27:28 2011 -0600 summary: Issue #13591: import_module potentially imports a module twice. files: Lib/importlib/_bootstrap.py | 4 +++- Lib/importlib/test/test_api.py | 17 +++++++++++++++++ Lib/importlib/test/util.py | 7 ++++++- Misc/NEWS | 3 +++ 4 files changed, 29 insertions(+), 2 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -837,7 +837,9 @@ for finder in meta_path: loader = finder.find_module(name, path) if loader is not None: - loader.load_module(name) + # The parent import may have already imported this module. + if name not in sys.modules: + loader.load_module(name) break else: raise ImportError(_ERR_MSG.format(name)) diff --git a/Lib/importlib/test/test_api.py b/Lib/importlib/test/test_api.py --- a/Lib/importlib/test/test_api.py +++ b/Lib/importlib/test/test_api.py @@ -67,6 +67,23 @@ importlib.import_module('.support') + def test_loaded_once(self): + # Issue #13591: Modules should only be loaded once when + # initializing the parent package attempts to import the + # module currently being imported. + b_load_count = 0 + def load_a(): + importlib.import_module('a.b') + def load_b(): + nonlocal b_load_count + b_load_count += 1 + code = {'a': load_a, 'a.b': load_b} + modules = ['a.__init__', 'a.b'] + with util.mock_modules(*modules, module_code=code) as mock: + with util.import_state(meta_path=[mock]): + importlib.import_module('a.b') + self.assertEqual(b_load_count, 1) + def test_main(): from test.support import run_unittest run_unittest(ImportModuleTests) diff --git a/Lib/importlib/test/util.py b/Lib/importlib/test/util.py --- a/Lib/importlib/test/util.py +++ b/Lib/importlib/test/util.py @@ -84,8 +84,9 @@ """A mock importer/loader.""" - def __init__(self, *names): + def __init__(self, *names, module_code={}): self.modules = {} + self.module_code = {} for name in names: if not name.endswith('.__init__'): import_name = name @@ -105,6 +106,8 @@ if import_name != name: module.__path__ = ['<mock __path__>'] self.modules[import_name] = module + if import_name in module_code: + self.module_code[import_name] = module_code[import_name] def __getitem__(self, name): return self.modules[name] @@ -120,6 +123,8 @@ raise ImportError else: sys.modules[fullname] = self.modules[fullname] + if fullname in self.module_code: + self.module_code[fullname]() return self.modules[fullname] def __enter__(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2548,6 +2548,9 @@ Library ------- +- Issue #13591: A bug in importlib has been fixed that caused import_module + to load a module twice. + - logging: added "handler of last resort". See http://bit.ly/last-resort-handler - test.support: Added TestHandler and Matcher classes for better support of -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13591: import_module potentially imports a module twice.
by meador.inge Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/d2504d30f259 changeset: 73973:d2504d30f259 branch: 3.2 parent: 73971:f1fe411bfd6b user: Meador Inge <meadori(a)gmail.com> date: Wed Dec 14 22:23:46 2011 -0600 summary: Issue #13591: import_module potentially imports a module twice. files: Lib/importlib/_bootstrap.py | 4 +++- Lib/importlib/test/test_api.py | 17 +++++++++++++++++ Lib/importlib/test/util.py | 7 ++++++- Misc/NEWS | 3 +++ 4 files changed, 29 insertions(+), 2 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -816,7 +816,9 @@ for finder in meta_path: loader = finder.find_module(name, path) if loader is not None: - loader.load_module(name) + # The parent import may have already imported this module. + if name not in sys.modules: + loader.load_module(name) break else: raise ImportError(_ERR_MSG.format(name)) diff --git a/Lib/importlib/test/test_api.py b/Lib/importlib/test/test_api.py --- a/Lib/importlib/test/test_api.py +++ b/Lib/importlib/test/test_api.py @@ -67,6 +67,23 @@ importlib.import_module('.support') + def test_loaded_once(self): + # Issue #13591: Modules should only be loaded once when + # initializing the parent package attempts to import the + # module currently being imported. + b_load_count = 0 + def load_a(): + importlib.import_module('a.b') + def load_b(): + nonlocal b_load_count + b_load_count += 1 + code = {'a': load_a, 'a.b': load_b} + modules = ['a.__init__', 'a.b'] + with util.mock_modules(*modules, module_code=code) as mock: + with util.import_state(meta_path=[mock]): + importlib.import_module('a.b') + self.assertEqual(b_load_count, 1) + def test_main(): from test.support import run_unittest run_unittest(ImportModuleTests) diff --git a/Lib/importlib/test/util.py b/Lib/importlib/test/util.py --- a/Lib/importlib/test/util.py +++ b/Lib/importlib/test/util.py @@ -84,8 +84,9 @@ """A mock importer/loader.""" - def __init__(self, *names): + def __init__(self, *names, module_code={}): self.modules = {} + self.module_code = {} for name in names: if not name.endswith('.__init__'): import_name = name @@ -105,6 +106,8 @@ if import_name != name: module.__path__ = ['<mock __path__>'] self.modules[import_name] = module + if import_name in module_code: + self.module_code[import_name] = module_code[import_name] def __getitem__(self, name): return self.modules[name] @@ -120,6 +123,8 @@ raise ImportError else: sys.modules[fullname] = self.modules[fullname] + if fullname in self.module_code: + self.module_code[fullname]() return self.modules[fullname] def __enter__(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1868,6 +1868,9 @@ Library ------- +- Issue #13591: A bug in importlib has been fixed that caused import_module + to load a module twice. + - logging: added "handler of last resort". See http://bit.ly/last-resort-handler - test.support: Added TestHandler and Matcher classes for better support of -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #4625: Add NEWS entry.
by ned.deily Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/f6510cdf4ada changeset: 73972:f6510cdf4ada parent: 73969:ba832d42cd9e parent: 73971:f1fe411bfd6b user: Ned Deily <nad(a)acm.org> date: Wed Dec 14 15:14:52 2011 -0800 summary: Issue #4625: Add NEWS entry. files: Misc/NEWS | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -409,6 +409,10 @@ Library ------- +- Issue #4625: If IDLE cannot write to its recent file or breakpoint + files, display a message popup and continue rather than crash. + (original patch by Roger Serwy) + - Issue #13449 sched.scheduler.run() method has a new "blocking" parameter which when set to False makes run() execute the scheduled events due to expire soonest (if any) and then return. Patch by Giampaolo Rodolà. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #4625: add NEWS entry.
by ned.deily Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/f1fe411bfd6b changeset: 73971:f1fe411bfd6b branch: 3.2 parent: 73968:26e3e542d20d user: Ned Deily <nad(a)acm.org> date: Wed Dec 14 15:06:50 2011 -0800 summary: Issue #4625: add NEWS entry. files: Misc/NEWS | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -93,6 +93,10 @@ Library ------- +- Issue #4625: If IDLE cannot write to its recent file or breakpoint + files, display a message popup and continue rather than crash. + (original patch by Roger Serwy) + - Issue #5905: time.strftime() is now using the locale encoding, instead of UTF-8, if the wcsftime() function is not available. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #4625: add NEWS entry.
by ned.deily Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/f6ca22bbf138 changeset: 73970:f6ca22bbf138 branch: 2.7 parent: 73967:7857d1f4ce79 user: Ned Deily <nad(a)acm.org> date: Wed Dec 14 15:05:42 2011 -0800 summary: Issue #4625: add NEWS entry. files: Misc/NEWS | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -82,6 +82,10 @@ Library ------- +- Issue #4625: If IDLE cannot write to its recent file or breakpoint + files, display a message popup and continue rather than crash. + (original patch by Roger Serwy) + - tarfile.py: Correctly detect bzip2 compressed streams with blocksizes other than 900k. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #4625: If IDLE cannot write to its recent file or breakpoint
by ned.deily Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/ba832d42cd9e changeset: 73969:ba832d42cd9e parent: 73966:1eae8154c109 parent: 73968:26e3e542d20d user: Ned Deily <nad(a)acm.org> date: Wed Dec 14 15:03:31 2011 -0800 summary: Issue #4625: If IDLE cannot write to its recent file or breakpoint files, display a message popup and continue rather than crash. (original patch by Roger Serwy) files: Lib/idlelib/EditorWindow.py | 15 ++++++++++----- Lib/idlelib/PyShell.py | 24 ++++++++++++++++-------- 2 files changed, 26 insertions(+), 13 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -799,12 +799,17 @@ rf_list = [path for path in rf_list if path not in bad_paths] ulchars = "1234567890ABCDEFGHIJK" rf_list = rf_list[0:len(ulchars)] - rf_file = open(self.recent_files_path, 'w', - encoding='utf_8', errors='replace') try: - rf_file.writelines(rf_list) - finally: - rf_file.close() + with open(self.recent_files_path, 'w', + encoding='utf_8', errors='replace') as rf_file: + rf_file.writelines(rf_list) + except IOError as err: + if not getattr(self.root, "recentfilelist_error_displayed", False): + self.root.recentfilelist_error_displayed = True + tkMessageBox.showerror(title='IDLE Error', + message='Unable to update Recent Files list:\n%s' + % str(err), + parent=self.text) # for each edit window instance, construct the recent files menu for instance in self.top.instance_dict: menu = instance.recent_files_menu diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -206,14 +206,22 @@ lines = fp.readlines() except IOError: lines = [] - with open(self.breakpointPath, "w") as new_file: - for line in lines: - if not line.startswith(filename + '='): - new_file.write(line) - self.update_breakpoints() - breaks = self.breakpoints - if breaks: - new_file.write(filename + '=' + str(breaks) + '\n') + try: + with open(self.breakpointPath, "w") as new_file: + for line in lines: + if not line.startswith(filename + '='): + new_file.write(line) + self.update_breakpoints() + breaks = self.breakpoints + if breaks: + new_file.write(filename + '=' + str(breaks) + '\n') + except IOError as err: + if not getattr(self.root, "breakpoint_error_displayed", False): + self.root.breakpoint_error_displayed = True + tkMessageBox.showerror(title='IDLE Error', + message='Unable to update breakpoint list:\n%s' + % str(err), + parent=self.text) def restore_file_breaks(self): self.text.update() # this enables setting "BREAK" tags to be visible -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #4625: If IDLE cannot write to its recent file or breakpoint
by ned.deily Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/7857d1f4ce79 changeset: 73967:7857d1f4ce79 branch: 2.7 parent: 73964:fbcaeb4a8654 user: Ned Deily <nad(a)acm.org> date: Wed Dec 14 14:57:43 2011 -0800 summary: Issue #4625: If IDLE cannot write to its recent file or breakpoint files, display a message popup and continue rather than crash. (original patch by Roger Serwy) files: Lib/idlelib/EditorWindow.py | 13 ++++++++--- Lib/idlelib/PyShell.py | 28 ++++++++++++++++-------- 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -796,11 +796,16 @@ rf_list = [path for path in rf_list if path not in bad_paths] ulchars = "1234567890ABCDEFGHIJK" rf_list = rf_list[0:len(ulchars)] - rf_file = open(self.recent_files_path, 'w') try: - rf_file.writelines(rf_list) - finally: - rf_file.close() + with open(self.recent_files_path, 'w') as rf_file: + rf_file.writelines(rf_list) + except IOError as err: + if not getattr(self.root, "recentfilelist_error_displayed", False): + self.root.recentfilelist_error_displayed = True + tkMessageBox.showerror(title='IDLE Error', + message='Unable to update Recent Files list:\n%s' + % str(err), + parent=self.text) # for each edit window instance, construct the recent files menu for instance in self.top.instance_dict.keys(): menu = instance.recent_files_menu diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -207,18 +207,26 @@ breaks = self.breakpoints filename = self.io.filename try: - lines = open(self.breakpointPath,"r").readlines() + with open(self.breakpointPath,"r") as old_file: + lines = old_file.readlines() except IOError: lines = [] - new_file = open(self.breakpointPath,"w") - for line in lines: - if not line.startswith(filename + '='): - new_file.write(line) - self.update_breakpoints() - breaks = self.breakpoints - if breaks: - new_file.write(filename + '=' + str(breaks) + '\n') - new_file.close() + try: + with open(self.breakpointPath,"w") as new_file: + for line in lines: + if not line.startswith(filename + '='): + new_file.write(line) + self.update_breakpoints() + breaks = self.breakpoints + if breaks: + new_file.write(filename + '=' + str(breaks) + '\n') + except IOError as err: + if not getattr(self.root, "breakpoint_error_displayed", False): + self.root.breakpoint_error_displayed = True + tkMessageBox.showerror(title='IDLE Error', + message='Unable to update breakpoint list:\n%s' + % str(err), + parent=self.text) def restore_file_breaks(self): self.text.update() # this enables setting "BREAK" tags to be visible -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #4625: If IDLE cannot write to its recent file or breakpoint
by ned.deily Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/26e3e542d20d changeset: 73968:26e3e542d20d branch: 3.2 parent: 73965:e497a3ed9beb user: Ned Deily <nad(a)acm.org> date: Wed Dec 14 14:58:24 2011 -0800 summary: Issue #4625: If IDLE cannot write to its recent file or breakpoint files, display a message popup and continue rather than crash. (original patch by Roger Serwy) files: Lib/idlelib/EditorWindow.py | 15 ++++++++++----- Lib/idlelib/PyShell.py | 24 ++++++++++++++++-------- 2 files changed, 26 insertions(+), 13 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -799,12 +799,17 @@ rf_list = [path for path in rf_list if path not in bad_paths] ulchars = "1234567890ABCDEFGHIJK" rf_list = rf_list[0:len(ulchars)] - rf_file = open(self.recent_files_path, 'w', - encoding='utf_8', errors='replace') try: - rf_file.writelines(rf_list) - finally: - rf_file.close() + with open(self.recent_files_path, 'w', + encoding='utf_8', errors='replace') as rf_file: + rf_file.writelines(rf_list) + except IOError as err: + if not getattr(self.root, "recentfilelist_error_displayed", False): + self.root.recentfilelist_error_displayed = True + tkMessageBox.showerror(title='IDLE Error', + message='Unable to update Recent Files list:\n%s' + % str(err), + parent=self.text) # for each edit window instance, construct the recent files menu for instance in self.top.instance_dict: menu = instance.recent_files_menu diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -206,14 +206,22 @@ lines = fp.readlines() except IOError: lines = [] - with open(self.breakpointPath, "w") as new_file: - for line in lines: - if not line.startswith(filename + '='): - new_file.write(line) - self.update_breakpoints() - breaks = self.breakpoints - if breaks: - new_file.write(filename + '=' + str(breaks) + '\n') + try: + with open(self.breakpointPath, "w") as new_file: + for line in lines: + if not line.startswith(filename + '='): + new_file.write(line) + self.update_breakpoints() + breaks = self.breakpoints + if breaks: + new_file.write(filename + '=' + str(breaks) + '\n') + except IOError as err: + if not getattr(self.root, "breakpoint_error_displayed", False): + self.root.breakpoint_error_displayed = True + tkMessageBox.showerror(title='IDLE Error', + message='Unable to update breakpoint list:\n%s' + % str(err), + parent=self.text) def restore_file_breaks(self): self.text.update() # this enables setting "BREAK" tags to be visible -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13453: Fix a race condition in test_poplib.
by charles-francois.natali Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/1eae8154c109 changeset: 73966:1eae8154c109 parent: 73963:4a27b5ab2710 parent: 73965:e497a3ed9beb user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 14 19:29:56 2011 +0100 summary: Issue #13453: Fix a race condition in test_poplib. files: Lib/test/test_poplib.py | 22 ++++++++++++---------- 1 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -320,32 +320,34 @@ def setUp(self): self.evt = threading.Event() self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.sock.settimeout(3) + self.sock.settimeout(60) # Safety net. Look issue 11812 self.port = test_support.bind_port(self.sock) - threading.Thread(target=self.server, args=(self.evt,self.sock)).start() - time.sleep(.1) + self.thread = threading.Thread(target=self.server, args=(self.evt,self.sock)) + self.thread.setDaemon(True) + self.thread.start() + self.evt.wait() def tearDown(self): - self.evt.wait() + self.thread.join() + del self.thread # Clear out any dangling Thread objects. def server(self, evt, serv): serv.listen(5) + evt.set() try: conn, addr = serv.accept() + conn.send(b"+ Hola mundo\n") + conn.close() except socket.timeout: pass - else: - conn.send(b"+ Hola mundo\n") - conn.close() finally: serv.close() - evt.set() def testTimeoutDefault(self): self.assertTrue(socket.getdefaulttimeout() is None) socket.setdefaulttimeout(30) try: - pop = poplib.POP3("localhost", self.port) + pop = poplib.POP3(HOST, self.port) finally: socket.setdefaulttimeout(None) self.assertEqual(pop.sock.gettimeout(), 30) @@ -362,7 +364,7 @@ pop.sock.close() def testTimeoutValue(self): - pop = poplib.POP3("localhost", self.port, timeout=30) + pop = poplib.POP3(HOST, self.port, timeout=30) self.assertEqual(pop.sock.gettimeout(), 30) pop.sock.close() -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13453: Fix a race condition in test_poplib.
by charles-francois.natali Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/e497a3ed9beb changeset: 73965:e497a3ed9beb branch: 3.2 parent: 73962:49e82c885d6b user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 14 19:28:56 2011 +0100 summary: Issue #13453: Fix a race condition in test_poplib. files: Lib/test/test_poplib.py | 22 ++++++++++++---------- 1 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -309,32 +309,34 @@ def setUp(self): self.evt = threading.Event() self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.sock.settimeout(3) + self.sock.settimeout(60) # Safety net. Look issue 11812 self.port = test_support.bind_port(self.sock) - threading.Thread(target=self.server, args=(self.evt,self.sock)).start() - time.sleep(.1) + self.thread = threading.Thread(target=self.server, args=(self.evt,self.sock)) + self.thread.setDaemon(True) + self.thread.start() + self.evt.wait() def tearDown(self): - self.evt.wait() + self.thread.join() + del self.thread # Clear out any dangling Thread objects. def server(self, evt, serv): serv.listen(5) + evt.set() try: conn, addr = serv.accept() + conn.send(b"+ Hola mundo\n") + conn.close() except socket.timeout: pass - else: - conn.send(b"+ Hola mundo\n") - conn.close() finally: serv.close() - evt.set() def testTimeoutDefault(self): self.assertTrue(socket.getdefaulttimeout() is None) socket.setdefaulttimeout(30) try: - pop = poplib.POP3("localhost", self.port) + pop = poplib.POP3(HOST, self.port) finally: socket.setdefaulttimeout(None) self.assertEqual(pop.sock.gettimeout(), 30) @@ -351,7 +353,7 @@ pop.sock.close() def testTimeoutValue(self): - pop = poplib.POP3("localhost", self.port, timeout=30) + pop = poplib.POP3(HOST, self.port, timeout=30) self.assertEqual(pop.sock.gettimeout(), 30) pop.sock.close() -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13453: Fix a race condition in test_poplib.
by charles-francois.natali Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/fbcaeb4a8654 changeset: 73964:fbcaeb4a8654 branch: 2.7 parent: 73961:8c658b625475 user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 14 19:28:08 2011 +0100 summary: Issue #13453: Fix a race condition in test_poplib. files: Lib/test/test_poplib.py | 22 ++++++++++++---------- 1 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -281,32 +281,34 @@ def setUp(self): self.evt = threading.Event() self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.sock.settimeout(3) + self.sock.settimeout(60) # Safety net. Look issue 11812 self.port = test_support.bind_port(self.sock) - threading.Thread(target=self.server, args=(self.evt,self.sock)).start() - time.sleep(.1) + self.thread = threading.Thread(target=self.server, args=(self.evt,self.sock)) + self.thread.setDaemon(True) + self.thread.start() + self.evt.wait() def tearDown(self): - self.evt.wait() + self.thread.join() + del self.thread # Clear out any dangling Thread objects. def server(self, evt, serv): serv.listen(5) + evt.set() try: conn, addr = serv.accept() + conn.send("+ Hola mundo\n") + conn.close() except socket.timeout: pass - else: - conn.send("+ Hola mundo\n") - conn.close() finally: serv.close() - evt.set() def testTimeoutDefault(self): self.assertTrue(socket.getdefaulttimeout() is None) socket.setdefaulttimeout(30) try: - pop = poplib.POP3("localhost", self.port) + pop = poplib.POP3(HOST, self.port) finally: socket.setdefaulttimeout(None) self.assertEqual(pop.sock.gettimeout(), 30) @@ -323,7 +325,7 @@ pop.sock.close() def testTimeoutValue(self): - pop = poplib.POP3("localhost", self.port, timeout=30) + pop = poplib.POP3(HOST, self.port, timeout=30) self.assertEqual(pop.sock.gettimeout(), 30) pop.sock.close() -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Null merge - Issue #4028: Make multiprocessing build on SunOS.
by charles-francois.natali Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/4a27b5ab2710 changeset: 73963:4a27b5ab2710 parent: 73960:2975618965c0 parent: 73962:49e82c885d6b user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 14 18:40:35 2011 +0100 summary: Null merge - Issue #4028: Make multiprocessing build on SunOS. files: -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #4028: Make multiprocessing build on SunOS.
by charles-francois.natali Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/49e82c885d6b changeset: 73962:49e82c885d6b branch: 3.2 parent: 73957:14695b4825dc user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 14 18:39:09 2011 +0100 summary: Issue #4028: Make multiprocessing build on SunOS. files: Modules/_multiprocessing/multiprocessing.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c --- a/Modules/_multiprocessing/multiprocessing.c +++ b/Modules/_multiprocessing/multiprocessing.c @@ -8,7 +8,7 @@ #include "multiprocessing.h" -#ifdef SCM_RIGHTS +#if (defined(CMSG_LEN) && defined(SCM_RIGHTS)) #define HAVE_FD_TRANSFER 1 #else #define HAVE_FD_TRANSFER 0 -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #4028: Make multiprocessing build on SunOS.
by charles-francois.natali Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/8c658b625475 changeset: 73961:8c658b625475 branch: 2.7 parent: 73954:d60856651139 user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 14 18:35:55 2011 +0100 summary: Issue #4028: Make multiprocessing build on SunOS. files: Modules/_multiprocessing/multiprocessing.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c --- a/Modules/_multiprocessing/multiprocessing.c +++ b/Modules/_multiprocessing/multiprocessing.c @@ -8,7 +8,7 @@ #include "multiprocessing.h" -#ifdef SCM_RIGHTS +#if (defined(CMSG_LEN) && defined(SCM_RIGHTS)) #define HAVE_FD_TRANSFER 1 #else #define HAVE_FD_TRANSFER 0 -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix #13449: add 'blocking' parameter to sched.scheduler.run() so that the
by giampaolo.rodola Dec. 15, 2011

Dec. 15, 2011

http://hg.python.org/cpython/rev/2975618965c0 changeset: 73960:2975618965c0 user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Wed Dec 14 14:38:45 2011 +0100 summary: Fix #13449: add 'blocking' parameter to sched.scheduler.run() so that the scheduler can be used in non-blocking applications files: Doc/library/sched.rst | 10 ++++++++-- Doc/whatsnew/3.3.rst | 6 ++++++ Lib/sched.py | 6 +++++- Lib/test/test_sched.py | 10 ++++++++++ Misc/NEWS | 6 +++++- 5 files changed, 34 insertions(+), 4 deletions(-) diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -102,12 +102,15 @@ Return true if the event queue is empty. -.. method:: scheduler.run() +.. method:: scheduler.run(blocking=True) - Run all scheduled events. This function will wait (using the :func:`delayfunc` + Run all scheduled events. This method will wait (using the :func:`delayfunc` function passed to the constructor) for the next event, then execute it and so on until there are no more scheduled events. + If *blocking* is False executes the scheduled events due to expire soonest + (if any) and then return. + Either *action* or *delayfunc* can raise an exception. In either case, the scheduler will maintain a consistent state and propagate the exception. If an exception is raised by *action*, the event will not be attempted in future calls @@ -118,6 +121,9 @@ the calling code is responsible for canceling events which are no longer pertinent. + .. versionadded:: 3.3 + *blocking* parameter was added. + .. attribute:: scheduler.queue Read-only attribute returning a list of upcoming events in the order they diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -662,6 +662,12 @@ sched ----- +* :meth:`~sched.scheduler.run` now accepts a *blocking* parameter which when + set to False makes the method execute the scheduled events due to expire + soonest (if any) and then return immediately. + This is useful in case you want to use the :class:`~sched.scheduler` in + non-blocking applications. (Contributed by Giampaolo Rodolà in :issue:`13449`) + * :class:`~sched.scheduler` class can now be safely used in multi-threaded environments. (Contributed by Josiah Carlson and Giampaolo Rodolà in :issue:`8684`) diff --git a/Lib/sched.py b/Lib/sched.py --- a/Lib/sched.py +++ b/Lib/sched.py @@ -91,8 +91,10 @@ with self._lock: return not self._queue - def run(self): + def run(self, blocking=True): """Execute events until the queue is empty. + If blocking is False executes the scheduled events due to + expire soonest (if any) and then return. When there is a positive delay until the first event, the delay function is called and the event is left in the queue; @@ -123,6 +125,8 @@ time, priority, action, argument, kwargs = checked_event = q[0] now = timefunc() if now < time: + if not blocking: + return delayfunc(time - now) else: event = pop(q) diff --git a/Lib/test/test_sched.py b/Lib/test/test_sched.py --- a/Lib/test/test_sched.py +++ b/Lib/test/test_sched.py @@ -86,6 +86,16 @@ scheduler.run() self.assertEqual(flag, [None]) + def test_run_non_blocking(self): + l = [] + fun = lambda x: l.append(x) + scheduler = sched.scheduler(time.time, time.sleep) + for x in [10, 9, 8, 7, 6]: + scheduler.enter(x, 1, fun, (x,)) + scheduler.run(blocking=False) + self.assertEqual(l, []) + + def test_main(): support.run_unittest(TestCase) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -409,8 +409,12 @@ Library ------- +- Issue #13449 sched.scheduler.run() method has a new "blocking" parameter which + when set to False makes run() execute the scheduled events due to expire + soonest (if any) and then return. Patch by Giampaolo Rodolà. + - Issue #8684 sched.scheduler class can be safely used in multi-threaded - environments. + environments. Patch by Josiah Carlson and Giampaolo Rodolà. - Alias resource.error to OSError ala PEP 3151. -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix #8684: make sched.scheduler class thread-safe
by giampaolo.rodola Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/f5aed0dba844 changeset: 73959:f5aed0dba844 user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Wed Dec 14 13:34:26 2011 +0100 summary: Fix #8684: make sched.scheduler class thread-safe files: Doc/library/sched.rst | 30 +------------- Doc/whatsnew/3.3.rst | 4 ++ Lib/sched.py | 62 +++++++++++++++++------------- Misc/NEWS | 3 + 4 files changed, 45 insertions(+), 54 deletions(-) diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -27,6 +27,9 @@ .. versionchanged:: 3.3 *timefunc* and *delayfunc* parameters are optional. + .. versionchanged:: 3.3 + :class:`scheduler` class can be safely used in multi-threaded + environments. Example:: @@ -47,33 +50,6 @@ From print_time 930343700.273 930343700.276 -In multi-threaded environments, the :class:`scheduler` class has limitations -with respect to thread-safety, inability to insert a new task before -the one currently pending in a running scheduler, and holding up the main -thread until the event queue is empty. Instead, the preferred approach -is to use the :class:`threading.Timer` class instead. - -Example:: - - >>> import time - >>> from threading import Timer - >>> def print_time(): - ... print("From print_time", time.time()) - ... - >>> def print_some_times(): - ... print(time.time()) - ... Timer(5, print_time, ()).start() - ... Timer(10, print_time, ()).start() - ... time.sleep(11) # sleep while time-delay events execute - ... print(time.time()) - ... - >>> print_some_times() - 930343690.257 - From print_time 930343695.274 - From print_time 930343700.273 - 930343701.301 - - .. _scheduler-objects: Scheduler Objects diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -662,6 +662,10 @@ sched ----- +* :class:`~sched.scheduler` class can now be safely used in multi-threaded + environments. (Contributed by Josiah Carlson and Giampaolo Rodolà in + :issue:`8684`) + * *timefunc* and *delayfunct* parameters of :class:`~sched.scheduler` class constructor are now optional and defaults to :func:`time.time` and :func:`time.sleep` respectively. (Contributed by Chris Clark in diff --git a/Lib/sched.py b/Lib/sched.py --- a/Lib/sched.py +++ b/Lib/sched.py @@ -30,6 +30,7 @@ import time import heapq +import threading from collections import namedtuple __all__ = ["scheduler"] @@ -48,6 +49,7 @@ """Initialize a new instance, passing the time and delay functions""" self._queue = [] + self._lock = threading.RLock() self.timefunc = timefunc self.delayfunc = delayfunc @@ -58,9 +60,10 @@ if necessary. """ - event = Event(time, priority, action, argument, kwargs) - heapq.heappush(self._queue, event) - return event # The ID + with self._lock: + event = Event(time, priority, action, argument, kwargs) + heapq.heappush(self._queue, event) + return event # The ID def enter(self, delay, priority, action, argument=[], kwargs={}): """A variant that specifies the time as a relative time. @@ -68,8 +71,9 @@ This is actually the more commonly used interface. """ - time = self.timefunc() + delay - return self.enterabs(time, priority, action, argument, kwargs) + with self._lock: + time = self.timefunc() + delay + return self.enterabs(time, priority, action, argument, kwargs) def cancel(self, event): """Remove an event from the queue. @@ -78,12 +82,14 @@ If the event is not in the queue, this raises ValueError. """ - self._queue.remove(event) - heapq.heapify(self._queue) + with self._lock: + self._queue.remove(event) + heapq.heapify(self._queue) def empty(self): """Check whether the queue is empty.""" - return not self._queue + with self._lock: + return not self._queue def run(self): """Execute events until the queue is empty. @@ -108,24 +114,25 @@ """ # localize variable access to minimize overhead # and to improve thread safety - q = self._queue - delayfunc = self.delayfunc - timefunc = self.timefunc - pop = heapq.heappop - while q: - time, priority, action, argument, kwargs = checked_event = q[0] - now = timefunc() - if now < time: - delayfunc(time - now) - else: - event = pop(q) - # Verify that the event was not removed or altered - # by another thread after we last looked at q[0]. - if event is checked_event: - action(*argument, **kwargs) - delayfunc(0) # Let other threads run + with self._lock: + q = self._queue + delayfunc = self.delayfunc + timefunc = self.timefunc + pop = heapq.heappop + while q: + time, priority, action, argument, kwargs = checked_event = q[0] + now = timefunc() + if now < time: + delayfunc(time - now) else: - heapq.heappush(q, event) + event = pop(q) + # Verify that the event was not removed or altered + # by another thread after we last looked at q[0]. + if event is checked_event: + action(*argument, **kwargs) + delayfunc(0) # Let other threads run + else: + heapq.heappush(q, event) @property def queue(self): @@ -138,5 +145,6 @@ # Use heapq to sort the queue rather than using 'sorted(self._queue)'. # With heapq, two events scheduled at the same time will show in # the actual order they would be retrieved. - events = self._queue[:] - return map(heapq.heappop, [events]*len(events)) + with self._lock: + events = self._queue[:] + return map(heapq.heappop, [events]*len(events)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -409,6 +409,9 @@ Library ------- +- Issue #8684 sched.scheduler class can be safely used in multi-threaded + environments. + - Alias resource.error to OSError ala PEP 3151. - Issue #5689: Add support for lzma compression to the tarfile module. -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (98df29d51e12): sum=0
by solipsis＠pitrou.net Dec. 14, 2011

Dec. 14, 2011

results for 98df29d51e12 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKR3zCb', '-x']

1 0

cpython (merge 3.2 -> default): Merge 3.2
by alexandre.vassalotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/98df29d51e12 changeset: 73958:98df29d51e12 parent: 73956:207408428242 parent: 73957:14695b4825dc user: Alexandre Vassalotti <alexandre(a)peadrop.com> date: Tue Dec 13 13:22:42 2011 -0500 summary: Merge 3.2 files: Lib/pickle.py | 6 ++- Lib/pickletools.py | 70 ++++++++++++--------------- Lib/test/pickletester.py | 12 +++- Misc/NEWS | 3 + Modules/_pickle.c | 60 ++++++++++++++++++----- 5 files changed, 94 insertions(+), 57 deletions(-) diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -485,7 +485,11 @@ def save_bytes(self, obj, pack=struct.pack): if self.proto < 3: - self.save_reduce(bytes, (list(obj),), obj=obj) + if len(obj) == 0: + self.save_reduce(bytes, (), obj=obj) + else: + self.save_reduce(codecs.encode, + (str(obj, 'latin1'), 'latin1'), obj=obj) return n = len(obj) if n < 256: diff --git a/Lib/pickletools.py b/Lib/pickletools.py --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -2082,27 +2082,22 @@ 29: ( MARK 30: d DICT (MARK at 29) 31: p PUT 2 - 34: c GLOBAL '__builtin__ bytes' - 53: p PUT 3 - 56: ( MARK - 57: ( MARK - 58: l LIST (MARK at 57) + 34: c GLOBAL '_codecs encode' + 50: p PUT 3 + 53: ( MARK + 54: V UNICODE 'abc' 59: p PUT 4 - 62: L LONG 97 - 67: a APPEND - 68: L LONG 98 - 73: a APPEND - 74: L LONG 99 - 79: a APPEND - 80: t TUPLE (MARK at 56) - 81: p PUT 5 - 84: R REDUCE - 85: p PUT 6 - 88: V UNICODE 'def' - 93: p PUT 7 - 96: s SETITEM - 97: a APPEND - 98: . STOP + 62: V UNICODE 'latin1' + 70: p PUT 5 + 73: t TUPLE (MARK at 53) + 74: p PUT 6 + 77: R REDUCE + 78: p PUT 7 + 81: V UNICODE 'def' + 86: p PUT 8 + 89: s SETITEM + 90: a APPEND + 91: . STOP highest protocol among opcodes = 0 Try again with a "binary" pickle. @@ -2121,25 +2116,22 @@ 14: q BINPUT 1 16: } EMPTY_DICT 17: q BINPUT 2 - 19: c GLOBAL '__builtin__ bytes' - 38: q BINPUT 3 - 40: ( MARK - 41: ] EMPTY_LIST - 42: q BINPUT 4 - 44: ( MARK - 45: K BININT1 97 - 47: K BININT1 98 - 49: K BININT1 99 - 51: e APPENDS (MARK at 44) - 52: t TUPLE (MARK at 40) - 53: q BINPUT 5 - 55: R REDUCE - 56: q BINPUT 6 - 58: X BINUNICODE 'def' - 66: q BINPUT 7 - 68: s SETITEM - 69: e APPENDS (MARK at 3) - 70: . STOP + 19: c GLOBAL '_codecs encode' + 35: q BINPUT 3 + 37: ( MARK + 38: X BINUNICODE 'abc' + 46: q BINPUT 4 + 48: X BINUNICODE 'latin1' + 59: q BINPUT 5 + 61: t TUPLE (MARK at 37) + 62: q BINPUT 6 + 64: R REDUCE + 65: q BINPUT 7 + 67: X BINUNICODE 'def' + 75: q BINPUT 8 + 77: s SETITEM + 78: e APPENDS (MARK at 3) + 79: . STOP highest protocol among opcodes = 1 Exercise the INST/OBJ/BUILD family. diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -637,9 +637,15 @@ def test_bytes(self): for proto in protocols: - for u in b'', b'xyz', b'xyz'*100: - p = self.dumps(u) - self.assertEqual(self.loads(p), u) + for s in b'', b'xyz', b'xyz'*100: + p = self.dumps(s) + self.assertEqual(self.loads(p), s) + for s in [bytes([i]) for i in range(256)]: + p = self.dumps(s) + self.assertEqual(self.loads(p), s) + for s in [bytes([i, i]) for i in range(256)]: + p = self.dumps(s) + self.assertEqual(self.loads(p), s) def test_ints(self): import sys diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #13505: Pickle bytes objects in a way that is compatible with + Python 2 when using protocols <= 2. + - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix given by Campbell Barton). diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -369,7 +369,7 @@ char *errors; /* Name of errors handling scheme to used when decoding strings. The default value is "strict". */ - Py_ssize_t *marks; /* Mark stack, used for unpickling container + Py_ssize_t *marks; /* Mark stack, used for unpickling container objects. */ Py_ssize_t num_marks; /* Number of marks in the mark stack. */ Py_ssize_t marks_size; /* Current allocated size of the mark stack. */ @@ -1708,26 +1708,58 @@ if (self->proto < 3) { /* Older pickle protocols do not have an opcode for pickling bytes objects. Therefore, we need to fake the copy protocol (i.e., - the __reduce__ method) to permit bytes object unpickling. */ + the __reduce__ method) to permit bytes object unpickling. + + Here we use a hack to be compatible with Python 2. Since in Python + 2 'bytes' is just an alias for 'str' (which has different + parameters than the actual bytes object), we use codecs.encode + to create the appropriate 'str' object when unpickled using + Python 2 *and* the appropriate 'bytes' object when unpickled + using Python 3. Again this is a hack and we don't need to do this + with newer protocols. */ + static PyObject *codecs_encode = NULL; PyObject *reduce_value = NULL; - PyObject *bytelist = NULL; int status; - bytelist = PySequence_List(obj); - if (bytelist == NULL) + if (codecs_encode == NULL) { + PyObject *codecs_module = PyImport_ImportModule("codecs"); + if (codecs_module == NULL) { + return -1; + } + codecs_encode = PyObject_GetAttrString(codecs_module, "encode"); + Py_DECREF(codecs_module); + if (codecs_encode == NULL) { + return -1; + } + } + + if (PyBytes_GET_SIZE(obj) == 0) { + reduce_value = Py_BuildValue("(O())", (PyObject*)&PyBytes_Type); + } + else { + static PyObject *latin1 = NULL; + PyObject *unicode_str = + PyUnicode_DecodeLatin1(PyBytes_AS_STRING(obj), + PyBytes_GET_SIZE(obj), + "strict"); + if (unicode_str == NULL) + return -1; + if (latin1 == NULL) { + latin1 = PyUnicode_InternFromString("latin1"); + if (latin1 == NULL) + return -1; + } + reduce_value = Py_BuildValue("(O(OO))", + codecs_encode, unicode_str, latin1); + Py_DECREF(unicode_str); + } + + if (reduce_value == NULL) return -1; - reduce_value = Py_BuildValue("(O(O))", (PyObject *)&PyBytes_Type, - bytelist); - if (reduce_value == NULL) { - Py_DECREF(bytelist); - return -1; - } - /* save_reduce() will memoize the object automatically. */ status = save_reduce(self, reduce_value, obj); Py_DECREF(reduce_value); - Py_DECREF(bytelist); return status; } else { @@ -1735,7 +1767,7 @@ char header[5]; Py_ssize_t len; - size = PyBytes_Size(obj); + size = PyBytes_GET_SIZE(obj); if (size < 0) return -1; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13505: Make pickling of bytes object compatible with Python 2.
by alexandre.vassalotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/14695b4825dc changeset: 73957:14695b4825dc branch: 3.2 parent: 73955:44ca4264dc88 user: Alexandre Vassalotti <alexandre(a)peadrop.com> date: Tue Dec 13 13:08:09 2011 -0500 summary: Issue #13505: Make pickling of bytes object compatible with Python 2. Initial patch by sbt. files: Lib/pickle.py | 6 ++- Lib/pickletools.py | 70 ++++++++++++--------------- Lib/test/pickletester.py | 12 +++- Misc/NEWS | 3 + Modules/_pickle.c | 60 ++++++++++++++++++----- 5 files changed, 94 insertions(+), 57 deletions(-) diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -487,7 +487,11 @@ def save_bytes(self, obj, pack=struct.pack): if self.proto < 3: - self.save_reduce(bytes, (list(obj),), obj=obj) + if len(obj) == 0: + self.save_reduce(bytes, (), obj=obj) + else: + self.save_reduce(codecs.encode, + (str(obj, 'latin1'), 'latin1'), obj=obj) return n = len(obj) if n < 256: diff --git a/Lib/pickletools.py b/Lib/pickletools.py --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -2083,27 +2083,22 @@ 29: ( MARK 30: d DICT (MARK at 29) 31: p PUT 2 - 34: c GLOBAL '__builtin__ bytes' - 53: p PUT 3 - 56: ( MARK - 57: ( MARK - 58: l LIST (MARK at 57) + 34: c GLOBAL '_codecs encode' + 50: p PUT 3 + 53: ( MARK + 54: V UNICODE 'abc' 59: p PUT 4 - 62: L LONG 97 - 67: a APPEND - 68: L LONG 98 - 73: a APPEND - 74: L LONG 99 - 79: a APPEND - 80: t TUPLE (MARK at 56) - 81: p PUT 5 - 84: R REDUCE - 85: p PUT 6 - 88: V UNICODE 'def' - 93: p PUT 7 - 96: s SETITEM - 97: a APPEND - 98: . STOP + 62: V UNICODE 'latin1' + 70: p PUT 5 + 73: t TUPLE (MARK at 53) + 74: p PUT 6 + 77: R REDUCE + 78: p PUT 7 + 81: V UNICODE 'def' + 86: p PUT 8 + 89: s SETITEM + 90: a APPEND + 91: . STOP highest protocol among opcodes = 0 Try again with a "binary" pickle. @@ -2122,25 +2117,22 @@ 14: q BINPUT 1 16: } EMPTY_DICT 17: q BINPUT 2 - 19: c GLOBAL '__builtin__ bytes' - 38: q BINPUT 3 - 40: ( MARK - 41: ] EMPTY_LIST - 42: q BINPUT 4 - 44: ( MARK - 45: K BININT1 97 - 47: K BININT1 98 - 49: K BININT1 99 - 51: e APPENDS (MARK at 44) - 52: t TUPLE (MARK at 40) - 53: q BINPUT 5 - 55: R REDUCE - 56: q BINPUT 6 - 58: X BINUNICODE 'def' - 66: q BINPUT 7 - 68: s SETITEM - 69: e APPENDS (MARK at 3) - 70: . STOP + 19: c GLOBAL '_codecs encode' + 35: q BINPUT 3 + 37: ( MARK + 38: X BINUNICODE 'abc' + 46: q BINPUT 4 + 48: X BINUNICODE 'latin1' + 59: q BINPUT 5 + 61: t TUPLE (MARK at 37) + 62: q BINPUT 6 + 64: R REDUCE + 65: q BINPUT 7 + 67: X BINUNICODE 'def' + 75: q BINPUT 8 + 77: s SETITEM + 78: e APPENDS (MARK at 3) + 79: . STOP highest protocol among opcodes = 1 Exercise the INST/OBJ/BUILD family. diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -636,9 +636,15 @@ def test_bytes(self): for proto in protocols: - for u in b'', b'xyz', b'xyz'*100: - p = self.dumps(u) - self.assertEqual(self.loads(p), u) + for s in b'', b'xyz', b'xyz'*100: + p = self.dumps(s) + self.assertEqual(self.loads(p), s) + for s in [bytes([i]) for i in range(256)]: + p = self.dumps(s) + self.assertEqual(self.loads(p), s) + for s in [bytes([i, i]) for i in range(256)]: + p = self.dumps(s) + self.assertEqual(self.loads(p), s) def test_ints(self): import sys diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #13505: Pickle bytes objects in a way that is compatible with + Python 2 when using protocols <= 2. + - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix given by Campbell Barton). diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -369,7 +369,7 @@ char *errors; /* Name of errors handling scheme to used when decoding strings. The default value is "strict". */ - Py_ssize_t *marks; /* Mark stack, used for unpickling container + Py_ssize_t *marks; /* Mark stack, used for unpickling container objects. */ Py_ssize_t num_marks; /* Number of marks in the mark stack. */ Py_ssize_t marks_size; /* Current allocated size of the mark stack. */ @@ -1700,26 +1700,58 @@ if (self->proto < 3) { /* Older pickle protocols do not have an opcode for pickling bytes objects. Therefore, we need to fake the copy protocol (i.e., - the __reduce__ method) to permit bytes object unpickling. */ + the __reduce__ method) to permit bytes object unpickling. + + Here we use a hack to be compatible with Python 2. Since in Python + 2 'bytes' is just an alias for 'str' (which has different + parameters than the actual bytes object), we use codecs.encode + to create the appropriate 'str' object when unpickled using + Python 2 *and* the appropriate 'bytes' object when unpickled + using Python 3. Again this is a hack and we don't need to do this + with newer protocols. */ + static PyObject *codecs_encode = NULL; PyObject *reduce_value = NULL; - PyObject *bytelist = NULL; int status; - bytelist = PySequence_List(obj); - if (bytelist == NULL) + if (codecs_encode == NULL) { + PyObject *codecs_module = PyImport_ImportModule("codecs"); + if (codecs_module == NULL) { + return -1; + } + codecs_encode = PyObject_GetAttrString(codecs_module, "encode"); + Py_DECREF(codecs_module); + if (codecs_encode == NULL) { + return -1; + } + } + + if (PyBytes_GET_SIZE(obj) == 0) { + reduce_value = Py_BuildValue("(O())", (PyObject*)&PyBytes_Type); + } + else { + static PyObject *latin1 = NULL; + PyObject *unicode_str = + PyUnicode_DecodeLatin1(PyBytes_AS_STRING(obj), + PyBytes_GET_SIZE(obj), + "strict"); + if (unicode_str == NULL) + return -1; + if (latin1 == NULL) { + latin1 = PyUnicode_InternFromString("latin1"); + if (latin1 == NULL) + return -1; + } + reduce_value = Py_BuildValue("(O(OO))", + codecs_encode, unicode_str, latin1); + Py_DECREF(unicode_str); + } + + if (reduce_value == NULL) return -1; - reduce_value = Py_BuildValue("(O(O))", (PyObject *)&PyBytes_Type, - bytelist); - if (reduce_value == NULL) { - Py_DECREF(bytelist); - return -1; - } - /* save_reduce() will memoize the object automatically. */ status = save_reduce(self, reduce_value, obj); Py_DECREF(reduce_value); - Py_DECREF(bytelist); return status; } else { @@ -1727,7 +1759,7 @@ char header[5]; Py_ssize_t len; - size = PyBytes_Size(obj); + size = PyBytes_GET_SIZE(obj); if (size < 0) return -1; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): #6570: merge with 3.2.
by ezio.melotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/207408428242 changeset: 73956:207408428242 parent: 73953:37094a1454ab parent: 73955:44ca4264dc88 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Tue Dec 13 15:53:47 2011 +0200 summary: #6570: merge with 3.2. files: Doc/tutorial/controlflow.rst | 39 +++++++++++++---------- 1 files changed, 22 insertions(+), 17 deletions(-) diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -412,8 +412,8 @@ Keyword Arguments ----------------- -Functions can also be called using keyword arguments of the form ``keyword = -value``. For instance, the following function:: +Functions can also be called using :term:`keyword arguments <keyword argument>` +of the form ``kwarg=value``. For instance, the following function:: def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'): print("-- This parrot wouldn't", action, end=' ') @@ -421,26 +421,31 @@ print("-- Lovely plumage, the", type) print("-- It's", state, "!") -could be called in any of the following ways:: +accepts one required argument (``voltage``) and three optional arguments +(``state``, ``action``, and ``type``). This function can be called in any +of the following ways:: - parrot(1000) - parrot(action = 'VOOOOOM', voltage = 1000000) - parrot('a thousand', state = 'pushing up the daisies') - parrot('a million', 'bereft of life', 'jump') + parrot(1000) # 1 positional argument + parrot(voltage=1000) # 1 keyword argument + parrot(voltage=1000000, action='VOOOOOM') # 2 keyword arguments + parrot(action='VOOOOOM', voltage=1000000) # 2 keyword arguments + parrot('a million', 'bereft of life', 'jump') # 3 positional arguments + parrot('a thousand', state='pushing up the daisies') # 1 positional, 1 keyword -but the following calls would all be invalid:: +but all the following calls would be invalid:: parrot() # required argument missing - parrot(voltage=5.0, 'dead') # non-keyword argument following keyword - parrot(110, voltage=220) # duplicate value for argument - parrot(actor='John Cleese') # unknown keyword + parrot(voltage=5.0, 'dead') # non-keyword argument after a keyword argument + parrot(110, voltage=220) # duplicate value for the same argument + parrot(actor='John Cleese') # unknown keyword argument -In general, an argument list must have any positional arguments followed by any -keyword arguments, where the keywords must be chosen from the formal parameter -names. It's not important whether a formal parameter has a default value or -not. No argument may receive a value more than once --- formal parameter names -corresponding to positional arguments cannot be used as keywords in the same -calls. Here's an example that fails due to this restriction:: +In a function call, keyword arguments must follow positional arguments. +All the keyword arguments passed must match one of the arguments +accepted by the function (e.g. ``actor`` is not a valid argument for the +``parrot`` function), and their order is not important. This also includes +non-optional arguments (e.g. ``parrot(voltage=1000)`` is valid too). +No argument may receive a value more than once. +Here's an example that fails due to this restriction:: >>> def function(a): ... pass -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): #6570: clarify tutorial section about keyword arguments.
by ezio.melotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/44ca4264dc88 changeset: 73955:44ca4264dc88 branch: 3.2 parent: 73952:ad5c70296c7b user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Tue Dec 13 15:49:22 2011 +0200 summary: #6570: clarify tutorial section about keyword arguments. files: Doc/tutorial/controlflow.rst | 39 +++++++++++++---------- 1 files changed, 22 insertions(+), 17 deletions(-) diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -412,8 +412,8 @@ Keyword Arguments ----------------- -Functions can also be called using keyword arguments of the form ``keyword = -value``. For instance, the following function:: +Functions can also be called using :term:`keyword arguments <keyword argument>` +of the form ``kwarg=value``. For instance, the following function:: def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'): print("-- This parrot wouldn't", action, end=' ') @@ -421,26 +421,31 @@ print("-- Lovely plumage, the", type) print("-- It's", state, "!") -could be called in any of the following ways:: +accepts one required argument (``voltage``) and three optional arguments +(``state``, ``action``, and ``type``). This function can be called in any +of the following ways:: - parrot(1000) - parrot(action = 'VOOOOOM', voltage = 1000000) - parrot('a thousand', state = 'pushing up the daisies') - parrot('a million', 'bereft of life', 'jump') + parrot(1000) # 1 positional argument + parrot(voltage=1000) # 1 keyword argument + parrot(voltage=1000000, action='VOOOOOM') # 2 keyword arguments + parrot(action='VOOOOOM', voltage=1000000) # 2 keyword arguments + parrot('a million', 'bereft of life', 'jump') # 3 positional arguments + parrot('a thousand', state='pushing up the daisies') # 1 positional, 1 keyword -but the following calls would all be invalid:: +but all the following calls would be invalid:: parrot() # required argument missing - parrot(voltage=5.0, 'dead') # non-keyword argument following keyword - parrot(110, voltage=220) # duplicate value for argument - parrot(actor='John Cleese') # unknown keyword + parrot(voltage=5.0, 'dead') # non-keyword argument after a keyword argument + parrot(110, voltage=220) # duplicate value for the same argument + parrot(actor='John Cleese') # unknown keyword argument -In general, an argument list must have any positional arguments followed by any -keyword arguments, where the keywords must be chosen from the formal parameter -names. It's not important whether a formal parameter has a default value or -not. No argument may receive a value more than once --- formal parameter names -corresponding to positional arguments cannot be used as keywords in the same -calls. Here's an example that fails due to this restriction:: +In a function call, keyword arguments must follow positional arguments. +All the keyword arguments passed must match one of the arguments +accepted by the function (e.g. ``actor`` is not a valid argument for the +``parrot`` function), and their order is not important. This also includes +non-optional arguments (e.g. ``parrot(voltage=1000)`` is valid too). +No argument may receive a value more than once. +Here's an example that fails due to this restriction:: >>> def function(a): ... pass -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): #6570: clarify tutorial section about keyword arguments.
by ezio.melotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/d60856651139 changeset: 73954:d60856651139 branch: 2.7 parent: 73951:132158b287d7 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Tue Dec 13 15:49:22 2011 +0200 summary: #6570: clarify tutorial section about keyword arguments. files: Doc/tutorial/controlflow.rst | 39 +++++++++++++---------- 1 files changed, 22 insertions(+), 17 deletions(-) diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -383,8 +383,8 @@ Keyword Arguments ----------------- -Functions can also be called using keyword arguments of the form ``keyword = -value``. For instance, the following function:: +Functions can also be called using :term:`keyword arguments <keyword argument>` +of the form ``kwarg=value``. For instance, the following function:: def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'): print "-- This parrot wouldn't", action, @@ -392,26 +392,31 @@ print "-- Lovely plumage, the", type print "-- It's", state, "!" -could be called in any of the following ways:: +accepts one required argument (``voltage``) and three optional arguments +(``state``, ``action``, and ``type``). This function can be called in any +of the following ways:: - parrot(1000) - parrot(action = 'VOOOOOM', voltage = 1000000) - parrot('a thousand', state = 'pushing up the daisies') - parrot('a million', 'bereft of life', 'jump') + parrot(1000) # 1 positional argument + parrot(voltage=1000) # 1 keyword argument + parrot(voltage=1000000, action='VOOOOOM') # 2 keyword arguments + parrot(action='VOOOOOM', voltage=1000000) # 2 keyword arguments + parrot('a million', 'bereft of life', 'jump') # 3 positional arguments + parrot('a thousand', state='pushing up the daisies') # 1 positional, 1 keyword -but the following calls would all be invalid:: +but all the following calls would be invalid:: parrot() # required argument missing - parrot(voltage=5.0, 'dead') # non-keyword argument following keyword - parrot(110, voltage=220) # duplicate value for argument - parrot(actor='John Cleese') # unknown keyword + parrot(voltage=5.0, 'dead') # non-keyword argument after a keyword argument + parrot(110, voltage=220) # duplicate value for the same argument + parrot(actor='John Cleese') # unknown keyword argument -In general, an argument list must have any positional arguments followed by any -keyword arguments, where the keywords must be chosen from the formal parameter -names. It's not important whether a formal parameter has a default value or -not. No argument may receive a value more than once --- formal parameter names -corresponding to positional arguments cannot be used as keywords in the same -calls. Here's an example that fails due to this restriction:: +In a function call, keyword arguments must follow positional arguments. +All the keyword arguments passed must match one of the arguments +accepted by the function (e.g. ``actor`` is not a valid argument for the +``parrot`` function), and their order is not important. This also includes +non-optional arguments (e.g. ``parrot(voltage=1000)`` is valid too). +No argument may receive a value more than once. +Here's an example that fails due to this restriction:: >>> def function(a): ... pass -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): #13549: merge with 3.2.
by ezio.melotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/37094a1454ab changeset: 73953:37094a1454ab parent: 73950:6f097ff9ac04 parent: 73952:ad5c70296c7b user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Tue Dec 13 15:38:13 2011 +0200 summary: #13549: merge with 3.2. files: Doc/tutorial/datastructures.rst | 166 +++++++++++-------- 1 files changed, 98 insertions(+), 68 deletions(-) diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -168,107 +168,137 @@ List Comprehensions ------------------- -List comprehensions provide a concise way to create lists from sequences. -Common applications are to make lists where each element is the result of -some operations applied to each member of the sequence, or to create a -subsequence of those elements that satisfy a certain condition. +List comprehensions provide a concise way to create lists. +Common applications are to make new lists where each element is the result of +some operations applied to each member of another sequence or iterable, or to +create a subsequence of those elements that satisfy a certain condition. + +For example, assume we want to create a list of squares, like:: + + >>> squares = [] + >>> for x in range(10): + ... squares.append(x**2) + ... + >>> squares + [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] + +We can obtain the same result with:: + + squares = [x**2 for x in range(10)] + +This is also equivalent to ``squares = map(lambda x: x**2, range(10))``, +but it's more concise and readable. A list comprehension consists of brackets containing an expression followed by a :keyword:`for` clause, then zero or more :keyword:`for` or :keyword:`if` -clauses. The result will be a list resulting from evaluating the expression in -the context of the :keyword:`for` and :keyword:`if` clauses which follow it. If -the expression would evaluate to a tuple, it must be parenthesized. +clauses. The result will be a new list resulting from evaluating the expression +in the context of the :keyword:`for` and :keyword:`if` clauses which follow it. +For example, this listcomp combines the elements of two lists if they are not +equal:: -Here we take a list of numbers and return a list of three times each number:: + >>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] - >>> vec = [2, 4, 6] - >>> [3*x for x in vec] - [6, 12, 18] +and it's equivalent to:: -Now we get a little fancier:: + >>> combs = [] + >>> for x in [1,2,3]: + ... for y in [3,1,4]: + ... if x != y: + ... combs.append((x, y)) + ... + >>> combs + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] - >>> [[x, x**2] for x in vec] - [[2, 4], [4, 16], [6, 36]] +Note how the order of the :keyword:`for` and :keyword:`if` statements is the +same in both these snippets. -Here we apply a method call to each item in a sequence:: +If the expression is a tuple (e.g. the ``(x, y)`` in the previous example), +it must be parenthesized. :: + >>> vec = [-4, -2, 0, 2, 4] + >>> # create a new list with the values doubled + >>> [x*2 for x in vec] + [-8, -4, 0, 4, 8] + >>> # filter the list to exclude negative numbers + >>> [x for x in vec if x >= 0] + [0, 2, 4] + >>> # apply a function to all the elements + >>> [abs(x) for x in vec] + [4, 2, 0, 2, 4] + >>> # call a method on each element >>> freshfruit = [' banana', ' loganberry ', 'passion fruit '] >>> [weapon.strip() for weapon in freshfruit] ['banana', 'loganberry', 'passion fruit'] - -Using the :keyword:`if` clause we can filter the stream:: - - >>> [3*x for x in vec if x > 3] - [12, 18] - >>> [3*x for x in vec if x < 2] - [] - -Tuples can often be created without their parentheses, but not here:: - - >>> [x, x**2 for x in vec] # error - parens required for tuples + >>> # create a list of 2-tuples like (number, square) + >>> [(x, x**2) for x in range(6)] + [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)] + >>> # the tuple must be parenthesized, otherwise an error is raised + >>> [x, x**2 for x in range(6)] File "<stdin>", line 1, in ? - [x, x**2 for x in vec] + [x, x**2 for x in range(6)] ^ SyntaxError: invalid syntax - >>> [(x, x**2) for x in vec] - [(2, 4), (4, 16), (6, 36)] + >>> # flatten a list using a listcomp with two 'for' + >>> vec = [[1,2,3], [4,5,6], [7,8,9]] + >>> [num for elem in vec for num in elem] + [1, 2, 3, 4, 5, 6, 7, 8, 9] -Here are some nested for loops and other fancy behavior:: +List comprehensions can contain complex expressions and nested functions:: - >>> vec1 = [2, 4, 6] - >>> vec2 = [4, 3, -9] - >>> [x*y for x in vec1 for y in vec2] - [8, 6, -18, 16, 12, -36, 24, 18, -54] - >>> [x+y for x in vec1 for y in vec2] - [6, 5, -7, 8, 7, -5, 10, 9, -3] - >>> [vec1[i]*vec2[i] for i in range(len(vec1))] - [8, 12, -54] - -List comprehensions can be applied to complex expressions and nested functions:: - - >>> [str(round(355/113, i)) for i in range(1, 6)] + >>> from math import pi + >>> [str(round(pi, i)) for i in range(1, 6)] ['3.1', '3.14', '3.142', '3.1416', '3.14159'] - Nested List Comprehensions -------------------------- -If you've got the stomach for it, list comprehensions can be nested. They are a -powerful tool but -- like all powerful tools -- they need to be used carefully, -if at all. +The initial expression in a list comprehension can be any arbitrary expression, +including another list comprehension. -Consider the following example of a 3x3 matrix held as a list containing three -lists, one list per row:: +Consider the following example of a 3x4 matrix implemented as a list of +3 lists of length 4:: - >>> mat = [ - ... [1, 2, 3], - ... [4, 5, 6], - ... [7, 8, 9], - ... ] + >>> matrix = [ + ... [1, 2, 3, 4], + ... [5, 6, 7, 8], + ... [9, 10, 11, 12], + ... ] -Now, if you wanted to swap rows and columns, you could use a list -comprehension:: +The following list comprehension will transpose rows and columns:: - >>> print([[row[i] for row in mat] for i in [0, 1, 2]]) - [[1, 4, 7], [2, 5, 8], [3, 6, 9]] + >>> [[row[i] for row in matrix] for i in range(4)] + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -Special care has to be taken for the *nested* list comprehension: +As we saw in the previous section, the nested listcomp is evaluated in +the context of the :keyword:`for` that follows it, so this example is +equivalent to:: - To avoid apprehension when nesting list comprehensions, read from right to - left. + >>> transposed = [] + >>> for i in range(4): + ... transposed.append([row[i] for row in matrix]) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -A more verbose version of this snippet shows the flow explicitly:: +which, in turn, is the same as:: - for i in [0, 1, 2]: - for row in mat: - print(row[i], end="") - print() + >>> transposed = [] + >>> for i in range(4): + ... # the following 3 lines implement the nested listcomp + ... transposed_row = [] + ... for row in matrix: + ... transposed_row.append(row[i]) + ... transposed.append(transposed_row) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -In real world, you should prefer built-in functions to complex flow statements. +In the real world, you should prefer built-in functions to complex flow statements. The :func:`zip` function would do a great job for this use case:: - >>> list(zip(*mat)) - [(1, 4, 7), (2, 5, 8), (3, 6, 9)] + >>> zip(*matrix) + [(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)] See :ref:`tut-unpacking-arguments` for details on the asterisk in this line. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): #13549: improve tutorial section about listcomps.
by ezio.melotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/ad5c70296c7b changeset: 73952:ad5c70296c7b branch: 3.2 parent: 73925:6040c248a199 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Tue Dec 13 15:36:19 2011 +0200 summary: #13549: improve tutorial section about listcomps. files: Doc/tutorial/datastructures.rst | 166 +++++++++++-------- 1 files changed, 98 insertions(+), 68 deletions(-) diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -163,107 +163,137 @@ List Comprehensions ------------------- -List comprehensions provide a concise way to create lists from sequences. -Common applications are to make lists where each element is the result of -some operations applied to each member of the sequence, or to create a -subsequence of those elements that satisfy a certain condition. +List comprehensions provide a concise way to create lists. +Common applications are to make new lists where each element is the result of +some operations applied to each member of another sequence or iterable, or to +create a subsequence of those elements that satisfy a certain condition. + +For example, assume we want to create a list of squares, like:: + + >>> squares = [] + >>> for x in range(10): + ... squares.append(x**2) + ... + >>> squares + [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] + +We can obtain the same result with:: + + squares = [x**2 for x in range(10)] + +This is also equivalent to ``squares = map(lambda x: x**2, range(10))``, +but it's more concise and readable. A list comprehension consists of brackets containing an expression followed by a :keyword:`for` clause, then zero or more :keyword:`for` or :keyword:`if` -clauses. The result will be a list resulting from evaluating the expression in -the context of the :keyword:`for` and :keyword:`if` clauses which follow it. If -the expression would evaluate to a tuple, it must be parenthesized. +clauses. The result will be a new list resulting from evaluating the expression +in the context of the :keyword:`for` and :keyword:`if` clauses which follow it. +For example, this listcomp combines the elements of two lists if they are not +equal:: -Here we take a list of numbers and return a list of three times each number:: + >>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] - >>> vec = [2, 4, 6] - >>> [3*x for x in vec] - [6, 12, 18] +and it's equivalent to:: -Now we get a little fancier:: + >>> combs = [] + >>> for x in [1,2,3]: + ... for y in [3,1,4]: + ... if x != y: + ... combs.append((x, y)) + ... + >>> combs + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] - >>> [[x, x**2] for x in vec] - [[2, 4], [4, 16], [6, 36]] +Note how the order of the :keyword:`for` and :keyword:`if` statements is the +same in both these snippets. -Here we apply a method call to each item in a sequence:: +If the expression is a tuple (e.g. the ``(x, y)`` in the previous example), +it must be parenthesized. :: + >>> vec = [-4, -2, 0, 2, 4] + >>> # create a new list with the values doubled + >>> [x*2 for x in vec] + [-8, -4, 0, 4, 8] + >>> # filter the list to exclude negative numbers + >>> [x for x in vec if x >= 0] + [0, 2, 4] + >>> # apply a function to all the elements + >>> [abs(x) for x in vec] + [4, 2, 0, 2, 4] + >>> # call a method on each element >>> freshfruit = [' banana', ' loganberry ', 'passion fruit '] >>> [weapon.strip() for weapon in freshfruit] ['banana', 'loganberry', 'passion fruit'] - -Using the :keyword:`if` clause we can filter the stream:: - - >>> [3*x for x in vec if x > 3] - [12, 18] - >>> [3*x for x in vec if x < 2] - [] - -Tuples can often be created without their parentheses, but not here:: - - >>> [x, x**2 for x in vec] # error - parens required for tuples + >>> # create a list of 2-tuples like (number, square) + >>> [(x, x**2) for x in range(6)] + [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)] + >>> # the tuple must be parenthesized, otherwise an error is raised + >>> [x, x**2 for x in range(6)] File "<stdin>", line 1, in ? - [x, x**2 for x in vec] + [x, x**2 for x in range(6)] ^ SyntaxError: invalid syntax - >>> [(x, x**2) for x in vec] - [(2, 4), (4, 16), (6, 36)] + >>> # flatten a list using a listcomp with two 'for' + >>> vec = [[1,2,3], [4,5,6], [7,8,9]] + >>> [num for elem in vec for num in elem] + [1, 2, 3, 4, 5, 6, 7, 8, 9] -Here are some nested for loops and other fancy behavior:: +List comprehensions can contain complex expressions and nested functions:: - >>> vec1 = [2, 4, 6] - >>> vec2 = [4, 3, -9] - >>> [x*y for x in vec1 for y in vec2] - [8, 6, -18, 16, 12, -36, 24, 18, -54] - >>> [x+y for x in vec1 for y in vec2] - [6, 5, -7, 8, 7, -5, 10, 9, -3] - >>> [vec1[i]*vec2[i] for i in range(len(vec1))] - [8, 12, -54] - -List comprehensions can be applied to complex expressions and nested functions:: - - >>> [str(round(355/113, i)) for i in range(1, 6)] + >>> from math import pi + >>> [str(round(pi, i)) for i in range(1, 6)] ['3.1', '3.14', '3.142', '3.1416', '3.14159'] - Nested List Comprehensions -------------------------- -If you've got the stomach for it, list comprehensions can be nested. They are a -powerful tool but -- like all powerful tools -- they need to be used carefully, -if at all. +The initial expression in a list comprehension can be any arbitrary expression, +including another list comprehension. -Consider the following example of a 3x3 matrix held as a list containing three -lists, one list per row:: +Consider the following example of a 3x4 matrix implemented as a list of +3 lists of length 4:: - >>> mat = [ - ... [1, 2, 3], - ... [4, 5, 6], - ... [7, 8, 9], - ... ] + >>> matrix = [ + ... [1, 2, 3, 4], + ... [5, 6, 7, 8], + ... [9, 10, 11, 12], + ... ] -Now, if you wanted to swap rows and columns, you could use a list -comprehension:: +The following list comprehension will transpose rows and columns:: - >>> print([[row[i] for row in mat] for i in [0, 1, 2]]) - [[1, 4, 7], [2, 5, 8], [3, 6, 9]] + >>> [[row[i] for row in matrix] for i in range(4)] + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -Special care has to be taken for the *nested* list comprehension: +As we saw in the previous section, the nested listcomp is evaluated in +the context of the :keyword:`for` that follows it, so this example is +equivalent to:: - To avoid apprehension when nesting list comprehensions, read from right to - left. + >>> transposed = [] + >>> for i in range(4): + ... transposed.append([row[i] for row in matrix]) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -A more verbose version of this snippet shows the flow explicitly:: +which, in turn, is the same as:: - for i in [0, 1, 2]: - for row in mat: - print(row[i], end="") - print() + >>> transposed = [] + >>> for i in range(4): + ... # the following 3 lines implement the nested listcomp + ... transposed_row = [] + ... for row in matrix: + ... transposed_row.append(row[i]) + ... transposed.append(transposed_row) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -In real world, you should prefer built-in functions to complex flow statements. +In the real world, you should prefer built-in functions to complex flow statements. The :func:`zip` function would do a great job for this use case:: - >>> list(zip(*mat)) - [(1, 4, 7), (2, 5, 8), (3, 6, 9)] + >>> zip(*matrix) + [(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)] See :ref:`tut-unpacking-arguments` for details on the asterisk in this line. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): #13549: improve tutorial section about listcomps.
by ezio.melotti Dec. 14, 2011

Dec. 14, 2011

http://hg.python.org/cpython/rev/132158b287d7 changeset: 73951:132158b287d7 branch: 2.7 parent: 73944:bf7329190ca6 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Tue Dec 13 14:50:21 2011 +0200 summary: #13549: improve tutorial section about listcomps. files: Doc/tutorial/datastructures.rst | 167 +++++++++++++------ 1 files changed, 109 insertions(+), 58 deletions(-) diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -235,89 +235,139 @@ List Comprehensions ------------------- -List comprehensions provide a concise way to create lists without resorting to -use of :func:`map`, :func:`filter` and/or :keyword:`lambda`. The resulting list -definition tends often to be clearer than lists built using those constructs. -Each list comprehension consists of an expression followed by a :keyword:`for` -clause, then zero or more :keyword:`for` or :keyword:`if` clauses. The result -will be a list resulting from evaluating the expression in the context of the -:keyword:`for` and :keyword:`if` clauses which follow it. If the expression -would evaluate to a tuple, it must be parenthesized. :: +List comprehensions provide a concise way to create lists. +Common applications are to make new lists where each element is the result of +some operations applied to each member of another sequence or iterable, or to +create a subsequence of those elements that satisfy a certain condition. +For example, assume we want to create a list of squares, like:: + + >>> squares = [] + >>> for x in range(10): + ... squares.append(x**2) + ... + >>> squares + [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] + +We can obtain the same result with:: + + squares = [x**2 for x in range(10)] + +This is also equivalent to ``squares = map(lambda x: x**2, range(10))``, +but it's more concise and readable. + +A list comprehension consists of brackets containing an expression followed +by a :keyword:`for` clause, then zero or more :keyword:`for` or :keyword:`if` +clauses. The result will be a new list resulting from evaluating the expression +in the context of the :keyword:`for` and :keyword:`if` clauses which follow it. +For example, this listcomp combines the elements of two lists if they are not +equal:: + + >>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] + +and it's equivalent to: + + >>> combs = [] + >>> for x in [1,2,3]: + ... for y in [3,1,4]: + ... if x != y: + ... combs.append((x, y)) + ... + >>> combs + [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)] + +Note how the order of the :keyword:`for` and :keyword:`if` statements is the +same in both these snippets. + +If the expression is a tuple (e.g. the ``(x, y)`` in the previous example), +it must be parenthesized. :: + + >>> vec = [-4, -2, 0, 2, 4] + >>> # create a new list with the values doubled + >>> [x*2 for x in vec] + [-8, -4, 0, 4, 8] + >>> # filter the list to exclude negative numbers + >>> [x for x in vec if x >= 0] + [0, 2, 4] + >>> # apply a function to all the elements + >>> [abs(x) for x in vec] + [4, 2, 0, 2, 4] + >>> # call a method on each element >>> freshfruit = [' banana', ' loganberry ', 'passion fruit '] >>> [weapon.strip() for weapon in freshfruit] ['banana', 'loganberry', 'passion fruit'] - >>> vec = [2, 4, 6] - >>> [3*x for x in vec] - [6, 12, 18] - >>> [3*x for x in vec if x > 3] - [12, 18] - >>> [3*x for x in vec if x < 2] - [] - >>> [[x,x**2] for x in vec] - [[2, 4], [4, 16], [6, 36]] - >>> [x, x**2 for x in vec] # error - parens required for tuples - File "<stdin>", line 1, in ? - [x, x**2 for x in vec] + >>> # create a list of 2-tuples like (number, square) + >>> [(x, x**2) for x in range(6)] + [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)] + >>> # the tuple must be parenthesized, otherwise an error is raised + >>> [x, x**2 for x in range(6)] + File "<stdin>", line 1 + [x, x**2 for x in range(6)] ^ SyntaxError: invalid syntax - >>> [(x, x**2) for x in vec] - [(2, 4), (4, 16), (6, 36)] - >>> vec1 = [2, 4, 6] - >>> vec2 = [4, 3, -9] - >>> [x*y for x in vec1 for y in vec2] - [8, 6, -18, 16, 12, -36, 24, 18, -54] - >>> [x+y for x in vec1 for y in vec2] - [6, 5, -7, 8, 7, -5, 10, 9, -3] - >>> [vec1[i]*vec2[i] for i in range(len(vec1))] - [8, 12, -54] + >>> # flatten a list using a listcomp with two 'for' + >>> vec = [[1,2,3], [4,5,6], [7,8,9]] + >>> [num for elem in vec for num in elem] + [1, 2, 3, 4, 5, 6, 7, 8, 9] -List comprehensions are much more flexible than :func:`map` and can be applied -to complex expressions and nested functions:: +List comprehensions can contain complex expressions and nested functions:: - >>> [str(round(355/113.0, i)) for i in range(1,6)] + >>> from math import pi + >>> [str(round(pi, i)) for i in range(1, 6)] ['3.1', '3.14', '3.142', '3.1416', '3.14159'] Nested List Comprehensions --------------------------- +'''''''''''''''''''''''''' -If you've got the stomach for it, list comprehensions can be nested. They are a -powerful tool but -- like all powerful tools -- they need to be used carefully, -if at all. +The initial expression in a list comprehension can be any arbitrary expression, +including another list comprehension. -Consider the following example of a 3x3 matrix held as a list containing three -lists, one list per row:: +Consider the following example of a 3x4 matrix implemented as a list of +3 lists of length 4:: - >>> mat = [ - ... [1, 2, 3], - ... [4, 5, 6], - ... [7, 8, 9], - ... ] + >>> matrix = [ + ... [1, 2, 3, 4], + ... [5, 6, 7, 8], + ... [9, 10, 11, 12], + ... ] -Now, if you wanted to swap rows and columns, you could use a list -comprehension:: +The following list comprehension will transpose rows and columns:: - >>> print [[row[i] for row in mat] for i in [0, 1, 2]] - [[1, 4, 7], [2, 5, 8], [3, 6, 9]] + >>> [[row[i] for row in matrix] for i in range(4)] + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -Special care has to be taken for the *nested* list comprehension: +As we saw in the previous section, the nested listcomp is evaluated in +the context of the :keyword:`for` that follows it, so this example is +equivalent to:: - To avoid apprehension when nesting list comprehensions, read from right to - left. + >>> transposed = [] + >>> for i in range(4): + ... transposed.append([row[i] for row in matrix]) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -A more verbose version of this snippet shows the flow explicitly:: +which, in turn, is the same as:: - for i in [0, 1, 2]: - for row in mat: - print row[i], - print + >>> transposed = [] + >>> for i in range(4): + ... # the following 3 lines implement the nested listcomp + ... transposed_row = [] + ... for row in matrix: + ... transposed_row.append(row[i]) + ... transposed.append(transposed_row) + ... + >>> transposed + [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -In real world, you should prefer built-in functions to complex flow statements. + +In the real world, you should prefer built-in functions to complex flow statements. The :func:`zip` function would do a great job for this use case:: - >>> zip(*mat) - [(1, 4, 7), (2, 5, 8), (3, 6, 9)] + >>> zip(*matrix) + [(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)] See :ref:`tut-unpacking-arguments` for details on the asterisk in this line. -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (6f097ff9ac04): sum=0
by solipsis＠pitrou.net Dec. 13, 2011

Dec. 13, 2011

results for 6f097ff9ac04 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogq7CYkf', '-x']

1 0

cpython: Fix test_gdb failure
by antoine.pitrou Dec. 13, 2011

Dec. 13, 2011

http://hg.python.org/cpython/rev/6f097ff9ac04 changeset: 73950:6f097ff9ac04 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 12 19:18:24 2011 +0100 summary: Fix test_gdb failure files: Tools/gdb/libpython.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -328,7 +328,6 @@ name_map = {'bool': PyBoolObjectPtr, 'classobj': PyClassObjectPtr, - 'instance': PyInstanceObjectPtr, 'NoneType': PyNoneStructPtr, 'frame': PyFrameObjectPtr, 'set' : PySetObjectPtr, -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13575: there is only one class type.
by florent.xicluna Dec. 13, 2011

Dec. 13, 2011

http://hg.python.org/cpython/rev/021e5bb297d1 changeset: 73949:021e5bb297d1 user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Mon Dec 12 18:54:29 2011 +0100 summary: Issue #13575: there is only one class type. files: Doc/howto/descriptor.rst | 12 +------ Lib/pickle.py | 2 +- Lib/pickletools.py | 2 + Modules/gc_weakref.txt | 6 +- Objects/typeobject.c | 31 +++++++------------ Python/errors.c | 2 +- Python/pythonrun.c | 7 +--- Tools/gdb/libpython.py | 44 +-------------------------- 8 files changed, 25 insertions(+), 81 deletions(-) diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -36,9 +36,7 @@ looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods -were defined. Note that descriptors are only invoked for new style objects or -classes (a class is new style if it inherits from :class:`object` or -:class:`type`). +were defined. Descriptors are a powerful, general purpose protocol. They are the mechanism behind properties, methods, static methods, class methods, and :func:`super()`. @@ -89,8 +87,6 @@ is invoked according to the precedence rules listed below. The details of invocation depend on whether ``obj`` is an object or a class. -Either way, descriptors only work for new style objects and classes. A class is -new style if it is a subclass of :class:`object`. For objects, the machinery is in :meth:`object.__getattribute__` which transforms ``b.x`` into ``type(b).__dict__['x'].__get__(b, type(b))``. The @@ -115,7 +111,6 @@ * descriptors are invoked by the :meth:`__getattribute__` method * overriding :meth:`__getattribute__` prevents automatic descriptor calls -* :meth:`__getattribute__` is only available with new style classes and objects * :meth:`object.__getattribute__` and :meth:`type.__getattribute__` make different calls to :meth:`__get__`. * data descriptors always override instance dictionaries. @@ -128,10 +123,7 @@ ``m`` is returned unchanged. If not in the dictionary, ``m`` reverts to a search using :meth:`object.__getattribute__`. -Note, in Python 2.2, ``super(B, obj).m()`` would only invoke :meth:`__get__` if -``m`` was a data descriptor. In Python 2.3, non-data descriptors also get -invoked unless an old-style class is involved. The implementation details are -in :c:func:`super_getattro()` in +The implementation details are in :c:func:`super_getattro()` in `Objects/typeobject.c <http://svn.python.org/view/python/trunk/Objects/typeobject.c?view=markup>`_ and a pure Python equivalent can be found in `Guido's Tutorial`_. diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -375,7 +375,7 @@ # allowing protocol 0 and 1 to work normally. For this to # work, the function returned by __reduce__ should be # called __newobj__, and its first argument should be a - # new-style class. The implementation for __newobj__ + # class. The implementation for __newobj__ # should be as follows, although pickle has no way to # verify this: # diff --git a/Lib/pickletools.py b/Lib/pickletools.py --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -1639,6 +1639,8 @@ is pushed on the stack. NOTE: checks for __safe_for_unpickling__ went away in Python 2.3. + NOTE: the distinction between old-style and new-style classes does + not make sense in Python 3. """), I(name='OBJ', diff --git a/Modules/gc_weakref.txt b/Modules/gc_weakref.txt --- a/Modules/gc_weakref.txt +++ b/Modules/gc_weakref.txt @@ -15,8 +15,8 @@ historically common bug), tp_clear empties an instance's __dict__, and "impossible" AttributeErrors result. At worst, tp_clear leaves behind an insane object at the C level, and segfaults result (historically, most -often by setting a new-style class's mro pointer to NULL, after which -attribute lookups performed by the class can segfault). +often by setting a class's mro pointer to NULL, after which attribute +lookups performed by the class can segfault). OTOH, it's OK to run Python-level code that can't access unreachable objects, and sometimes that's necessary. The chief example is the callback @@ -119,7 +119,7 @@ it took quite a while to dream up failing test cases. Zope3 saw segfaults during shutdown, during the second call of gc in Py_Finalize, after most modules had been torn down. That creates many trash cycles (esp. those -involving new-style classes), making the problem much more likely. Once you +involving classes), making the problem much more likely. Once you know what's required to provoke the problem, though, it's easy to create tests that segfault before shutdown. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -100,15 +100,13 @@ static void type_mro_modified(PyTypeObject *type, PyObject *bases) { /* - Check that all base classes or elements of the mro of type are + Check that all base classes or elements of the MRO of type are able to be cached. This function is called after the base classes or mro of the type are altered. Unset HAVE_VERSION_TAG and VALID_VERSION_TAG if the type - inherits from an old-style class, either directly or if it - appears in the MRO of a new-style class. No support either for - custom MROs that include types that are not officially super - types. + has a custom MRO that includes a type which is not officially + super type. Called from mro_internal, which will subsequently be called on each subclass when their mro is recursively updated. @@ -124,11 +122,7 @@ PyObject *b = PyTuple_GET_ITEM(bases, i); PyTypeObject *cls; - if (!PyType_Check(b) ) { - clear = 1; - break; - } - + assert(PyType_Check(b)); cls = (PyTypeObject *)b; if (!PyType_HasFeature(cls, Py_TPFLAGS_HAVE_VERSION_TAG) || @@ -488,7 +482,7 @@ if (!PyType_Check(ob)) { PyErr_Format( PyExc_TypeError, - "%s.__bases__ must be tuple of old- or new-style classes, not '%s'", + "%s.__bases__ must be tuple of classes, not '%s'", type->tp_name, Py_TYPE(ob)->tp_name); return -1; } @@ -1619,7 +1613,7 @@ type->tp_mro = tuple; type_mro_modified(type, type->tp_mro); - /* corner case: the old-style super class might have been hidden + /* corner case: the super class might have been hidden from the custom MRO */ type_mro_modified(type, type->tp_bases); @@ -1676,9 +1670,8 @@ return NULL; } } - if (base == NULL) - PyErr_SetString(PyExc_TypeError, - "a new-style class can't have only classic bases"); + assert (base != NULL); + return base; } @@ -3196,7 +3189,7 @@ } if (!PyType_Check(value)) { PyErr_Format(PyExc_TypeError, - "__class__ must be set to new-style class, not '%s' object", + "__class__ must be set to a class, not '%s' object", Py_TYPE(value)->tp_name); return -1; } @@ -3811,8 +3804,8 @@ that the extension type's own factory function ensures). Heap types, of course, are under our control, so they do inherit tp_new; static extension types that specify some - other built-in type as the default are considered - new-style-aware so they also inherit object.__new__. */ + other built-in type as the default also + inherit object.__new__. */ if (base != &PyBaseObject_Type || (type->tp_flags & Py_TPFLAGS_HEAPTYPE)) { if (type->tp_new == NULL) @@ -6352,7 +6345,7 @@ { /* Check that a super() call makes sense. Return a type object. - obj can be a new-style class, or an instance of one: + obj can be a class, or an instance of one: - If it is a class, it must be a subclass of 'type'. This case is used for class methods; the return value is obj. diff --git a/Python/errors.c b/Python/errors.c --- a/Python/errors.c +++ b/Python/errors.c @@ -665,7 +665,7 @@ if (bases == NULL) goto failure; } - /* Create a real new-style class. */ + /* Create a real class. */ result = PyObject_CallFunction((PyObject *)&PyType_Type, "sOO", dot+1, bases, dict); failure: diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -474,7 +474,7 @@ flush_std_files(); /* Collect final garbage. This disposes of cycles created by - * new-style class definitions, for example. + * class definitions, for example. * XXX This is disabled because it caused too many problems. If * XXX a __del__ or weakref callback triggers here, Python code has * XXX a hard time running, because even the sys module has been @@ -1348,11 +1348,6 @@ _Py_IDENTIFIER(offset); _Py_IDENTIFIER(text); - /* old style errors */ - if (PyTuple_Check(err)) - return PyArg_ParseTuple(err, "O(ziiz)", message, filename, - lineno, offset, text); - /* new style errors. `err' is an instance */ if (! (v = _PyObject_GetAttrId(err, &PyId_msg))) diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -402,7 +402,7 @@ def _write_instance_repr(out, visited, name, pyop_attrdict, address): - '''Shared code for use by old-style and new-style classes: + '''Shared code for use by all classes: write a representation to file-like object "out"''' out.write('<') out.write(name) @@ -481,7 +481,7 @@ def proxyval(self, visited): ''' - Support for new-style classes. + Support for classes. Currently we just locate the dictionary using a transliteration to python of _PyObject_GetDictPtr, ignoring descriptors @@ -498,7 +498,7 @@ attr_dict = {} tp_name = self.safe_tp_name() - # New-style class: + # Class: return InstanceProxy(tp_name, attr_dict, long(self._gdbval)) def write_repr(self, out, visited): @@ -670,44 +670,6 @@ pyop_value.write_repr(out, visited) out.write('}') -class PyInstanceObjectPtr(PyObjectPtr): - _typename = 'PyInstanceObject' - - def proxyval(self, visited): - # Guard against infinite loops: - if self.as_address() in visited: - return ProxyAlreadyVisited('<...>') - visited.add(self.as_address()) - - # Get name of class: - in_class = self.pyop_field('in_class') - cl_name = in_class.pyop_field('cl_name').proxyval(visited) - - # Get dictionary of instance attributes: - in_dict = self.pyop_field('in_dict').proxyval(visited) - - # Old-style class: - return InstanceProxy(cl_name, in_dict, long(self._gdbval)) - - def write_repr(self, out, visited): - # Guard against infinite loops: - if self.as_address() in visited: - out.write('<...>') - return - visited.add(self.as_address()) - - # Old-style class: - - # Get name of class: - in_class = self.pyop_field('in_class') - cl_name = in_class.pyop_field('cl_name').proxyval(visited) - - # Get dictionary of instance attributes: - pyop_in_dict = self.pyop_field('in_dict') - - _write_instance_repr(out, visited, - cl_name, pyop_in_dict, self.as_address()) - class PyListObjectPtr(PyObjectPtr): _typename = 'PyListObject' -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13577: various kinds of descriptors now have a __qualname__ attribute.
by antoine.pitrou Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/24238e89f938 changeset: 73948:24238e89f938 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 12 13:47:25 2011 +0100 summary: Issue #13577: various kinds of descriptors now have a __qualname__ attribute. Patch by sbt. files: Include/descrobject.h | 1 + Lib/test/test_descr.py | 18 ++++++++++ Lib/test/test_sys.py | 8 ++-- Objects/descrobject.c | 51 ++++++++++++++++++++++++++++++ 4 files changed, 74 insertions(+), 4 deletions(-) diff --git a/Include/descrobject.h b/Include/descrobject.h --- a/Include/descrobject.h +++ b/Include/descrobject.h @@ -42,6 +42,7 @@ PyObject_HEAD PyTypeObject *d_type; PyObject *d_name; + PyObject *d_qualname; } PyDescrObject; #define PyDescr_COMMON PyDescrObject d_common diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4442,6 +4442,24 @@ self.assertIn("can't delete X.__doc__", str(cm.exception)) self.assertEqual(X.__doc__, "banana") + def test_qualname(self): + descriptors = [str.lower, complex.real, float.real, int.__add__] + types = ['method', 'member', 'getset', 'wrapper'] + + # make sure we have an example of each type of descriptor + for d, n in zip(descriptors, types): + self.assertEqual(type(d).__name__, n + '_descriptor') + + for d in descriptors: + qualname = d.__objclass__.__qualname__ + '.' + d.__name__ + self.assertEqual(d.__qualname__, qualname) + + self.assertEqual(str.lower.__qualname__, 'str.lower') + self.assertEqual(complex.real.__qualname__, 'complex.real') + self.assertEqual(float.real.__qualname__, 'float.real') + self.assertEqual(int.__add__.__qualname__, 'int.__add__') + + class DictProxyTests(unittest.TestCase): def setUp(self): class C(object): diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -670,17 +670,17 @@ # complex check(complex(0,1), size(h + '2d')) # method_descriptor (descriptor object) - check(str.lower, size(h + '2PP')) + check(str.lower, size(h + '3PP')) # classmethod_descriptor (descriptor object) # XXX # member_descriptor (descriptor object) import datetime - check(datetime.timedelta.days, size(h + '2PP')) + check(datetime.timedelta.days, size(h + '3PP')) # getset_descriptor (descriptor object) import collections - check(collections.defaultdict.default_factory, size(h + '2PP')) + check(collections.defaultdict.default_factory, size(h + '3PP')) # wrapper_descriptor (descriptor object) - check(int.__add__, size(h + '2P2P')) + check(int.__add__, size(h + '3P2P')) # method-wrapper (descriptor object) check({}.__iter__, size(h + '2P')) # dict diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -9,6 +9,7 @@ _PyObject_GC_UNTRACK(descr); Py_XDECREF(descr->d_type); Py_XDECREF(descr->d_name); + Py_XDECREF(descr->d_qualname); PyObject_GC_Del(descr); } @@ -321,6 +322,44 @@ return PyUnicode_FromString(descr->d_method->ml_doc); } +static PyObject * +calculate_qualname(PyDescrObject *descr) +{ + PyObject *type_qualname, *res; + _Py_IDENTIFIER(__qualname__); + + if (descr->d_name == NULL || !PyUnicode_Check(descr->d_name)) { + PyErr_SetString(PyExc_TypeError, + "<descriptor>.__name__ is not a unicode object"); + return NULL; + } + + type_qualname = _PyObject_GetAttrId((PyObject *)descr->d_type, + &PyId___qualname__); + if (type_qualname == NULL) + return NULL; + + if (!PyUnicode_Check(type_qualname)) { + PyErr_SetString(PyExc_TypeError, "<descriptor>.__objclass__." + "__qualname__ is not a unicode object"); + Py_XDECREF(type_qualname); + return NULL; + } + + res = PyUnicode_FromFormat("%S.%S", type_qualname, descr->d_name); + Py_DECREF(type_qualname); + return res; +} + +static PyObject * +descr_get_qualname(PyDescrObject *descr) +{ + if (descr->d_qualname == NULL) + descr->d_qualname = calculate_qualname(descr); + Py_XINCREF(descr->d_qualname); + return descr->d_qualname; +} + static PyMemberDef descr_members[] = { {"__objclass__", T_OBJECT, offsetof(PyDescrObject, d_type), READONLY}, {"__name__", T_OBJECT, offsetof(PyDescrObject, d_name), READONLY}, @@ -329,6 +368,7 @@ static PyGetSetDef method_getset[] = { {"__doc__", (getter)method_get_doc}, + {"__qualname__", (getter)descr_get_qualname}, {0} }; @@ -344,6 +384,7 @@ static PyGetSetDef member_getset[] = { {"__doc__", (getter)member_get_doc}, + {"__qualname__", (getter)descr_get_qualname}, {0} }; @@ -359,6 +400,7 @@ static PyGetSetDef getset_getset[] = { {"__doc__", (getter)getset_get_doc}, + {"__qualname__", (getter)descr_get_qualname}, {0} }; @@ -374,6 +416,7 @@ static PyGetSetDef wrapperdescr_getset[] = { {"__doc__", (getter)wrapperdescr_get_doc}, + {"__qualname__", (getter)descr_get_qualname}, {0} }; @@ -585,6 +628,7 @@ Py_DECREF(descr); descr = NULL; } + descr->d_qualname = NULL; } return descr; } @@ -987,9 +1031,16 @@ } } +static PyObject * +wrapper_qualname(wrapperobject *wp) +{ + return descr_get_qualname((PyDescrObject *)wp->descr); +} + static PyGetSetDef wrapper_getsets[] = { {"__objclass__", (getter)wrapper_objclass}, {"__name__", (getter)wrapper_name}, + {"__qualname__", (getter)wrapper_qualname}, {"__doc__", (getter)wrapper_doc}, {0} }; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: PyUnicode_Resize(): warn about canonical representation
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/3ad7d01acbf4 changeset: 73947:3ad7d01acbf4 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Mon Dec 12 13:24:15 2011 +0100 summary: PyUnicode_Resize(): warn about canonical representation Call also directly unicode_resize() in unicodeobject.c files: Include/unicodeobject.h | 5 ++++- Objects/unicodeobject.c | 25 +++++++++++++------------ 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -779,7 +779,10 @@ a new string and copy characters), or create a new string. Error handling is implemented as follows: an exception is set, -1 - is returned and *unicode left untouched. */ + is returned and *unicode left untouched. + + WARNING: The function doesn't check string content, the result may not be a + string in canonical representation. */ PyAPI_FUNC(int) PyUnicode_Resize( PyObject **unicode, /* Pointer to the Unicode object */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5040,7 +5040,7 @@ *consumed = (const char *)q-starts; /* Adjust length */ - if (PyUnicode_Resize(&unicode, outpos) < 0) + if (unicode_resize(&unicode, outpos) < 0) goto onError; Py_XDECREF(errorHandler); @@ -5404,7 +5404,7 @@ *consumed = (const char *)q-starts; /* Adjust length */ - if (PyUnicode_Resize(&unicode, outpos) < 0) + if (unicode_resize(&unicode, outpos) < 0) goto onError; Py_XDECREF(errorHandler); @@ -5824,7 +5824,7 @@ } #undef WRITECHAR - if (PyUnicode_Resize(&v, i) < 0) + if (unicode_resize(&v, i) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -6086,7 +6086,7 @@ nextByte: ; } - if (PyUnicode_Resize(&v, outpos) < 0) + if (unicode_resize(&v, outpos) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -6273,7 +6273,7 @@ goto onError; } - if (PyUnicode_Resize(&v, outpos) < 0) + if (unicode_resize(&v, outpos) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -6727,7 +6727,7 @@ data = PyUnicode_DATA(v); } } - if (PyUnicode_Resize(&v, outpos) < 0) + if (unicode_resize(&v, outpos) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -6874,7 +6874,7 @@ else { /* Extend unicode object */ Py_ssize_t n = PyUnicode_GET_SIZE(*v); - if (PyUnicode_Resize(v, n + outsize) < 0) + if (unicode_resize(v, n + outsize) < 0) return -1; out = PyUnicode_AS_UNICODE(*v) + n; } @@ -6958,7 +6958,7 @@ PyErr_NoMemory(); goto error; } - if (PyUnicode_Resize(v, n + size * Py_ARRAY_LENGTH(buffer)) < 0) + if (unicode_resize(v, n + size * Py_ARRAY_LENGTH(buffer)) < 0) goto error; startout = PyUnicode_AS_UNICODE(*v) + n; } @@ -7017,7 +7017,7 @@ /* Extend unicode object */ outsize = out - startout; assert(outsize <= PyUnicode_WSTR_LENGTH(*v)); - if (PyUnicode_Resize(v, outsize) < 0) + if (unicode_resize(v, outsize) < 0) goto error; ret = size; @@ -7664,8 +7664,9 @@ (targetsize << 2); extrachars += needed; /* XXX overflow detection missing */ - if (PyUnicode_Resize(&v, - PyUnicode_GET_LENGTH(v) + needed) < 0) { + if (unicode_resize(&v, + PyUnicode_GET_LENGTH(v) + needed) < 0) + { Py_DECREF(x); goto onError; } @@ -7689,7 +7690,7 @@ ++s; } } - if (PyUnicode_Resize(&v, outpos) < 0) + if (unicode_resize(&v, outpos) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix PyUnicode_Resize() for compact string: leave the string unchanged on error
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/61a2cae5308c changeset: 73946:61a2cae5308c user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Mon Dec 12 13:08:33 2011 +0100 summary: Fix PyUnicode_Resize() for compact string: leave the string unchanged on error Fix also PyUnicode_Resize() doc files: Include/unicodeobject.h | 13 ++++------- Objects/unicodeobject.c | 31 +++++++++------------------- 2 files changed, 15 insertions(+), 29 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -768,18 +768,15 @@ PyAPI_FUNC(Py_UNICODE) PyUnicode_GetMax(void); #endif -/* Resize an Unicode object allocated by the legacy API (e.g. - PyUnicode_FromUnicode). Unicode objects allocated by the new API (e.g. - PyUnicode_New) cannot be resized by this function. - - The length is a number of characters (and not the number of Py_UNICODE characters). +/* Resize an Unicode object. The length is the number of characters, except + if the kind of the string is PyUnicode_WCHAR_KIND: in this case, the length + is the number of Py_UNICODE characters. *unicode is modified to point to the new (resized) object and 0 returned on success. - If the refcount on the object is 1, the function resizes the string in - place, which is usually faster than allocating a new string (and copy - characters). + Try to resize the string in place (which is usually faster than allocating + a new string and copy characters), or create a new string. Error handling is implemented as follows: an exception is set, -1 is returned and *unicode left untouched. */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -655,7 +655,6 @@ share_wstr = _PyUnicode_SHARE_WSTR(unicode); if (length > ((PY_SSIZE_T_MAX - struct_size) / char_size - 1)) { - Py_DECREF(unicode); PyErr_NoMemory(); return NULL; } @@ -666,7 +665,7 @@ new_unicode = (PyObject *)PyObject_REALLOC((char *)unicode, new_size); if (new_unicode == NULL) { - PyObject_Del(unicode); + _Py_NewReference(unicode); PyErr_NoMemory(); return NULL; } @@ -834,8 +833,9 @@ new_size = sizeof(Py_UNICODE) * ((size_t)length + 1); _PyUnicode_WSTR(unicode) = (Py_UNICODE*) PyObject_MALLOC(new_size); if (!_PyUnicode_WSTR(unicode)) { + Py_DECREF(unicode); PyErr_NoMemory(); - goto onError; + return NULL; } /* Initialize the first element to guard against cases where @@ -860,13 +860,6 @@ _PyUnicode_UTF8_LENGTH(unicode) = 0; assert(_PyUnicode_CheckConsistency((PyObject *)unicode, 0)); return unicode; - - onError: - /* XXX UNREF/NEWREF interface should be more symmetrical */ - _Py_DEC_REFTOTAL; - _Py_ForgetReference((PyObject *)unicode); - PyObject_Del(unicode); - return NULL; } static const char* @@ -1506,15 +1499,10 @@ PyObject_DEL(_PyUnicode_WSTR(unicode)); if (_PyUnicode_HAS_UTF8_MEMORY(unicode)) PyObject_DEL(_PyUnicode_UTF8(unicode)); - - if (PyUnicode_IS_COMPACT(unicode)) { - Py_TYPE(unicode)->tp_free(unicode); - } - else { - if (_PyUnicode_DATA_ANY(unicode)) - PyObject_DEL(_PyUnicode_DATA_ANY(unicode)); - Py_TYPE(unicode)->tp_free(unicode); - } + if (!PyUnicode_IS_COMPACT(unicode) && _PyUnicode_DATA_ANY(unicode)) + PyObject_DEL(_PyUnicode_DATA_ANY(unicode)); + + Py_TYPE(unicode)->tp_free(unicode); } #ifdef Py_DEBUG @@ -1590,9 +1578,10 @@ } if (PyUnicode_IS_COMPACT(unicode)) { - *p_unicode = resize_compact(unicode, length); - if (*p_unicode == NULL) + PyObject *new_unicode = resize_compact(unicode, length); + if (new_unicode == NULL) return -1; + *p_unicode = new_unicode; assert(_PyUnicode_CheckConsistency(*p_unicode, 0)); return 0; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Add missing versionchanged information.
by lars.gustaebel Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/e9a57c3ef635 changeset: 73945:e9a57c3ef635 parent: 73943:963e98f5ad31 user: Lars Gustäbel <lars(a)gustaebel.de> date: Mon Dec 12 10:22:56 2011 +0100 summary: Add missing versionchanged information. files: Doc/library/tarfile.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -32,6 +32,9 @@ character devices and block devices and is able to acquire and restore file information like timestamp, access permissions and owner. +.. versionchanged:: 3.3 + Added support for :mod:`lzma` compression. + .. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, \*\*kwargs) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13573: The csv.writer now uses the repr() for floats rather than str().
by raymond.hettinger Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/bf7329190ca6 changeset: 73944:bf7329190ca6 branch: 2.7 parent: 73922:fd5f43737546 user: Raymond Hettinger <python(a)rcn.com> date: Sun Dec 11 22:31:09 2011 -0800 summary: Issue #13573: The csv.writer now uses the repr() for floats rather than str(). files: Lib/test/test_csv.py | 16 ++++++++++++++-- Misc/NEWS | 3 +++ Modules/_csv.c | 6 +++++- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -207,6 +207,18 @@ fileobj.close() os.unlink(name) + def test_write_float(self): + # Issue 13573: loss of precision because csv.writer + # uses str() for floats instead of repr() + orig_row = [1.234567890123, 1.0/7.0, 'abc'] + f = StringIO() + c = csv.writer(f, quoting=csv.QUOTE_NONNUMERIC) + c.writerow(orig_row) + f.seek(0) + c = csv.reader(f, quoting=csv.QUOTE_NONNUMERIC) + new_row = next(c) + self.assertEqual(orig_row, new_row) + def _read_test(self, input, expect, **kwargs): reader = csv.reader(input, **kwargs) result = list(reader) @@ -784,7 +796,7 @@ try: writer = csv.writer(fileobj, dialect="excel") writer.writerow(a) - expected = ",".join([str(i) for i in a])+"\r\n" + expected = ",".join([repr(i) for i in a])+"\r\n" fileobj.seek(0) self.assertEqual(fileobj.read(), expected) finally: @@ -800,7 +812,7 @@ try: writer = csv.writer(fileobj, dialect="excel") writer.writerow(a) - expected = ",".join([str(i) for i in a])+"\r\n" + expected = ",".join([repr(i) for i in a])+"\r\n" fileobj.seek(0) self.assertEqual(fileobj.read(), expected) finally: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -85,6 +85,9 @@ - tarfile.py: Correctly detect bzip2 compressed streams with blocksizes other than 900k. +- Issue #13573: The csv.writer now uses the repr() for floats rather than str(). + This allows floats to round-trip without loss of precision. + - Issue #13439: Fix many errors in turtle docstrings. - Issue #12856: Ensure child processes do not inherit the parent's random diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -1184,7 +1184,11 @@ else { PyObject *str; - str = PyObject_Str(field); + if (PyFloat_Check(field)) { + str = PyObject_Repr(field); + } else { + str = PyObject_Str(field); + } Py_DECREF(field); if (str == NULL) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13544: Add __qualname__ to functools.WRAPPER_ASSIGNMENTS.
by meador.inge Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/963e98f5ad31 changeset: 73943:963e98f5ad31 user: Meador Inge <meadori(a)gmail.com> date: Sun Dec 11 22:37:31 2011 -0600 summary: Issue #13544: Add __qualname__ to functools.WRAPPER_ASSIGNMENTS. Patch by Filip Gruszczyński. files: Lib/functools.py | 3 ++- Lib/test/test_functools.py | 12 +++++++++--- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -21,7 +21,8 @@ # update_wrapper() and wraps() are tools to help write # wrapper functions that can handle naive introspection -WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__doc__', '__annotations__') +WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__', + '__annotations__') WRAPPER_UPDATES = ('__dict__',) def update_wrapper(wrapper, wrapped, diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -226,6 +226,7 @@ self.check_wrapper(wrapper, f) self.assertIs(wrapper.__wrapped__, f) self.assertEqual(wrapper.__name__, 'f') + self.assertEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.attr, 'This is also a test') self.assertEqual(wrapper.__annotations__['a'], 'This is a new annotation') self.assertNotIn('b', wrapper.__annotations__) @@ -246,6 +247,7 @@ functools.update_wrapper(wrapper, f, (), ()) self.check_wrapper(wrapper, f, (), ()) self.assertEqual(wrapper.__name__, 'wrapper') + self.assertNotEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.__doc__, None) self.assertEqual(wrapper.__annotations__, {}) self.assertFalse(hasattr(wrapper, 'attr')) @@ -263,6 +265,7 @@ functools.update_wrapper(wrapper, f, assign, update) self.check_wrapper(wrapper, f, assign, update) self.assertEqual(wrapper.__name__, 'wrapper') + self.assertNotEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.__doc__, None) self.assertEqual(wrapper.attr, 'This is a different test') self.assertEqual(wrapper.dict_attr, f.dict_attr) @@ -309,17 +312,18 @@ def wrapper(): pass self.check_wrapper(wrapper, f) - return wrapper + return wrapper, f def test_default_update(self): - wrapper = self._default_update() + wrapper, f = self._default_update() self.assertEqual(wrapper.__name__, 'f') + self.assertEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.attr, 'This is also a test') @unittest.skipIf(not sys.flags.optimize <= 1, "Docstrings are omitted with -O2 and above") def test_default_update_doc(self): - wrapper = self._default_update() + wrapper, _ = self._default_update() self.assertEqual(wrapper.__doc__, 'This is a test') def test_no_update(self): @@ -332,6 +336,7 @@ pass self.check_wrapper(wrapper, f, (), ()) self.assertEqual(wrapper.__name__, 'wrapper') + self.assertNotEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.__doc__, None) self.assertFalse(hasattr(wrapper, 'attr')) @@ -351,6 +356,7 @@ pass self.check_wrapper(wrapper, f, assign, update) self.assertEqual(wrapper.__name__, 'wrapper') + self.assertNotEqual(wrapper.__qualname__, f.__qualname__) self.assertEqual(wrapper.__doc__, None) self.assertEqual(wrapper.attr, 'This is a different test') self.assertEqual(wrapper.dict_attr, f.dict_attr) -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (9c76bb654809): sum=0
by solipsis＠pitrou.net Dec. 12, 2011

Dec. 12, 2011

results for 9c76bb654809 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog64t0OT', '-x']

1 0

cpython: Make PyUnicode_Copy() private => _PyUnicode_Copy()
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/9c76bb654809 changeset: 73942:9c76bb654809 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Mon Dec 12 01:53:47 2011 +0100 summary: Make PyUnicode_Copy() private => _PyUnicode_Copy() Undocument the function. Make also decode_utf8_errors() as private (static). files: Doc/c-api/unicode.rst | 7 ------- Doc/whatsnew/3.3.rst | 4 ++-- Include/unicodeobject.h | 4 +++- Objects/codeobject.c | 2 +- Objects/unicodeobject.c | 12 ++++++------ 5 files changed, 12 insertions(+), 17 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -386,13 +386,6 @@ .. versionadded:: 3.3 -.. c:function:: PyObject* PyUnicode_Copy(PyObject *unicode) - - Get a new copy of a Unicode object. - - .. versionadded:: 3.3 - - .. c:function:: PyObject* PyUnicode_FromKindAndData(int kind, const void *buffer, \ Py_ssize_t size) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -769,8 +769,8 @@ * :c:macro:`PyUnicode_GET_DATA_SIZE`: use ``PyUnicode_GET_LENGTH(str) * PyUnicode_KIND(str)`` (only work on ready strings) - * :c:func:`PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy`, - :c:func:`PyUnicode_AsWideCharString` or :c:func:`PyUnicode_Copy` + * :c:func:`PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy` or + :c:func:`PyUnicode_AsWideCharString` Functions and macros manipulating Py_UNICODE* strings: diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -616,9 +616,11 @@ #endif /* Get a copy of a Unicode string. */ -PyAPI_FUNC(PyObject*) PyUnicode_Copy( +#ifndef Py_LIMITED_API +PyAPI_FUNC(PyObject*) _PyUnicode_Copy( PyObject *unicode ); +#endif /* Copy character from one unicode object into another, this function performs character conversion when necessary and falls back to memcpy if possible. diff --git a/Objects/codeobject.c b/Objects/codeobject.c --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -249,7 +249,7 @@ return NULL; } else { - item = PyUnicode_Copy(item); + item = _PyUnicode_Copy(item); if (item == NULL) { Py_DECREF(newtuple); return NULL; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -497,7 +497,7 @@ } else /* Subtype -- return genuine unicode string with the same value. */ - return PyUnicode_Copy(unicode); + return _PyUnicode_Copy(unicode); } #ifdef HAVE_MBCS @@ -1961,7 +1961,7 @@ } PyObject* -PyUnicode_Copy(PyObject *unicode) +_PyUnicode_Copy(PyObject *unicode) { Py_ssize_t length; PyObject *copy; @@ -2832,7 +2832,7 @@ if (PyUnicode_Check(obj)) { /* For a Unicode subtype that's not a Unicode object, return a true Unicode object with the same data. */ - return PyUnicode_Copy(obj); + return _PyUnicode_Copy(obj); } PyErr_Format(PyExc_TypeError, "Can't convert '%.100s' object to str implicitly", @@ -4333,7 +4333,7 @@ goto onError; \ } while (0) -PyObject * +static PyObject * decode_utf8_errors(const char *starts, Py_ssize_t size, const char *errors, @@ -9231,7 +9231,7 @@ Py_UCS4 maxchar_old, maxchar_new = 0; PyObject *v; - u = PyUnicode_Copy(self); + u = _PyUnicode_Copy(self); if (u == NULL) return NULL; maxchar_old = PyUnicode_MAX_CHAR_VALUE(u); @@ -12753,7 +12753,7 @@ static PyObject * unicode_getnewargs(PyObject *v) { - PyObject *copy = PyUnicode_Copy(v); + PyObject *copy = _PyUnicode_Copy(v); if (!copy) return NULL; return Py_BuildValue("(N)", copy); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: resize_copy() now supports legacy ready strings
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/0eea55925d1e changeset: 73941:0eea55925d1e user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Mon Dec 12 00:13:42 2011 +0100 summary: resize_copy() now supports legacy ready strings files: Include/unicodeobject.h | 4 ++++ Objects/unicodeobject.c | 28 +++++++++++++++------------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -220,6 +220,7 @@ - compact ascii: * structure = PyASCIIObject + * test: PyUnicode_IS_COMPACT_ASCII(op) * kind = PyUnicode_1BYTE_KIND * compact = 1 * ascii = 1 @@ -231,6 +232,7 @@ - compact: * structure = PyCompactUnicodeObject + * test: PyUnicode_IS_ASCII(op) && !PyUnicode_IS_COMPACT(op) * kind = PyUnicode_1BYTE_KIND, PyUnicode_2BYTE_KIND or PyUnicode_4BYTE_KIND * compact = 1 @@ -247,6 +249,7 @@ - legacy string, not ready: * structure = PyUnicodeObject + * test: kind == PyUnicode_WCHAR_KIND * length = 0 (use wstr_length) * hash = -1 * kind = PyUnicode_WCHAR_KIND @@ -262,6 +265,7 @@ - legacy string, ready: * structure = PyUnicodeObject structure + * test: !PyUnicode_IS_COMPACT(op) && kind != PyUnicode_WCHAR_KIND * kind = PyUnicode_1BYTE_KIND, PyUnicode_2BYTE_KIND or PyUnicode_4BYTE_KIND * compact = 0 diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -688,20 +688,26 @@ resize_inplace(PyObject *unicode, Py_ssize_t length) { wchar_t *wstr; + Py_ssize_t new_size; assert(!PyUnicode_IS_COMPACT(unicode)); assert(Py_REFCNT(unicode) == 1); if (PyUnicode_IS_READY(unicode)) { Py_ssize_t char_size; - Py_ssize_t new_size; int share_wstr, share_utf8; void *data; data = _PyUnicode_DATA_ANY(unicode); - assert(data != NULL); char_size = PyUnicode_KIND(unicode); share_wstr = _PyUnicode_SHARE_WSTR(unicode); share_utf8 = _PyUnicode_SHARE_UTF8(unicode); + + if (length > (PY_SSIZE_T_MAX / char_size - 1)) { + PyErr_NoMemory(); + return -1; + } + new_size = (length + 1) * char_size; + if (!share_utf8 && _PyUnicode_HAS_UTF8_MEMORY(unicode)) { PyObject_DEL(_PyUnicode_UTF8(unicode)); @@ -709,12 +715,6 @@ _PyUnicode_UTF8_LENGTH(unicode) = 0; } - if (length > (PY_SSIZE_T_MAX / char_size - 1)) { - PyErr_NoMemory(); - return -1; - } - new_size = (length + 1) * char_size; - data = (PyObject *)PyObject_REALLOC(data, new_size); if (data == NULL) { PyErr_NoMemory(); @@ -743,8 +743,9 @@ PyErr_NoMemory(); return -1; } + new_size = sizeof(wchar_t) * (length + 1); wstr = _PyUnicode_WSTR(unicode); - wstr = PyObject_REALLOC(wstr, sizeof(wchar_t) * (length + 1)); + wstr = PyObject_REALLOC(wstr, new_size); if (!wstr) { PyErr_NoMemory(); return -1; @@ -760,9 +761,11 @@ resize_copy(PyObject *unicode, Py_ssize_t length) { Py_ssize_t copy_length; - if (PyUnicode_IS_COMPACT(unicode)) { + if (_PyUnicode_KIND(unicode) != PyUnicode_WCHAR_KIND) { PyObject *copy; - assert(PyUnicode_IS_READY(unicode)); + + if (PyUnicode_READY(unicode) < 0) + return NULL; copy = PyUnicode_New(length, PyUnicode_MAX_CHAR_VALUE(unicode)); if (copy == NULL) @@ -774,8 +777,7 @@ } else { PyObject *w; - assert(_PyUnicode_WSTR(unicode) != NULL); - assert(_PyUnicode_DATA_ANY(unicode) == NULL); + w = (PyObject*)_PyUnicode_New(length); if (w == NULL) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: PyUnicode_IS_ASCII() macro ensures that the string is ready
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/d36c4ec5342b changeset: 73939:d36c4ec5342b user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Mon Dec 12 01:24:20 2011 +0100 summary: PyUnicode_IS_ASCII() macro ensures that the string is ready It has no sense to check if a not ready string is ASCII or not. files: Include/unicodeobject.h | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -424,10 +424,12 @@ #define SSTATE_INTERNED_IMMORTAL 2 /* Return true if the string contains only ASCII characters, or 0 if not. The - string may be compact (PyUnicode_IS_COMPACT_ASCII) or not. No type checks - or Ready calls are performed. */ -#define PyUnicode_IS_ASCII(op) \ - (((PyASCIIObject*)op)->state.ascii) + string may be compact (PyUnicode_IS_COMPACT_ASCII) or not, but must be + ready. */ +#define PyUnicode_IS_ASCII(op) \ + (assert(PyUnicode_Check(op)), \ + assert(PyUnicode_IS_READY(op)), \ + ((PyASCIIObject*)op)->state.ascii) /* Return true if the string is compact or 0 if not. No type checks or Ready calls are performed. */ @@ -437,7 +439,7 @@ /* Return true if the string is a compact ASCII string (use PyASCIIObject structure), or 0 if not. No type checks or Ready calls are performed. */ #define PyUnicode_IS_COMPACT_ASCII(op) \ - (PyUnicode_IS_ASCII(op) && PyUnicode_IS_COMPACT(op)) + (((PyASCIIObject*)op)->state.ascii && PyUnicode_IS_COMPACT(op)) enum PyUnicode_Kind { /* String contains only wstr byte characters. This is only possible -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Rewrite PyUnicode_Append(); unicode_modifiable() is more strict
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/561f5e86fa47 changeset: 73940:561f5e86fa47 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Mon Dec 12 00:01:39 2011 +0100 summary: Rewrite PyUnicode_Append(); unicode_modifiable() is more strict * Rename unicode_resizable() to unicode_modifiable() * Rename _PyUnicode_Dirty() to unicode_check_modifiable() to make it clear that the function is private * Inline PyUnicode_Concat() and unicode_append_inplace() in PyUnicode_Append() to simplify the code * unicode_modifiable() return 0 if the hash has been computed or if the string is not an exact unicode string * Remove _PyUnicode_DIRTY(): no need to reset the hash anymore, because if the hash has already been computed, you cannot modify a string inplace anymore * PyUnicode_Concat() checks for integer overflow files: Objects/unicodeobject.c | 197 ++++++++++++++------------- 1 files changed, 101 insertions(+), 96 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -165,9 +165,6 @@ *_to++ = (to_type) *_iter++; \ } while (0) -/* The Unicode string has been modified: reset the hash */ -#define _PyUnicode_DIRTY(op) do { _PyUnicode_HASH(op) = -1; } while (0) - /* This dictionary holds all interned unicode strings. Note that references to strings in this dictionary are *not* counted in the string's ob_refcnt. When the interned string reaches a refcnt of 0 the string deallocation @@ -226,6 +223,8 @@ PyObject *to, Py_ssize_t to_start, PyObject *from, Py_ssize_t from_start, Py_ssize_t how_many); +static int unicode_modifiable(PyObject *unicode); + static PyObject * unicode_fromascii(const unsigned char *s, Py_ssize_t size); @@ -645,10 +644,11 @@ Py_ssize_t new_size; int share_wstr; PyObject *new_unicode; - assert(PyUnicode_IS_READY(unicode)); + assert(PyUnicode_IS_COMPACT(unicode)); + char_size = PyUnicode_KIND(unicode); - if (PyUnicode_IS_COMPACT_ASCII(unicode)) + if (PyUnicode_IS_ASCII(unicode)) struct_size = sizeof(PyASCIIObject); else struct_size = sizeof(PyCompactUnicodeObject); @@ -676,7 +676,7 @@ _PyUnicode_LENGTH(unicode) = length; if (share_wstr) { _PyUnicode_WSTR(unicode) = PyUnicode_DATA(unicode); - if (!PyUnicode_IS_COMPACT_ASCII(unicode)) + if (!PyUnicode_IS_ASCII(unicode)) _PyUnicode_WSTR_LENGTH(unicode) = length; } PyUnicode_WRITE(PyUnicode_KIND(unicode), PyUnicode_DATA(unicode), @@ -691,8 +691,6 @@ assert(!PyUnicode_IS_COMPACT(unicode)); assert(Py_REFCNT(unicode) == 1); - _PyUnicode_DIRTY(unicode); - if (PyUnicode_IS_READY(unicode)) { Py_ssize_t char_size; Py_ssize_t new_size; @@ -1115,15 +1113,13 @@ #endif static int -_PyUnicode_Dirty(PyObject *unicode) -{ - assert(_PyUnicode_CHECK(unicode)); - if (Py_REFCNT(unicode) != 1) { +unicode_check_modifiable(PyObject *unicode) +{ + if (!unicode_modifiable(unicode)) { PyErr_SetString(PyExc_SystemError, - "Cannot modify a string having more than 1 reference"); + "Cannot modify a string currently used"); return -1; } - _PyUnicode_DIRTY(unicode); return 0; } @@ -1289,7 +1285,7 @@ if (how_many == 0) return 0; - if (_PyUnicode_Dirty(to)) + if (unicode_check_modifiable(to)) return -1; err = _copy_characters(to, to_start, from, from_start, how_many, 1); @@ -1537,12 +1533,17 @@ #endif static int -unicode_resizable(PyObject *unicode) -{ +unicode_modifiable(PyObject *unicode) +{ + assert(_PyUnicode_CHECK(unicode)); if (Py_REFCNT(unicode) != 1) return 0; + if (_PyUnicode_HASH(unicode) != -1) + return 0; if (PyUnicode_CHECK_INTERNED(unicode)) return 0; + if (!PyUnicode_CheckExact(unicode)) + return 0; #ifdef Py_DEBUG /* singleton refcount is greater than 1 */ assert(!unicode_is_singleton(unicode)); @@ -1577,7 +1578,7 @@ return 0; } - if (!unicode_resizable(unicode)) { + if (!unicode_modifiable(unicode)) { PyObject *copy = resize_copy(unicode, length); if (copy == NULL) return -1; @@ -3591,11 +3592,12 @@ PyErr_BadArgument(); return -1; } + assert(PyUnicode_IS_READY(unicode)); if (index < 0 || index >= PyUnicode_GET_LENGTH(unicode)) { PyErr_SetString(PyExc_IndexError, "string index out of range"); return -1; } - if (_PyUnicode_Dirty(unicode)) + if (unicode_check_modifiable(unicode)) return -1; PyUnicode_WRITE(PyUnicode_KIND(unicode), PyUnicode_DATA(unicode), index, ch); @@ -10566,6 +10568,7 @@ { PyObject *u = NULL, *v = NULL, *w; Py_UCS4 maxchar, maxchar2; + Py_ssize_t u_len, v_len, new_len; /* Coerce the two arguments */ u = PyUnicode_FromObject(left); @@ -10585,18 +10588,25 @@ return v; } + u_len = PyUnicode_GET_LENGTH(u); + v_len = PyUnicode_GET_LENGTH(v); + if (u_len > PY_SSIZE_T_MAX - v_len) { + PyErr_SetString(PyExc_OverflowError, + "strings are too large to concat"); + goto onError; + } + new_len = u_len + v_len; + maxchar = PyUnicode_MAX_CHAR_VALUE(u); maxchar2 = PyUnicode_MAX_CHAR_VALUE(v); maxchar = Py_MAX(maxchar, maxchar2); /* Concat the two Unicode strings */ - w = PyUnicode_New( - PyUnicode_GET_LENGTH(u) + PyUnicode_GET_LENGTH(v), - maxchar); + w = PyUnicode_New(new_len, maxchar); if (w == NULL) goto onError; - copy_characters(w, 0, u, 0, PyUnicode_GET_LENGTH(u)); - copy_characters(w, PyUnicode_GET_LENGTH(u), v, 0, PyUnicode_GET_LENGTH(v)); + copy_characters(w, 0, u, 0, u_len); + copy_characters(w, u_len, v, 0, v_len); Py_DECREF(u); Py_DECREF(v); assert(_PyUnicode_CheckConsistency(w, 1)); @@ -10608,15 +10618,41 @@ return NULL; } -static void -unicode_append_inplace(PyObject **p_left, PyObject *right) -{ +void +PyUnicode_Append(PyObject **p_left, PyObject *right) +{ + PyObject *left, *res; + Py_UCS4 maxchar, maxchar2; Py_ssize_t left_len, right_len, new_len; - assert(PyUnicode_IS_READY(*p_left)); - assert(PyUnicode_IS_READY(right)); - - left_len = PyUnicode_GET_LENGTH(*p_left); + if (p_left == NULL) { + if (!PyErr_Occurred()) + PyErr_BadInternalCall(); + return; + } + left = *p_left; + if (right == NULL || !PyUnicode_Check(left)) { + if (!PyErr_Occurred()) + PyErr_BadInternalCall(); + goto error; + } + + if (PyUnicode_READY(left)) + goto error; + if (PyUnicode_READY(right)) + goto error; + + /* Shortcuts */ + if (left == unicode_empty) { + Py_DECREF(left); + Py_INCREF(right); + *p_left = right; + return; + } + if (right == unicode_empty) + return; + + left_len = PyUnicode_GET_LENGTH(left); right_len = PyUnicode_GET_LENGTH(right); if (left_len > PY_SSIZE_T_MAX - right_len) { PyErr_SetString(PyExc_OverflowError, @@ -10625,78 +10661,47 @@ } new_len = left_len + right_len; - /* Now we own the last reference to 'left', so we can resize it - * in-place. - */ - if (unicode_resize(p_left, new_len) != 0) { - /* XXX if _PyUnicode_Resize() fails, 'left' has been - * deallocated so it cannot be put back into - * 'variable'. The MemoryError is raised when there - * is no value in 'variable', which might (very - * remotely) be a cause of incompatibilities. - */ - goto error; - } - /* copy 'right' into the newly allocated area of 'left' */ - copy_characters(*p_left, left_len, right, 0, right_len); - _PyUnicode_DIRTY(*p_left); - return; - -error: - Py_DECREF(*p_left); - *p_left = NULL; -} - -void -PyUnicode_Append(PyObject **p_left, PyObject *right) -{ - PyObject *left, *res; - - if (p_left == NULL) { - if (!PyErr_Occurred()) - PyErr_BadInternalCall(); - return; - } - left = *p_left; - if (right == NULL || !PyUnicode_Check(left)) { - if (!PyErr_Occurred()) - PyErr_BadInternalCall(); - goto error; - } - - if (PyUnicode_READY(left)) - goto error; - if (PyUnicode_READY(right)) - goto error; - - if (PyUnicode_CheckExact(left) && left != unicode_empty - && PyUnicode_CheckExact(right) && right != unicode_empty - && unicode_resizable(left) - && (_PyUnicode_KIND(right) <= _PyUnicode_KIND(left) - || _PyUnicode_WSTR(left) != NULL)) - { + if (unicode_modifiable(left) + && PyUnicode_CheckExact(right) + && PyUnicode_KIND(right) <= PyUnicode_KIND(left) /* Don't resize for ascii += latin1. Convert ascii to latin1 requires to change the structure size, but characters are stored just after the structure, and so it requires to move all characters which is not so different than duplicating the string. */ - if (!(PyUnicode_IS_ASCII(left) && !PyUnicode_IS_ASCII(right))) - { - unicode_append_inplace(p_left, right); - assert(p_left == NULL || _PyUnicode_CheckConsistency(*p_left, 1)); - return; - } - } - - res = PyUnicode_Concat(left, right); - if (res == NULL) - goto error; - Py_DECREF(left); - *p_left = res; + && !(PyUnicode_IS_ASCII(left) && !PyUnicode_IS_ASCII(right))) + { + /* append inplace */ + if (unicode_resize(p_left, new_len) != 0) { + /* XXX if _PyUnicode_Resize() fails, 'left' has been + * deallocated so it cannot be put back into + * 'variable'. The MemoryError is raised when there + * is no value in 'variable', which might (very + * remotely) be a cause of incompatibilities. + */ + goto error; + } + /* copy 'right' into the newly allocated area of 'left' */ + copy_characters(*p_left, left_len, right, 0, right_len); + } + else { + maxchar = PyUnicode_MAX_CHAR_VALUE(left); + maxchar2 = PyUnicode_MAX_CHAR_VALUE(right); + maxchar = Py_MAX(maxchar, maxchar2); + + /* Concat the two Unicode strings */ + res = PyUnicode_New(new_len, maxchar); + if (res == NULL) + goto error; + copy_characters(res, 0, left, 0, left_len); + copy_characters(res, left_len, right, 0, right_len); + Py_DECREF(left); + *p_left = res; + } + assert(_PyUnicode_CheckConsistency(*p_left, 1)); return; error: - Py_DECREF(*p_left); - *p_left = NULL; + Py_CLEAR(*p_left); } void -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Create unicode_result_unchanged() subfunction
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/6d498f469377 changeset: 73938:6d498f469377 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 22:44:26 2011 +0100 summary: Create unicode_result_unchanged() subfunction files: Objects/unicodeobject.c | 139 +++++++++++---------------- 1 files changed, 59 insertions(+), 80 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -487,6 +487,20 @@ return unicode_result_wchar(unicode); } +static PyObject* +unicode_result_unchanged(PyObject *unicode) +{ + if (PyUnicode_CheckExact(unicode)) { + if (PyUnicode_READY(unicode) < 0) + return NULL; + Py_INCREF(unicode); + return unicode; + } + else + /* Subtype -- return genuine unicode string with the same value. */ + return PyUnicode_Copy(unicode); +} + #ifdef HAVE_MBCS static OSVERSIONINFOEX winver; #endif @@ -3563,7 +3577,7 @@ PyErr_BadArgument(); return (Py_UCS4)-1; } - if (index < 0 || index >= _PyUnicode_LENGTH(unicode)) { + if (index < 0 || index >= PyUnicode_GET_LENGTH(unicode)) { PyErr_SetString(PyExc_IndexError, "string index out of range"); return (Py_UCS4)-1; } @@ -3577,7 +3591,7 @@ PyErr_BadArgument(); return -1; } - if (index < 0 || index >= _PyUnicode_LENGTH(unicode)) { + if (index < 0 || index >= PyUnicode_GET_LENGTH(unicode)) { PyErr_SetString(PyExc_IndexError, "string index out of range"); return -1; } @@ -9675,10 +9689,8 @@ if (right < 0) right = 0; - if (left == 0 && right == 0 && PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } + if (left == 0 && right == 0) + return unicode_result_unchanged(self); if (left > PY_SSIZE_T_MAX - _PyUnicode_LENGTH(self) || right > PY_SSIZE_T_MAX - (left + _PyUnicode_LENGTH(self))) { @@ -10214,11 +10226,8 @@ PyMem_FREE(buf1); if (release2) PyMem_FREE(buf2); - if (PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } - return PyUnicode_Copy(self); + return unicode_result_unchanged(self); + error: if (srelease && sbuf) PyMem_FREE(sbuf); @@ -10334,15 +10343,13 @@ if (!PyArg_ParseTuple(args, "n|O&:center", &width, convert_uc, &fillchar)) return NULL; - if (PyUnicode_READY(self) == -1) - return NULL; - - if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } - - marg = width - _PyUnicode_LENGTH(self); + if (PyUnicode_READY(self) < 0) + return NULL; + + if (PyUnicode_GET_LENGTH(self) >= width) + return unicode_result_unchanged(self); + + marg = width - PyUnicode_GET_LENGTH(self); left = marg / 2 + (marg & width & 1); return pad(self, left, marg - left, fillchar); @@ -10851,10 +10858,8 @@ line_pos = 0; } } - if (!found && PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } + if (!found) + return unicode_result_unchanged(self); /* Second pass: create output string and fill it */ u = PyUnicode_New(j, PyUnicode_MAX_CHAR_VALUE(self)); @@ -11489,18 +11494,16 @@ Py_ssize_t width; Py_UCS4 fillchar = ' '; - if (PyUnicode_READY(self) == -1) - return NULL; - if (!PyArg_ParseTuple(args, "n|O&:ljust", &width, convert_uc, &fillchar)) return NULL; - if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } - - return pad(self, 0, width - _PyUnicode_LENGTH(self), fillchar); + if (PyUnicode_READY(self) < 0) + return NULL; + + if (PyUnicode_GET_LENGTH(self) >= width) + return unicode_result_unchanged(self); + + return pad(self, 0, width - PyUnicode_GET_LENGTH(self), fillchar); } PyDoc_STRVAR(lower__doc__, @@ -11575,14 +11578,7 @@ end = Py_MIN(end, PyUnicode_GET_LENGTH(self)); if (start == 0 && end == PyUnicode_GET_LENGTH(self)) - { - if (PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } - else - return PyUnicode_Copy(self); - } + return unicode_result_unchanged(self); length = end - start; if (length == 1) @@ -11723,13 +11719,11 @@ return unicode_empty; } - if (len == 1 && PyUnicode_CheckExact(str)) { - /* no repeat, return original string */ - Py_INCREF(str); - return str; - } - - if (PyUnicode_READY(str) == -1) + /* no repeat, return original string */ + if (len == 1) + return unicode_result_unchanged(str); + + if (PyUnicode_READY(str) < 0) return NULL; if (PyUnicode_GET_LENGTH(str) > PY_SSIZE_T_MAX / len) { @@ -12079,15 +12073,13 @@ if (!PyArg_ParseTuple(args, "n|O&:rjust", &width, convert_uc, &fillchar)) return NULL; - if (PyUnicode_READY(self) == -1) - return NULL; - - if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } - - return pad(self, width - _PyUnicode_LENGTH(self), 0, fillchar); + if (PyUnicode_READY(self) < 0) + return NULL; + + if (PyUnicode_GET_LENGTH(self) >= width) + return unicode_result_unchanged(self); + + return pad(self, width - PyUnicode_GET_LENGTH(self), 0, fillchar); } PyObject * @@ -12380,12 +12372,7 @@ static PyObject *unicode_str(PyObject *self) { - if (PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } else - /* Subtype -- return genuine unicode string with the same value. */ - return PyUnicode_Copy(self); + return unicode_result_unchanged(self); } PyDoc_STRVAR(swapcase__doc__, @@ -12558,22 +12545,16 @@ void *data; Py_UCS4 chr; - if (PyUnicode_READY(self) == -1) - return NULL; - if (!PyArg_ParseTuple(args, "n:zfill", &width)) return NULL; - if (PyUnicode_GET_LENGTH(self) >= width) { - if (PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; - } - else - return PyUnicode_Copy(self); - } - - fill = width - _PyUnicode_LENGTH(self); + if (PyUnicode_READY(self) < 0) + return NULL; + + if (PyUnicode_GET_LENGTH(self) >= width) + return unicode_result_unchanged(self); + + fill = width - PyUnicode_GET_LENGTH(self); u = pad(self, fill, 0, '0'); @@ -12890,10 +12871,8 @@ Py_INCREF(unicode_empty); return unicode_empty; } else if (start == 0 && step == 1 && - slicelength == PyUnicode_GET_LENGTH(self) && - PyUnicode_CheckExact(self)) { - Py_INCREF(self); - return self; + slicelength == PyUnicode_GET_LENGTH(self)) { + return unicode_result_unchanged(self); } else if (step == 1) { return PyUnicode_Substring(self, start, start + slicelength); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix fixup() for unchanged unicode subtype
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/7a6ff1275b16 changeset: 73937:7a6ff1275b16 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 22:22:39 2011 +0100 summary: Fix fixup() for unchanged unicode subtype If maxchar_new == 0 and self is a unicode subtype, return u instead of duplicating u. files: Objects/unicodeobject.c | 70 ++++++++++++++-------------- 1 files changed, 35 insertions(+), 35 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9211,6 +9211,7 @@ { PyObject *u; Py_UCS4 maxchar_old, maxchar_new = 0; + PyObject *v; u = PyUnicode_Copy(self); if (u == NULL) @@ -9222,9 +9223,19 @@ everything is fine. Otherwise we need to change the string kind and re-run the fix function. */ maxchar_new = fixfct(u); - if (maxchar_new == 0) - /* do nothing, keep maxchar_new at 0 which means no changes. */; - else if (maxchar_new <= 127) + + if (maxchar_new == 0) { + /* no changes */; + if (PyUnicode_CheckExact(self)) { + Py_DECREF(u); + Py_INCREF(self); + return self; + } + else + return u; + } + + if (maxchar_new <= 127) maxchar_new = 127; else if (maxchar_new <= 255) maxchar_new = 255; @@ -9233,41 +9244,30 @@ else maxchar_new = MAX_UNICODE; - if (!maxchar_new && PyUnicode_CheckExact(self)) { - /* fixfct should return TRUE if it modified the buffer. If - FALSE, return a reference to the original buffer instead - (to save space, not time) */ - Py_INCREF(self); + if (maxchar_new == maxchar_old) + return u; + + /* In case the maximum character changed, we need to + convert the string to the new category. */ + v = PyUnicode_New(PyUnicode_GET_LENGTH(self), maxchar_new); + if (v == NULL) { Py_DECREF(u); - return self; - } - else if (maxchar_new == maxchar_old) { - return u; + return NULL; + } + if (maxchar_new > maxchar_old) { + /* If the maxchar increased so that the kind changed, not all + characters are representable anymore and we need to fix the + string again. This only happens in very few cases. */ + copy_characters(v, 0, self, 0, PyUnicode_GET_LENGTH(self)); + maxchar_old = fixfct(v); + assert(maxchar_old > 0 && maxchar_old <= maxchar_new); } else { - /* In case the maximum character changed, we need to - convert the string to the new category. */ - PyObject *v = PyUnicode_New(PyUnicode_GET_LENGTH(self), maxchar_new); - if (v == NULL) { - Py_DECREF(u); - return NULL; - } - if (maxchar_new > maxchar_old) { - /* If the maxchar increased so that the kind changed, not all - characters are representable anymore and we need to fix the - string again. This only happens in very few cases. */ - copy_characters(v, 0, self, 0, PyUnicode_GET_LENGTH(self)); - maxchar_old = fixfct(v); - assert(maxchar_old > 0 && maxchar_old <= maxchar_new); - } - else { - copy_characters(v, 0, u, 0, PyUnicode_GET_LENGTH(self)); - } - - Py_DECREF(u); - assert(_PyUnicode_CheckConsistency(v, 1)); - return v; - } + copy_characters(v, 0, u, 0, PyUnicode_GET_LENGTH(self)); + } + Py_DECREF(u); + assert(_PyUnicode_CheckConsistency(v, 1)); + return v; } static Py_UCS4 -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Call directly PyUnicode_DecodeUTF8Stateful() instead of PyUnicode_DecodeUTF8()
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/9538a4e2ec08 changeset: 73935:9538a4e2ec08 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 21:53:09 2011 +0100 summary: Call directly PyUnicode_DecodeUTF8Stateful() instead of PyUnicode_DecodeUTF8() * Remove micro-optimization from PyUnicode_FromStringAndSize(): PyUnicode_DecodeUTF8Stateful() has already these optimizations (for size=0 and one ascii char). * Rename utf8_max_char_size_and_char_count() to utf8_scanner(), and remove an useless variable files: Objects/unicodeobject.c | 47 ++++++++-------------------- 1 files changed, 14 insertions(+), 33 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1717,28 +1717,10 @@ "Negative size passed to PyUnicode_FromStringAndSize"); return NULL; } - - /* If the Unicode data is known at construction time, we can apply - some optimizations which share commonly used objects. - Also, this means the input must be UTF-8, so fall back to the - UTF-8 decoder at the end. */ - if (u != NULL) { - - /* Optimization for empty strings */ - if (size == 0 && unicode_empty != NULL) { - Py_INCREF(unicode_empty); - return unicode_empty; - } - - /* Single characters are shared when using this constructor. - Restrict to ASCII, since the input must be UTF-8. */ - if (size == 1 && (unsigned char)*u < 128) - return get_latin1_char((unsigned char)*u); - - return PyUnicode_DecodeUTF8(u, size, NULL); - } - - return (PyObject *)_PyUnicode_New(size); + if (u != NULL) + return PyUnicode_DecodeUTF8Stateful(u, size, NULL, NULL); + else + return (PyObject *)_PyUnicode_New(size); } PyObject * @@ -1749,15 +1731,16 @@ PyErr_SetString(PyExc_OverflowError, "input too long"); return NULL; } - - return PyUnicode_FromStringAndSize(u, size); + return PyUnicode_DecodeUTF8Stateful(u, (Py_ssize_t)size, NULL, NULL); } PyObject * _PyUnicode_FromId(_Py_Identifier *id) { if (!id->object) { - id->object = PyUnicode_FromString(id->string); + id->object = PyUnicode_DecodeUTF8Stateful(id->string, + strlen(id->string), + NULL, NULL); if (!id->object) return NULL; PyUnicode_InternInPlace(&id->object); @@ -2443,7 +2426,7 @@ { /* UTF-8 */ const char *s = va_arg(count, const char*); - PyObject *str = PyUnicode_DecodeUTF8(s, strlen(s), "replace"); + PyObject *str = PyUnicode_DecodeUTF8Stateful(s, strlen(s), "replace", NULL); if (!str) goto fail; /* since PyUnicode_DecodeUTF8 returns already flexible @@ -2482,7 +2465,7 @@ *callresult++ = NULL; } else { - str_obj = PyUnicode_DecodeUTF8(str, strlen(str), "replace"); + str_obj = PyUnicode_DecodeUTF8Stateful(str, strlen(str), "replace", NULL); if (!str_obj) goto fail; if (PyUnicode_READY(str_obj)) { @@ -2947,7 +2930,7 @@ if (normalize_encoding(encoding, lower, sizeof(lower))) { if ((strcmp(lower, "utf-8") == 0) || (strcmp(lower, "utf8") == 0)) - return PyUnicode_DecodeUTF8(s, size, errors); + return PyUnicode_DecodeUTF8Stateful(s, size, errors, NULL); else if ((strcmp(lower, "latin-1") == 0) || (strcmp(lower, "latin1") == 0) || (strcmp(lower, "iso-8859-1") == 0)) @@ -3260,7 +3243,7 @@ #ifdef HAVE_MBCS return PyUnicode_DecodeMBCS(s, size, NULL); #elif defined(__APPLE__) - return PyUnicode_DecodeUTF8(s, size, "surrogateescape"); + return PyUnicode_DecodeUTF8Stateful(s, size, "surrogateescape", NULL); #else PyInterpreterState *interp = PyThreadState_GET()->interp; /* Bootstrap check: if the filesystem codec is implemented in Python, we @@ -4240,11 +4223,9 @@ PyUnicode_DecodeUTF8Stateful. */ static Py_UCS4 -utf8_max_char_size_and_char_count(const char *s, Py_ssize_t string_size, - Py_ssize_t *unicode_size) +utf8_scanner(const unsigned char *p, Py_ssize_t string_size, Py_ssize_t *unicode_size) { Py_ssize_t char_count = 0; - const unsigned char *p = (const unsigned char *)s; const unsigned char *end = p + string_size; const unsigned char *aligned_end = (const unsigned char *) ((size_t) end & ~LONG_PTR_MASK); @@ -4563,7 +4544,7 @@ return unicode_empty; } - maxchar = utf8_max_char_size_and_char_count(s, size, &unicode_size); + maxchar = utf8_scanner((const unsigned char *)s, size, &unicode_size); /* When the string is ASCII only, just use memcpy and return. unicode_size may be != size if there is an incomplete UTF-8 -- Repository URL: http://hg.python.org/cpython

1 0

cpython: unicode_fromascii() doesn't check string content twice in debug mode
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/d728637f5d78 changeset: 73936:d728637f5d78 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 21:54:30 2011 +0100 summary: unicode_fromascii() doesn't check string content twice in debug mode _PyUnicode_CheckConsistency() also checks string content. files: Objects/unicodeobject.c | 11 ++++------- 1 files changed, 4 insertions(+), 7 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1768,15 +1768,12 @@ unicode_fromascii(const unsigned char* s, Py_ssize_t size) { PyObject *unicode; + if (size == 1) { #ifdef Py_DEBUG - const unsigned char *p; - const unsigned char *end = s + size; - for (p=s; p < end; p++) { - assert(*p < 128); - } -#endif - if (size == 1) + assert(s[0] < 128); +#endif return get_latin1_char(s[0]); + } unicode = PyUnicode_New(size, 127); if (!unicode) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Use directly unicode_empty instead of PyUnicode_New(0, 0)
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/7d4051e620fe changeset: 73934:7d4051e620fe user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 21:44:00 2011 +0100 summary: Use directly unicode_empty instead of PyUnicode_New(0, 0) files: Objects/unicodeobject.c | 18 ++++++++++++------ 1 files changed, 12 insertions(+), 6 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2122,8 +2122,10 @@ PyUnicode_FromWideChar(register const wchar_t *w, Py_ssize_t size) { if (w == NULL) { - if (size == 0) - return PyUnicode_New(0, 0); + if (size == 0) { + Py_INCREF(unicode_empty); + return unicode_empty; + } PyErr_BadInternalCall(); return NULL; } @@ -4557,7 +4559,8 @@ if (size == 0) { if (consumed) *consumed = 0; - return (PyObject *)PyUnicode_New(0, 0); + Py_INCREF(unicode_empty); + return unicode_empty; } maxchar = utf8_max_char_size_and_char_count(s, size, &unicode_size); @@ -12906,7 +12909,8 @@ } if (slicelength <= 0) { - return PyUnicode_New(0, 0); + Py_INCREF(unicode_empty); + return unicode_empty; } else if (start == 0 && step == 1 && slicelength == PyUnicode_GET_LENGTH(self) && PyUnicode_CheckExact(self)) { @@ -13582,8 +13586,10 @@ if (!PyArg_ParseTupleAndKeywords(args, kwds, "|Oss:str", kwlist, &x, &encoding, &errors)) return NULL; - if (x == NULL) - return PyUnicode_New(0, 0); + if (x == NULL) { + Py_INCREF(unicode_empty); + return unicode_empty; + } if (encoding == NULL && errors == NULL) return PyObject_Str(x); else -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Move the slowest UTF-8 decoder to its own subfunction
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/8815966e881a changeset: 73933:8815966e881a user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 20:09:03 2011 +0100 summary: Move the slowest UTF-8 decoder to its own subfunction * Create decode_utf8_errors() * Reuse unicode_fromascii() * decode_utf8_errors() doesn't refit at the beginning * Remove refit_partial_string(), use unicode_adjust_maxchar() instead files: Objects/unicodeobject.c | 230 ++++++++++++--------------- 1 files changed, 100 insertions(+), 130 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1784,7 +1784,7 @@ static PyObject* unicode_fromascii(const unsigned char* s, Py_ssize_t size) { - PyObject *res; + PyObject *unicode; #ifdef Py_DEBUG const unsigned char *p; const unsigned char *end = s + size; @@ -1794,11 +1794,12 @@ #endif if (size == 1) return get_latin1_char(s[0]); - res = PyUnicode_New(size, 127); - if (!res) - return NULL; - memcpy(PyUnicode_1BYTE_DATA(res), s, size); - return res; + unicode = PyUnicode_New(size, 127); + if (!unicode) + return NULL; + memcpy(PyUnicode_1BYTE_DATA(unicode), s, size); + assert(_PyUnicode_CheckConsistency(unicode, 1)); + return unicode; } static Py_UCS4 @@ -4320,126 +4321,38 @@ return 65537; } -/* Called when we encountered some error that wasn't detected in the original - scan, e.g. an encoded surrogate character. The original maxchar computation - may have been incorrect, so redo it. */ -static int -refit_partial_string(PyObject **unicode, int kind, void *data, Py_ssize_t n) -{ - PyObject *tmp; - Py_ssize_t k; - Py_UCS4 maxchar; - for (k = 0, maxchar = 0; k < n; k++) - maxchar = Py_MAX(maxchar, PyUnicode_READ(kind, data, k)); - tmp = PyUnicode_New(PyUnicode_GET_LENGTH(*unicode), maxchar); - if (tmp == NULL) - return -1; - PyUnicode_CopyCharacters(tmp, 0, *unicode, 0, n); - Py_DECREF(*unicode); - *unicode = tmp; - return 0; -} - /* Similar to PyUnicode_WRITE but may attempt to widen and resize the string - in case of errors. Implicit parameters: unicode, kind, data, has_errors, - onError. Potential resizing overallocates, so the result needs to shrink - at the end. + in case of errors. Implicit parameters: unicode, kind, data, onError. + Potential resizing overallocates, so the result needs to shrink at the end. */ -#define WRITE_MAYBE_FAIL(index, value) \ - do { \ - if (has_errors) { \ - Py_ssize_t pos = index; \ - if (pos > PyUnicode_GET_LENGTH(unicode) && \ - unicode_resize(&unicode, pos + pos/8) < 0) \ - goto onError; \ - if (unicode_putchar(&unicode, &pos, value) < 0) \ - goto onError; \ - } \ - else \ - PyUnicode_WRITE(kind, data, index, value); \ +#define WRITE_MAYBE_FAIL(index, value) \ + do { \ + Py_ssize_t pos = index; \ + if (pos > PyUnicode_GET_LENGTH(unicode) && \ + unicode_resize(&unicode, pos + pos/8) < 0) \ + goto onError; \ + if (unicode_putchar(&unicode, &pos, value) < 0) \ + goto onError; \ } while (0) PyObject * -PyUnicode_DecodeUTF8Stateful(const char *s, - Py_ssize_t size, - const char *errors, - Py_ssize_t *consumed) -{ - const char *starts = s; +decode_utf8_errors(const char *starts, + Py_ssize_t size, + const char *errors, + Py_ssize_t *consumed, + const char *s, + PyObject *unicode, + Py_ssize_t i) +{ int n; int k; Py_ssize_t startinpos; Py_ssize_t endinpos; - const char *e, *aligned_end; - PyObject *unicode; + const char *e = starts + size; + const char *aligned_end; const char *errmsg = ""; PyObject *errorHandler = NULL; PyObject *exc = NULL; - Py_UCS4 maxchar = 0; - Py_ssize_t unicode_size; - Py_ssize_t i; - int kind; - void *data; - int has_errors = 0; - - if (size == 0) { - if (consumed) - *consumed = 0; - return (PyObject *)PyUnicode_New(0, 0); - } - maxchar = utf8_max_char_size_and_char_count(s, size, &unicode_size); - /* When the string is ASCII only, just use memcpy and return. - unicode_size may be != size if there is an incomplete UTF-8 - sequence at the end of the ASCII block. */ - if (maxchar < 128 && size == unicode_size) { - if (consumed) - *consumed = size; - - if (size == 1) - return get_latin1_char((unsigned char)s[0]); - - unicode = PyUnicode_New(unicode_size, maxchar); - if (!unicode) - return NULL; - Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size); - assert(_PyUnicode_CheckConsistency(unicode, 1)); - return unicode; - } - - /* In case of errors, maxchar and size computation might be incorrect; - code below refits and resizes as necessary. */ - unicode = PyUnicode_New(unicode_size, maxchar); - if (!unicode) - return NULL; - kind = PyUnicode_KIND(unicode); - data = PyUnicode_DATA(unicode); - - /* Unpack UTF-8 encoded data */ - i = 0; - e = s + size; - switch (kind) { - case PyUnicode_1BYTE_KIND: - has_errors = ucs1lib_utf8_try_decode(s, e, (Py_UCS1 *) data, &s, &i); - break; - case PyUnicode_2BYTE_KIND: - has_errors = ucs2lib_utf8_try_decode(s, e, (Py_UCS2 *) data, &s, &i); - break; - case PyUnicode_4BYTE_KIND: - has_errors = ucs4lib_utf8_try_decode(s, e, (Py_UCS4 *) data, &s, &i); - break; - } - if (!has_errors) { - /* Ensure the unicode size calculation was correct */ - assert(i == unicode_size); - assert(s == e); - if (consumed) - *consumed = s-starts; - return unicode; - } - /* Fall through to the generic decoding loop for the rest of - the string */ - if (refit_partial_string(&unicode, kind, data, i) < 0) - goto onError; aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK); @@ -4591,11 +4504,6 @@ continue; utf8Error: - if (!has_errors) { - if (refit_partial_string(&unicode, kind, data, i) < 0) - goto onError; - has_errors = 1; - } if (unicode_decode_call_errorhandler( errors, &errorHandler, "utf8", errmsg, @@ -4604,22 +4512,18 @@ goto onError; /* Update data because unicode_decode_call_errorhandler might have re-created or resized the unicode object. */ - data = PyUnicode_DATA(unicode); - kind = PyUnicode_KIND(unicode); aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK); } - /* Ensure the unicode_size calculation above was correct: */ - assert(has_errors || i == unicode_size); - if (consumed) *consumed = s-starts; /* Adjust length and ready string when it contained errors and is of the old resizable kind. */ - if (has_errors) { - if (PyUnicode_Resize(&unicode, i) < 0) - goto onError; - } + if (unicode_resize(&unicode, i) < 0) + goto onError; + unicode_adjust_maxchar(&unicode); + if (unicode == NULL) + goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); @@ -4629,12 +4533,78 @@ onError: Py_XDECREF(errorHandler); Py_XDECREF(exc); - Py_DECREF(unicode); + Py_XDECREF(unicode); return NULL; } - #undef WRITE_MAYBE_FAIL +PyObject * +PyUnicode_DecodeUTF8Stateful(const char *s, + Py_ssize_t size, + const char *errors, + Py_ssize_t *consumed) +{ + Py_UCS4 maxchar = 0; + Py_ssize_t unicode_size; + int has_errors = 0; + PyObject *unicode; + int kind; + void *data; + const char *starts = s; + const char *e; + Py_ssize_t i; + + if (size == 0) { + if (consumed) + *consumed = 0; + return (PyObject *)PyUnicode_New(0, 0); + } + + maxchar = utf8_max_char_size_and_char_count(s, size, &unicode_size); + + /* When the string is ASCII only, just use memcpy and return. + unicode_size may be != size if there is an incomplete UTF-8 + sequence at the end of the ASCII block. */ + if (maxchar < 128 && size == unicode_size) { + if (consumed) + *consumed = size; + return unicode_fromascii(s, size); + } + + unicode = PyUnicode_New(unicode_size, maxchar); + if (!unicode) + return NULL; + kind = PyUnicode_KIND(unicode); + data = PyUnicode_DATA(unicode); + + /* Unpack UTF-8 encoded data */ + i = 0; + e = starts + size; + switch (kind) { + case PyUnicode_1BYTE_KIND: + has_errors = ucs1lib_utf8_try_decode(s, e, (Py_UCS1 *) data, &s, &i); + break; + case PyUnicode_2BYTE_KIND: + has_errors = ucs2lib_utf8_try_decode(s, e, (Py_UCS2 *) data, &s, &i); + break; + case PyUnicode_4BYTE_KIND: + has_errors = ucs4lib_utf8_try_decode(s, e, (Py_UCS4 *) data, &s, &i); + break; + } + if (!has_errors) { + /* Ensure the unicode size calculation was correct */ + assert(i == unicode_size); + assert(s == e); + if (consumed) + *consumed = size; + return unicode; + } + + /* In case of errors, maxchar and size computation might be incorrect; + code below refits and resizes as necessary. */ + return decode_utf8_errors(starts, size, errors, consumed, s, unicode, i); +} + #ifdef __APPLE__ /* Simplified UTF-8 decoder using surrogateescape error handler, -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix error handling in resize_compact()
by victor.stinner Dec. 12, 2011

Dec. 12, 2011

http://hg.python.org/cpython/rev/21d7f5fdd34c changeset: 73932:21d7f5fdd34c user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sun Dec 11 20:04:56 2011 +0100 summary: Fix error handling in resize_compact() files: Objects/unicodeobject.c | 20 ++++++++++++-------- 1 files changed, 12 insertions(+), 8 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -630,6 +630,7 @@ Py_ssize_t struct_size; Py_ssize_t new_size; int share_wstr; + PyObject *new_unicode; assert(PyUnicode_IS_READY(unicode)); char_size = PyUnicode_KIND(unicode); @@ -639,22 +640,25 @@ struct_size = sizeof(PyCompactUnicodeObject); share_wstr = _PyUnicode_SHARE_WSTR(unicode); + if (length > ((PY_SSIZE_T_MAX - struct_size) / char_size - 1)) { + Py_DECREF(unicode); + PyErr_NoMemory(); + return NULL; + } + new_size = (struct_size + (length + 1) * char_size); + _Py_DEC_REFTOTAL; _Py_ForgetReference(unicode); - if (length > ((PY_SSIZE_T_MAX - struct_size) / char_size - 1)) { - PyErr_NoMemory(); - return NULL; - } - new_size = (struct_size + (length + 1) * char_size); - - unicode = (PyObject *)PyObject_REALLOC((char *)unicode, new_size); - if (unicode == NULL) { + new_unicode = (PyObject *)PyObject_REALLOC((char *)unicode, new_size); + if (new_unicode == NULL) { PyObject_Del(unicode); PyErr_NoMemory(); return NULL; } + unicode = new_unicode; _Py_NewReference(unicode); + _PyUnicode_LENGTH(unicode) = length; if (share_wstr) { _PyUnicode_WSTR(unicode) = PyUnicode_DATA(unicode); -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (2bbe7dc920de): sum=0
by solipsis＠pitrou.net Dec. 11, 2011

Dec. 11, 2011

results for 2bbe7dc920de on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogbSizE7', '-x']

1 0

cpython (merge default -> default): merge heads
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/2bbe7dc920de changeset: 73931:2bbe7dc920de parent: 73930:8d670cb0d64a parent: 73929:6cd736239b8a user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 17:55:31 2011 -0500 summary: merge heads files: Doc/library/tarfile.rst | 22 +++++-- Lib/ftplib.py | 25 +------- Lib/tarfile.py | 66 +++++++++++++++++++++-- Lib/test/test_minidom.py | 2 +- Lib/test/test_tarfile.py | 78 ++++++++++++++++++++++++--- Lib/xml/dom/pulldom.py | 6 +- Misc/NEWS | 2 + 7 files changed, 153 insertions(+), 48 deletions(-) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -13,12 +13,12 @@ -------------- The :mod:`tarfile` module makes it possible to read and write tar -archives, including those using gzip or bz2 compression. +archives, including those using gzip, bz2 and lzma compression. (:file:`.zip` files can be read and written using the :mod:`zipfile` module.) Some facts and figures: -* reads and writes :mod:`gzip` and :mod:`bz2` compressed archives. +* reads and writes :mod:`gzip`, :mod:`bz2` and :mod:`lzma` compressed archives. * read/write support for the POSIX.1-1988 (ustar) format. @@ -55,6 +55,8 @@ +------------------+---------------------------------------------+ | ``'r:bz2'`` | Open for reading with bzip2 compression. | +------------------+---------------------------------------------+ + | ``'r:xz'`` | Open for reading with lzma compression. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -64,11 +66,13 @@ +------------------+---------------------------------------------+ | ``'w:bz2'`` | Open for bzip2 compressed writing. | +------------------+---------------------------------------------+ + | ``'w:xz'`` | Open for lzma compressed writing. | + +------------------+---------------------------------------------+ - Note that ``'a:gz'`` or ``'a:bz2'`` is not possible. If *mode* is not suitable - to open a certain (compressed) file for reading, :exc:`ReadError` is raised. Use - *mode* ``'r'`` to avoid this. If a compression method is not supported, - :exc:`CompressionError` is raised. + Note that ``'a:gz'``, ``'a:bz2'`` or ``'a:xz'`` is not possible. If *mode* + is not suitable to open a certain (compressed) file for reading, + :exc:`ReadError` is raised. Use *mode* ``'r'`` to avoid this. If a + compression method is not supported, :exc:`CompressionError` is raised. If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. @@ -99,6 +103,9 @@ | ``'r|bz2'`` | Open a bzip2 compressed *stream* for | | | reading. | +-------------+--------------------------------------------+ + | ``'r|xz'`` | Open a lzma compressed *stream* for | + | | reading. | + +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | +-------------+--------------------------------------------+ | ``'w|gz'`` | Open a gzip compressed *stream* for | @@ -107,6 +114,9 @@ | ``'w|bz2'`` | Open a bzip2 compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ + | ``'w|xz'`` | Open an lzma compressed *stream* for | + | | writing. | + +-------------+--------------------------------------------+ .. class:: TarFile diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -359,8 +359,7 @@ conn.close() raise else: - sock = self.makeport() - try: + with self.makeport() as sock: if rest is not None: self.sendcmd("REST %s" % rest) resp = self.sendcmd(cmd) @@ -372,8 +371,6 @@ conn, sockaddr = sock.accept() if self.timeout is not _GLOBAL_DEFAULT_TIMEOUT: conn.settimeout(self.timeout) - finally: - sock.close() if resp[:3] == '150': # this is conditional in case we received a 125 size = parse150(resp) @@ -753,8 +750,7 @@ def retrbinary(self, cmd, callback, blocksize=8192, rest=None): self.voidcmd('TYPE I') - conn = self.transfercmd(cmd, rest) - try: + with self.transfercmd(cmd, rest) as conn: while 1: data = conn.recv(blocksize) if not data: @@ -763,8 +759,6 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - conn.close() return self.voidresp() def retrlines(self, cmd, callback = None): @@ -772,7 +766,7 @@ resp = self.sendcmd('TYPE A') conn = self.transfercmd(cmd) fp = conn.makefile('r', encoding=self.encoding) - try: + with fp, conn: while 1: line = fp.readline() if self.debugging > 2: print('*retr*', repr(line)) @@ -786,15 +780,11 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - fp.close() - conn.close() return self.voidresp() def storbinary(self, cmd, fp, blocksize=8192, callback=None, rest=None): self.voidcmd('TYPE I') - conn = self.transfercmd(cmd, rest) - try: + with self.transfercmd(cmd, rest) as conn: while 1: buf = fp.read(blocksize) if not buf: break @@ -803,14 +793,11 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - conn.close() return self.voidresp() def storlines(self, cmd, fp, callback=None): self.voidcmd('TYPE A') - conn = self.transfercmd(cmd) - try: + with self.transfercmd(cmd) as conn: while 1: buf = fp.readline() if not buf: break @@ -822,8 +809,6 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - conn.close() return self.voidresp() def abort(self): diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -420,10 +420,11 @@ self.crc = zlib.crc32(b"") if mode == "r": self._init_read_gz() + self.exception = zlib.error else: self._init_write_gz() - if comptype == "bz2": + elif comptype == "bz2": try: import bz2 except ImportError: @@ -431,8 +432,25 @@ if mode == "r": self.dbuf = b"" self.cmp = bz2.BZ2Decompressor() + self.exception = IOError else: self.cmp = bz2.BZ2Compressor() + + elif comptype == "xz": + try: + import lzma + except ImportError: + raise CompressionError("lzma module is not available") + if mode == "r": + self.dbuf = b"" + self.cmp = lzma.LZMADecompressor() + self.exception = lzma.LZMAError + else: + self.cmp = lzma.LZMACompressor() + + elif comptype != "tar": + raise CompressionError("unknown compression type %r" % comptype) + except: if not self._extfileobj: self.fileobj.close() @@ -584,7 +602,7 @@ break try: buf = self.cmp.decompress(buf) - except IOError: + except self.exception: raise ReadError("invalid compressed data") self.dbuf += buf c += len(buf) @@ -622,11 +640,14 @@ return self.buf def getcomptype(self): - if self.buf.startswith(b"\037\213\010"): + if self.buf.startswith(b"\x1f\x8b\x08"): return "gz" - if self.buf[0:3] == b"BZh" and self.buf[4:10] == b"1AY&SY": + elif self.buf[0:3] == b"BZh" and self.buf[4:10] == b"1AY&SY": return "bz2" - return "tar" + elif self.buf.startswith((b"\x5d\x00\x00\x80", b"\xfd7zXZ")): + return "xz" + else: + return "tar" def close(self): self.fileobj.close() @@ -1651,18 +1672,22 @@ 'r:' open for reading exclusively uncompressed 'r:gz' open for reading with gzip compression 'r:bz2' open for reading with bzip2 compression + 'r:xz' open for reading with lzma compression 'a' or 'a:' open for appending, creating the file if necessary 'w' or 'w:' open for writing without compression 'w:gz' open for writing with gzip compression 'w:bz2' open for writing with bzip2 compression + 'w:xz' open for writing with lzma compression 'r|*' open a stream of tar blocks with transparent compression 'r|' open an uncompressed stream of tar blocks for reading 'r|gz' open a gzip compressed stream of tar blocks 'r|bz2' open a bzip2 compressed stream of tar blocks + 'r|xz' open an lzma compressed stream of tar blocks 'w|' open an uncompressed stream for writing 'w|gz' open a gzip compressed stream for writing 'w|bz2' open a bzip2 compressed stream for writing + 'w|xz' open an lzma compressed stream for writing """ if not name and not fileobj: @@ -1780,11 +1805,40 @@ t._extfileobj = False return t + @classmethod + def xzopen(cls, name, mode="r", fileobj=None, preset=9, **kwargs): + """Open lzma compressed tar archive name for reading or writing. + Appending is not allowed. + """ + if mode not in ("r", "w"): + raise ValueError("mode must be 'r' or 'w'") + + try: + import lzma + except ImportError: + raise CompressionError("lzma module is not available") + + if mode == "r": + # LZMAFile complains about a preset argument in read mode. + preset = None + + fileobj = lzma.LZMAFile(filename=name if fileobj is None else None, + mode=mode, fileobj=fileobj, preset=preset) + + try: + t = cls.taropen(name, mode, fileobj, **kwargs) + except (lzma.LZMAError, EOFError): + fileobj.close() + raise ReadError("not an lzma file") + t._extfileobj = False + return t + # All *open() methods are registered here. OPEN_METH = { "tar": "taropen", # uncompressed tar "gz": "gzopen", # gzip compressed tar - "bz2": "bz2open" # bzip2 compressed tar + "bz2": "bz2open", # bzip2 compressed tar + "xz": "xzopen" # lzma compressed tar } #-------------------------------------------------------------------------- diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -47,7 +47,7 @@ def checkWholeText(self, node, s): t = node.wholeText - self.confirm(t == s, "looking for %s, found %s" % (repr(s), repr(t))) + self.confirm(t == s, "looking for %r, found %r" % (s, t)) def testParseFromFile(self): with open(tstfile) as file: diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -21,6 +21,10 @@ import bz2 except ImportError: bz2 = None +try: + import lzma +except ImportError: + lzma = None def md5sum(data): return md5(data).hexdigest() @@ -29,6 +33,7 @@ tarname = support.findfile("testtar.tar") gzipname = os.path.join(TEMPDIR, "testtar.tar.gz") bz2name = os.path.join(TEMPDIR, "testtar.tar.bz2") +xzname = os.path.join(TEMPDIR, "testtar.tar.xz") tmpname = os.path.join(TEMPDIR, "tmp.tar") md5_regtype = "65f477c818ad9e15f7feab0c6d37742f" @@ -201,13 +206,15 @@ _open = gzip.GzipFile elif self.mode.endswith(":bz2"): _open = bz2.BZ2File + elif self.mode.endswith(":xz"): + _open = lzma.LZMAFile else: - _open = open + _open = io.FileIO for char in (b'\0', b'a'): # Test if EOFHeaderError ('\0') and InvalidHeaderError ('a') # are ignored correctly. - with _open(tmpname, "wb") as fobj: + with _open(tmpname, "w") as fobj: fobj.write(char * 1024) fobj.write(tarfile.TarInfo("foo").tobuf()) @@ -222,9 +229,10 @@ class MiscReadTest(CommonReadTest): def test_no_name_argument(self): - if self.mode.endswith("bz2"): - # BZ2File has no name attribute. - return + if self.mode.endswith(("bz2", "xz")): + # BZ2File and LZMAFile have no name attribute. + self.skipTest("no name attribute") + with open(self.tarname, "rb") as fobj: tar = tarfile.open(fileobj=fobj, mode=self.mode) self.assertEqual(tar.name, os.path.abspath(fobj.name)) @@ -265,10 +273,12 @@ _open = gzip.GzipFile elif self.mode.endswith(":bz2"): _open = bz2.BZ2File + elif self.mode.endswith(":xz"): + _open = lzma.LZMAFile else: - _open = open - fobj = _open(self.tarname, "rb") - try: + _open = io.FileIO + + with _open(self.tarname) as fobj: fobj.seek(offset) # Test if the tarfile starts with the second member. @@ -281,8 +291,6 @@ self.assertEqual(tar.extractfile(t).read(), data, "seek back did not work") tar.close() - finally: - fobj.close() def test_fail_comp(self): # For Gzip and Bz2 Tests: fail with a ReadError on an uncompressed file. @@ -526,6 +534,18 @@ testfunc(bz2name, "r|*") testfunc(bz2name, "r|bz2") + if lzma: + self.assertRaises(tarfile.ReadError, tarfile.open, tarname, mode="r:xz") + self.assertRaises(tarfile.ReadError, tarfile.open, tarname, mode="r|xz") + self.assertRaises(tarfile.ReadError, tarfile.open, xzname, mode="r:") + self.assertRaises(tarfile.ReadError, tarfile.open, xzname, mode="r|") + + testfunc(xzname, "r") + testfunc(xzname, "r:*") + testfunc(xzname, "r:xz") + testfunc(xzname, "r|*") + testfunc(xzname, "r|xz") + def test_detect_file(self): self._test_modes(self._testfunc_file) @@ -1096,6 +1116,9 @@ data = dec.decompress(data) self.assertTrue(len(dec.unused_data) == 0, "found trailing data") + elif self.mode.endswith("xz"): + with lzma.LZMAFile(tmpname) as fobj: + data = fobj.read() else: with open(tmpname, "rb") as fobj: data = fobj.read() @@ -1510,6 +1533,12 @@ self._create_testtar("w:bz2") self.assertRaises(tarfile.ReadError, tarfile.open, tmpname, "a") + def test_append_lzma(self): + if lzma is None: + self.skipTest("lzma module not available") + self._create_testtar("w:xz") + self.assertRaises(tarfile.ReadError, tarfile.open, tmpname, "a") + # Append mode is supposed to fail if the tarfile to append to # does not end with a zero block. def _test_error(self, data): @@ -1788,6 +1817,21 @@ self._test_partial_input("r:bz2") +class LzmaMiscReadTest(MiscReadTest): + tarname = xzname + mode = "r:xz" +class LzmaUstarReadTest(UstarReadTest): + tarname = xzname + mode = "r:xz" +class LzmaStreamReadTest(StreamReadTest): + tarname = xzname + mode = "r|xz" +class LzmaWriteTest(WriteTest): + mode = "w:xz" +class LzmaStreamWriteTest(StreamWriteTest): + mode = "w|xz" + + def test_main(): support.unlink(TEMPDIR) os.makedirs(TEMPDIR) @@ -1850,6 +1894,20 @@ Bz2PartialReadTest, ] + if lzma: + # Create testtar.tar.xz and add lzma-specific tests. + support.unlink(xzname) + with lzma.LZMAFile(xzname, "w") as tar: + tar.write(data) + + tests += [ + LzmaMiscReadTest, + LzmaUstarReadTest, + LzmaStreamReadTest, + LzmaWriteTest, + LzmaStreamWriteTest, + ] + try: support.run_unittest(*tests) finally: diff --git a/Lib/xml/dom/pulldom.py b/Lib/xml/dom/pulldom.py --- a/Lib/xml/dom/pulldom.py +++ b/Lib/xml/dom/pulldom.py @@ -1,6 +1,5 @@ import xml.sax import xml.sax.handler -import types START_ELEMENT = "START_ELEMENT" END_ELEMENT = "END_ELEMENT" @@ -334,10 +333,7 @@ return DOMEventStream(stream, parser, bufsize) def parseString(string, parser=None): - try: - from io import StringIO - except ImportError: - from io import StringIO + from io import StringIO bufsize = len(string) buf = StringIO(string) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -408,6 +408,8 @@ - Alias resource.error to OSError ala PEP 3151. +- Issue #5689: Add support for lzma compression to the tarfile module. + - Issue #13248: Turn 3.2's PendingDeprecationWarning into 3.3's DeprecationWarning. It covers 'cgi.escape', 'importlib.abc.PyLoader', 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', -- Repository URL: http://hg.python.org/cpython

1 0

cpython: alias resource.error to OSError
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/8d670cb0d64a changeset: 73930:8d670cb0d64a parent: 73926:aab45b904141 user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 17:50:22 2011 -0500 summary: alias resource.error to OSError files: Doc/library/resource.rst | 8 +++++--- Misc/NEWS | 2 ++ Modules/resource.c | 16 +++++----------- 3 files changed, 12 insertions(+), 14 deletions(-) diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -14,13 +14,15 @@ Symbolic constants are used to specify particular system resources and to request usage information about either the current process or its children. -A single exception is defined for errors: +An :exc:`OSError` is raised on syscall failure. .. exception:: error - The functions described below may raise this error if the underlying system call - failures unexpectedly. + A deprecated alias of :exc:`OSError`. + + .. versionchanged:: 3.3 + Following :pep:`3151`, this class was made an alias of :exc:`OSError`. Resource Limits diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,8 @@ Library ------- +- Alias resource.error to OSError ala PEP 3151. + - Issue #13248: Turn 3.2's PendingDeprecationWarning into 3.3's DeprecationWarning. It covers 'cgi.escape', 'importlib.abc.PyLoader', 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', diff --git a/Modules/resource.c b/Modules/resource.c --- a/Modules/resource.c +++ b/Modules/resource.c @@ -18,8 +18,6 @@ #define doubletime(TV) ((double)(TV).tv_sec + (TV).tv_usec * 0.000001) -static PyObject *ResourceError; - PyDoc_STRVAR(struct_rusage__doc__, "struct_rusage: Result from getrusage.\n\n" "This object may be accessed either as a tuple of\n" @@ -73,7 +71,7 @@ "invalid who parameter"); return NULL; } - PyErr_SetFromErrno(ResourceError); + PyErr_SetFromErrno(PyExc_OSError); return NULL; } @@ -125,7 +123,7 @@ } if (getrlimit(resource, &rl) == -1) { - PyErr_SetFromErrno(ResourceError); + PyErr_SetFromErrno(PyExc_OSError); return NULL; } @@ -183,7 +181,7 @@ PyErr_SetString(PyExc_ValueError, "not allowed to raise maximum limit"); else - PyErr_SetFromErrno(ResourceError); + PyErr_SetFromErrno(PyExc_OSError); return NULL; } Py_INCREF(Py_None); @@ -246,12 +244,8 @@ return NULL; /* Add some symbolic constants to the module */ - if (ResourceError == NULL) { - ResourceError = PyErr_NewException("resource.error", - NULL, NULL); - } - Py_INCREF(ResourceError); - PyModule_AddObject(m, "error", ResourceError); + Py_INCREF(PyExc_OSError); + PyModule_AddObject(m, "error", PyExc_OSError); if (!initialized) PyStructSequence_InitType(&StructRUsageType, &struct_rusage_desc); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: fix #13563: make use of with statement in ftplib.py where needed
by giampaolo.rodola Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/6cd736239b8a changeset: 73929:6cd736239b8a user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Sat Dec 10 21:25:04 2011 +0100 summary: fix #13563: make use of with statement in ftplib.py where needed files: Lib/ftplib.py | 25 +++++-------------------- 1 files changed, 5 insertions(+), 20 deletions(-) diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -359,8 +359,7 @@ conn.close() raise else: - sock = self.makeport() - try: + with self.makeport() as sock: if rest is not None: self.sendcmd("REST %s" % rest) resp = self.sendcmd(cmd) @@ -372,8 +371,6 @@ conn, sockaddr = sock.accept() if self.timeout is not _GLOBAL_DEFAULT_TIMEOUT: conn.settimeout(self.timeout) - finally: - sock.close() if resp[:3] == '150': # this is conditional in case we received a 125 size = parse150(resp) @@ -753,8 +750,7 @@ def retrbinary(self, cmd, callback, blocksize=8192, rest=None): self.voidcmd('TYPE I') - conn = self.transfercmd(cmd, rest) - try: + with self.transfercmd(cmd, rest) as conn: while 1: data = conn.recv(blocksize) if not data: @@ -763,8 +759,6 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - conn.close() return self.voidresp() def retrlines(self, cmd, callback = None): @@ -772,7 +766,7 @@ resp = self.sendcmd('TYPE A') conn = self.transfercmd(cmd) fp = conn.makefile('r', encoding=self.encoding) - try: + with fp, conn: while 1: line = fp.readline() if self.debugging > 2: print('*retr*', repr(line)) @@ -786,15 +780,11 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - fp.close() - conn.close() return self.voidresp() def storbinary(self, cmd, fp, blocksize=8192, callback=None, rest=None): self.voidcmd('TYPE I') - conn = self.transfercmd(cmd, rest) - try: + with self.transfercmd(cmd, rest) as conn: while 1: buf = fp.read(blocksize) if not buf: break @@ -803,14 +793,11 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - conn.close() return self.voidresp() def storlines(self, cmd, fp, callback=None): self.voidcmd('TYPE A') - conn = self.transfercmd(cmd) - try: + with self.transfercmd(cmd) as conn: while 1: buf = fp.readline() if not buf: break @@ -822,8 +809,6 @@ # shutdown ssl layer if isinstance(conn, ssl.SSLSocket): conn.unwrap() - finally: - conn.close() return self.voidresp() def abort(self): -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix imports in xml.dom.
by florent.xicluna Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/ed6ddd59bf2c changeset: 73928:ed6ddd59bf2c user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Sat Dec 10 21:14:53 2011 +0100 summary: Fix imports in xml.dom. files: Lib/test/test_minidom.py | 2 +- Lib/xml/dom/pulldom.py | 6 +----- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -47,7 +47,7 @@ def checkWholeText(self, node, s): t = node.wholeText - self.confirm(t == s, "looking for %s, found %s" % (repr(s), repr(t))) + self.confirm(t == s, "looking for %r, found %r" % (s, t)) def testParseFromFile(self): with open(tstfile) as file: diff --git a/Lib/xml/dom/pulldom.py b/Lib/xml/dom/pulldom.py --- a/Lib/xml/dom/pulldom.py +++ b/Lib/xml/dom/pulldom.py @@ -1,6 +1,5 @@ import xml.sax import xml.sax.handler -import types START_ELEMENT = "START_ELEMENT" END_ELEMENT = "END_ELEMENT" @@ -334,10 +333,7 @@ return DOMEventStream(stream, parser, bufsize) def parseString(string, parser=None): - try: - from io import StringIO - except ImportError: - from io import StringIO + from io import StringIO bufsize = len(string) buf = StringIO(string) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #5689: Add support for lzma compression to the tarfile module.
by lars.gustaebel Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/899a8c7b2310 changeset: 73927:899a8c7b2310 user: Lars Gustäbel <lars(a)gustaebel.de> date: Sat Dec 10 20:38:14 2011 +0100 summary: Issue #5689: Add support for lzma compression to the tarfile module. files: Doc/library/tarfile.rst | 22 +++++-- Lib/tarfile.py | 66 +++++++++++++++++++++-- Lib/test/test_tarfile.py | 78 ++++++++++++++++++++++++--- Misc/NEWS | 2 + 4 files changed, 146 insertions(+), 22 deletions(-) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -13,12 +13,12 @@ -------------- The :mod:`tarfile` module makes it possible to read and write tar -archives, including those using gzip or bz2 compression. +archives, including those using gzip, bz2 and lzma compression. (:file:`.zip` files can be read and written using the :mod:`zipfile` module.) Some facts and figures: -* reads and writes :mod:`gzip` and :mod:`bz2` compressed archives. +* reads and writes :mod:`gzip`, :mod:`bz2` and :mod:`lzma` compressed archives. * read/write support for the POSIX.1-1988 (ustar) format. @@ -55,6 +55,8 @@ +------------------+---------------------------------------------+ | ``'r:bz2'`` | Open for reading with bzip2 compression. | +------------------+---------------------------------------------+ + | ``'r:xz'`` | Open for reading with lzma compression. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -64,11 +66,13 @@ +------------------+---------------------------------------------+ | ``'w:bz2'`` | Open for bzip2 compressed writing. | +------------------+---------------------------------------------+ + | ``'w:xz'`` | Open for lzma compressed writing. | + +------------------+---------------------------------------------+ - Note that ``'a:gz'`` or ``'a:bz2'`` is not possible. If *mode* is not suitable - to open a certain (compressed) file for reading, :exc:`ReadError` is raised. Use - *mode* ``'r'`` to avoid this. If a compression method is not supported, - :exc:`CompressionError` is raised. + Note that ``'a:gz'``, ``'a:bz2'`` or ``'a:xz'`` is not possible. If *mode* + is not suitable to open a certain (compressed) file for reading, + :exc:`ReadError` is raised. Use *mode* ``'r'`` to avoid this. If a + compression method is not supported, :exc:`CompressionError` is raised. If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. @@ -99,6 +103,9 @@ | ``'r|bz2'`` | Open a bzip2 compressed *stream* for | | | reading. | +-------------+--------------------------------------------+ + | ``'r|xz'`` | Open a lzma compressed *stream* for | + | | reading. | + +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | +-------------+--------------------------------------------+ | ``'w|gz'`` | Open a gzip compressed *stream* for | @@ -107,6 +114,9 @@ | ``'w|bz2'`` | Open a bzip2 compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ + | ``'w|xz'`` | Open an lzma compressed *stream* for | + | | writing. | + +-------------+--------------------------------------------+ .. class:: TarFile diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -420,10 +420,11 @@ self.crc = zlib.crc32(b"") if mode == "r": self._init_read_gz() + self.exception = zlib.error else: self._init_write_gz() - if comptype == "bz2": + elif comptype == "bz2": try: import bz2 except ImportError: @@ -431,8 +432,25 @@ if mode == "r": self.dbuf = b"" self.cmp = bz2.BZ2Decompressor() + self.exception = IOError else: self.cmp = bz2.BZ2Compressor() + + elif comptype == "xz": + try: + import lzma + except ImportError: + raise CompressionError("lzma module is not available") + if mode == "r": + self.dbuf = b"" + self.cmp = lzma.LZMADecompressor() + self.exception = lzma.LZMAError + else: + self.cmp = lzma.LZMACompressor() + + elif comptype != "tar": + raise CompressionError("unknown compression type %r" % comptype) + except: if not self._extfileobj: self.fileobj.close() @@ -584,7 +602,7 @@ break try: buf = self.cmp.decompress(buf) - except IOError: + except self.exception: raise ReadError("invalid compressed data") self.dbuf += buf c += len(buf) @@ -622,11 +640,14 @@ return self.buf def getcomptype(self): - if self.buf.startswith(b"\037\213\010"): + if self.buf.startswith(b"\x1f\x8b\x08"): return "gz" - if self.buf[0:3] == b"BZh" and self.buf[4:10] == b"1AY&SY": + elif self.buf[0:3] == b"BZh" and self.buf[4:10] == b"1AY&SY": return "bz2" - return "tar" + elif self.buf.startswith((b"\x5d\x00\x00\x80", b"\xfd7zXZ")): + return "xz" + else: + return "tar" def close(self): self.fileobj.close() @@ -1651,18 +1672,22 @@ 'r:' open for reading exclusively uncompressed 'r:gz' open for reading with gzip compression 'r:bz2' open for reading with bzip2 compression + 'r:xz' open for reading with lzma compression 'a' or 'a:' open for appending, creating the file if necessary 'w' or 'w:' open for writing without compression 'w:gz' open for writing with gzip compression 'w:bz2' open for writing with bzip2 compression + 'w:xz' open for writing with lzma compression 'r|*' open a stream of tar blocks with transparent compression 'r|' open an uncompressed stream of tar blocks for reading 'r|gz' open a gzip compressed stream of tar blocks 'r|bz2' open a bzip2 compressed stream of tar blocks + 'r|xz' open an lzma compressed stream of tar blocks 'w|' open an uncompressed stream for writing 'w|gz' open a gzip compressed stream for writing 'w|bz2' open a bzip2 compressed stream for writing + 'w|xz' open an lzma compressed stream for writing """ if not name and not fileobj: @@ -1780,11 +1805,40 @@ t._extfileobj = False return t + @classmethod + def xzopen(cls, name, mode="r", fileobj=None, preset=9, **kwargs): + """Open lzma compressed tar archive name for reading or writing. + Appending is not allowed. + """ + if mode not in ("r", "w"): + raise ValueError("mode must be 'r' or 'w'") + + try: + import lzma + except ImportError: + raise CompressionError("lzma module is not available") + + if mode == "r": + # LZMAFile complains about a preset argument in read mode. + preset = None + + fileobj = lzma.LZMAFile(filename=name if fileobj is None else None, + mode=mode, fileobj=fileobj, preset=preset) + + try: + t = cls.taropen(name, mode, fileobj, **kwargs) + except (lzma.LZMAError, EOFError): + fileobj.close() + raise ReadError("not an lzma file") + t._extfileobj = False + return t + # All *open() methods are registered here. OPEN_METH = { "tar": "taropen", # uncompressed tar "gz": "gzopen", # gzip compressed tar - "bz2": "bz2open" # bzip2 compressed tar + "bz2": "bz2open", # bzip2 compressed tar + "xz": "xzopen" # lzma compressed tar } #-------------------------------------------------------------------------- diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -21,6 +21,10 @@ import bz2 except ImportError: bz2 = None +try: + import lzma +except ImportError: + lzma = None def md5sum(data): return md5(data).hexdigest() @@ -29,6 +33,7 @@ tarname = support.findfile("testtar.tar") gzipname = os.path.join(TEMPDIR, "testtar.tar.gz") bz2name = os.path.join(TEMPDIR, "testtar.tar.bz2") +xzname = os.path.join(TEMPDIR, "testtar.tar.xz") tmpname = os.path.join(TEMPDIR, "tmp.tar") md5_regtype = "65f477c818ad9e15f7feab0c6d37742f" @@ -201,13 +206,15 @@ _open = gzip.GzipFile elif self.mode.endswith(":bz2"): _open = bz2.BZ2File + elif self.mode.endswith(":xz"): + _open = lzma.LZMAFile else: - _open = open + _open = io.FileIO for char in (b'\0', b'a'): # Test if EOFHeaderError ('\0') and InvalidHeaderError ('a') # are ignored correctly. - with _open(tmpname, "wb") as fobj: + with _open(tmpname, "w") as fobj: fobj.write(char * 1024) fobj.write(tarfile.TarInfo("foo").tobuf()) @@ -222,9 +229,10 @@ class MiscReadTest(CommonReadTest): def test_no_name_argument(self): - if self.mode.endswith("bz2"): - # BZ2File has no name attribute. - return + if self.mode.endswith(("bz2", "xz")): + # BZ2File and LZMAFile have no name attribute. + self.skipTest("no name attribute") + with open(self.tarname, "rb") as fobj: tar = tarfile.open(fileobj=fobj, mode=self.mode) self.assertEqual(tar.name, os.path.abspath(fobj.name)) @@ -265,10 +273,12 @@ _open = gzip.GzipFile elif self.mode.endswith(":bz2"): _open = bz2.BZ2File + elif self.mode.endswith(":xz"): + _open = lzma.LZMAFile else: - _open = open - fobj = _open(self.tarname, "rb") - try: + _open = io.FileIO + + with _open(self.tarname) as fobj: fobj.seek(offset) # Test if the tarfile starts with the second member. @@ -281,8 +291,6 @@ self.assertEqual(tar.extractfile(t).read(), data, "seek back did not work") tar.close() - finally: - fobj.close() def test_fail_comp(self): # For Gzip and Bz2 Tests: fail with a ReadError on an uncompressed file. @@ -526,6 +534,18 @@ testfunc(bz2name, "r|*") testfunc(bz2name, "r|bz2") + if lzma: + self.assertRaises(tarfile.ReadError, tarfile.open, tarname, mode="r:xz") + self.assertRaises(tarfile.ReadError, tarfile.open, tarname, mode="r|xz") + self.assertRaises(tarfile.ReadError, tarfile.open, xzname, mode="r:") + self.assertRaises(tarfile.ReadError, tarfile.open, xzname, mode="r|") + + testfunc(xzname, "r") + testfunc(xzname, "r:*") + testfunc(xzname, "r:xz") + testfunc(xzname, "r|*") + testfunc(xzname, "r|xz") + def test_detect_file(self): self._test_modes(self._testfunc_file) @@ -1096,6 +1116,9 @@ data = dec.decompress(data) self.assertTrue(len(dec.unused_data) == 0, "found trailing data") + elif self.mode.endswith("xz"): + with lzma.LZMAFile(tmpname) as fobj: + data = fobj.read() else: with open(tmpname, "rb") as fobj: data = fobj.read() @@ -1510,6 +1533,12 @@ self._create_testtar("w:bz2") self.assertRaises(tarfile.ReadError, tarfile.open, tmpname, "a") + def test_append_lzma(self): + if lzma is None: + self.skipTest("lzma module not available") + self._create_testtar("w:xz") + self.assertRaises(tarfile.ReadError, tarfile.open, tmpname, "a") + # Append mode is supposed to fail if the tarfile to append to # does not end with a zero block. def _test_error(self, data): @@ -1788,6 +1817,21 @@ self._test_partial_input("r:bz2") +class LzmaMiscReadTest(MiscReadTest): + tarname = xzname + mode = "r:xz" +class LzmaUstarReadTest(UstarReadTest): + tarname = xzname + mode = "r:xz" +class LzmaStreamReadTest(StreamReadTest): + tarname = xzname + mode = "r|xz" +class LzmaWriteTest(WriteTest): + mode = "w:xz" +class LzmaStreamWriteTest(StreamWriteTest): + mode = "w|xz" + + def test_main(): support.unlink(TEMPDIR) os.makedirs(TEMPDIR) @@ -1850,6 +1894,20 @@ Bz2PartialReadTest, ] + if lzma: + # Create testtar.tar.xz and add lzma-specific tests. + support.unlink(xzname) + with lzma.LZMAFile(xzname, "w") as tar: + tar.write(data) + + tests += [ + LzmaMiscReadTest, + LzmaUstarReadTest, + LzmaStreamReadTest, + LzmaWriteTest, + LzmaStreamWriteTest, + ] + try: support.run_unittest(*tests) finally: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,8 @@ Library ------- +- Issue #5689: Add support for lzma compression to the tarfile module. + - Issue #13248: Turn 3.2's PendingDeprecationWarning into 3.3's DeprecationWarning. It covers 'cgi.escape', 'importlib.abc.PyLoader', 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): merge 3.2
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/aab45b904141 changeset: 73926:aab45b904141 parent: 73924:423f47499e4a parent: 73925:6040c248a199 user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 12:44:37 2011 -0500 summary: merge 3.2 files: Lib/test/test_subprocess.py | 7 +------ 1 files changed, 1 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -13,11 +13,7 @@ import warnings import select import shutil -try: - import gc -except ImportError: - gc = None - +import gc try: import resource @@ -951,7 +947,6 @@ self.fail("Exception raised by preexec_fn did not make it " "to the parent process.") - @unittest.skipUnless(gc, "Requires a gc module.") def test_preexec_gc_module_failure(self): # This tests the code that disables garbage collection if the child # process will execute any Python. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): we're always going to have gc
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/6040c248a199 changeset: 73925:6040c248a199 branch: 3.2 parent: 73923:245f6094ccba user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 12:44:25 2011 -0500 summary: we're always going to have gc files: Lib/test/test_subprocess.py | 7 +------ 1 files changed, 1 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -13,11 +13,7 @@ import warnings import select import shutil -try: - import gc -except ImportError: - gc = None - +import gc try: import resource @@ -859,7 +855,6 @@ self.fail("Exception raised by preexec_fn did not make it " "to the parent process.") - @unittest.skipUnless(gc, "Requires a gc module.") def test_preexec_gc_module_failure(self): # This tests the code that disables garbage collection if the child # process will execute any Python. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): merge 3.2
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/423f47499e4a changeset: 73924:423f47499e4a parent: 73921:e37a7dc8944e parent: 73923:245f6094ccba user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 12:38:52 2011 -0500 summary: merge 3.2 files: Lib/test/test_subprocess.py | 28 +++++++++++++++--------- 1 files changed, 17 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -18,6 +18,12 @@ except ImportError: gc = None + +try: + import resource +except ImportError: + resource = None + mswindows = (sys.platform == "win32") # @@ -824,12 +830,12 @@ def __enter__(self): """Try to save previous ulimit, then set it to (0, 0).""" - try: - import resource - self.old_limit = resource.getrlimit(resource.RLIMIT_CORE) - resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) - except (ImportError, ValueError, resource.error): - pass + if resource is not None: + try: + self.old_limit = resource.getrlimit(resource.RLIMIT_CORE) + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + except (ValueError, resource.error): + pass if sys.platform == 'darwin': # Check if the 'Crash Reporter' on OSX was configured @@ -850,11 +856,11 @@ """Return core file behavior to default.""" if self.old_limit is None: return - try: - import resource - resource.setrlimit(resource.RLIMIT_CORE, self.old_limit) - except (ImportError, ValueError, resource.error): - pass + if resource is not None: + try: + resource.setrlimit(resource.RLIMIT_CORE, self.old_limit) + except (ValueError, resource.error): + pass @unittest.skipIf(mswindows, "POSIX specific tests") -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): you can't get resource.error if you can't import resource
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/245f6094ccba changeset: 73923:245f6094ccba branch: 3.2 parent: 73919:c998c6f5464b user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 12:31:42 2011 -0500 summary: you can't get resource.error if you can't import resource files: Lib/test/test_subprocess.py | 28 +++++++++++++++--------- 1 files changed, 17 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -18,6 +18,12 @@ except ImportError: gc = None + +try: + import resource +except ImportError: + resource = None + mswindows = (sys.platform == "win32") # @@ -732,12 +738,12 @@ def __enter__(self): """Try to save previous ulimit, then set it to (0, 0).""" - try: - import resource - self.old_limit = resource.getrlimit(resource.RLIMIT_CORE) - resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) - except (ImportError, ValueError, resource.error): - pass + if resource is not None: + try: + self.old_limit = resource.getrlimit(resource.RLIMIT_CORE) + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + except (ValueError, resource.error): + pass if sys.platform == 'darwin': # Check if the 'Crash Reporter' on OSX was configured @@ -758,11 +764,11 @@ """Return core file behavior to default.""" if self.old_limit is None: return - try: - import resource - resource.setrlimit(resource.RLIMIT_CORE, self.old_limit) - except (ImportError, ValueError, resource.error): - pass + if resource is not None: + try: + resource.setrlimit(resource.RLIMIT_CORE, self.old_limit) + except (ValueError, resource.error): + pass @unittest.skipIf(mswindows, "POSIX specific tests") -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): you can't get resource.error if you can't import resource
by benjamin.peterson Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/fd5f43737546 changeset: 73922:fd5f43737546 branch: 2.7 parent: 73918:5ba1a22c8988 user: Benjamin Peterson <benjamin(a)python.org> date: Sat Dec 10 12:31:42 2011 -0500 summary: you can't get resource.error if you can't import resource files: Lib/test/test_subprocess.py | 27 ++++++++++++++---------- 1 files changed, 16 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -10,6 +10,11 @@ import re import sysconfig +try: + import resource +except ImportError: + resource = None + mswindows = (sys.platform == "win32") # @@ -633,12 +638,12 @@ def __enter__(self): """Try to save previous ulimit, then set it to (0, 0).""" - try: - import resource - self.old_limit = resource.getrlimit(resource.RLIMIT_CORE) - resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) - except (ImportError, ValueError, resource.error): - pass + if resource is not None: + try: + self.old_limit = resource.getrlimit(resource.RLIMIT_CORE) + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + except (ValueError, resource.error): + pass if sys.platform == 'darwin': # Check if the 'Crash Reporter' on OSX was configured @@ -658,11 +663,11 @@ """Return core file behavior to default.""" if self.old_limit is None: return - try: - import resource - resource.setrlimit(resource.RLIMIT_CORE, self.old_limit) - except (ImportError, ValueError, resource.error): - pass + if resource is not None: + try: + resource.setrlimit(resource.RLIMIT_CORE, self.old_limit) + except (ValueError, resource.error): + pass @unittest.skipUnless(hasattr(signal, 'SIGALRM'), "Requires signal.SIGALRM") -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #11886: Fix also test_time for the non-DST timezone name (EST/AEST)
by victor.stinner Dec. 11, 2011

Dec. 11, 2011

http://hg.python.org/cpython/rev/e37a7dc8944e changeset: 73921:e37a7dc8944e user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Dec 10 14:37:53 2011 +0100 summary: Issue #11886: Fix also test_time for the non-DST timezone name (EST/AEST) files: Lib/test/test_time.py | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -252,11 +252,13 @@ self.assertNotEqual(time.gmtime(xmas2002), time.localtime(xmas2002)) # Issue #11886: Australian Eastern Standard Time (UTC+10) is called - # "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some - # operating systems (e.g. FreeBSD), which is wrong. See for example - # this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 + # "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" + # (non-DST timezone), and "EDT" instead of "AEDT" (DST timezone), + # on some operating systems (e.g. FreeBSD), which is wrong. See for + # example this bug: + # http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 self.assertIn(time.tzname[0], ('AEST' 'EST'), time.tzname[0]) - self.assertTrue(time.tzname[1] == 'AEDT', str(time.tzname[1])) + self.assertTrue(time.tzname[1] in ('AEDT', 'EDT'), str(time.tzname[1])) self.assertEqual(len(time.tzname), 2) self.assertEqual(time.daylight, 1) self.assertEqual(time.timezone, -36000) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13453: Catch EAI_FAIL in support.transient_internet.
by charles-francois.natali Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/767badea87c3 changeset: 73920:767badea87c3 parent: 73917:a3309634f38c parent: 73919:c998c6f5464b user: Charles-François Natali <neologix(a)free.fr> date: Sat Dec 10 13:17:46 2011 +0100 summary: Issue #13453: Catch EAI_FAIL in support.transient_internet. files: Lib/test/support.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -951,6 +951,7 @@ ] default_gai_errnos = [ ('EAI_AGAIN', -3), + ('EAI_FAIL', -4), ('EAI_NONAME', -2), ('EAI_NODATA', -5), # Encountered when trying to resolve IPv6-only hostnames -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13453: Catch EAI_FAIL in support.transient_internet.
by charles-francois.natali Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/c998c6f5464b changeset: 73919:c998c6f5464b branch: 3.2 parent: 73915:caeec3e3606f user: Charles-François Natali <neologix(a)free.fr> date: Sat Dec 10 13:16:44 2011 +0100 summary: Issue #13453: Catch EAI_FAIL in support.transient_internet. files: Lib/test/support.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -877,6 +877,7 @@ ] default_gai_errnos = [ ('EAI_AGAIN', -3), + ('EAI_FAIL', -4), ('EAI_NONAME', -2), ('EAI_NODATA', -5), # Encountered when trying to resolve IPv6-only hostnames -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13453: Catch EAI_FAIL in support.transient_internet.
by charles-francois.natali Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/5ba1a22c8988 changeset: 73918:5ba1a22c8988 branch: 2.7 parent: 73910:78a7d5f8f054 user: Charles-François Natali <neologix(a)free.fr> date: Sat Dec 10 13:16:02 2011 +0100 summary: Issue #13453: Catch EAI_FAIL in support.transient_internet. files: Lib/test/test_support.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -764,6 +764,7 @@ ] default_gai_errnos = [ ('EAI_AGAIN', -3), + ('EAI_FAIL', -4), ('EAI_NONAME', -2), ('EAI_NODATA', -5), ] -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix comment in difflib.
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/a3309634f38c changeset: 73917:a3309634f38c user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Sat Dec 10 13:02:17 2011 +0100 summary: Fix comment in difflib. files: Lib/difflib.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/difflib.py b/Lib/difflib.py --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -204,7 +204,7 @@ # returning true iff the element is "junk" -- this has # subtle but helpful effects on the algorithm, which I'll # get around to writing up someday <0.9 wink>. - # DON'T USE! Only __chain_b uses this. Use isbjunk. + # DON'T USE! Only __chain_b uses this. Use "in self.bjunk". # bjunk # the items in b for which isjunk is True. # bpopular @@ -287,7 +287,6 @@ # when self.isjunk is defined, junk elements don't show up in this # map at all, which stops the central find_longest_match method # from starting any matching block at a junk element ... - # also creates the fast isbjunk function ... # b2j also does not contain entries for "popular" elements, meaning # elements that account for more than 1 + 1% of the total elements, and # when the sequence is reasonably large (>= 200 elements); this can -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge with 3.2: Fix doc typo.
by lars.gustaebel Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/5da2c242025f changeset: 73916:5da2c242025f parent: 73914:f82ebf9b3a52 parent: 73915:caeec3e3606f user: Lars Gustäbel <lars(a)gustaebel.de> date: Sat Dec 10 12:48:03 2011 +0100 summary: Merge with 3.2: Fix doc typo. files: Doc/library/tarfile.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -101,10 +101,10 @@ +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | +-------------+--------------------------------------------+ - | ``'w|gz'`` | Open an gzip compressed *stream* for | + | ``'w|gz'`` | Open a gzip compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ - | ``'w|bz2'`` | Open an bzip2 compressed *stream* for | + | ``'w|bz2'`` | Open a bzip2 compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix doc typo.
by lars.gustaebel Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/caeec3e3606f changeset: 73915:caeec3e3606f branch: 3.2 parent: 73912:ee0c9ca09c84 user: Lars Gustäbel <lars(a)gustaebel.de> date: Sat Dec 10 12:45:45 2011 +0100 summary: Fix doc typo. files: Doc/library/tarfile.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -101,10 +101,10 @@ +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | +-------------+--------------------------------------------+ - | ``'w|gz'`` | Open an gzip compressed *stream* for | + | ``'w|gz'`` | Open a gzip compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ - | ``'w|bz2'`` | Open an bzip2 compressed *stream* for | + | ``'w|bz2'`` | Open a bzip2 compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13248: turn 3.2's PendingDeprecationWarning into 3.3's
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/f82ebf9b3a52 changeset: 73914:f82ebf9b3a52 user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Sat Dec 10 11:07:42 2011 +0100 summary: Issue #13248: turn 3.2's PendingDeprecationWarning into 3.3's DeprecationWarning (cgi, importlib, nntplib, smtpd). files: Doc/library/nntplib.rst | 3 + Lib/cgi.py | 2 +- Lib/importlib/abc.py | 4 +- Lib/importlib/test/source/test_abc_loader.py | 4 +- Lib/nntplib.py | 4 +- Lib/smtpd.py | 44 +++++----- Lib/test/test_smtpd.py | 44 +++++----- Misc/NEWS | 5 + 8 files changed, 59 insertions(+), 51 deletions(-) diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -517,6 +517,9 @@ article with message ID *id*. Most of the time, this extension is not enabled by NNTP server administrators. + .. deprecated:: 3.3 + The XPATH extension is not actively used. + .. XXX deprecated: diff --git a/Lib/cgi.py b/Lib/cgi.py --- a/Lib/cgi.py +++ b/Lib/cgi.py @@ -1012,7 +1012,7 @@ def escape(s, quote=None): """Deprecated API.""" warn("cgi.escape is deprecated, use html.escape instead", - PendingDeprecationWarning, stacklevel=2) + DeprecationWarning, stacklevel=2) s = s.replace("&", "&") # Must be done first! s = s.replace("<", "<") s = s.replace(">", ">") diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py --- a/Lib/importlib/abc.py +++ b/Lib/importlib/abc.py @@ -195,7 +195,7 @@ "use SourceLoader instead. " "See the importlib documentation on how to be " "compatible with Python 3.1 onwards.", - PendingDeprecationWarning) + DeprecationWarning) path = self.source_path(fullname) if path is None: raise ImportError @@ -234,7 +234,7 @@ "removal in Python 3.4; use SourceLoader instead. " "If Python 3.1 compatibility is required, see the " "latest documentation for PyLoader.", - PendingDeprecationWarning) + DeprecationWarning) source_timestamp = self.source_mtime(fullname) # Try to use bytecode if it is available. bytecode_path = self.bytecode_path(fullname) diff --git a/Lib/importlib/test/source/test_abc_loader.py b/Lib/importlib/test/source/test_abc_loader.py --- a/Lib/importlib/test/source/test_abc_loader.py +++ b/Lib/importlib/test/source/test_abc_loader.py @@ -102,7 +102,7 @@ warnings.simplefilter("always") path = super().get_filename(name) assert len(w) == 1 - assert issubclass(w[0].category, PendingDeprecationWarning) + assert issubclass(w[0].category, DeprecationWarning) return path @@ -198,7 +198,7 @@ warnings.simplefilter("always") code_object = super().get_code(name) assert len(w) == 1 - assert issubclass(w[0].category, PendingDeprecationWarning) + assert issubclass(w[0].category, DeprecationWarning) return code_object class PyLoaderTests(testing_abc.LoaderTests): diff --git a/Lib/nntplib.py b/Lib/nntplib.py --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -828,7 +828,7 @@ - list: list of (name,title) strings""" warnings.warn("The XGTITLE extension is not actively used, " "use descriptions() instead", - PendingDeprecationWarning, 2) + DeprecationWarning, 2) line_pat = re.compile('^([^ \t]+)[ \t]+(.*)$') resp, raw_lines = self._longcmdstring('XGTITLE ' + group, file) lines = [] @@ -846,7 +846,7 @@ path: directory path to article """ warnings.warn("The XPATH extension is not actively used", - PendingDeprecationWarning, 2) + DeprecationWarning, 2) resp = self._shortcmd('XPATH {0}'.format(id)) if not resp.startswith('223'): diff --git a/Lib/smtpd.py b/Lib/smtpd.py --- a/Lib/smtpd.py +++ b/Lib/smtpd.py @@ -142,122 +142,122 @@ @property def __server(self): warn("Access to __server attribute on SMTPChannel is deprecated, " - "use 'smtp_server' instead", PendingDeprecationWarning, 2) + "use 'smtp_server' instead", DeprecationWarning, 2) return self.smtp_server @__server.setter def __server(self, value): warn("Setting __server attribute on SMTPChannel is deprecated, " - "set 'smtp_server' instead", PendingDeprecationWarning, 2) + "set 'smtp_server' instead", DeprecationWarning, 2) self.smtp_server = value @property def __line(self): warn("Access to __line attribute on SMTPChannel is deprecated, " - "use 'received_lines' instead", PendingDeprecationWarning, 2) + "use 'received_lines' instead", DeprecationWarning, 2) return self.received_lines @__line.setter def __line(self, value): warn("Setting __line attribute on SMTPChannel is deprecated, " - "set 'received_lines' instead", PendingDeprecationWarning, 2) + "set 'received_lines' instead", DeprecationWarning, 2) self.received_lines = value @property def __state(self): warn("Access to __state attribute on SMTPChannel is deprecated, " - "use 'smtp_state' instead", PendingDeprecationWarning, 2) + "use 'smtp_state' instead", DeprecationWarning, 2) return self.smtp_state @__state.setter def __state(self, value): warn("Setting __state attribute on SMTPChannel is deprecated, " - "set 'smtp_state' instead", PendingDeprecationWarning, 2) + "set 'smtp_state' instead", DeprecationWarning, 2) self.smtp_state = value @property def __greeting(self): warn("Access to __greeting attribute on SMTPChannel is deprecated, " - "use 'seen_greeting' instead", PendingDeprecationWarning, 2) + "use 'seen_greeting' instead", DeprecationWarning, 2) return self.seen_greeting @__greeting.setter def __greeting(self, value): warn("Setting __greeting attribute on SMTPChannel is deprecated, " - "set 'seen_greeting' instead", PendingDeprecationWarning, 2) + "set 'seen_greeting' instead", DeprecationWarning, 2) self.seen_greeting = value @property def __mailfrom(self): warn("Access to __mailfrom attribute on SMTPChannel is deprecated, " - "use 'mailfrom' instead", PendingDeprecationWarning, 2) + "use 'mailfrom' instead", DeprecationWarning, 2) return self.mailfrom @__mailfrom.setter def __mailfrom(self, value): warn("Setting __mailfrom attribute on SMTPChannel is deprecated, " - "set 'mailfrom' instead", PendingDeprecationWarning, 2) + "set 'mailfrom' instead", DeprecationWarning, 2) self.mailfrom = value @property def __rcpttos(self): warn("Access to __rcpttos attribute on SMTPChannel is deprecated, " - "use 'rcpttos' instead", PendingDeprecationWarning, 2) + "use 'rcpttos' instead", DeprecationWarning, 2) return self.rcpttos @__rcpttos.setter def __rcpttos(self, value): warn("Setting __rcpttos attribute on SMTPChannel is deprecated, " - "set 'rcpttos' instead", PendingDeprecationWarning, 2) + "set 'rcpttos' instead", DeprecationWarning, 2) self.rcpttos = value @property def __data(self): warn("Access to __data attribute on SMTPChannel is deprecated, " - "use 'received_data' instead", PendingDeprecationWarning, 2) + "use 'received_data' instead", DeprecationWarning, 2) return self.received_data @__data.setter def __data(self, value): warn("Setting __data attribute on SMTPChannel is deprecated, " - "set 'received_data' instead", PendingDeprecationWarning, 2) + "set 'received_data' instead", DeprecationWarning, 2) self.received_data = value @property def __fqdn(self): warn("Access to __fqdn attribute on SMTPChannel is deprecated, " - "use 'fqdn' instead", PendingDeprecationWarning, 2) + "use 'fqdn' instead", DeprecationWarning, 2) return self.fqdn @__fqdn.setter def __fqdn(self, value): warn("Setting __fqdn attribute on SMTPChannel is deprecated, " - "set 'fqdn' instead", PendingDeprecationWarning, 2) + "set 'fqdn' instead", DeprecationWarning, 2) self.fqdn = value @property def __peer(self): warn("Access to __peer attribute on SMTPChannel is deprecated, " - "use 'peer' instead", PendingDeprecationWarning, 2) + "use 'peer' instead", DeprecationWarning, 2) return self.peer @__peer.setter def __peer(self, value): warn("Setting __peer attribute on SMTPChannel is deprecated, " - "set 'peer' instead", PendingDeprecationWarning, 2) + "set 'peer' instead", DeprecationWarning, 2) self.peer = value @property def __conn(self): warn("Access to __conn attribute on SMTPChannel is deprecated, " - "use 'conn' instead", PendingDeprecationWarning, 2) + "use 'conn' instead", DeprecationWarning, 2) return self.conn @__conn.setter def __conn(self, value): warn("Setting __conn attribute on SMTPChannel is deprecated, " - "set 'conn' instead", PendingDeprecationWarning, 2) + "set 'conn' instead", DeprecationWarning, 2) self.conn = value @property def __addr(self): warn("Access to __addr attribute on SMTPChannel is deprecated, " - "use 'addr' instead", PendingDeprecationWarning, 2) + "use 'addr' instead", DeprecationWarning, 2) return self.addr @__addr.setter def __addr(self, value): warn("Setting __addr attribute on SMTPChannel is deprecated, " - "set 'addr' instead", PendingDeprecationWarning, 2) + "set 'addr' instead", DeprecationWarning, 2) self.addr = value # Overrides base class for convenience diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py --- a/Lib/test/test_smtpd.py +++ b/Lib/test/test_smtpd.py @@ -239,49 +239,49 @@ self.assertEqual(self.channel.socket.last, b'501 Syntax: RSET\r\n') def test_attribute_deprecations(self): - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__server - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__server = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__line - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__line = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__state - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__state = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__greeting - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__greeting = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__mailfrom - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__mailfrom = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__rcpttos - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__rcpttos = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__data - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__data = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__fqdn - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__fqdn = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__peer - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__peer = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__conn - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__conn = 'spam' - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__addr - with support.check_warnings(('', PendingDeprecationWarning)): + with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__addr = 'spam' def test_main(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,11 @@ Library ------- +- Issue #13248: Turn 3.2's PendingDeprecationWarning into 3.3's + DeprecationWarning. It covers 'cgi.escape', 'importlib.abc.PyLoader', + 'importlib.abc.PyPycLoader', 'nntplib.NNTP.xgtitle', 'nntplib.NNTP.xpath', + and private attributes of 'smtpd.SMTPChannel'. + - Issue #5905: time.strftime() is now using the locale encoding, instead of UTF-8, if the wcsftime() function is not available. -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (b7c5e30582d1): sum=0
by solipsis＠pitrou.net Dec. 10, 2011

Dec. 10, 2011

results for b7c5e30582d1 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogwfwWLq', '-x']

1 0

cpython (merge 3.2 -> default): Merge 3.2
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/b7c5e30582d1 changeset: 73913:b7c5e30582d1 parent: 73909:9fe28f52eaaa parent: 73912:ee0c9ca09c84 user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Fri Dec 09 23:42:29 2011 +0100 summary: Merge 3.2 files: Modules/arraymodule.c | 2 +- Python/_warnings.c | 1 - 2 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2619,7 +2619,7 @@ \n\ Return a new array whose items are restricted by typecode, and\n\ initialized from the optional initializer value, which must be a list,\n\ -string. or iterable over elements of the appropriate type.\n\ +string or iterable over elements of the appropriate type.\n\ \n\ Arrays represent basic values and behave very much like lists, except\n\ the type of objects stored in them is constrained.\n\ diff --git a/Python/_warnings.c b/Python/_warnings.c --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -895,7 +895,6 @@ static PyObject * init_filters(void) { - /* Don't silence DeprecationWarning if -3 was used. */ PyObject *filters = PyList_New(5); unsigned int pos = 0; /* Post-incremented in each use. */ unsigned int x; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix docstring typo.
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/1e2880250610 changeset: 73911:1e2880250610 branch: 3.2 parent: 73908:eb30f2becb79 user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Fri Dec 09 23:41:19 2011 +0100 summary: Fix docstring typo. files: Modules/arraymodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2543,7 +2543,7 @@ \n\ Return a new array whose items are restricted by typecode, and\n\ initialized from the optional initializer value, which must be a list,\n\ -string. or iterable over elements of the appropriate type.\n\ +string or iterable over elements of the appropriate type.\n\ \n\ Arrays represent basic values and behave very much like lists, except\n\ the type of objects stored in them is constrained.\n\ -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Remove obsolete py3k comment.
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/ee0c9ca09c84 changeset: 73912:ee0c9ca09c84 branch: 3.2 user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Fri Dec 09 23:41:21 2011 +0100 summary: Remove obsolete py3k comment. files: Python/_warnings.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Python/_warnings.c b/Python/_warnings.c --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -888,7 +888,6 @@ static PyObject * init_filters(void) { - /* Don't silence DeprecationWarning if -3 was used. */ PyObject *filters = PyList_New(5); unsigned int pos = 0; /* Post-incremented in each use. */ unsigned int x; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Fix docstring typo.
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/78a7d5f8f054 changeset: 73910:78a7d5f8f054 branch: 2.7 parent: 73901:70337a6d5dde user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Fri Dec 09 23:40:27 2011 +0100 summary: Fix docstring typo. files: Modules/arraymodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2050,7 +2050,7 @@ \n\ Return a new array whose items are restricted by typecode, and\n\ initialized from the optional initializer value, which must be a list,\n\ -string. or iterable over elements of the appropriate type.\n\ +string or iterable over elements of the appropriate type.\n\ \n\ Arrays represent basic values and behave very much like lists, except\n\ the type of objects stored in them is constrained.\n\ -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13528: rework the performance question in the programming FAQ
by antoine.pitrou Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/9fe28f52eaaa changeset: 73909:9fe28f52eaaa parent: 73907:b3c1a504ebc1 parent: 73908:eb30f2becb79 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 09 23:11:16 2011 +0100 summary: Issue #13528: rework the performance question in the programming FAQ files: Doc/faq/programming.rst | 214 ++++++++------------------- 1 files changed, 62 insertions(+), 152 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -115,159 +115,6 @@ :pep:`8`. -My program is too slow. How do I speed it up? ---------------------------------------------- - -That's a tough one, in general. There are many tricks to speed up Python code; -consider rewriting parts in C as a last resort. - -`Cython <http://cython.org>`_ and `Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ -can compile a slightly modified version of Python code into a C extension, and -can be used on many different platforms. Depending on your code, Cython -may be able to make it significantly faster than when run by the Python -interpreter. - -The rest of this answer will discuss various tricks for squeezing a bit more -speed out of Python code. *Never* apply any optimization tricks unless you know -you need them, after profiling has indicated that a particular function is the -heavily executed hot spot in the code. Optimizations almost always make the -code less clear, and you shouldn't pay the costs of reduced clarity (increased -development time, greater likelihood of bugs) unless the resulting performance -benefit is worth it. - -There is a page on the wiki devoted to `performance tips -<http://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. - -Guido van Rossum has written up an anecdote related to optimization at -http://www.python.org/doc/essays/list2str.html. - -One thing to notice is that function and (especially) method calls are rather -expensive; if you have designed a purely OO interface with lots of tiny -functions that don't do much more than get or set an instance variable or call -another method, you might consider using a more direct way such as directly -accessing instance variables. Also see the standard module :mod:`profile` which -makes it possible to find out where your program is spending most of its time -(if you have some patience -- the profiling itself can slow your program down by -an order of magnitude). - -Remember that many standard optimization heuristics you may know from other -programming experience may well apply to Python. For example it may be faster -to send output to output devices using larger writes rather than smaller ones in -order to reduce the overhead of kernel system calls. Thus CGI scripts that -write all output in "one shot" may be faster than those that write lots of small -pieces of output. - -Also, be sure to use Python's core features where appropriate. For example, -slicing allows programs to chop up lists and other sequence objects in a single -tick of the interpreter's mainloop using highly optimized C implementations. -Thus to get the same effect as:: - - L2 = [] - for i in range(3): - L2.append(L1[i]) - -it is much shorter and far faster to use :: - - L2 = list(L1[:3]) # "list" is redundant if L1 is a list. - -Note that the functionally-oriented built-in functions such as :func:`map`, -:func:`zip`, and friends can be a convenient accelerator for loops that -perform a single task. For example to pair the elements of two lists -together:: - - >>> list(zip([1, 2, 3], [4, 5, 6])) - [(1, 4), (2, 5), (3, 6)] - -or to compute a number of sines:: - - >>> list(map(math.sin, (1, 2, 3, 4))) - [0.841470984808, 0.909297426826, 0.14112000806, -0.756802495308] - -The operation completes very quickly in such cases. - -Other examples include the ``join()`` and ``split()`` :ref:`methods -of string objects <string-methods>`. - -For example if s1..s7 are large (10K+) strings then -``"".join([s1,s2,s3,s4,s5,s6,s7])`` may be far faster than the more obvious -``s1+s2+s3+s4+s5+s6+s7``, since the "summation" will compute many -subexpressions, whereas ``join()`` does all the copying in one pass. For -manipulating strings, use the ``replace()`` and the ``format()`` :ref:`methods -on string objects <string-methods>`. Use regular expressions only when you're -not dealing with constant string patterns. - -Be sure to use the :meth:`list.sort` built-in method to do sorting, and see the -`sorting mini-HOWTO <http://wiki.python.org/moin/HowTo/Sorting>`_ for examples -of moderately advanced usage. :meth:`list.sort` beats other techniques for -sorting in all but the most extreme circumstances. - -Another common trick is to "push loops into functions or methods." For example -suppose you have a program that runs slowly and you use the profiler to -determine that a Python function ``ff()`` is being called lots of times. If you -notice that ``ff()``:: - - def ff(x): - ... # do something with x computing result... - return result - -tends to be called in loops like:: - - list = map(ff, oldlist) - -or:: - - for x in sequence: - value = ff(x) - ... # do something with value... - -then you can often eliminate function call overhead by rewriting ``ff()`` to:: - - def ffseq(seq): - resultseq = [] - for x in seq: - ... # do something with x computing result... - resultseq.append(result) - return resultseq - -and rewrite the two examples to ``list = ffseq(oldlist)`` and to:: - - for value in ffseq(sequence): - ... # do something with value... - -Single calls to ``ff(x)`` translate to ``ffseq([x])[0]`` with little penalty. -Of course this technique is not always appropriate and there are other variants -which you can figure out. - -You can gain some performance by explicitly storing the results of a function or -method lookup into a local variable. A loop like:: - - for key in token: - dict[key] = dict.get(key, 0) + 1 - -resolves ``dict.get`` every iteration. If the method isn't going to change, a -slightly faster implementation is:: - - dict_get = dict.get # look up the method once - for key in token: - dict[key] = dict_get(key, 0) + 1 - -Default arguments can be used to determine values once, at compile time instead -of at run time. This can only be done for functions or objects which will not -be changed during program execution, such as replacing :: - - def degree_sin(deg): - return math.sin(deg * math.pi / 180.0) - -with :: - - def degree_sin(deg, factor=math.pi/180.0, sin=math.sin): - return sin(deg * factor) - -Because this trick uses default arguments for terms which should not be changed, -it should only be used when you are not concerned with presenting a possibly -confusing API to your users. - - Core Language ============= @@ -938,6 +785,68 @@ See the :ref:`unicode-howto`. +Performance +=========== + +My program is too slow. How do I speed it up? +--------------------------------------------- + +That's a tough one, in general. First, here are a list of things to +remember before diving further: + +* Performance characteristics vary accross Python implementations. This FAQ + focusses on :term:`CPython`. +* Behaviour can vary accross operating systems, especially when talking about + I/O or multi-threading. +* You should always find the hot spots in your program *before* attempting to + optimize any code (see the :mod:`profile` module). +* Writing benchmark scripts will allow you to iterate quickly when searching + for improvements (see the :mod:`timeit` module). +* It is highly recommended to have good code coverage (through unit testing + or any other technique) before potentially introducing regressions hidden + in sophisticated optimizations. + +That being said, there are many tricks to speed up Python code. Here are +some general principles which go a long way towards reaching acceptable +performance levels: + +* Making your algorithms faster (or changing to faster ones) can yield + much larger benefits than trying to sprinkle micro-optimization tricks + all over your code. + +* Use the right data structures. Study documentation for the :ref:`bltin-types` + and the :mod:`collections` module. + +* When the standard library provides a primitive for doing something, it is + likely (although not guaranteed) to be faster than any alternative you + may come up with. This is doubly true for primitives written in C, such + as builtins and some extension types. For example, be sure to use + either the :meth:`list.sort` built-in method or the related :func:`sorted` + function to do sorting (and see the + `sorting mini-HOWTO <http://wiki.python.org/moin/HowTo/Sorting>`_ for examples + of moderately advanced usage). + +* Abstractions tend to create indirections and force the interpreter to work + more. If the levels of indirection outweigh the amount of useful work + done, your program will be slower. You should avoid excessive abstraction, + especially under the form of tiny functions or methods (which are also often + detrimental to readability). + +If you have reached the limit of what pure Python can allow, there are tools +to take you further away. For example, `Cython <http://cython.org>`_ can +compile a slightly modified version of Python code into a C extension, and +can be used on many different platforms. Cython can take advantage of +compilation (and optional type annotations) to make your code significantly +faster than when interpreted. If you are confident in your C programming +skills, you can also :ref:`write a C extension module <extending-index>` +yourself. + +.. seealso:: + The wiki page devoted to `performance tips + <http://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. + +.. _efficient_string_concatenation: + What is the most efficient way to concatenate many strings together? -------------------------------------------------------------------- -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13528: rework the performance question in the programming FAQ
by antoine.pitrou Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/eb30f2becb79 changeset: 73908:eb30f2becb79 branch: 3.2 parent: 73904:8620e6901e58 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Dec 09 23:10:31 2011 +0100 summary: Issue #13528: rework the performance question in the programming FAQ files: Doc/faq/programming.rst | 214 ++++++++------------------- 1 files changed, 62 insertions(+), 152 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -115,159 +115,6 @@ :pep:`8`. -My program is too slow. How do I speed it up? ---------------------------------------------- - -That's a tough one, in general. There are many tricks to speed up Python code; -consider rewriting parts in C as a last resort. - -`Cython <http://cython.org>`_ and `Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ -can compile a slightly modified version of Python code into a C extension, and -can be used on many different platforms. Depending on your code, Cython -may be able to make it significantly faster than when run by the Python -interpreter. - -The rest of this answer will discuss various tricks for squeezing a bit more -speed out of Python code. *Never* apply any optimization tricks unless you know -you need them, after profiling has indicated that a particular function is the -heavily executed hot spot in the code. Optimizations almost always make the -code less clear, and you shouldn't pay the costs of reduced clarity (increased -development time, greater likelihood of bugs) unless the resulting performance -benefit is worth it. - -There is a page on the wiki devoted to `performance tips -<http://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. - -Guido van Rossum has written up an anecdote related to optimization at -http://www.python.org/doc/essays/list2str.html. - -One thing to notice is that function and (especially) method calls are rather -expensive; if you have designed a purely OO interface with lots of tiny -functions that don't do much more than get or set an instance variable or call -another method, you might consider using a more direct way such as directly -accessing instance variables. Also see the standard module :mod:`profile` which -makes it possible to find out where your program is spending most of its time -(if you have some patience -- the profiling itself can slow your program down by -an order of magnitude). - -Remember that many standard optimization heuristics you may know from other -programming experience may well apply to Python. For example it may be faster -to send output to output devices using larger writes rather than smaller ones in -order to reduce the overhead of kernel system calls. Thus CGI scripts that -write all output in "one shot" may be faster than those that write lots of small -pieces of output. - -Also, be sure to use Python's core features where appropriate. For example, -slicing allows programs to chop up lists and other sequence objects in a single -tick of the interpreter's mainloop using highly optimized C implementations. -Thus to get the same effect as:: - - L2 = [] - for i in range(3): - L2.append(L1[i]) - -it is much shorter and far faster to use :: - - L2 = list(L1[:3]) # "list" is redundant if L1 is a list. - -Note that the functionally-oriented built-in functions such as :func:`map`, -:func:`zip`, and friends can be a convenient accelerator for loops that -perform a single task. For example to pair the elements of two lists -together:: - - >>> list(zip([1, 2, 3], [4, 5, 6])) - [(1, 4), (2, 5), (3, 6)] - -or to compute a number of sines:: - - >>> list(map(math.sin, (1, 2, 3, 4))) - [0.841470984808, 0.909297426826, 0.14112000806, -0.756802495308] - -The operation completes very quickly in such cases. - -Other examples include the ``join()`` and ``split()`` :ref:`methods -of string objects <string-methods>`. - -For example if s1..s7 are large (10K+) strings then -``"".join([s1,s2,s3,s4,s5,s6,s7])`` may be far faster than the more obvious -``s1+s2+s3+s4+s5+s6+s7``, since the "summation" will compute many -subexpressions, whereas ``join()`` does all the copying in one pass. For -manipulating strings, use the ``replace()`` and the ``format()`` :ref:`methods -on string objects <string-methods>`. Use regular expressions only when you're -not dealing with constant string patterns. - -Be sure to use the :meth:`list.sort` built-in method to do sorting, and see the -`sorting mini-HOWTO <http://wiki.python.org/moin/HowTo/Sorting>`_ for examples -of moderately advanced usage. :meth:`list.sort` beats other techniques for -sorting in all but the most extreme circumstances. - -Another common trick is to "push loops into functions or methods." For example -suppose you have a program that runs slowly and you use the profiler to -determine that a Python function ``ff()`` is being called lots of times. If you -notice that ``ff()``:: - - def ff(x): - ... # do something with x computing result... - return result - -tends to be called in loops like:: - - list = map(ff, oldlist) - -or:: - - for x in sequence: - value = ff(x) - ... # do something with value... - -then you can often eliminate function call overhead by rewriting ``ff()`` to:: - - def ffseq(seq): - resultseq = [] - for x in seq: - ... # do something with x computing result... - resultseq.append(result) - return resultseq - -and rewrite the two examples to ``list = ffseq(oldlist)`` and to:: - - for value in ffseq(sequence): - ... # do something with value... - -Single calls to ``ff(x)`` translate to ``ffseq([x])[0]`` with little penalty. -Of course this technique is not always appropriate and there are other variants -which you can figure out. - -You can gain some performance by explicitly storing the results of a function or -method lookup into a local variable. A loop like:: - - for key in token: - dict[key] = dict.get(key, 0) + 1 - -resolves ``dict.get`` every iteration. If the method isn't going to change, a -slightly faster implementation is:: - - dict_get = dict.get # look up the method once - for key in token: - dict[key] = dict_get(key, 0) + 1 - -Default arguments can be used to determine values once, at compile time instead -of at run time. This can only be done for functions or objects which will not -be changed during program execution, such as replacing :: - - def degree_sin(deg): - return math.sin(deg * math.pi / 180.0) - -with :: - - def degree_sin(deg, factor=math.pi/180.0, sin=math.sin): - return sin(deg * factor) - -Because this trick uses default arguments for terms which should not be changed, -it should only be used when you are not concerned with presenting a possibly -confusing API to your users. - - Core Language ============= @@ -938,6 +785,68 @@ See the :ref:`unicode-howto`. +Performance +=========== + +My program is too slow. How do I speed it up? +--------------------------------------------- + +That's a tough one, in general. First, here are a list of things to +remember before diving further: + +* Performance characteristics vary accross Python implementations. This FAQ + focusses on :term:`CPython`. +* Behaviour can vary accross operating systems, especially when talking about + I/O or multi-threading. +* You should always find the hot spots in your program *before* attempting to + optimize any code (see the :mod:`profile` module). +* Writing benchmark scripts will allow you to iterate quickly when searching + for improvements (see the :mod:`timeit` module). +* It is highly recommended to have good code coverage (through unit testing + or any other technique) before potentially introducing regressions hidden + in sophisticated optimizations. + +That being said, there are many tricks to speed up Python code. Here are +some general principles which go a long way towards reaching acceptable +performance levels: + +* Making your algorithms faster (or changing to faster ones) can yield + much larger benefits than trying to sprinkle micro-optimization tricks + all over your code. + +* Use the right data structures. Study documentation for the :ref:`bltin-types` + and the :mod:`collections` module. + +* When the standard library provides a primitive for doing something, it is + likely (although not guaranteed) to be faster than any alternative you + may come up with. This is doubly true for primitives written in C, such + as builtins and some extension types. For example, be sure to use + either the :meth:`list.sort` built-in method or the related :func:`sorted` + function to do sorting (and see the + `sorting mini-HOWTO <http://wiki.python.org/moin/HowTo/Sorting>`_ for examples + of moderately advanced usage). + +* Abstractions tend to create indirections and force the interpreter to work + more. If the levels of indirection outweigh the amount of useful work + done, your program will be slower. You should avoid excessive abstraction, + especially under the form of tiny functions or methods (which are also often + detrimental to readability). + +If you have reached the limit of what pure Python can allow, there are tools +to take you further away. For example, `Cython <http://cython.org>`_ can +compile a slightly modified version of Python code into a C extension, and +can be used on many different platforms. Cython can take advantage of +compilation (and optional type annotations) to make your code significantly +faster than when interpreted. If you are confident in your C programming +skills, you can also :ref:`write a C extension module <extending-index>` +yourself. + +.. seealso:: + The wiki page devoted to `performance tips + <http://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. + +.. _efficient_string_concatenation: + What is the most efficient way to concatenate many strings together? -------------------------------------------------------------------- -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Closes #2979: add parameter 'use_builtin_types' to the SimpleXMLRPCServer.
by florent.xicluna Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/b3c1a504ebc1 changeset: 73907:b3c1a504ebc1 user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Fri Dec 09 22:35:06 2011 +0100 summary: Closes #2979: add parameter 'use_builtin_types' to the SimpleXMLRPCServer. files: Doc/library/xmlrpc.server.rst | 28 ++++++++++++++++-- Lib/test/test_xmlrpc.py | 34 +++++++++++++++++++++++ Lib/xmlrpc/server.py | 25 ++++++++++------ 3 files changed, 73 insertions(+), 14 deletions(-) diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -16,7 +16,9 @@ :class:`CGIXMLRPCRequestHandler`. -.. class:: SimpleXMLRPCServer(addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True) +.. class:: SimpleXMLRPCServer(addr, requestHandler=SimpleXMLRPCRequestHandler,\ + logRequests=True, allow_none=False, encoding=None,\ + bind_and_activate=True, use_builtin_types=False) Create a new server instance. This class provides methods for registration of functions that can be called by the XML-RPC protocol. The *requestHandler* @@ -25,18 +27,31 @@ are passed to the :class:`socketserver.TCPServer` constructor. If *logRequests* is true (the default), requests will be logged; setting this parameter to false will turn off logging. The *allow_none* and *encoding* parameters are passed - on to :mod:`xmlrpc.client` and control the XML-RPC responses that will be returned + on to :mod:`xmlrpc.client` and control the XML-RPC responses that will be returned from the server. The *bind_and_activate* parameter controls whether :meth:`server_bind` and :meth:`server_activate` are called immediately by the constructor; it defaults to true. Setting it to false allows code to manipulate the *allow_reuse_address* class variable before the address is bound. + The *use_builtin_types* parameter is passed to the + :func:`~xmlrpc.client.loads` function and controls which types are processed + when date/times values or binary data are received; it defaults to false. + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. -.. class:: CGIXMLRPCRequestHandler(allow_none=False, encoding=None) + +.. class:: CGIXMLRPCRequestHandler(allow_none=False, encoding=None,\ + use_builtin_types=False) Create a new instance to handle XML-RPC requests in a CGI environment. The *allow_none* and *encoding* parameters are passed on to :mod:`xmlrpc.client` and control the XML-RPC responses that will be returned from the server. + The *use_builtin_types* parameter is passed to the + :func:`~xmlrpc.client.loads` function and controls which types are processed + when date/times values or binary data are received; it defaults to false. + + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. .. class:: SimpleXMLRPCRequestHandler() @@ -233,12 +248,17 @@ :class:`DocCGIXMLRPCRequestHandler`. -.. class:: DocXMLRPCServer(addr, requestHandler=DocXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True) +.. class:: DocXMLRPCServer(addr, requestHandler=DocXMLRPCRequestHandler,\ + logRequests=True, allow_none=False, encoding=None,\ + bind_and_activate=True, use_builtin_types=True) Create a new server instance. All parameters have the same meaning as for :class:`SimpleXMLRPCServer`; *requestHandler* defaults to :class:`DocXMLRPCRequestHandler`. + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. + .. class:: DocCGIXMLRPCRequestHandler() diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -1023,10 +1023,44 @@ len(content)) +class UseBuiltinTypesTestCase(unittest.TestCase): + + def test_use_builtin_types(self): + # SimpleXMLRPCDispatcher.__init__ accepts use_builtin_types, which + # makes all dispatch of binary data as bytes instances, and all + # dispatch of datetime argument as datetime.datetime instances. + self.log = [] + expected_bytes = b"my dog has fleas" + expected_date = datetime.datetime(2008, 5, 26, 18, 25, 12) + marshaled = xmlrpclib.dumps((expected_bytes, expected_date), 'foobar') + def foobar(*args): + self.log.extend(args) + handler = xmlrpc.server.SimpleXMLRPCDispatcher( + allow_none=True, encoding=None, use_builtin_types=True) + handler.register_function(foobar) + handler._marshaled_dispatch(marshaled) + self.assertEqual(len(self.log), 2) + mybytes, mydate = self.log + self.assertEqual(self.log, [expected_bytes, expected_date]) + self.assertIs(type(mydate), datetime.datetime) + self.assertIs(type(mybytes), bytes) + + def test_cgihandler_has_use_builtin_types_flag(self): + handler = xmlrpc.server.CGIXMLRPCRequestHandler(use_builtin_types=True) + self.assertTrue(handler.use_builtin_types) + + def test_xmlrpcserver_has_use_builtin_types_flag(self): + server = xmlrpc.server.SimpleXMLRPCServer(("localhost", 0), + use_builtin_types=True) + server.server_close() + self.assertTrue(server.use_builtin_types) + + @support.reap_threads def test_main(): xmlrpc_tests = [XMLRPCTestCase, HelperTestCase, DateTimeTestCase, BinaryTestCase, FaultTestCase] + xmlrpc_tests.append(UseBuiltinTypesTestCase) xmlrpc_tests.append(SimpleServerTestCase) xmlrpc_tests.append(KeepaliveServerTestCase1) xmlrpc_tests.append(KeepaliveServerTestCase2) diff --git a/Lib/xmlrpc/server.py b/Lib/xmlrpc/server.py --- a/Lib/xmlrpc/server.py +++ b/Lib/xmlrpc/server.py @@ -160,11 +160,13 @@ can be instanced when used by the MultiPathXMLRPCServer """ - def __init__(self, allow_none=False, encoding=None): + def __init__(self, allow_none=False, encoding=None, + use_builtin_types=False): self.funcs = {} self.instance = None self.allow_none = allow_none self.encoding = encoding or 'utf-8' + self.use_builtin_types = use_builtin_types def register_instance(self, instance, allow_dotted_names=False): """Registers an instance to respond to XML-RPC requests. @@ -245,7 +247,7 @@ """ try: - params, method = loads(data) + params, method = loads(data, use_builtin_types=self.use_builtin_types) # generate response if dispatch_method is not None: @@ -572,10 +574,11 @@ _send_traceback_header = False def __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, - logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): + logRequests=True, allow_none=False, encoding=None, + bind_and_activate=True, use_builtin_types=False): self.logRequests = logRequests - SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) + SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding, use_builtin_types) socketserver.TCPServer.__init__(self, addr, requestHandler, bind_and_activate) # [Bug #1222790] If possible, set close-on-exec flag; if a @@ -595,10 +598,11 @@ Make sure that the requestHandler accepts the paths in question. """ def __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler, - logRequests=True, allow_none=False, encoding=None, bind_and_activate=True): + logRequests=True, allow_none=False, encoding=None, + bind_and_activate=True, use_builtin_types=False): SimpleXMLRPCServer.__init__(self, addr, requestHandler, logRequests, allow_none, - encoding, bind_and_activate) + encoding, bind_and_activate, use_builtin_types) self.dispatchers = {} self.allow_none = allow_none self.encoding = encoding or 'utf-8' @@ -628,8 +632,8 @@ class CGIXMLRPCRequestHandler(SimpleXMLRPCDispatcher): """Simple handler for XML-RPC data passed through CGI.""" - def __init__(self, allow_none=False, encoding=None): - SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding) + def __init__(self, allow_none=False, encoding=None, use_builtin_types=False): + SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding, use_builtin_types) def handle_xmlrpc(self, request_text): """Handle a single XML-RPC request""" @@ -924,9 +928,10 @@ def __init__(self, addr, requestHandler=DocXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, - bind_and_activate=True): + bind_and_activate=True, use_builtin_types=False): SimpleXMLRPCServer.__init__(self, addr, requestHandler, logRequests, - allow_none, encoding, bind_and_activate) + allow_none, encoding, bind_and_activate, + use_builtin_types) XMLRPCDocGenerator.__init__(self) class DocCGIXMLRPCRequestHandler( CGIXMLRPCRequestHandler, -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Write tests for invalid characters (U+00110000)
by victor.stinner Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/bfa9d1ba36ae changeset: 73906:bfa9d1ba36ae user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 09 20:49:49 2011 +0100 summary: Write tests for invalid characters (U+00110000) Test the following functions: * codecs.raw_unicode_escape_decode() * PyUnicode_FromWideChar() * PyUnicode_FromUnicode() * "unicode_internal" and "unicode_escape" decoders files: Lib/test/test_codecs.py | 16 ++++++++++++++++ Modules/_testcapimodule.c | 18 ++++++++++++++++++ 2 files changed, 34 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1034,6 +1034,16 @@ 'deprecated', DeprecationWarning)): self.assertRaises(UnicodeDecodeError, internal.decode, "unicode_internal") + if sys.byteorder == "little": + invalid = b"\x00\x00\x11\x00" + else: + invalid = b"\x00\x11\x00\x00" + with support.check_warnings(): + self.assertRaises(UnicodeDecodeError, + invalid.decode, "unicode_internal") + with support.check_warnings(): + self.assertEqual(invalid.decode("unicode_internal", "replace"), + '\ufffd') @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t') def test_decode_error_attributes(self): @@ -1729,6 +1739,12 @@ self.assertEqual(codecs.raw_unicode_escape_decode(r"\u1234"), ("\u1234", 6)) self.assertEqual(codecs.raw_unicode_escape_decode(br"\u1234"), ("\u1234", 6)) + self.assertRaises(UnicodeDecodeError, codecs.unicode_escape_decode, br"\U00110000") + self.assertEqual(codecs.unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10)) + + self.assertRaises(UnicodeDecodeError, codecs.raw_unicode_escape_decode, br"\U00110000") + self.assertEqual(codecs.raw_unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10)) + class SurrogateEscapeTest(unittest.TestCase): def test_utf8(self): diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1409,6 +1409,7 @@ #if defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4) const wchar_t wtext[2] = {(wchar_t)0x10ABCDu}; size_t wtextlen = 1; + const wchar_t invalid[1] = {(wchar_t)0x110000u}; #else const wchar_t wtext[3] = {(wchar_t)0xDBEAu, (wchar_t)0xDFCDu}; size_t wtextlen = 2; @@ -1444,6 +1445,23 @@ Py_DECREF(wide); Py_DECREF(utf8); + +#if defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4) + wide = PyUnicode_FromWideChar(invalid, 1); + if (wide == NULL) + PyErr_Clear(); + else + return raiseTestError("test_widechar", + "PyUnicode_FromWideChar(L\"\\U00110000\", 1) didn't fail"); + + wide = PyUnicode_FromUnicode(invalid, 1); + if (wide == NULL) + PyErr_Clear(); + else + return raiseTestError("test_widechar", + "PyUnicode_FromUnicode(L\"\\U00110000\", 1) didn't fail"); +#endif + Py_RETURN_NONE; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): (Merge 3.2) Issue #5905: time.strftime() is now using the locale encoding,
by victor.stinner Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/bee7694988a4 changeset: 73905:bee7694988a4 parent: 73903:7ffe3d304487 parent: 73904:8620e6901e58 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 09 20:21:17 2011 +0100 summary: (Merge 3.2) Issue #5905: time.strftime() is now using the locale encoding, instead of UTF-8, if the wcsftime() function is not available. files: Misc/NEWS | 3 +++ Modules/timemodule.c | 15 ++++----------- 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,9 @@ Library ------- +- Issue #5905: time.strftime() is now using the locale encoding, instead of + UTF-8, if the wcsftime() function is not available. + - Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". Patch by Tal Einat. diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -30,12 +30,6 @@ #endif /* MS_WINDOWS */ #endif /* !__WATCOMC__ || __QNX__ */ -#if defined(HAVE_MBCS) -# define TZNAME_ENCODING "mbcs" -#else -# define TZNAME_ENCODING "utf-8" -#endif - #if defined(PYOS_OS2) #define INCL_DOS #define INCL_ERRORS @@ -492,7 +486,7 @@ fmt = format; #else /* Convert the unicode string to an ascii one */ - format = PyUnicode_AsEncodedString(format_arg, TZNAME_ENCODING, NULL); + format = PyUnicode_EncodeFSDefault(format_arg); if (format == NULL) return NULL; fmt = PyBytes_AS_STRING(format); @@ -536,8 +530,7 @@ #ifdef HAVE_WCSFTIME ret = PyUnicode_FromWideChar(outbuf, buflen); #else - ret = PyUnicode_Decode(outbuf, buflen, - TZNAME_ENCODING, NULL); + ret = PyUnicode_DecodeFSDefaultAndSize(outbuf, buflen); #endif PyMem_Free(outbuf); break; @@ -769,8 +762,8 @@ #endif /* PYOS_OS2 */ #endif PyModule_AddIntConstant(m, "daylight", daylight); - otz0 = PyUnicode_Decode(tzname[0], strlen(tzname[0]), TZNAME_ENCODING, NULL); - otz1 = PyUnicode_Decode(tzname[1], strlen(tzname[1]), TZNAME_ENCODING, NULL); + otz0 = PyUnicode_DecodeFSDefaultAndSize(tzname[0], strlen(tzname[0])); + otz1 = PyUnicode_DecodeFSDefaultAndSize(tzname[1], strlen(tzname[1])); PyModule_AddObject(m, "tzname", Py_BuildValue("(NN)", otz0, otz1)); #else /* !HAVE_TZNAME || __GLIBC__ || __CYGWIN__*/ #ifdef HAVE_STRUCT_TM_TM_ZONE -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #5905: time.strftime() is now using the locale encoding, instead of
by victor.stinner Dec. 10, 2011

Dec. 10, 2011

http://hg.python.org/cpython/rev/8620e6901e58 changeset: 73904:8620e6901e58 branch: 3.2 parent: 73894:8ed2c4d4df62 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 09 20:19:24 2011 +0100 summary: Issue #5905: time.strftime() is now using the locale encoding, instead of UTF-8, if the wcsftime() function is not available. files: Misc/NEWS | 3 +++ Modules/timemodule.c | 13 ++++--------- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -90,6 +90,9 @@ Library ------- +- Issue #5905: time.strftime() is now using the locale encoding, instead of + UTF-8, if the wcsftime() function is not available. + - Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". Patch by Tal Einat. diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -3,8 +3,6 @@ #include "Python.h" #include "_time.h" -#define TZNAME_ENCODING "utf-8" - #include <ctype.h> #ifdef HAVE_SYS_TYPES_H @@ -48,8 +46,6 @@ #if defined(MS_WINDOWS) && !defined(__BORLANDC__) /* Win32 has better clock replacement; we have our own version below. */ #undef HAVE_CLOCK -#undef TZNAME_ENCODING -#define TZNAME_ENCODING "mbcs" #endif /* MS_WINDOWS && !defined(__BORLANDC__) */ #if defined(PYOS_OS2) @@ -502,7 +498,7 @@ fmt = format; #else /* Convert the unicode string to an ascii one */ - format = PyUnicode_AsEncodedString(format_arg, TZNAME_ENCODING, NULL); + format = PyUnicode_EncodeFSDefault(format_arg); if (format == NULL) return NULL; fmt = PyBytes_AS_STRING(format); @@ -546,8 +542,7 @@ #ifdef HAVE_WCSFTIME ret = PyUnicode_FromWideChar(outbuf, buflen); #else - ret = PyUnicode_Decode(outbuf, buflen, - TZNAME_ENCODING, NULL); + ret = PyUnicode_DecodeFSDefaultAndSize(outbuf, buflen); #endif PyMem_Free(outbuf); break; @@ -789,8 +784,8 @@ #endif /* PYOS_OS2 */ #endif PyModule_AddIntConstant(m, "daylight", daylight); - otz0 = PyUnicode_Decode(tzname[0], strlen(tzname[0]), TZNAME_ENCODING, NULL); - otz1 = PyUnicode_Decode(tzname[1], strlen(tzname[1]), TZNAME_ENCODING, NULL); + otz0 = PyUnicode_DecodeFSDefaultAndSize(tzname[0], strlen(tzname[0])); + otz1 = PyUnicode_DecodeFSDefaultAndSize(tzname[1], strlen(tzname[1])); PyModule_AddObject(m, "tzname", Py_BuildValue("(NN)", otz0, otz1)); #else /* !HAVE_TZNAME || __GLIBC__ || __CYGWIN__*/ #ifdef HAVE_STRUCT_TM_TM_ZONE -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13441: Enable the workaround for Solaris locale bug
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/7ffe3d304487 changeset: 73903:7ffe3d304487 user: Victor Stinner <vstinner(a)wyplay.com> date: Fri Dec 09 11:29:44 2011 +0100 summary: Issue #13441: Enable the workaround for Solaris locale bug Skip locales triggering the mbstowcs() bug. I collected the locale list thanks my previous commit: * hu_HU (ISO8859-2): character U+30000020 * de_AT (ISO8859-1): character U+30000076 * cs_CZ (ISO8859-2): character U+30000020 * sk_SK (ISO8859-2): character U+30000020 * pl_PL (ISO8859-2): character U+30000020 * fr_CA (ISO8859-1): character U+30000020 files: Lib/test/test__locale.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py --- a/Lib/test/test__locale.py +++ b/Lib/test/test__locale.py @@ -48,7 +48,7 @@ % (loc, encoding, type(err), err)) else: locales.append(loc) - #candidate_locales = locales + candidate_locales = locales finally: locale.setlocale(locale.LC_ALL, old_locale) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13441: Skip some locales (e.g. cs_CZ and hu_HU) on Solaris to workaround
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/2a2d0872d993 changeset: 73902:2a2d0872d993 parent: 73900:87c6be1e393a user: Victor Stinner <vstinner(a)wyplay.com> date: Fri Dec 09 10:28:45 2011 +0100 summary: Issue #13441: Skip some locales (e.g. cs_CZ and hu_HU) on Solaris to workaround a mbstowcs() bug. For example, on Solaris, the hu_HU locale uses the locale encoding ISO-8859-2, the thousauds separator is b'\xA0' and it is decoded as U+30000020 (an invalid character) by mbstowcs(). The workaround is not enabled yet (commented): I would like first to get more information about the failing locales. files: Lib/test/test__locale.py | 43 +++++++++++++++++++-------- 1 files changed, 30 insertions(+), 13 deletions(-) diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py --- a/Lib/test/test__locale.py +++ b/Lib/test/test__locale.py @@ -1,13 +1,15 @@ -from test.support import run_unittest from _locale import (setlocale, LC_ALL, LC_CTYPE, LC_NUMERIC, localeconv, Error) try: from _locale import (RADIXCHAR, THOUSEP, nl_langinfo) except ImportError: nl_langinfo = None +import codecs +import locale +import sys import unittest -import sys from platform import uname +from test.support import run_unittest if uname()[0] == "Darwin": maj, min, mic = [int(part) for part in uname()[2].split(".")] @@ -17,7 +19,7 @@ candidate_locales = ['es_UY', 'fr_FR', 'fi_FI', 'es_CO', 'pt_PT', 'it_IT', 'et_EE', 'es_PY', 'no_NO', 'nl_NL', 'lv_LV', 'el_GR', 'be_BY', 'fr_BE', 'ro_RO', 'ru_UA', 'ru_RU', 'es_VE', 'ca_ES', 'se_NO', 'es_EC', 'id_ID', - 'ka_GE', 'es_CL', 'wa_BE', 'lt_LT', 'sl_SI', 'hr_HR', 'es_AR', + 'ka_GE', 'es_CL', 'wa_BE', 'hu_HU', 'lt_LT', 'sl_SI', 'hr_HR', 'es_AR', 'es_ES', 'oc_FR', 'gl_ES', 'bg_BG', 'is_IS', 'mk_MK', 'de_AT', 'pt_BR', 'da_DK', 'nn_NO', 'cs_CZ', 'de_LU', 'es_BO', 'sq_AL', 'sk_SK', 'fr_CH', 'de_DE', 'sr_YU', 'br_FR', 'nl_BE', 'sv_FI', 'pl_PL', 'fr_CA', 'fo_FO', @@ -25,12 +27,30 @@ 'eu_ES', 'vi_VN', 'af_ZA', 'nb_NO', 'en_DK', 'tg_TJ', 'en_US', 'es_ES.ISO8859-1', 'fr_FR.ISO8859-15', 'ru_RU.KOI8-R', 'ko_KR.eucKR'] -# Issue #13441: Don't test the hu_HU locale on Solaris to workaround a -# mbstowcs() bug. On Solaris, if the locale is hu_HU (and if the locale -# encoding is not UTF-8), the thousauds separator is b'\xA0' which is decoded -# as U+30000020 instead of U+0020 by mbstowcs(). -if sys.platform != 'sunos5': - candidate_locales.append('hu_HU') +# Issue #13441: Skip some locales (e.g. cs_CZ and hu_HU) on Solaris to +# workaround a mbstowcs() bug. For example, on Solaris, the hu_HU locale uses +# the locale encoding ISO-8859-2, the thousauds separator is b'\xA0' and it is +# decoded as U+30000020 (an invalid character) by mbstowcs(). +if sys.platform == 'sunos5': + old_locale = locale.setlocale(locale.LC_ALL) + try: + locales = [] + for loc in candidate_locales: + try: + locale.setlocale(locale.LC_ALL, loc) + except Error: + continue + encoding = locale.getpreferredencoding(False) + try: + localeconv() + except Exception as err: + print("WARNING: Skip locale %s (encoding %s): [%s] %s" + % (loc, encoding, type(err), err)) + else: + locales.append(loc) + #candidate_locales = locales + finally: + locale.setlocale(locale.LC_ALL, old_locale) # Workaround for MSVC6(debug) crash bug if "MSC v.1200" in sys.version: @@ -93,10 +113,7 @@ setlocale(LC_CTYPE, loc) except Error: continue - try: - formatting = localeconv() - except Exception as err: - self.fail("localeconv() failed with %s locale: %s" % (loc, err)) + formatting = localeconv() for lc in ("decimal_point", "thousands_sep"): self.numeric_tester('localeconv', formatting[lc], lc, loc) -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (87c6be1e393a): sum=0
by solipsis＠pitrou.net Dec. 9, 2011

Dec. 9, 2011

results for 87c6be1e393a on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogf1EUl2', '-x']

1 0

cpython (2.7): Implemented suggested improvements for pdb test by Éric Araujo
by jason.coombs Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/70337a6d5dde changeset: 73901:70337a6d5dde branch: 2.7 parent: 73892:9d329adbbb01 user: Jason R. Coombs <jaraco(a)jaraco.com> date: Thu Dec 08 22:14:56 2011 -0500 summary: Implemented suggested improvements for pdb test by Éric Araujo files: Lib/test/test_pdb.py | 35 ++++++++++++++++--------------- 1 files changed, 18 insertions(+), 17 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -280,35 +280,36 @@ 4 """ -class Tester7750(unittest.TestCase): - # if the filename has something that resolves to a python - # escape character (such as \t), it will fail - test_fn = '.\\test7750.py' +class ModuleInitTester(unittest.TestCase): - msg = "issue7750 only applies when os.sep is a backslash" - @unittest.skipUnless(os.path.sep == '\\', msg) - def test_issue7750(self): - with open(self.test_fn, 'w') as f: - f.write('print("hello world")') - cmd = [sys.executable, '-m', 'pdb', self.test_fn,] + def test_filename_correct(self): + """ + In issue 7750, it was found that if the filename has a sequence that + resolves to an escape character in a Python string (such as \t), it + will be treated as the escaped character. + """ + # the test_fn must contain something like \t + # on Windows, this will create 'test_mod.py' in the current directory. + # on Unix, this will create '.\test_mod.py' in the current directory. + test_fn = '.\\test_mod.py' + code = 'print("testing pdb")' + with open(test_fn, 'w') as f: + f.write(code) + self.addCleanup(os.remove, test_fn) + cmd = [sys.executable, '-m', 'pdb', test_fn,] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, ) stdout, stderr = proc.communicate('quit\n') - self.assertNotIn('IOError', stdout, "pdb munged the filename") - - def tearDown(self): - if os.path.isfile(self.test_fn): - os.remove(self.test_fn) + self.assertIn(code, stdout, "pdb munged the filename") def test_main(): from test import test_pdb test_support.run_doctest(test_pdb, verbosity=True) - + test_support.run_unittest(ModuleInitTester) if __name__ == '__main__': test_main() - unittest.main() -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13441: Don't test the hu_HU locale on Solaris to workaround a mbstowcs()
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/87c6be1e393a changeset: 73900:87c6be1e393a user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 09 01:20:03 2011 +0100 summary: Issue #13441: Don't test the hu_HU locale on Solaris to workaround a mbstowcs() bug. On Solaris, if the locale is hu_HU (and if the locale encoding is not UTF-8), the thousauds separator is b'\xA0' which is decoded as U+30000020 instead of U+0020 by mbstowcs(). files: Lib/test/test__locale.py | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py --- a/Lib/test/test__locale.py +++ b/Lib/test/test__locale.py @@ -17,7 +17,7 @@ candidate_locales = ['es_UY', 'fr_FR', 'fi_FI', 'es_CO', 'pt_PT', 'it_IT', 'et_EE', 'es_PY', 'no_NO', 'nl_NL', 'lv_LV', 'el_GR', 'be_BY', 'fr_BE', 'ro_RO', 'ru_UA', 'ru_RU', 'es_VE', 'ca_ES', 'se_NO', 'es_EC', 'id_ID', - 'ka_GE', 'es_CL', 'hu_HU', 'wa_BE', 'lt_LT', 'sl_SI', 'hr_HR', 'es_AR', + 'ka_GE', 'es_CL', 'wa_BE', 'lt_LT', 'sl_SI', 'hr_HR', 'es_AR', 'es_ES', 'oc_FR', 'gl_ES', 'bg_BG', 'is_IS', 'mk_MK', 'de_AT', 'pt_BR', 'da_DK', 'nn_NO', 'cs_CZ', 'de_LU', 'es_BO', 'sq_AL', 'sk_SK', 'fr_CH', 'de_DE', 'sr_YU', 'br_FR', 'nl_BE', 'sv_FI', 'pl_PL', 'fr_CA', 'fo_FO', @@ -25,6 +25,13 @@ 'eu_ES', 'vi_VN', 'af_ZA', 'nb_NO', 'en_DK', 'tg_TJ', 'en_US', 'es_ES.ISO8859-1', 'fr_FR.ISO8859-15', 'ru_RU.KOI8-R', 'ko_KR.eucKR'] +# Issue #13441: Don't test the hu_HU locale on Solaris to workaround a +# mbstowcs() bug. On Solaris, if the locale is hu_HU (and if the locale +# encoding is not UTF-8), the thousauds separator is b'\xA0' which is decoded +# as U+30000020 instead of U+0020 by mbstowcs(). +if sys.platform != 'sunos5': + candidate_locales.append('hu_HU') + # Workaround for MSVC6(debug) crash bug if "MSC v.1200" in sys.version: def accept(loc): -- Repository URL: http://hg.python.org/cpython

1 0

cpython: What's New in Python 3.3: Add entry for lzma module (issue #6715).
by nadeem.vawda Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/66df5ace0eee changeset: 73899:66df5ace0eee user: Nadeem Vawda <nadeem.vawda(a)gmail.com> date: Fri Dec 09 01:32:46 2011 +0200 summary: What's New in Python 3.3: Add entry for lzma module (issue #6715). files: Doc/whatsnew/3.3.rst | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -391,6 +391,16 @@ (Contributed by Sijin Joseph in :issue:`8808`) +lzma +---- + +The newly-added :mod:`lzma` module provides data compression and decompression +using the LZMA algorithm, including support for the ``.xz`` and ``.lzma`` +file formats. + +(Contributed by Nadeem Vawda and Per Øyvind Karlsen in :issue:`6715`) + + math ---- -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Document PyUnicode_Copy() and PyUnicode_EncodeCodePage()
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/e7d94d35506b changeset: 73898:e7d94d35506b user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 09 00:18:11 2011 +0100 summary: Document PyUnicode_Copy() and PyUnicode_EncodeCodePage() files: Doc/c-api/unicode.rst | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -386,6 +386,13 @@ .. versionadded:: 3.3 +.. c:function:: PyObject* PyUnicode_Copy(PyObject *unicode) + + Get a new copy of a Unicode object. + + .. versionadded:: 3.3 + + .. c:function:: PyObject* PyUnicode_FromKindAndData(int kind, const void *buffer, \ Py_ssize_t size) @@ -1379,6 +1386,15 @@ raised by the codec. +.. c:function:: PyObject* PyUnicode_EncodeCodePage(int code_page, PyObject *unicode, const char *errors) + + Encode the Unicode object using the specified code page and return a Python + bytes object. Return *NULL* if an exception was raised by the codec. Use + :c:data:`CP_ACP` code page to get the MBCS encoder. + + .. versionadded:: 3.3 + + .. c:function:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors) Encode the :c:type:`Py_UNICODE` buffer of the given *size* using MBCS and return @@ -1387,7 +1403,7 @@ .. deprecated-removed:: 3.3 4.0 Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using - :c:func:`PyUnicode_AsMBCSString`. + :c:func:`PyUnicode_AsMBCSString` or :c:func:`PyUnicode_EncodeCodePage`. Methods & Slots -- Repository URL: http://hg.python.org/cpython

1 0

cpython: What's New in Python 3.3: Add a Deprecated section
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/0846fc6bf6a8 changeset: 73897:0846fc6bf6a8 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Dec 09 00:10:41 2011 +0100 summary: What's New in Python 3.3: Add a Deprecated section files: Doc/whatsnew/3.3.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -711,8 +711,11 @@ +Deprecated +========== + Unsupported Operating Systems -============================= +----------------------------- OS/2 and VMS are no longer supported due to the lack of a maintainer. @@ -721,7 +724,7 @@ Deprecated Python modules, functions and methods -================================================ +------------------------------------------------ * The :mod:`packaging` module replaces the :mod:`distutils` module * The ``unicode_internal`` codec has been deprecated because of the @@ -737,7 +740,7 @@ Deprecated functions and types of the C API -=========================================== +------------------------------------------- The :c:type:`Py_UNICODE` has been deprecated by the :pep:`393` and will be removed in Python 4. All functions using this type are deprecated: -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13441: Log the locale when localeconv() fails
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/93bab8400ca5 changeset: 73896:93bab8400ca5 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 08 23:42:52 2011 +0100 summary: Issue #13441: Log the locale when localeconv() fails files: Lib/test/test__locale.py | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py --- a/Lib/test/test__locale.py +++ b/Lib/test/test__locale.py @@ -86,9 +86,13 @@ setlocale(LC_CTYPE, loc) except Error: continue + try: + formatting = localeconv() + except Exception as err: + self.fail("localeconv() failed with %s locale: %s" % (loc, err)) for lc in ("decimal_point", "thousands_sep"): - self.numeric_tester('localeconv', localeconv()[lc], lc, loc) + self.numeric_tester('localeconv', formatting[lc], lc, loc) @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") def test_lc_numeric_basic(self): -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge fix for issue #13547.
by stefan.krah Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/053c95ad09cf changeset: 73895:053c95ad09cf parent: 73893:090574ed8db1 parent: 73894:8ed2c4d4df62 user: Stefan Krah <skrah(a)bytereef.org> date: Thu Dec 08 23:31:40 2011 +0100 summary: Merge fix for issue #13547. files: Makefile.pre.in | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1315,6 +1315,8 @@ find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name 'fficonfig.py' -exec rm -f {} ';' || true -rm -f Lib/lib2to3/*Grammar*.pickle + -rm -f Lib/_sysconfigdata.py + -rm -f Modules/_testembed profile-removal: find . -name '*.gc??' -exec rm -f {} ';' -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13547: clean Lib/_sysconfigdata.py and Modules/_testembed
by stefan.krah Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/8ed2c4d4df62 changeset: 73894:8ed2c4d4df62 branch: 3.2 parent: 73890:7efad6256e58 user: Stefan Krah <skrah(a)bytereef.org> date: Thu Dec 08 23:25:15 2011 +0100 summary: Issue #13547: clean Lib/_sysconfigdata.py and Modules/_testembed files: Makefile.pre.in | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1252,6 +1252,7 @@ find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name 'fficonfig.py' -exec rm -f {} ';' || true -rm -f Lib/lib2to3/*Grammar*.pickle + -rm -f Modules/_testembed profile-removal: find . -name '*.gc??' -exec rm -f {} ';' -- Repository URL: http://hg.python.org/cpython

1 0

cpython: PyUnicode_FromWideChar() and PyUnicode_FromUnicode() raise a ValueError if a
by victor.stinner Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/489ea02ed351 changeset: 73889:489ea02ed351 parent: 73887:c7638be1e430 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 08 22:14:11 2011 +0100 summary: PyUnicode_FromWideChar() and PyUnicode_FromUnicode() raise a ValueError if a character in not in range [U+0000; U+10ffff]. files: Objects/unicodeobject.c | 67 ++++++++++++++-------------- 1 files changed, 34 insertions(+), 33 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -66,6 +66,9 @@ extern "C" { #endif +/* Maximum code point of Unicode 6.0: 0x10ffff (1,114,111) */ +#define MAX_UNICODE 0x10ffff + #ifdef Py_DEBUG # define _PyUnicode_CHECK(op) _PyUnicode_CheckConsistency(op, 0) #else @@ -393,9 +396,7 @@ } else { assert(maxchar >= 0x10000); - /* FIXME: Issue #13441: on Solaris, localeconv() and strxfrm() - return characters outside the range U+0000-U+10FFFF. */ - /* assert(maxchar <= 0x10FFFF); */ + assert(maxchar <= MAX_UNICODE); } } return 1; @@ -1295,36 +1296,37 @@ Py_UCS4 *maxchar, Py_ssize_t *num_surrogates) { const wchar_t *iter; + Py_UCS4 ch; assert(num_surrogates != NULL && maxchar != NULL); *num_surrogates = 0; *maxchar = 0; for (iter = begin; iter < end; ) { - if (*iter > *maxchar) { - *maxchar = *iter; -#if SIZEOF_WCHAR_T != 2 - if (*maxchar >= 0x10000) - return 0; -#endif - } #if SIZEOF_WCHAR_T == 2 if (Py_UNICODE_IS_HIGH_SURROGATE(iter[0]) && (iter+1) < end && Py_UNICODE_IS_LOW_SURROGATE(iter[1])) { - Py_UCS4 surrogate_val; - surrogate_val = Py_UNICODE_JOIN_SURROGATES(iter[0], iter[1]); + ch = Py_UNICODE_JOIN_SURROGATES(iter[0], iter[1]); ++(*num_surrogates); - if (surrogate_val > *maxchar) - *maxchar = surrogate_val; iter += 2; } else +#endif + { + ch = *iter; iter++; -#else - iter++; -#endif + } + if (ch > *maxchar) { + *maxchar = ch; + if (*maxchar > MAX_UNICODE) { + PyErr_Format(PyExc_ValueError, + "character U+%x is not in range [U+0000; U+10ffff]", + ch); + return -1; + } + } } return 0; } @@ -1669,8 +1671,7 @@ &maxchar, &num_surrogates) == -1) return NULL; - unicode = PyUnicode_New(size - num_surrogates, - maxchar); + unicode = PyUnicode_New(size - num_surrogates, maxchar); if (!unicode) return NULL; @@ -1808,7 +1809,7 @@ return 0x10000; default: assert(0 && "invalid kind"); - return 0x10ffff; + return MAX_UNICODE; } } @@ -2796,7 +2797,7 @@ PyUnicode_FromOrdinal(int ordinal) { PyObject *v; - if (ordinal < 0 || ordinal > 0x10ffff) { + if (ordinal < 0 || ordinal > MAX_UNICODE) { PyErr_SetString(PyExc_ValueError, "chr() arg not in range(0x110000)"); return NULL; @@ -3472,7 +3473,7 @@ four_bytes = PyUnicode_4BYTE_DATA(unicode); for (; four_bytes < ucs4_end; ++four_bytes, ++w) { if (*four_bytes > 0xFFFF) { - assert(*four_bytes <= 0x10FFFF); + assert(*four_bytes <= MAX_UNICODE); /* encode surrogate pair in this case */ *w++ = Py_UNICODE_HIGH_SURROGATE(*four_bytes); *w = Py_UNICODE_LOW_SURROGATE(*four_bytes); @@ -4118,7 +4119,7 @@ continue; encode_char: if (ch >= 0x10000) { - assert(ch <= 0x10FFFF); + assert(ch <= MAX_UNICODE); /* code first surrogate */ base64bits += 16; @@ -4577,7 +4578,7 @@ } ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) + ((s[2] & 0x3f) << 6) + (s[3] & 0x3f); - assert ((ch > 0xFFFF) && (ch <= 0x10ffff)); + assert ((ch > 0xFFFF) && (ch <= MAX_UNICODE)); WRITE_MAYBE_FAIL(i++, ch); break; @@ -4714,7 +4715,7 @@ } ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) + ((s[2] & 0x3f) << 6) + (s[3] & 0x3f); - assert ((ch > 0xFFFF) && (ch <= 0x10ffff)); + assert ((ch > 0xFFFF) && (ch <= MAX_UNICODE)); #if SIZEOF_WCHAR_T == 4 *p++ = (wchar_t)ch; @@ -4884,7 +4885,7 @@ *p++ = (char)(0x80 | ((ch >> 6) & 0x3f)); *p++ = (char)(0x80 | (ch & 0x3f)); } else /* ch >= 0x10000 */ { - assert(ch <= 0x10FFFF); + assert(ch <= MAX_UNICODE); /* Encode UCS4 Unicode ordinals */ *p++ = (char)(0xf0 | (ch >> 18)); *p++ = (char)(0x80 | ((ch >> 12) & 0x3f)); @@ -5792,7 +5793,7 @@ break; store: /* when we get here, chr is a 32-bit unicode character */ - if (chr <= 0x10ffff) { + if (chr <= MAX_UNICODE) { WRITECHAR(chr); } else { endinpos = s-starts; @@ -5957,7 +5958,7 @@ /* Map 21-bit characters to '\U00xxxxxx' */ else if (ch >= 0x10000) { - assert(ch <= 0x10FFFF); + assert(ch <= MAX_UNICODE); *p++ = '\\'; *p++ = 'U'; *p++ = Py_hexdigits[(ch >> 28) & 0x0000000F]; @@ -6108,7 +6109,7 @@ else x += 10 + c - 'A'; } - if (x <= 0x10ffff) { + if (x <= MAX_UNICODE) { if (unicode_putchar(&v, &outpos, x) < 0) goto onError; } else { @@ -6175,7 +6176,7 @@ Py_UCS4 ch = PyUnicode_READ(kind, data, pos); /* Map 32-bit characters to '\Uxxxxxxxx' */ if (ch >= 0x10000) { - assert(ch <= 0x10FFFF); + assert(ch <= MAX_UNICODE); *p++ = '\\'; *p++ = 'U'; *p++ = Py_hexdigits[(ch >> 28) & 0xf]; @@ -6536,7 +6537,7 @@ else if (ch < 1000000) repsize += 2+6+1; else { - assert(ch <= 0x10FFFF); + assert(ch <= MAX_UNICODE); repsize += 2+7+1; } } @@ -9275,7 +9276,7 @@ else if (maxchar_new <= 65535) maxchar_new = 65535; else - maxchar_new = 1114111; /* 0x10ffff */ + maxchar_new = MAX_UNICODE; if (!maxchar_new && PyUnicode_CheckExact(self)) { /* fixfct should return TRUE if it modified the buffer. If @@ -13059,7 +13060,7 @@ if (x == -1 && PyErr_Occurred()) goto onError; - if (x < 0 || x > 0x10ffff) { + if (x < 0 || x > MAX_UNICODE) { PyErr_SetString(PyExc_OverflowError, "%c arg not in range(0x110000)"); return (Py_UCS4) -1; -- Repository URL: http://hg.python.org/cpython

2 1

cpython (merge default -> default): Merge.
by stefan.krah Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/090574ed8db1 changeset: 73893:090574ed8db1 parent: 73889:489ea02ed351 parent: 73891:e48df59af394 user: Stefan Krah <skrah(a)bytereef.org> date: Thu Dec 08 22:30:18 2011 +0100 summary: Merge. files: configure | 7 ++++--- configure.in | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -5450,9 +5450,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) diff --git a/configure.in b/configure.in --- a/configure.in +++ b/configure.in @@ -907,9 +907,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Backport second fix for issue #11149.
by stefan.krah Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/9d329adbbb01 changeset: 73892:9d329adbbb01 branch: 2.7 parent: 73888:2bca2cee79a1 user: Stefan Krah <skrah(a)bytereef.org> date: Thu Dec 08 22:26:06 2011 +0100 summary: Backport second fix for issue #11149. files: configure | 7 ++++--- configure.in | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -5413,9 +5413,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) diff --git a/configure.in b/configure.in --- a/configure.in +++ b/configure.in @@ -934,9 +934,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge second fix for issue #11149.
by stefan.krah Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/e48df59af394 changeset: 73891:e48df59af394 parent: 73887:c7638be1e430 parent: 73890:7efad6256e58 user: Stefan Krah <skrah(a)bytereef.org> date: Thu Dec 08 22:22:58 2011 +0100 summary: Merge second fix for issue #11149. files: configure | 7 ++++--- configure.in | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -5450,9 +5450,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) diff --git a/configure.in b/configure.in --- a/configure.in +++ b/configure.in @@ -907,9 +907,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #11149: Also enable -fwrapv if $CC is a full path
by stefan.krah Dec. 9, 2011

Dec. 9, 2011

http://hg.python.org/cpython/rev/7efad6256e58 changeset: 73890:7efad6256e58 branch: 3.2 parent: 73886:c143e66e5efe user: Stefan Krah <skrah(a)bytereef.org> date: Thu Dec 08 22:20:31 2011 +0100 summary: Issue #11149: Also enable -fwrapv if $CC is a full path or has a trailing version number. files: configure | 7 ++++--- configure.in | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -5498,9 +5498,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) diff --git a/configure.in b/configure.in --- a/configure.in +++ b/configure.in @@ -928,9 +928,10 @@ fi # Clang also needs -fwrapv - if test "$CC" = "clang" ; then - WRAP="-fwrapv" - fi + case $CC in + *clang*) WRAP="-fwrapv" + ;; + esac case $ac_cv_prog_cc_g in yes) -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (c7638be1e430): sum=0
by solipsis＠pitrou.net Dec. 8, 2011

Dec. 8, 2011

results for c7638be1e430 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog2HGBjZ', '-x']

1 0

cpython (2.7): Issue #11886: workaround an OS bug (time zone data) in test_time
by victor.stinner Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/2bca2cee79a1 changeset: 73888:2bca2cee79a1 branch: 2.7 parent: 73881:a8deeb549e1a user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 08 00:32:51 2011 +0100 summary: Issue #11886: workaround an OS bug (time zone data) in test_time Australian Eastern Standard Time (UTC+10) is called "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some operating systems (e.g. FreeBSD), which is wrong. See for example this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 files: Lib/test/test_time.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -184,7 +184,12 @@ environ['TZ'] = victoria time.tzset() self.assertNotEqual(time.gmtime(xmas2002), time.localtime(xmas2002)) - self.assertTrue(time.tzname[0] == 'AEST', str(time.tzname[0])) + + # Issue #11886: Australian Eastern Standard Time (UTC+10) is called + # "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some + # operating systems (e.g. FreeBSD), which is wrong. See for example + # this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 + self.assertIn(time.tzname[0], ('AEST' 'EST'), time.tzname[0]) self.assertTrue(time.tzname[1] == 'AEDT', str(time.tzname[1])) self.assertEqual(len(time.tzname), 2) self.assertEqual(time.daylight, 1) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): (Merge 3.2) Issue #11886: workaround an OS bug (time zone data) in test_time
by victor.stinner Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/c7638be1e430 changeset: 73887:c7638be1e430 parent: 73885:50c7ac2fe13c parent: 73886:c143e66e5efe user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 08 00:33:14 2011 +0100 summary: (Merge 3.2) Issue #11886: workaround an OS bug (time zone data) in test_time Australian Eastern Standard Time (UTC+10) is called "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some operating systems (e.g. FreeBSD), which is wrong. See for example this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 files: Lib/test/test_time.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -250,7 +250,12 @@ environ['TZ'] = victoria time.tzset() self.assertNotEqual(time.gmtime(xmas2002), time.localtime(xmas2002)) - self.assertTrue(time.tzname[0] == 'AEST', str(time.tzname[0])) + + # Issue #11886: Australian Eastern Standard Time (UTC+10) is called + # "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some + # operating systems (e.g. FreeBSD), which is wrong. See for example + # this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 + self.assertIn(time.tzname[0], ('AEST' 'EST'), time.tzname[0]) self.assertTrue(time.tzname[1] == 'AEDT', str(time.tzname[1])) self.assertEqual(len(time.tzname), 2) self.assertEqual(time.daylight, 1) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #11886: workaround an OS bug (time zone data) in test_time
by victor.stinner Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/c143e66e5efe changeset: 73886:c143e66e5efe branch: 3.2 parent: 73882:17ceebc61b65 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 08 00:32:51 2011 +0100 summary: Issue #11886: workaround an OS bug (time zone data) in test_time Australian Eastern Standard Time (UTC+10) is called "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some operating systems (e.g. FreeBSD), which is wrong. See for example this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 files: Lib/test/test_time.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -206,7 +206,12 @@ environ['TZ'] = victoria time.tzset() self.assertNotEqual(time.gmtime(xmas2002), time.localtime(xmas2002)) - self.assertTrue(time.tzname[0] == 'AEST', str(time.tzname[0])) + + # Issue #11886: Australian Eastern Standard Time (UTC+10) is called + # "EST" (as Eastern Standard Time, UTC-5) instead of "AEST" on some + # operating systems (e.g. FreeBSD), which is wrong. See for example + # this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=93810 + self.assertIn(time.tzname[0], ('AEST' 'EST'), time.tzname[0]) self.assertTrue(time.tzname[1] == 'AEDT', str(time.tzname[1])) self.assertEqual(len(time.tzname), 2) self.assertEqual(time.daylight, 1) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: libpython.py: defer call to gdb.lookup_type('PyUnicodeObject')
by victor.stinner Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/50c7ac2fe13c changeset: 73885:50c7ac2fe13c user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Dec 08 00:08:22 2011 +0100 summary: libpython.py: defer call to gdb.lookup_type('PyUnicodeObject') The lookup fails at startup if Python is linked to a shared library. files: Tools/gdb/libpython.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -53,7 +53,8 @@ _type_unsigned_short_ptr = gdb.lookup_type('unsigned short').pointer() _type_unsigned_int_ptr = gdb.lookup_type('unsigned int').pointer() -_is_pep393 = 'data' in [f.name for f in gdb.lookup_type('PyUnicodeObject').target().fields()] +# value computed later, see PyUnicodeObjectPtr.proxy() +_is_pep393 = None SIZEOF_VOID_P = _type_void_ptr.sizeof @@ -1123,6 +1124,10 @@ return _type_Py_UNICODE.sizeof def proxyval(self, visited): + global _is_pep393 + if _is_pep393 is None: + fields = gdb.lookup_type('PyUnicodeObject').target().fields() + _is_pep393 = 'data' in [f.name for f in fields] if _is_pep393: # Python 3.3 and newer may_have_surrogates = False -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Followup to a541bda2f5e2: Add a short comment.
by charles-francois.natali Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/174fbbed8747 changeset: 73884:174fbbed8747 user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 07 23:17:58 2011 +0100 summary: Followup to a541bda2f5e2: Add a short comment. files: Python/import.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1946,6 +1946,7 @@ if (_Py_stat(filename, &statbuf) != 0 || S_ISDIR(statbuf.st_mode)) { + /* it doesn't exist, or it's a directory */ Py_DECREF(filename); continue; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): #13531: add a test for defaultdict with a non-callable arg. Patch by Mike
by ezio.melotti Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/17ceebc61b65 changeset: 73882:17ceebc61b65 branch: 3.2 parent: 73877:3822c8087d70 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Thu Dec 08 00:02:00 2011 +0200 summary: #13531: add a test for defaultdict with a non-callable arg. Patch by Mike Cheng. files: Lib/test/test_defaultdict.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_defaultdict.py b/Lib/test/test_defaultdict.py --- a/Lib/test/test_defaultdict.py +++ b/Lib/test/test_defaultdict.py @@ -172,6 +172,9 @@ finally: os.remove(tfn) + def test_callable_arg(self): + self.assertRaises(TypeError, defaultdict, {}) + def test_pickleing(self): d = defaultdict(int) d[1] -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): #13531: merge with 3.2.
by ezio.melotti Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/4180308547d9 changeset: 73883:4180308547d9 parent: 73879:a541bda2f5e2 parent: 73882:17ceebc61b65 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Thu Dec 08 00:03:59 2011 +0200 summary: #13531: merge with 3.2. files: Lib/test/test_defaultdict.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_defaultdict.py b/Lib/test/test_defaultdict.py --- a/Lib/test/test_defaultdict.py +++ b/Lib/test/test_defaultdict.py @@ -172,6 +172,9 @@ finally: os.remove(tfn) + def test_callable_arg(self): + self.assertRaises(TypeError, defaultdict, {}) + def test_pickleing(self): d = defaultdict(int) d[1] -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): #13531: add a test for defaultdict with a non-callable arg. Patch by Mike
by ezio.melotti Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/a8deeb549e1a changeset: 73881:a8deeb549e1a branch: 2.7 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Thu Dec 08 00:00:49 2011 +0200 summary: #13531: add a test for defaultdict with a non-callable arg. Patch by Mike Cheng. files: Lib/test/test_defaultdict.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_defaultdict.py b/Lib/test/test_defaultdict.py --- a/Lib/test/test_defaultdict.py +++ b/Lib/test/test_defaultdict.py @@ -171,6 +171,8 @@ finally: os.remove(tfn) + def test_callable_arg(self): + self.assertRaises(TypeError, defaultdict, {}) def test_main(): test_support.run_unittest(TestDefaultDict) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13546: Fixed an overflow issue that could crash the intepreter when
by amaury.forgeotdarc Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/57de1ad15c54 changeset: 73880:57de1ad15c54 branch: 2.7 parent: 73872:5910c385fab6 user: Amaury Forgeot d'Arc <amauryfa(a)gmail.com> date: Wed Dec 07 21:46:48 2011 +0100 summary: Issue #13546: Fixed an overflow issue that could crash the intepreter when calling sys.setrecursionlimit((1<<31)-1). 2.7 only. files: Lib/test/test_sys.py | 12 ++++++++++++ Misc/NEWS | 3 +++ Python/errors.c | 6 ++++-- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -224,6 +224,18 @@ self.assertEqual(sys.getrecursionlimit(), 10000) sys.setrecursionlimit(oldlimit) + self.assertRaises(OverflowError, sys.setrecursionlimit, 1 << 31) + try: + sys.setrecursionlimit((1 << 31) - 5) + try: + # issue13546: isinstance(e, ValueError) used to fail + # when the recursion limit is close to 1<<31 + raise ValueError() + except ValueError, e: + pass + finally: + sys.setrecursionlimit(oldlimit) + def test_getwindowsversion(self): # Raise SkipTest if sys doesn't have getwindowsversion attribute test.test_support.get_attribute(sys, "getwindowsversion") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,9 @@ Core and Builtins ----------------- +- Issue #13546: Fixed an overflow issue that could crash the intepreter when + calling sys.setrecursionlimit((1<<31)-1). + - Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder already accepts them). diff --git a/Python/errors.c b/Python/errors.c --- a/Python/errors.c +++ b/Python/errors.c @@ -111,9 +111,11 @@ PyErr_Fetch(&exception, &value, &tb); /* Temporarily bump the recursion limit, so that in the most common case PyObject_IsSubclass will not raise a recursion - error we have to ignore anyway. */ + error we have to ignore anyway. Don't do it when the limit + is already insanely high, to avoid overflow */ reclimit = Py_GetRecursionLimit(); - Py_SetRecursionLimit(reclimit + 5); + if (reclimit < (1 << 30)) + Py_SetRecursionLimit(reclimit + 5); res = PyObject_IsSubclass(err, exc); Py_SetRecursionLimit(reclimit); /* This function must not fail, so print the error here */ -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #11051: Reduce the number of syscalls per import.
by charles-francois.natali Dec. 8, 2011

Dec. 8, 2011

http://hg.python.org/cpython/rev/a541bda2f5e2 changeset: 73879:a541bda2f5e2 user: Charles-François Natali <neologix(a)free.fr> date: Wed Dec 07 19:16:01 2011 +0100 summary: Issue #11051: Reduce the number of syscalls per import. files: Python/import.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1944,8 +1944,7 @@ if (Py_VerboseFlag > 1) PySys_FormatStderr("# trying %R\n", filename); - if (_Py_stat(filename, &statbuf) == 0 && /* it exists */ - S_ISDIR(statbuf.st_mode)) /* it's a directory */ + if (_Py_stat(filename, &statbuf) != 0 || S_ISDIR(statbuf.st_mode)) { Py_DECREF(filename); continue; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..".
by ned.deily Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/e49220f4c31f changeset: 73878:e49220f4c31f parent: 73876:daaacc0ec584 parent: 73877:3822c8087d70 user: Ned Deily <nad(a)acm.org> date: Wed Dec 07 01:12:50 2011 -0800 summary: Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". (Patch by Tal Einat) files: Lib/idlelib/ColorDelegator.py | 8 ++++---- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Lib/idlelib/ColorDelegator.py b/Lib/idlelib/ColorDelegator.py --- a/Lib/idlelib/ColorDelegator.py +++ b/Lib/idlelib/ColorDelegator.py @@ -20,10 +20,10 @@ # 1st 'file' colorized normal, 2nd as builtin, 3rd as string builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b" comment = any("COMMENT", [r"#[^\n]*"]) - sqstring = r"(\b[rRuU])?'[^'\\\n]*(\\.[^'\\\n]*)*'?" - dqstring = r'(\b[rRuU])?"[^"\\\n]*(\\.[^"\\\n]*)*"?' - sq3string = r"(\b[rRuU])?'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" - dq3string = r'(\b[rRuU])?"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' + sqstring = r"(\b[rRbB])?'[^'\\\n]*(\\.[^'\\\n]*)*'?" + dqstring = r'(\b[rRbB])?"[^"\\\n]*(\\.[^"\\\n]*)*"?' + sq3string = r"(\b[rRbB])?'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" + dq3string = r'(\b[rRbB])?"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' string = any("STRING", [sq3string, dq3string, sqstring, dqstring]) return kw + "|" + builtin + "|" + comment + "|" + string +\ "|" + any("SYNC", [r"\n"]) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -275,6 +275,7 @@ John Edmonds Grant Edwards John Ehresman +Tal Einat Eric Eisner Andrew Eland Julien Élie diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,9 @@ Library ------- +- Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". + Patch by Tal Einat. + - Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch by Jon Kuhn. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..".
by ned.deily Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/3822c8087d70 changeset: 73877:3822c8087d70 branch: 3.2 parent: 73873:b6b4d74b8d42 user: Ned Deily <nad(a)acm.org> date: Wed Dec 07 01:08:35 2011 -0800 summary: Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". (Patch by Tal Einat) files: Lib/idlelib/ColorDelegator.py | 8 ++++---- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Lib/idlelib/ColorDelegator.py b/Lib/idlelib/ColorDelegator.py --- a/Lib/idlelib/ColorDelegator.py +++ b/Lib/idlelib/ColorDelegator.py @@ -20,10 +20,10 @@ # 1st 'file' colorized normal, 2nd as builtin, 3rd as string builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b" comment = any("COMMENT", [r"#[^\n]*"]) - sqstring = r"(\b[rRuU])?'[^'\\\n]*(\\.[^'\\\n]*)*'?" - dqstring = r'(\b[rRuU])?"[^"\\\n]*(\\.[^"\\\n]*)*"?' - sq3string = r"(\b[rRuU])?'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" - dq3string = r'(\b[rRuU])?"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' + sqstring = r"(\b[rRbB])?'[^'\\\n]*(\\.[^'\\\n]*)*'?" + dqstring = r'(\b[rRbB])?"[^"\\\n]*(\\.[^"\\\n]*)*"?' + sq3string = r"(\b[rRbB])?'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" + dq3string = r'(\b[rRbB])?"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' string = any("STRING", [sq3string, dq3string, sqstring, dqstring]) return kw + "|" + builtin + "|" + comment + "|" + string +\ "|" + any("SYNC", [r"\n"]) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -256,6 +256,7 @@ Rodolpho Eckhardt Grant Edwards John Ehresman +Tal Einat Eric Eisner Andrew Eland Julien Élie diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -90,6 +90,9 @@ Library ------- +- Issue #8641: Update IDLE 3 syntax coloring to recognize b".." and not u"..". + Patch by Tal Einat. + - tarfile.py: Correctly detect bzip2 compressed streams with blocksizes other than 900k. -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (daaacc0ec584): sum=0
by solipsis＠pitrou.net Dec. 7, 2011

Dec. 7, 2011

results for daaacc0ec584 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloglUfbkU', '-x']

1 0

cpython: Fix dangling whitespace
by antoine.pitrou Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/daaacc0ec584 changeset: 73876:daaacc0ec584 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Tue Dec 06 22:34:36 2011 +0100 summary: Fix dangling whitespace files: Lib/http/client.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -599,7 +599,7 @@ break except ValueError: raise IncompleteRead(bytes(b[0:total_bytes])) - + if len(mvb) < chunk_left: n = self._safe_readinto(mvb) self.chunk_left = chunk_left - n -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13464: Add a readinto() method to http.client.HTTPResponse.
by antoine.pitrou Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/806cfe39f729 changeset: 73875:806cfe39f729 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Tue Dec 06 22:33:57 2011 +0100 summary: Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch by Jon Kuhn. files: Doc/library/http.client.rst | 6 + Lib/http/client.py | 167 +++++++++++++++++------ Lib/test/test_httplib.py | 86 ++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 + 5 files changed, 215 insertions(+), 48 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -502,6 +502,12 @@ Reads and returns the response body, or up to the next *amt* bytes. +.. method:: HTTPResponse.readinto(b) + + Reads up to the next len(b) bytes of the response body into the buffer *b*. + Returns the number of bytes read. + + .. versionadded:: 3.3 .. method:: HTTPResponse.getheader(name, default=None) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -485,11 +485,17 @@ self.close() return b"" - if self.chunked: - return self._read_chunked(amt) + if amt is not None: + # Amount is given, so call base class version + # (which is implemented in terms of self.readinto) + return super(HTTPResponse, self).read(amt) + else: + # Amount is not given (unbounded read) so we must check self.length + # and self.chunked - if amt is None: - # unbounded read + if self.chunked: + return self._readall_chunked() + if self.length is None: s = self.fp.read() else: @@ -498,61 +504,49 @@ self.close() # we read everything return s + def readinto(self, b): + if self.fp is None: + return 0 + + if self._method == "HEAD": + self.close() + return 0 + + if self.chunked: + return self._readinto_chunked(b) + if self.length is not None: - if amt > self.length: + if len(b) > self.length: # clip the read to the "end of response" - amt = self.length + b = memoryview(b)[0:self.length] # we do not use _safe_read() here because this may be a .will_close # connection, and the user is reading more bytes than will be provided # (for example, reading in 1k chunks) - s = self.fp.read(amt) + n = self.fp.readinto(b) if self.length is not None: - self.length -= len(s) + self.length -= n if not self.length: self.close() - return s + return n - def _read_chunked(self, amt): - assert self.chunked != _UNKNOWN - chunk_left = self.chunk_left - value = [] - while True: - if chunk_left is None: - line = self.fp.readline(_MAXLINE + 1) - if len(line) > _MAXLINE: - raise LineTooLong("chunk size") - i = line.find(b";") - if i >= 0: - line = line[:i] # strip chunk-extensions - try: - chunk_left = int(line, 16) - except ValueError: - # close the connection as protocol synchronisation is - # probably lost - self.close() - raise IncompleteRead(b''.join(value)) - if chunk_left == 0: - break - if amt is None: - value.append(self._safe_read(chunk_left)) - elif amt < chunk_left: - value.append(self._safe_read(amt)) - self.chunk_left = chunk_left - amt - return b''.join(value) - elif amt == chunk_left: - value.append(self._safe_read(amt)) - self._safe_read(2) # toss the CRLF at the end of the chunk - self.chunk_left = None - return b''.join(value) - else: - value.append(self._safe_read(chunk_left)) - amt -= chunk_left + def _read_next_chunk_size(self): + # Read the next chunk size from the file + line = self.fp.readline(_MAXLINE + 1) + if len(line) > _MAXLINE: + raise LineTooLong("chunk size") + i = line.find(b";") + if i >= 0: + line = line[:i] # strip chunk-extensions + try: + return int(line, 16) + except ValueError: + # close the connection as protocol synchronisation is + # probably lost + self.close() + raise - # we read the whole chunk, get another - self._safe_read(2) # toss the CRLF at the end of the chunk - chunk_left = None - + def _read_and_discard_trailer(self): # read and discard trailer up to the CRLF terminator ### note: we shouldn't have any trailers! while True: @@ -566,11 +560,72 @@ if line == b"\r\n": break + def _readall_chunked(self): + assert self.chunked != _UNKNOWN + chunk_left = self.chunk_left + value = [] + while True: + if chunk_left is None: + try: + chunk_left = self._read_next_chunk_size() + if chunk_left == 0: + break + except ValueError: + raise IncompleteRead(b''.join(value)) + value.append(self._safe_read(chunk_left)) + + # we read the whole chunk, get another + self._safe_read(2) # toss the CRLF at the end of the chunk + chunk_left = None + + self._read_and_discard_trailer() + # we read everything; close the "file" self.close() return b''.join(value) + def _readinto_chunked(self, b): + assert self.chunked != _UNKNOWN + chunk_left = self.chunk_left + + total_bytes = 0 + mvb = memoryview(b) + while True: + if chunk_left is None: + try: + chunk_left = self._read_next_chunk_size() + if chunk_left == 0: + break + except ValueError: + raise IncompleteRead(bytes(b[0:total_bytes])) + + if len(mvb) < chunk_left: + n = self._safe_readinto(mvb) + self.chunk_left = chunk_left - n + return n + elif len(mvb) == chunk_left: + n = self._safe_readinto(mvb) + self._safe_read(2) # toss the CRLF at the end of the chunk + self.chunk_left = None + return n + else: + temp_mvb = mvb[0:chunk_left] + n = self._safe_readinto(temp_mvb) + mvb = mvb[n:] + total_bytes += n + + # we read the whole chunk, get another + self._safe_read(2) # toss the CRLF at the end of the chunk + chunk_left = None + + self._read_and_discard_trailer() + + # we read everything; close the "file" + self.close() + + return total_bytes + def _safe_read(self, amt): """Read the number of bytes requested, compensating for partial reads. @@ -594,6 +649,22 @@ amt -= len(chunk) return b"".join(s) + def _safe_readinto(self, b): + """Same as _safe_read, but for reading into a buffer.""" + total_bytes = 0 + mvb = memoryview(b) + while total_bytes < len(b): + if MAXAMOUNT < len(mvb): + temp_mvb = mvb[0:MAXAMOUNT] + n = self.fp.readinto(temp_mvb) + else: + n = self.fp.readinto(mvb) + if not n: + raise IncompleteRead(bytes(mvb[0:total_bytes]), len(b)) + mvb = mvb[n:] + total_bytes += n + return total_bytes + def fileno(self): return self.fp.fileno() diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -158,6 +158,23 @@ self.assertEqual(resp.read(2), b'xt') self.assertTrue(resp.isclosed()) + def test_partial_readintos(self): + # if we have a lenght, the system knows when to close itself + # same behaviour than when we read the whole thing with read() + body = "HTTP/1.1 200 Ok\r\nContent-Length: 4\r\n\r\nText" + sock = FakeSocket(body) + resp = client.HTTPResponse(sock) + resp.begin() + b = bytearray(2) + n = resp.readinto(b) + self.assertEqual(n, 2) + self.assertEqual(bytes(b), b'Te') + self.assertFalse(resp.isclosed()) + n = resp.readinto(b) + self.assertEqual(n, 2) + self.assertEqual(bytes(b), b'xt') + self.assertTrue(resp.isclosed()) + def test_host_port(self): # Check invalid host_port @@ -206,6 +223,21 @@ if resp.read(): self.fail("Did not expect response from HEAD request") + def test_readinto_head(self): + # Test that the library doesn't attempt to read any data + # from a HEAD request. (Tickles SF bug #622042.) + sock = FakeSocket( + 'HTTP/1.1 200 OK\r\n' + 'Content-Length: 14432\r\n' + '\r\n', + NoEOFStringIO) + resp = client.HTTPResponse(sock, method="HEAD") + resp.begin() + b = bytearray(5) + if resp.readinto(b) != 0: + self.fail("Did not expect response from HEAD request") + self.assertEqual(bytes(b), b'\x00'*5) + def test_send_file(self): expected = (b'GET /foo HTTP/1.1\r\nHost: example.com\r\n' b'Accept-Encoding: identity\r\nContent-Length:') @@ -285,6 +317,40 @@ finally: resp.close() + def test_readinto_chunked(self): + chunked_start = ( + 'HTTP/1.1 200 OK\r\n' + 'Transfer-Encoding: chunked\r\n\r\n' + 'a\r\n' + 'hello worl\r\n' + '1\r\n' + 'd\r\n' + ) + sock = FakeSocket(chunked_start + '0\r\n') + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + b = bytearray(16) + n = resp.readinto(b) + self.assertEqual(b[:11], b'hello world') + self.assertEqual(n, 11) + resp.close() + + for x in ('', 'foo\r\n'): + sock = FakeSocket(chunked_start + x) + resp = client.HTTPResponse(sock, method="GET") + resp.begin() + try: + b = bytearray(16) + n = resp.readinto(b) + except client.IncompleteRead as i: + self.assertEqual(i.partial, b'hello world') + self.assertEqual(repr(i),'IncompleteRead(11 bytes read)') + self.assertEqual(str(i),'IncompleteRead(11 bytes read)') + else: + self.fail('IncompleteRead expected') + finally: + resp.close() + def test_chunked_head(self): chunked_start = ( 'HTTP/1.1 200 OK\r\n' @@ -302,6 +368,26 @@ self.assertEqual(resp.reason, 'OK') self.assertTrue(resp.isclosed()) + def test_readinto_chunked_head(self): + chunked_start = ( + 'HTTP/1.1 200 OK\r\n' + 'Transfer-Encoding: chunked\r\n\r\n' + 'a\r\n' + 'hello world\r\n' + '1\r\n' + 'd\r\n' + ) + sock = FakeSocket(chunked_start + '0\r\n') + resp = client.HTTPResponse(sock, method="HEAD") + resp.begin() + b = bytearray(5) + n = resp.readinto(b) + self.assertEqual(n, 0) + self.assertEqual(bytes(b), b'\x00'*5) + self.assertEqual(resp.status, 200) + self.assertEqual(resp.reason, 'OK') + self.assertTrue(resp.isclosed()) + def test_negative_content_length(self): sock = FakeSocket( 'HTTP/1.1 200 OK\r\nContent-Length: -1\r\n\r\nHello\r\n') diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -547,6 +547,7 @@ Andrej Krpic Ivan Krstić Andrew Kuchling +Jon Kuhn Vladimir Kushnir Ross Lagerwall Cameron Laird diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,9 @@ Library ------- +- Issue #13464: Add a readinto() method to http.client.HTTPResponse. Patch + by Jon Kuhn. + - tarfile.py: Correctly detect bzip2 compressed streams with blocksizes other than 900k. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Close #13500: Hitting EOF gets cmd.py into a infinite EOF on return loop
by jesus.cea Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/b6b4d74b8d42 changeset: 73873:b6b4d74b8d42 branch: 3.2 parent: 73868:80876df8adce user: Jesus Cea <jcea(a)jcea.es> date: Tue Dec 06 20:46:57 2011 +0100 summary: Close #13500: Hitting EOF gets cmd.py into a infinite EOF on return loop files: Lib/cmd.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/cmd.py b/Lib/cmd.py --- a/Lib/cmd.py +++ b/Lib/cmd.py @@ -205,6 +205,8 @@ if cmd is None: return self.default(line) self.lastcmd = line + if line == 'EOF' : + self.lastcmd = '' if cmd == '': return self.default(line) else: -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): MERGE: Close #13500: Hitting EOF gets cmd.py into a infinite EOF on return loop
by jesus.cea Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/70ba352f9586 changeset: 73874:70ba352f9586 parent: 73871:56b7b1ecc240 parent: 73873:b6b4d74b8d42 user: Jesus Cea <jcea(a)jcea.es> date: Tue Dec 06 20:47:38 2011 +0100 summary: MERGE: Close #13500: Hitting EOF gets cmd.py into a infinite EOF on return loop files: Lib/cmd.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/cmd.py b/Lib/cmd.py --- a/Lib/cmd.py +++ b/Lib/cmd.py @@ -205,6 +205,8 @@ if cmd is None: return self.default(line) self.lastcmd = line + if line == 'EOF' : + self.lastcmd = '' if cmd == '': return self.default(line) else: -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Close #13500: Hitting EOF gets cmd.py into a infinite EOF on return loop
by jesus.cea Dec. 7, 2011

Dec. 7, 2011

http://hg.python.org/cpython/rev/5910c385fab6 changeset: 73872:5910c385fab6 branch: 2.7 parent: 73870:6d1a91e9f506 user: Jesus Cea <jcea(a)jcea.es> date: Tue Dec 06 20:46:04 2011 +0100 summary: Close #13500: Hitting EOF gets cmd.py into a infinite EOF on return loop files: Lib/cmd.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/cmd.py b/Lib/cmd.py --- a/Lib/cmd.py +++ b/Lib/cmd.py @@ -209,6 +209,8 @@ if cmd is None: return self.default(line) self.lastcmd = line + if line == 'EOF' : + self.lastcmd = '' if cmd == '': return self.default(line) else: -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Remove no longer needed work-around for bz2 file object support.
by lars.gustaebel Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/56b7b1ecc240 changeset: 73871:56b7b1ecc240 parent: 73869:9149aefff883 user: Lars Gustäbel <lars(a)gustaebel.de> date: Tue Dec 06 13:44:10 2011 +0100 summary: Remove no longer needed work-around for bz2 file object support. files: Lib/tarfile.py | 66 +--------------------------- Lib/test/test_tarfile.py | 3 + 2 files changed, 5 insertions(+), 64 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -632,66 +632,6 @@ self.fileobj.close() # class StreamProxy -class _BZ2Proxy(object): - """Small proxy class that enables external file object - support for "r:bz2" and "w:bz2" modes. This is actually - a workaround for a limitation in bz2 module's BZ2File - class which (unlike gzip.GzipFile) has no support for - a file object argument. - """ - - blocksize = 16 * 1024 - - def __init__(self, fileobj, mode): - self.fileobj = fileobj - self.mode = mode - self.name = getattr(self.fileobj, "name", None) - self.init() - - def init(self): - import bz2 - self.pos = 0 - if self.mode == "r": - self.bz2obj = bz2.BZ2Decompressor() - self.fileobj.seek(0) - self.buf = b"" - else: - self.bz2obj = bz2.BZ2Compressor() - - def read(self, size): - x = len(self.buf) - while x < size: - raw = self.fileobj.read(self.blocksize) - if not raw: - break - data = self.bz2obj.decompress(raw) - self.buf += data - x += len(data) - - buf = self.buf[:size] - self.buf = self.buf[size:] - self.pos += len(buf) - return buf - - def seek(self, pos): - if pos < self.pos: - self.init() - self.read(pos - self.pos) - - def tell(self): - return self.pos - - def write(self, data): - self.pos += len(data) - raw = self.bz2obj.compress(data) - self.fileobj.write(raw) - - def close(self): - if self.mode == "w": - raw = self.bz2obj.flush() - self.fileobj.write(raw) -# class _BZ2Proxy - #------------------------ # Extraction file object #------------------------ @@ -1829,10 +1769,8 @@ except ImportError: raise CompressionError("bz2 module is not available") - if fileobj is not None: - fileobj = _BZ2Proxy(fileobj, mode) - else: - fileobj = bz2.BZ2File(name, mode, compresslevel=compresslevel) + fileobj = bz2.BZ2File(filename=name if fileobj is None else None, + mode=mode, fileobj=fileobj, compresslevel=compresslevel) try: t = cls.taropen(name, mode, fileobj, **kwargs) diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -222,6 +222,9 @@ class MiscReadTest(CommonReadTest): def test_no_name_argument(self): + if self.mode.endswith("bz2"): + # BZ2File has no name attribute. + return with open(self.tarname, "rb") as fobj: tar = tarfile.open(fileobj=fobj, mode=self.mode) self.assertEqual(tar.name, os.path.abspath(fobj.name)) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Correctly detect bzip2 compressed streams with blocksizes other than 900k.
by lars.gustaebel Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/6d1a91e9f506 changeset: 73870:6d1a91e9f506 branch: 2.7 parent: 73853:478b4e9551fa user: Lars Gustäbel <lars(a)gustaebel.de> date: Tue Dec 06 13:07:09 2011 +0100 summary: Correctly detect bzip2 compressed streams with blocksizes other than 900k. files: Lib/tarfile.py | 2 +- Lib/test/test_tarfile.py | 17 +++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 21 insertions(+), 1 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -627,7 +627,7 @@ def getcomptype(self): if self.buf.startswith("\037\213\010"): return "gz" - if self.buf.startswith("BZh91"): + if self.buf[0:3] == "BZh" and self.buf[4:10] == "1AY&SY": return "bz2" return "tar" diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -440,6 +440,23 @@ def test_detect_fileobj(self): self._test_modes(self._testfunc_fileobj) + def test_detect_stream_bz2(self): + # Originally, tarfile's stream detection looked for the string + # "BZh91" at the start of the file. This is incorrect because + # the '9' represents the blocksize (900kB). If the file was + # compressed using another blocksize autodetection fails. + if not bz2: + return + + with open(tarname, "rb") as fobj: + data = fobj.read() + + # Compress with blocksize 100kB, the file starts with "BZh11". + with bz2.BZ2File(tmpname, "wb", compresslevel=1) as fobj: + fobj.write(data) + + self._testfunc_file(tmpname, "r|*") + class MemberReadTest(ReadTest): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -79,6 +79,9 @@ Library ------- +- tarfile.py: Correctly detect bzip2 compressed streams with blocksizes + other than 900k. + - Issue #13439: Fix many errors in turtle docstrings. - Issue #12856: Ensure child processes do not inherit the parent's random -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge with 3.2: Correctly detect bzip2 compressed streams with blocksizes other
by lars.gustaebel Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/9149aefff883 changeset: 73869:9149aefff883 parent: 73867:6b6c79eba944 parent: 73868:80876df8adce user: Lars Gustäbel <lars(a)gustaebel.de> date: Tue Dec 06 13:00:58 2011 +0100 summary: Merge with 3.2: Correctly detect bzip2 compressed streams with blocksizes other than 900k. files: Lib/tarfile.py | 2 +- Lib/test/test_tarfile.py | 22 ++++++++++++++++++---- Misc/NEWS | 3 +++ 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -624,7 +624,7 @@ def getcomptype(self): if self.buf.startswith(b"\037\213\010"): return "gz" - if self.buf.startswith(b"BZh91"): + if self.buf[0:3] == b"BZh" and self.buf[4:10] == b"1AY&SY": return "bz2" return "tar" diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -529,6 +529,23 @@ def test_detect_fileobj(self): self._test_modes(self._testfunc_fileobj) + def test_detect_stream_bz2(self): + # Originally, tarfile's stream detection looked for the string + # "BZh91" at the start of the file. This is incorrect because + # the '9' represents the blocksize (900kB). If the file was + # compressed using another blocksize autodetection fails. + if not bz2: + return + + with open(tarname, "rb") as fobj: + data = fobj.read() + + # Compress with blocksize 100kB, the file starts with "BZh11". + with bz2.BZ2File(tmpname, "wb", compresslevel=1) as fobj: + fobj.write(data) + + self._testfunc_file(tmpname, "r|*") + class MemberReadTest(ReadTest): @@ -1818,11 +1835,8 @@ if bz2: # Create testtar.tar.bz2 and add bz2-specific tests. support.unlink(bz2name) - tar = bz2.BZ2File(bz2name, "wb") - try: + with bz2.BZ2File(bz2name, "wb") as tar: tar.write(data) - finally: - tar.close() tests += [ Bz2MiscReadTest, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -406,6 +406,9 @@ Library ------- +- tarfile.py: Correctly detect bzip2 compressed streams with blocksizes + other than 900k. + - Issue #13439: Fix many errors in turtle docstrings. - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Correctly detect bzip2 compressed streams with blocksizes other than 900k.
by lars.gustaebel Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/80876df8adce changeset: 73868:80876df8adce branch: 3.2 parent: 73866:4579cd952156 user: Lars Gustäbel <lars(a)gustaebel.de> date: Tue Dec 06 12:56:38 2011 +0100 summary: Correctly detect bzip2 compressed streams with blocksizes other than 900k. files: Lib/tarfile.py | 2 +- Lib/test/test_tarfile.py | 22 ++++++++++++++++++---- Misc/NEWS | 3 +++ 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -627,7 +627,7 @@ def getcomptype(self): if self.buf.startswith(b"\037\213\010"): return "gz" - if self.buf.startswith(b"BZh91"): + if self.buf[0:3] == b"BZh" and self.buf[4:10] == b"1AY&SY": return "bz2" return "tar" diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -529,6 +529,23 @@ def test_detect_fileobj(self): self._test_modes(self._testfunc_fileobj) + def test_detect_stream_bz2(self): + # Originally, tarfile's stream detection looked for the string + # "BZh91" at the start of the file. This is incorrect because + # the '9' represents the blocksize (900kB). If the file was + # compressed using another blocksize autodetection fails. + if not bz2: + return + + with open(tarname, "rb") as fobj: + data = fobj.read() + + # Compress with blocksize 100kB, the file starts with "BZh11". + with bz2.BZ2File(tmpname, "wb", compresslevel=1) as fobj: + fobj.write(data) + + self._testfunc_file(tmpname, "r|*") + class MemberReadTest(ReadTest): @@ -1818,11 +1835,8 @@ if bz2: # Create testtar.tar.bz2 and add bz2-specific tests. support.unlink(bz2name) - tar = bz2.BZ2File(bz2name, "wb") - try: + with bz2.BZ2File(bz2name, "wb") as tar: tar.write(data) - finally: - tar.close() tests += [ Bz2MiscReadTest, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -90,6 +90,9 @@ Library ------- +- tarfile.py: Correctly detect bzip2 compressed streams with blocksizes + other than 900k. + - Issue #13439: Fix many errors in turtle docstrings. - Issue #13487: Make inspect.getmodule robust against changes done to -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (6b6c79eba944): sum=0
by solipsis＠pitrou.net Dec. 6, 2011

Dec. 6, 2011

results for 6b6c79eba944 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflognNz12r', '-x']

1 0

cpython (merge 3.2 -> default): - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix
by barry.warsaw Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/6b6c79eba944 changeset: 73867:6b6c79eba944 parent: 73865:e2959a6a1440 parent: 73866:4579cd952156 user: Barry Warsaw <barry(a)python.org> date: Mon Dec 05 16:50:41 2011 -0500 summary: - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix given by Campbell Barton). files: Include/pyatomic.h | 1 + Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Include/pyatomic.h b/Include/pyatomic.h --- a/Include/pyatomic.h +++ b/Include/pyatomic.h @@ -58,6 +58,7 @@ static __inline__ void _Py_ANNOTATE_MEMORY_ORDER(const volatile void *address, _Py_memory_order order) { + (void)address; /* shut up -Wunused-parameter */ switch(order) { case _Py_memory_order_release: case _Py_memory_order_acq_rel: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix + given by Campbell Barton). + - Issue #13503: Use a more efficient reduction format for bytearrays with pickle protocol >= 3. The old reduction format is kept with older protocols in order to allow unpickling under Python 2. Patch by Irmen de Jong. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix
by barry.warsaw Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/4579cd952156 changeset: 73866:4579cd952156 branch: 3.2 parent: 73863:c34188efb965 user: Barry Warsaw <barry(a)python.org> date: Mon Dec 05 16:45:02 2011 -0500 summary: - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix given by Campbell Barton). files: Include/pyatomic.h | 1 + Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Include/pyatomic.h b/Include/pyatomic.h --- a/Include/pyatomic.h +++ b/Include/pyatomic.h @@ -58,6 +58,7 @@ static __inline__ void _Py_ANNOTATE_MEMORY_ORDER(const volatile void *address, _Py_memory_order order) { + (void)address; /* shut up -Wunused-parameter */ switch(order) { case _Py_memory_order_release: case _Py_memory_order_acq_rel: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix + given by Campbell Barton). + - Issue #7111: Python can now be run without a stdin, stdout or stderr stream. It was already the case with Python 2. However, the corresponding sys module entries are now set to None (instead of an unusable file object). -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13503: Use a more efficient reduction format for bytearrays with
by antoine.pitrou Dec. 6, 2011

Dec. 6, 2011

http://hg.python.org/cpython/rev/e2959a6a1440 changeset: 73865:e2959a6a1440 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 20:40:08 2011 +0100 summary: Issue #13503: Use a more efficient reduction format for bytearrays with pickle protocol >= 3. The old reduction format is kept with older protocols in order to allow unpickling under Python 2. Patch by Irmen de Jong. files: Misc/NEWS | 4 ++ Objects/bytearrayobject.c | 52 +++++++++++++++++++++----- 2 files changed, 46 insertions(+), 10 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #13503: Use a more efficient reduction format for bytearrays with + pickle protocol >= 3. The old reduction format is kept with older protocols + in order to allow unpickling under Python 2. Patch by Irmen de Jong. + - Issue #7111: Python can now be run without a stdin, stdout or stderr stream. It was already the case with Python 2. However, the corresponding sys module entries are now set to None (instead of an unusable file object). diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -2725,20 +2725,13 @@ return NULL; } -PyDoc_STRVAR(reduce_doc, "Return state information for pickling."); static PyObject * -bytearray_reduce(PyByteArrayObject *self) +_common_reduce(PyByteArrayObject *self, int proto) { - PyObject *latin1, *dict; + PyObject *dict; _Py_IDENTIFIER(__dict__); - if (self->ob_bytes) - latin1 = PyUnicode_DecodeLatin1(self->ob_bytes, - Py_SIZE(self), NULL); - else - latin1 = PyUnicode_FromString(""); - dict = _PyObject_GetAttrId((PyObject *)self, &PyId___dict__); if (dict == NULL) { PyErr_Clear(); @@ -2746,7 +2739,45 @@ Py_INCREF(dict); } - return Py_BuildValue("(O(Ns)N)", Py_TYPE(self), latin1, "latin-1", dict); + if (proto < 3) { + /* use str based reduction for backwards compatibility with Python 2.x */ + PyObject *latin1; + if (self->ob_bytes) + latin1 = PyUnicode_DecodeLatin1(self->ob_bytes, Py_SIZE(self), NULL); + else + latin1 = PyUnicode_FromString(""); + return Py_BuildValue("(O(Ns)N)", Py_TYPE(self), latin1, "latin-1", dict); + } + else { + /* use more efficient byte based reduction */ + if (self->ob_bytes) { + return Py_BuildValue("(O(y#)N)", Py_TYPE(self), self->ob_bytes, Py_SIZE(self), dict); + } + else { + return Py_BuildValue("(O()N)", Py_TYPE(self), dict); + } + } +} + +PyDoc_STRVAR(reduce_doc, "Return state information for pickling."); + +static PyObject * +bytearray_reduce(PyByteArrayObject *self) +{ + return _common_reduce(self, 2); +} + +PyDoc_STRVAR(reduce_ex_doc, "Return state information for pickling."); + +static PyObject * +bytearray_reduce_ex(PyByteArrayObject *self, PyObject *args) +{ + int proto = 0; + + if (!PyArg_ParseTuple(args, "|i:__reduce_ex__", &proto)) + return NULL; + + return _common_reduce(self, proto); } PyDoc_STRVAR(sizeof_doc, @@ -2790,6 +2821,7 @@ bytearray_methods[] = { {"__alloc__", (PyCFunction)bytearray_alloc, METH_NOARGS, alloc_doc}, {"__reduce__", (PyCFunction)bytearray_reduce, METH_NOARGS, reduce_doc}, + {"__reduce_ex__", (PyCFunction)bytearray_reduce_ex, METH_VARARGS, reduce_ex_doc}, {"__sizeof__", (PyCFunction)bytearray_sizeof, METH_NOARGS, sizeof_doc}, {"append", (PyCFunction)bytearray_append, METH_O, append__doc__}, {"capitalize", (PyCFunction)stringlib_capitalize, METH_NOARGS, -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (f2be1e010cc8): sum=0
by solipsis＠pitrou.net Dec. 5, 2011

Dec. 5, 2011

results for f2be1e010cc8 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloglEA4bV', '-x']

1 0

cpython (merge 3.2 -> default): Merge assorted fixes from 3.2
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/f2be1e010cc8 changeset: 73864:f2be1e010cc8 parent: 73859:8701f6373d0b parent: 73863:c34188efb965 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:47:40 2011 +0100 summary: Merge assorted fixes from 3.2 files: Doc/howto/sockets.rst | 36 ++++++++++++++---------------- 1 files changed, 17 insertions(+), 19 deletions(-) diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -60,11 +60,10 @@ Roughly speaking, when you clicked on the link that brought you to this page, your browser did something like the following:: - #create an INET, STREAMing socket + # create an INET, STREAMing socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - #now connect to the web server on port 80 - # - the normal http port - s.connect(("www.mcmillan-inc.com", 80)) + # now connect to the web server on port 80 - the normal http port + s.connect(("www.python.org", 80)) When the ``connect`` completes, the socket ``s`` can be used to send in a request for the text of the page. The same socket will read the @@ -75,13 +74,11 @@ What happens in the web server is a bit more complex. First, the web server creates a "server socket":: - #create an INET, STREAMing socket - serversocket = socket.socket( - socket.AF_INET, socket.SOCK_STREAM) - #bind the socket to a public host, - # and a well-known port + # create an INET, STREAMing socket + serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + # bind the socket to a public host, and a well-known port serversocket.bind((socket.gethostname(), 80)) - #become a server socket + # become a server socket serversocket.listen(5) A couple things to notice: we used ``socket.gethostname()`` so that the socket @@ -101,10 +98,10 @@ mainloop of the web server:: while True: - #accept connections from outside + # accept connections from outside (clientsocket, address) = serversocket.accept() - #now do something with the clientsocket - #in this case, we'll pretend this is a threaded server + # now do something with the clientsocket + # in this case, we'll pretend this is a threaded server ct = client_thread(clientsocket) ct.run() @@ -126,12 +123,13 @@ --- If you need fast IPC between two processes on one machine, you should look into -whatever form of shared memory the platform offers. A simple protocol based -around shared memory and locks or semaphores is by far the fastest technique. +pipes or shared memory. If you do decide to use AF_INET sockets, bind the +"server" socket to ``'localhost'``. On most platforms, this will take a +shortcut around a couple of layers of network code and be quite a bit faster. -If you do decide to use sockets, bind the "server" socket to ``'localhost'``. On -most platforms, this will take a shortcut around a couple of layers of network -code and be quite a bit faster. +.. seealso:: + The :mod:`multiprocessing` integrates cross-platform IPC into a higher-level + API. Using a Socket @@ -300,7 +298,7 @@ Probably the worst thing about using blocking sockets is what happens when the other side comes down hard (without doing a ``close``). Your socket is likely to -hang. SOCKSTREAM is a reliable protocol, and it will wait a long, long time +hang. TCP is a reliable protocol, and it will wait a long, long time before giving up on a connection. If you're using threads, the entire thread is essentially dead. There's not much you can do about it. As long as you aren't doing something dumb, like holding a lock while doing a blocking read, the -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Reword IPC section
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/9d8a0cfcd6d9 changeset: 73862:9d8a0cfcd6d9 branch: 3.2 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:43:32 2011 +0100 summary: Reword IPC section files: Doc/howto/sockets.rst | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -123,12 +123,13 @@ --- If you need fast IPC between two processes on one machine, you should look into -whatever form of shared memory the platform offers. A simple protocol based -around shared memory and locks or semaphores is by far the fastest technique. +pipes or shared memory. If you do decide to use AF_INET sockets, bind the +"server" socket to ``'localhost'``. On most platforms, this will take a +shortcut around a couple of layers of network code and be quite a bit faster. -If you do decide to use sockets, bind the "server" socket to ``'localhost'``. On -most platforms, this will take a shortcut around a couple of layers of network -code and be quite a bit faster. +.. seealso:: + The :mod:`multiprocessing` integrates cross-platform IPC into a higher-level + API. Using a Socket -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): s/SOCKSTREAM/TCP/
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/c34188efb965 changeset: 73863:c34188efb965 branch: 3.2 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:46:35 2011 +0100 summary: s/SOCKSTREAM/TCP/ files: Doc/howto/sockets.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -298,7 +298,7 @@ Probably the worst thing about using blocking sockets is what happens when the other side comes down hard (without doing a ``close``). Your socket is likely to -hang. SOCKSTREAM is a reliable protocol, and it will wait a long, long time +hang. TCP is a reliable protocol, and it will wait a long, long time before giving up on a connection. If you're using threads, the entire thread is essentially dead. There's not much you can do about it. As long as you aren't doing something dumb, like holding a lock while doing a blocking read, the -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Use www.python.org instead of a hostname pointing to a parked (or squatted)
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/7e310c9cf46e changeset: 73861:7e310c9cf46e branch: 3.2 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:37:34 2011 +0100 summary: Use www.python.org instead of a hostname pointing to a parked (or squatted) domain. Also, reformat a bit. files: Doc/howto/sockets.rst | 23 ++++++++++------------- 1 files changed, 10 insertions(+), 13 deletions(-) diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -60,11 +60,10 @@ Roughly speaking, when you clicked on the link that brought you to this page, your browser did something like the following:: - #create an INET, STREAMing socket + # create an INET, STREAMing socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - #now connect to the web server on port 80 - # - the normal http port - s.connect(("www.mcmillan-inc.com", 80)) + # now connect to the web server on port 80 - the normal http port + s.connect(("www.python.org", 80)) When the ``connect`` completes, the socket ``s`` can be used to send in a request for the text of the page. The same socket will read the @@ -75,13 +74,11 @@ What happens in the web server is a bit more complex. First, the web server creates a "server socket":: - #create an INET, STREAMing socket - serversocket = socket.socket( - socket.AF_INET, socket.SOCK_STREAM) - #bind the socket to a public host, - # and a well-known port + # create an INET, STREAMing socket + serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + # bind the socket to a public host, and a well-known port serversocket.bind((socket.gethostname(), 80)) - #become a server socket + # become a server socket serversocket.listen(5) A couple things to notice: we used ``socket.gethostname()`` so that the socket @@ -101,10 +98,10 @@ mainloop of the web server:: while True: - #accept connections from outside + # accept connections from outside (clientsocket, address) = serversocket.accept() - #now do something with the clientsocket - #in this case, we'll pretend this is a threaded server + # now do something with the clientsocket + # in this case, we'll pretend this is a threaded server ct = client_thread(clientsocket) ct.run() -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Remove the outdated notion that multithreading doesn't work well on Unices.
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/64d980770571 changeset: 73860:64d980770571 branch: 3.2 parent: 73858:427b9dae1ae3 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:32:29 2011 +0100 summary: Remove the outdated notion that multithreading doesn't work well on Unices. files: Doc/howto/sockets.rst | 18 ++++++------------ 1 files changed, 6 insertions(+), 12 deletions(-) diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -395,19 +395,13 @@ There's no question that the fastest sockets code uses non-blocking sockets and select to multiplex them. You can put together something that will saturate a -LAN connection without putting any strain on the CPU. The trouble is that an app -written this way can't do much of anything else - it needs to be ready to -shuffle bytes around at all times. +LAN connection without putting any strain on the CPU. -Assuming that your app is actually supposed to do something more than that, -threading is the optimal solution, (and using non-blocking sockets will be -faster than using blocking sockets). Unfortunately, threading support in Unixes -varies both in API and quality. So the normal Unix solution is to fork a -subprocess to deal with each connection. The overhead for this is significant -(and don't do this on Windows - the overhead of process creation is enormous -there). It also means that unless each subprocess is completely independent, -you'll need to use another form of IPC, say a pipe, or shared memory and -semaphores, to communicate between the parent and child processes. +The trouble is that an app written this way can't do much of anything else - +it needs to be ready to shuffle bytes around at all times. Assuming that your +app is actually supposed to do something more than that, threading is the +optimal solution, (and using non-blocking sockets will be faster than using +blocking sockets). Finally, remember that even though blocking sockets are somewhat slower than non-blocking, in many cases they are the "right" solution. After all, if your -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Remove reference to the base64 encoding.
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/427b9dae1ae3 changeset: 73858:427b9dae1ae3 branch: 3.2 parent: 73856:2aeef275bec8 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:21:46 2011 +0100 summary: Remove reference to the base64 encoding. files: Doc/howto/unicode.rst | 27 +++++---------------------- 1 files changed, 5 insertions(+), 22 deletions(-) diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -552,7 +552,6 @@ i.e. Unix systems. - Tips for Writing Unicode-aware Programs --------------------------------------- @@ -572,28 +571,12 @@ When using data coming from a web browser or some other untrusted source, a common technique is to check for illegal characters in a string before using the string in a generated command line or storing it in a database. If you're doing -this, be careful to check the string once it's in the form that will be used or -stored; it's possible for encodings to be used to disguise characters. This is -especially true if the input data also specifies the encoding; many encodings -leave the commonly checked-for characters alone, but Python includes some -encodings such as ``'base64'`` that modify every single character. +this, be careful to check the decoded string, not the encoded bytes data; +some encodings may have interesting properties, such as not being bijective +or not being fully ASCII-compatible. This is especially true if the input +data also specifies the encoding, since the attacker can then choose a +clever way to hide malicious text in the encoded bytestream. -For example, let's say you have a content management system that takes a Unicode -filename, and you want to disallow paths with a '/' character. You might write -this code:: - - def read_file(filename, encoding): - if '/' in filename: - raise ValueError("'/' not allowed in filenames") - unicode_name = filename.decode(encoding) - with open(unicode_name, 'r') as f: - # ... return contents of file ... - -However, if an attacker could specify the ``'base64'`` encoding, they could pass -``'L2V0Yy9wYXNzd2Q='``, which is the base-64 encoded form of the string -``'/etc/passwd'``, to read a system file. The above code looks for ``'/'`` -characters in the encoded form and misses the dangerous character in the -resulting decoded form. References ---------- -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Remove reference to the base64 encoding.
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/8701f6373d0b changeset: 73859:8701f6373d0b parent: 73857:3828f81a64e7 parent: 73858:427b9dae1ae3 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:22:03 2011 +0100 summary: Remove reference to the base64 encoding. files: Doc/howto/unicode.rst | 27 +++++---------------------- 1 files changed, 5 insertions(+), 22 deletions(-) diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -552,7 +552,6 @@ i.e. Unix systems. - Tips for Writing Unicode-aware Programs --------------------------------------- @@ -572,28 +571,12 @@ When using data coming from a web browser or some other untrusted source, a common technique is to check for illegal characters in a string before using the string in a generated command line or storing it in a database. If you're doing -this, be careful to check the string once it's in the form that will be used or -stored; it's possible for encodings to be used to disguise characters. This is -especially true if the input data also specifies the encoding; many encodings -leave the commonly checked-for characters alone, but Python includes some -encodings such as ``'base64'`` that modify every single character. +this, be careful to check the decoded string, not the encoded bytes data; +some encodings may have interesting properties, such as not being bijective +or not being fully ASCII-compatible. This is especially true if the input +data also specifies the encoding, since the attacker can then choose a +clever way to hide malicious text in the encoded bytestream. -For example, let's say you have a content management system that takes a Unicode -filename, and you want to disallow paths with a '/' character. You might write -this code:: - - def read_file(filename, encoding): - if '/' in filename: - raise ValueError("'/' not allowed in filenames") - unicode_name = filename.decode(encoding) - with open(unicode_name, 'r') as f: - # ... return contents of file ... - -However, if an attacker could specify the ``'base64'`` encoding, they could pass -``'L2V0Yy9wYXNzd2Q='``, which is the base-64 encoded form of the string -``'/etc/passwd'``, to read a system file. The above code looks for ``'/'`` -characters in the encoded form and misses the dangerous character in the -resulting decoded form. References ---------- -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): The functional module hasn't been maintained since 2006 and doesn't work with
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/3828f81a64e7 changeset: 73857:3828f81a64e7 parent: 73855:e80a4e3b799b parent: 73856:2aeef275bec8 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:05:55 2011 +0100 summary: The functional module hasn't been maintained since 2006 and doesn't work with Python 3. Remove section about it from the functional programming FAQ. files: Doc/howto/functional.rst | 128 --------------------------- 1 files changed, 0 insertions(+), 128 deletions(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1010,135 +1010,6 @@ Consult the operator module's documentation for a complete list. - -The functional module ---------------------- - -Collin Winter's `functional module <http://oakwinter.com/code/functional/>`__ -provides a number of more advanced tools for functional programming. It also -reimplements several Python built-ins, trying to make them more intuitive to -those used to functional programming in other languages. - -This section contains an introduction to some of the most important functions in -``functional``; full documentation can be found at `the project's website -<http://oakwinter.com/code/functional/documentation/>`__. - -``compose(outer, inner, unpack=False)`` - -The ``compose()`` function implements function composition. In other words, it -returns a wrapper around the ``outer`` and ``inner`` callables, such that the -return value from ``inner`` is fed directly to ``outer``. That is, :: - - >>> def add(a, b): - ... return a + b - ... - >>> def double(a): - ... return 2 * a - ... - >>> compose(double, add)(5, 6) - 22 - -is equivalent to :: - - >>> double(add(5, 6)) - 22 - -The ``unpack`` keyword is provided to work around the fact that Python functions -are not always `fully curried <http://en.wikipedia.org/wiki/Currying>`__. By -default, it is expected that the ``inner`` function will return a single object -and that the ``outer`` function will take a single argument. Setting the -``unpack`` argument causes ``compose`` to expect a tuple from ``inner`` which -will be expanded before being passed to ``outer``. Put simply, :: - - compose(f, g)(5, 6) - -is equivalent to:: - - f(g(5, 6)) - -while :: - - compose(f, g, unpack=True)(5, 6) - -is equivalent to:: - - f(*g(5, 6)) - -Even though ``compose()`` only accepts two functions, it's trivial to build up a -version that will compose any number of functions. We'll use -:func:`functools.reduce`, ``compose()`` and ``partial()`` (the last of which is -provided by both ``functional`` and ``functools``). :: - - from functional import compose, partial - import functools - - - multi_compose = partial(functools.reduce, compose) - - -We can also use ``map()``, ``compose()`` and ``partial()`` to craft a version of -``"".join(...)`` that converts its arguments to string:: - - from functional import compose, partial - - join = compose("".join, partial(map, str)) - - -``flip(func)`` - -``flip()`` wraps the callable in ``func`` and causes it to receive its -non-keyword arguments in reverse order. :: - - >>> def triple(a, b, c): - ... return (a, b, c) - ... - >>> triple(5, 6, 7) - (5, 6, 7) - >>> - >>> flipped_triple = flip(triple) - >>> flipped_triple(5, 6, 7) - (7, 6, 5) - -``foldl(func, start, iterable)`` - -``foldl()`` takes a binary function, a starting value (usually some kind of -'zero'), and an iterable. The function is applied to the starting value and the -first element of the list, then the result of that and the second element of the -list, then the result of that and the third element of the list, and so on. - -This means that a call such as:: - - foldl(f, 0, [1, 2, 3]) - -is equivalent to:: - - f(f(f(0, 1), 2), 3) - - -``foldl()`` is roughly equivalent to the following recursive function:: - - def foldl(func, start, seq): - if len(seq) == 0: - return start - - return foldl(func, func(start, seq[0]), seq[1:]) - -Speaking of equivalence, the above ``foldl`` call can be expressed in terms of -the built-in :func:`functools.reduce` like so:: - - import functools - functools.reduce(f, [1, 2, 3], 0) - - -We can use ``foldl()``, ``operator.concat()`` and ``partial()`` to write a -cleaner, more aesthetically-pleasing version of Python's ``"".join(...)`` -idiom:: - - from functional import foldl, partial from operator import concat - - join = partial(foldl, concat, "") - - Small functions and the lambda expression ========================================= -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): The functional module hasn't been maintained since 2006 and doesn't work with
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/2aeef275bec8 changeset: 73856:2aeef275bec8 branch: 3.2 parent: 73854:1494e021cb9f user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 01:05:32 2011 +0100 summary: The functional module hasn't been maintained since 2006 and doesn't work with Python 3. Remove section about it from the functional programming FAQ. files: Doc/howto/functional.rst | 128 --------------------------- 1 files changed, 0 insertions(+), 128 deletions(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1010,135 +1010,6 @@ Consult the operator module's documentation for a complete list. - -The functional module ---------------------- - -Collin Winter's `functional module <http://oakwinter.com/code/functional/>`__ -provides a number of more advanced tools for functional programming. It also -reimplements several Python built-ins, trying to make them more intuitive to -those used to functional programming in other languages. - -This section contains an introduction to some of the most important functions in -``functional``; full documentation can be found at `the project's website -<http://oakwinter.com/code/functional/documentation/>`__. - -``compose(outer, inner, unpack=False)`` - -The ``compose()`` function implements function composition. In other words, it -returns a wrapper around the ``outer`` and ``inner`` callables, such that the -return value from ``inner`` is fed directly to ``outer``. That is, :: - - >>> def add(a, b): - ... return a + b - ... - >>> def double(a): - ... return 2 * a - ... - >>> compose(double, add)(5, 6) - 22 - -is equivalent to :: - - >>> double(add(5, 6)) - 22 - -The ``unpack`` keyword is provided to work around the fact that Python functions -are not always `fully curried <http://en.wikipedia.org/wiki/Currying>`__. By -default, it is expected that the ``inner`` function will return a single object -and that the ``outer`` function will take a single argument. Setting the -``unpack`` argument causes ``compose`` to expect a tuple from ``inner`` which -will be expanded before being passed to ``outer``. Put simply, :: - - compose(f, g)(5, 6) - -is equivalent to:: - - f(g(5, 6)) - -while :: - - compose(f, g, unpack=True)(5, 6) - -is equivalent to:: - - f(*g(5, 6)) - -Even though ``compose()`` only accepts two functions, it's trivial to build up a -version that will compose any number of functions. We'll use -:func:`functools.reduce`, ``compose()`` and ``partial()`` (the last of which is -provided by both ``functional`` and ``functools``). :: - - from functional import compose, partial - import functools - - - multi_compose = partial(functools.reduce, compose) - - -We can also use ``map()``, ``compose()`` and ``partial()`` to craft a version of -``"".join(...)`` that converts its arguments to string:: - - from functional import compose, partial - - join = compose("".join, partial(map, str)) - - -``flip(func)`` - -``flip()`` wraps the callable in ``func`` and causes it to receive its -non-keyword arguments in reverse order. :: - - >>> def triple(a, b, c): - ... return (a, b, c) - ... - >>> triple(5, 6, 7) - (5, 6, 7) - >>> - >>> flipped_triple = flip(triple) - >>> flipped_triple(5, 6, 7) - (7, 6, 5) - -``foldl(func, start, iterable)`` - -``foldl()`` takes a binary function, a starting value (usually some kind of -'zero'), and an iterable. The function is applied to the starting value and the -first element of the list, then the result of that and the second element of the -list, then the result of that and the third element of the list, and so on. - -This means that a call such as:: - - foldl(f, 0, [1, 2, 3]) - -is equivalent to:: - - f(f(f(0, 1), 2), 3) - - -``foldl()`` is roughly equivalent to the following recursive function:: - - def foldl(func, start, seq): - if len(seq) == 0: - return start - - return foldl(func, func(start, seq[0]), seq[1:]) - -Speaking of equivalence, the above ``foldl`` call can be expressed in terms of -the built-in :func:`functools.reduce` like so:: - - import functools - functools.reduce(f, [1, 2, 3], 0) - - -We can use ``foldl()``, ``operator.concat()`` and ``partial()`` to write a -cleaner, more aesthetically-pleasing version of Python's ``"".join(...)`` -idiom:: - - from functional import foldl, partial from operator import concat - - join = partial(foldl, concat, "") - - Small functions and the lambda expression ========================================= -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Remove obsolete references to bsddb
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/e80a4e3b799b changeset: 73855:e80a4e3b799b parent: 73852:f0008683585c parent: 73854:1494e021cb9f user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 00:41:51 2011 +0100 summary: Remove obsolete references to bsddb files: Doc/faq/library.rst | 44 --------------------------------- 1 files changed, 0 insertions(+), 44 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -814,52 +814,6 @@ general such as using gdbm with pickle/shelve. -If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come? ------------------------------------------------------------------------------------------- - -.. XXX move this FAQ entry elsewhere? - -.. note:: - - The bsddb module is now available as a standalone package `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_. - -Databases opened for write access with the bsddb module (and often by the anydbm -module, since it will preferentially use bsddb) must explicitly be closed using -the ``.close()`` method of the database. The underlying library caches database -contents which need to be converted to on-disk form and written. - -If you have initialized a new bsddb database but not written anything to it -before the program crashes, you will often wind up with a zero-length file and -encounter an exception the next time the file is opened. - - -I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data? ----------------------------------------------------------------------------------------------------------------------------- - -.. XXX move this FAQ entry elsewhere? - -.. note:: - - The bsddb module is now available as a standalone package `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_. - -Don't panic! Your data is probably intact. The most frequent cause for the error -is that you tried to open an earlier Berkeley DB file with a later version of -the Berkeley DB library. - -Many Linux systems now have all three versions of Berkeley DB available. If you -are migrating from version 1 to a newer version use db_dump185 to dump a plain -text version of the database. If you are migrating from version 2 to version 3 -use db2_dump to create a plain text version of the database. In either case, -use db_load to create a new native database for the latest version installed on -your computer. If you have version 3 of Berkeley DB installed, you should be -able to use db2_load to create a native version 2 database. - -You should move away from Berkeley DB version 1 files because the hash file code -contains known bugs that can corrupt your data. - - Mathematics and Numerics ======================== -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Remove obsolete references to bsddb
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/1494e021cb9f changeset: 73854:1494e021cb9f branch: 3.2 parent: 73851:2111bf7e5bca user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Dec 05 00:41:19 2011 +0100 summary: Remove obsolete references to bsddb files: Doc/faq/library.rst | 44 --------------------------------- 1 files changed, 0 insertions(+), 44 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -814,52 +814,6 @@ general such as using gdbm with pickle/shelve. -If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come? ------------------------------------------------------------------------------------------- - -.. XXX move this FAQ entry elsewhere? - -.. note:: - - The bsddb module is now available as a standalone package `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_. - -Databases opened for write access with the bsddb module (and often by the anydbm -module, since it will preferentially use bsddb) must explicitly be closed using -the ``.close()`` method of the database. The underlying library caches database -contents which need to be converted to on-disk form and written. - -If you have initialized a new bsddb database but not written anything to it -before the program crashes, you will often wind up with a zero-length file and -encounter an exception the next time the file is opened. - - -I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data? ----------------------------------------------------------------------------------------------------------------------------- - -.. XXX move this FAQ entry elsewhere? - -.. note:: - - The bsddb module is now available as a standalone package `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_. - -Don't panic! Your data is probably intact. The most frequent cause for the error -is that you tried to open an earlier Berkeley DB file with a later version of -the Berkeley DB library. - -Many Linux systems now have all three versions of Berkeley DB available. If you -are migrating from version 1 to a newer version use db_dump185 to dump a plain -text version of the database. If you are migrating from version 2 to version 3 -use db2_dump to create a plain text version of the database. In either case, -use db_load to create a new native database for the latest version installed on -your computer. If you have version 3 of Berkeley DB installed, you should be -able to use db2_load to create a native version 2 database. - -You should move away from Berkeley DB version 1 files because the hash file code -contains known bugs that can corrupt your data. - - Mathematics and Numerics ======================== -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #13527: remove mention of Python megawidgets and Tkinter3000 WCK
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/f0008683585c changeset: 73852:f0008683585c parent: 73850:8fa1dc66de5d parent: 73851:2111bf7e5bca user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sun Dec 04 23:57:55 2011 +0100 summary: Issue #13527: remove mention of Python megawidgets and Tkinter3000 WCK from the doc. These two projects appear dead. files: Doc/library/othergui.rst | 30 +-------------------------- 1 files changed, 2 insertions(+), 28 deletions(-) diff --git a/Doc/library/othergui.rst b/Doc/library/othergui.rst --- a/Doc/library/othergui.rst +++ b/Doc/library/othergui.rst @@ -3,34 +3,8 @@ Other Graphical User Interface Packages ======================================= -There are an number of extension widget sets to :mod:`tkinter`. - -.. seealso:: - - `Python megawidgets <http://pmw.sourceforge.net/>`_ - is a toolkit for building high-level compound widgets in Python using the - :mod:`tkinter` package. It consists of a set of base classes and a library of - flexible and extensible megawidgets built on this foundation. These megawidgets - include notebooks, comboboxes, selection widgets, paned widgets, scrolled - widgets, dialog windows, etc. Also, with the Pmw.Blt interface to BLT, the - busy, graph, stripchart, tabset and vector commands are be available. - - The initial ideas for Pmw were taken from the Tk ``itcl`` extensions ``[incr - Tk]`` by Michael McLennan and ``[incr Widgets]`` by Mark Ulferts. Several of the - megawidgets are direct translations from the itcl to Python. It offers most of - the range of widgets that ``[incr Widgets]`` does, and is almost as complete as - Tix, lacking however Tix's fast :class:`HList` widget for drawing trees. - - `Tkinter3000 Widget Construction Kit (WCK) <http://tkinter.effbot.org/>`_ - is a library that allows you to write new Tkinter widgets in pure Python. The - WCK framework gives you full control over widget creation, configuration, screen - appearance, and event handling. WCK widgets can be very fast and light-weight, - since they can operate directly on Python data structures, without having to - transfer data through the Tk/Tcl layer. - - -The major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits that are -also available for Python: +Major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits are +available for Python: .. seealso:: -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13527: remove mention of Python megawidgets and Tkinter3000 WCK
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/478b4e9551fa changeset: 73853:478b4e9551fa branch: 2.7 parent: 73826:fb8b6d310fb8 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sun Dec 04 23:56:30 2011 +0100 summary: Issue #13527: remove mention of Python megawidgets and Tkinter3000 WCK from the doc. These two projects appear dead. files: Doc/library/othergui.rst | 30 +-------------------------- 1 files changed, 2 insertions(+), 28 deletions(-) diff --git a/Doc/library/othergui.rst b/Doc/library/othergui.rst --- a/Doc/library/othergui.rst +++ b/Doc/library/othergui.rst @@ -3,34 +3,8 @@ Other Graphical User Interface Packages ======================================= -There are an number of extension widget sets to :mod:`Tkinter`. - -.. seealso:: - - `Python megawidgets <http://pmw.sourceforge.net/>`_ - is a toolkit for building high-level compound widgets in Python using the - :mod:`Tkinter` module. It consists of a set of base classes and a library of - flexible and extensible megawidgets built on this foundation. These megawidgets - include notebooks, comboboxes, selection widgets, paned widgets, scrolled - widgets, dialog windows, etc. Also, with the Pmw.Blt interface to BLT, the - busy, graph, stripchart, tabset and vector commands are be available. - - The initial ideas for Pmw were taken from the Tk ``itcl`` extensions ``[incr - Tk]`` by Michael McLennan and ``[incr Widgets]`` by Mark Ulferts. Several of the - megawidgets are direct translations from the itcl to Python. It offers most of - the range of widgets that ``[incr Widgets]`` does, and is almost as complete as - Tix, lacking however Tix's fast :class:`HList` widget for drawing trees. - - `Tkinter3000 Widget Construction Kit (WCK) <http://tkinter.effbot.org/>`_ - is a library that allows you to write new Tkinter widgets in pure Python. The - WCK framework gives you full control over widget creation, configuration, screen - appearance, and event handling. WCK widgets can be very fast and light-weight, - since they can operate directly on Python data structures, without having to - transfer data through the Tk/Tcl layer. - - -The major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits that are -also available for Python: +Major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits are +available for Python: .. seealso:: -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13527: remove mention of Python megawidgets and Tkinter3000 WCK
by antoine.pitrou Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/2111bf7e5bca changeset: 73851:2111bf7e5bca branch: 3.2 parent: 73848:0436ef8be253 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sun Dec 04 23:56:30 2011 +0100 summary: Issue #13527: remove mention of Python megawidgets and Tkinter3000 WCK from the doc. These two projects appear dead. files: Doc/library/othergui.rst | 30 +-------------------------- 1 files changed, 2 insertions(+), 28 deletions(-) diff --git a/Doc/library/othergui.rst b/Doc/library/othergui.rst --- a/Doc/library/othergui.rst +++ b/Doc/library/othergui.rst @@ -3,34 +3,8 @@ Other Graphical User Interface Packages ======================================= -There are an number of extension widget sets to :mod:`tkinter`. - -.. seealso:: - - `Python megawidgets <http://pmw.sourceforge.net/>`_ - is a toolkit for building high-level compound widgets in Python using the - :mod:`tkinter` package. It consists of a set of base classes and a library of - flexible and extensible megawidgets built on this foundation. These megawidgets - include notebooks, comboboxes, selection widgets, paned widgets, scrolled - widgets, dialog windows, etc. Also, with the Pmw.Blt interface to BLT, the - busy, graph, stripchart, tabset and vector commands are be available. - - The initial ideas for Pmw were taken from the Tk ``itcl`` extensions ``[incr - Tk]`` by Michael McLennan and ``[incr Widgets]`` by Mark Ulferts. Several of the - megawidgets are direct translations from the itcl to Python. It offers most of - the range of widgets that ``[incr Widgets]`` does, and is almost as complete as - Tix, lacking however Tix's fast :class:`HList` widget for drawing trees. - - `Tkinter3000 Widget Construction Kit (WCK) <http://tkinter.effbot.org/>`_ - is a library that allows you to write new Tkinter widgets in pure Python. The - WCK framework gives you full control over widget creation, configuration, screen - appearance, and event handling. WCK widgets can be very fast and light-weight, - since they can operate directly on Python data structures, without having to - transfer data through the Tk/Tcl layer. - - -The major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits that are -also available for Python: +Major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits are +available for Python: .. seealso:: -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Corrected order of parameters to HTTPError in test_urllib2.py.
by jason.coombs Dec. 5, 2011

Dec. 5, 2011

http://hg.python.org/cpython/rev/8fa1dc66de5d changeset: 73850:8fa1dc66de5d user: Jason R. Coombs <jaraco(a)jaraco.com> date: Sun Dec 04 08:14:18 2011 -0500 summary: Corrected order of parameters to HTTPError in test_urllib2.py. files: Lib/test/test_urllib2.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1456,7 +1456,7 @@ >>> msg = 'something bad happened' >>> url = code = hdrs = fp = None - >>> err = urllib.error.HTTPError(msg, url, code, hdrs, fp) + >>> err = urllib.error.HTTPError(url, code, msg, hdrs, fp) >>> assert hasattr(err, 'reason') >>> err.reason 'something bad happened' -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix typo.
by georg.brandl Dec. 4, 2011

Dec. 4, 2011

http://hg.python.org/cpython/rev/0436ef8be253 changeset: 73848:0436ef8be253 branch: 3.2 parent: 73845:bf0c5cd303e5 user: Georg Brandl <georg(a)python.org> date: Sun Dec 04 11:51:21 2011 +0100 summary: Fix typo. files: Doc/tools/sphinxext/static/copybutton.js | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/tools/sphinxext/static/copybutton.js b/Doc/tools/sphinxext/static/copybutton.js --- a/Doc/tools/sphinxext/static/copybutton.js +++ b/Doc/tools/sphinxext/static/copybutton.js @@ -8,8 +8,8 @@ // get the styles from the current theme pre.parent().parent().css('position', 'relative'); - var hide_text = 'Hide the prompts and ouput'; - var show_text = 'Show the prompts and ouput'; + var hide_text = 'Hide the prompts and output'; + var show_text = 'Show the prompts and output'; var border_width = pre.css('border-top-width'); var border_style = pre.css('border-top-style'); var border_color = pre.css('border-top-color'); -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge with 3.2.
by georg.brandl Dec. 4, 2011

Dec. 4, 2011

http://hg.python.org/cpython/rev/2e4fd707201c changeset: 73849:2e4fd707201c parent: 73847:a3ddee916808 parent: 73848:0436ef8be253 user: Georg Brandl <georg(a)python.org> date: Sun Dec 04 11:51:33 2011 +0100 summary: Merge with 3.2. files: Doc/tools/sphinxext/static/copybutton.js | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/tools/sphinxext/static/copybutton.js b/Doc/tools/sphinxext/static/copybutton.js --- a/Doc/tools/sphinxext/static/copybutton.js +++ b/Doc/tools/sphinxext/static/copybutton.js @@ -8,8 +8,8 @@ // get the styles from the current theme pre.parent().parent().css('position', 'relative'); - var hide_text = 'Hide the prompts and ouput'; - var show_text = 'Show the prompts and ouput'; + var hide_text = 'Hide the prompts and output'; + var show_text = 'Show the prompts and output'; var border_width = pre.css('border-top-width'); var border_style = pre.css('border-top-style'); var border_color = pre.css('border-top-color'); -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (f09477693434): sum=-417
by solipsis＠pitrou.net Dec. 4, 2011

Dec. 4, 2011

results for f09477693434 on branch "default" -------------------------------------------- test_urllib2net leaked [1405, -1822, 0] references, sum=-417 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogF7_xgV', '-x']

1 0