From python-checkins at python.org Fri Feb 1 00:20:15 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 1 Feb 2013 00:20:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A_update_the_impleme?= =?utf-8?q?ntation?= Message-ID: <3Yxy8v3V1hzRB2@mail.python.org> http://hg.python.org/peps/rev/a9b88df8cbab changeset: 4707:a9b88df8cbab user: Victor Stinner date: Fri Feb 01 00:19:01 2013 +0100 summary: PEP 433: update the implementation files: pep-0433.txt | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -532,12 +532,14 @@ os.dup() -------- + * Windows: ``DuplicateHandle()`` [atomic] * ``fcntl(fd, F_DUPFD_CLOEXEC)`` [atomic] * ``dup()`` + ``os.set_cloexec(fd, True)`` [best-effort] os.dup2() --------- + * ``fcntl(fd, F_DUP2FD_CLOEXEC, fd2)`` [atomic] * ``dup3()`` with ``O_CLOEXEC`` flag [atomic] * ``dup2()`` + ``os.set_cloexec(fd, True)`` [best-effort] -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Feb 1 00:49:33 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 1 Feb 2013 00:49:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A_subprocess_clears_?= =?utf-8?q?cloexec_flag_of_pass=5Ffds?= Message-ID: <3Yxypj1lkPzMPr@mail.python.org> http://hg.python.org/peps/rev/a9f8eb11b08b changeset: 4708:a9f8eb11b08b user: Victor Stinner date: Fri Feb 01 00:48:18 2013 +0100 summary: PEP 433: subprocess clears cloexec flag of pass_fds files: pep-0433.txt | 11 +++-------- 1 files changed, 3 insertions(+), 8 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -174,9 +174,6 @@ should be modified to conform to this PEP. The new ``os.set_cloexec()`` function can be used for example. -XXX Should ``subprocess.Popen`` clear the close-on-exec flag on file -XXX descriptors of the constructor the ``pass_fds`` parameter? - .. note:: See `Close file descriptors after fork`_ for a possible solution for ``fork()`` without ``exec()``. @@ -229,6 +226,9 @@ Add a new command line option ``-e`` and an environment variable ``PYTHONCLOEXEC`` to the set close-on-exec flag by default. +``subprocess`` clears the close-on-exec flag of file descriptors of the +``pass_fds`` parameter. + All functions creating file descriptors in the standard library must respect the default *cloexec* parameter (``sys.getdefaultcloexec()``). @@ -284,11 +284,6 @@ If a file must be inherited by child processes, ``cloexec=False`` parameter can be used. -``subprocess.Popen`` constructor has an ``pass_fds`` parameter to -specify which file descriptors must be inherited. The close-on-exec -flag of these file descriptors must be changed with -``os.set_cloexec()``. - Advantages of setting close-on-exec flag by default: * There are far more programs that are bitten by FD inheritance upon -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Feb 1 01:01:01 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 1 Feb 2013 01:01:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A_typo?= Message-ID: <3Yxz3x2bWfzRB2@mail.python.org> http://hg.python.org/peps/rev/c969d6ce3619 changeset: 4709:c969d6ce3619 user: Victor Stinner date: Fri Feb 01 00:59:47 2013 +0100 summary: PEP 433: typo files: pep-0433.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -230,7 +230,8 @@ ``pass_fds`` parameter. All functions creating file descriptors in the standard library must -respect the default *cloexec* parameter (``sys.getdefaultcloexec()``). +respect the default value of the *cloexec* parameter: +``sys.getdefaultcloexec()``. File descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are expected to be inherited, but Python does not handle them differently. When -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Feb 1 04:02:31 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 1 Feb 2013 04:02:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2317040=3A_document_that_?= =?utf-8?q?shelve=2Eopen=28=29_and_the_Shelf_object_can_be_used_as_context?= Message-ID: <3Yy35M5FxSzQ12@mail.python.org> http://hg.python.org/cpython/rev/935a286b8066 changeset: 81860:935a286b8066 parent: 81858:e6cc582cafce user: Ezio Melotti date: Fri Feb 01 05:01:50 2013 +0200 summary: #17040: document that shelve.open() and the Shelf object can be used as context managers. Initial patch by Berker Peksag. files: Doc/library/shelve.rst | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -44,8 +44,11 @@ .. note:: Do not rely on the shelf being closed automatically; always call - :meth:`close` explicitly when you don't need it any more, or use a - :keyword:`with` statement with :func:`contextlib.closing`. + :meth:`~Shelf.close` explicitly when you don't need it any more, or + use :func:`shelve.open` as a context manager:: + + with shelve.open('spam') as db: + db['eggs'] = 'eggs' .. warning:: @@ -118,10 +121,15 @@ The *keyencoding* parameter is the encoding used to encode keys before they are used with the underlying dict. - .. versionadded:: 3.2 - The *keyencoding* parameter; previously, keys were always encoded in + :class:`Shelf` objects can also be used as context managers. + + .. versionchanged:: 3.2 + Added the *keyencoding* parameter; previously, keys were always encoded in UTF-8. + .. versionchanged:: 3.4 + Added context manager support. + .. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 04:20:42 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 1 Feb 2013 04:20:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE2MTI4OiBjbGFy?= =?utf-8?q?ify_that_instances_of_user-defined_classes_compare_equal_with?= Message-ID: <3Yy3VL2XyVzRKB@mail.python.org> http://hg.python.org/cpython/rev/79a021beaf58 changeset: 81861:79a021beaf58 branch: 2.7 parent: 81859:8ee6d96a1019 user: Ezio Melotti date: Fri Feb 01 05:18:44 2013 +0200 summary: #16128: clarify that instances of user-defined classes compare equal with themselves. files: Doc/glossary.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -330,7 +330,8 @@ All of Python's immutable built-in objects are hashable, while no mutable containers (such as lists or dictionaries) are. Objects which are instances of user-defined classes are hashable by default; they all - compare unequal, and their hash value is their :func:`id`. + compare unequal (except with themselves), and their hash value is their + :func:`id`. IDLE An Integrated Development Environment for Python. IDLE is a basic editor -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 04:20:43 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 1 Feb 2013 04:20:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogIzE2MTI4OiBjbGFy?= =?utf-8?q?ify_that_instances_of_user-defined_classes_compare_equal_with?= Message-ID: <3Yy3VM54yjzRJm@mail.python.org> http://hg.python.org/cpython/rev/e84c5cf92b6f changeset: 81862:e84c5cf92b6f branch: 3.2 parent: 81856:9c0cd608464e user: Ezio Melotti date: Fri Feb 01 05:18:44 2013 +0200 summary: #16128: clarify that instances of user-defined classes compare equal with themselves. files: Doc/glossary.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -320,7 +320,8 @@ All of Python's immutable built-in objects are hashable, while no mutable containers (such as lists or dictionaries) are. Objects which are instances of user-defined classes are hashable by default; they all - compare unequal, and their hash value is their :func:`id`. + compare unequal (except with themselves), and their hash value is their + :func:`id`. IDLE An Integrated Development Environment for Python. IDLE is a basic editor -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 04:20:45 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 1 Feb 2013 04:20:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_=2316128=3A_merge_with_3=2E2=2E?= Message-ID: <3Yy3VP0TqKzRKL@mail.python.org> http://hg.python.org/cpython/rev/d9255c100971 changeset: 81863:d9255c100971 branch: 3.3 parent: 81857:886f48754f7e parent: 81862:e84c5cf92b6f user: Ezio Melotti date: Fri Feb 01 05:20:06 2013 +0200 summary: #16128: merge with 3.2. files: Doc/glossary.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -320,7 +320,8 @@ All of Python's immutable built-in objects are hashable, while no mutable containers (such as lists or dictionaries) are. Objects which are instances of user-defined classes are hashable by default; they all - compare unequal, and their hash value is their :func:`id`. + compare unequal (except with themselves), and their hash value is their + :func:`id`. IDLE An Integrated Development Environment for Python. IDLE is a basic editor -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 04:20:46 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 1 Feb 2013 04:20:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogIzE2MTI4OiBtZXJnZSB3aXRoIDMuMy4=?= Message-ID: <3Yy3VQ355CzRJ5@mail.python.org> http://hg.python.org/cpython/rev/1890c63f6153 changeset: 81864:1890c63f6153 parent: 81860:935a286b8066 parent: 81863:d9255c100971 user: Ezio Melotti date: Fri Feb 01 05:20:20 2013 +0200 summary: #16128: merge with 3.3. files: Doc/glossary.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -320,7 +320,8 @@ All of Python's immutable built-in objects are hashable, while no mutable containers (such as lists or dictionaries) are. Objects which are instances of user-defined classes are hashable by default; they all - compare unequal, and their hash value is their :func:`id`. + compare unequal (except with themselves), and their hash value is their + :func:`id`. IDLE An Integrated Development Environment for Python. IDLE is a basic editor -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Fri Feb 1 05:15:45 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 14:15:45 +1000 Subject: [Python-checkins] cpython (2.7): #16128: clarify that instances of user-defined classes compare equal with In-Reply-To: <3Yy3VL2XyVzRKB@mail.python.org> References: <3Yy3VL2XyVzRKB@mail.python.org> Message-ID: On 1 Feb 2013 13:22, "ezio.melotti" wrote: > > http://hg.python.org/cpython/rev/79a021beaf58 > changeset: 81861:79a021beaf58 > branch: 2.7 > parent: 81859:8ee6d96a1019 > user: Ezio Melotti > date: Fri Feb 01 05:18:44 2013 +0200 > summary: > #16128: clarify that instances of user-defined classes compare equal with themselves. > > files: > Doc/glossary.rst | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > > diff --git a/Doc/glossary.rst b/Doc/glossary.rst > --- a/Doc/glossary.rst > +++ b/Doc/glossary.rst > @@ -330,7 +330,8 @@ > All of Python's immutable built-in objects are hashable, while no mutable > containers (such as lists or dictionaries) are. Objects which are > instances of user-defined classes are hashable by default; they all > - compare unequal, and their hash value is their :func:`id`. > + compare unequal (except with themselves), and their hash value is their > + :func:`id`. The hash(x) == id(x) behaviour is a CPython implementation detail. It shouldn't be mentioned here. > > IDLE > An Integrated Development Environment for Python. IDLE is a basic editor > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 1 06:00:32 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 01 Feb 2013 06:00:32 +0100 Subject: [Python-checkins] Daily reference leaks (e6cc582cafce): sum=0 Message-ID: results for e6cc582cafce on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogRH6JFY', '-x'] From python-checkins at python.org Fri Feb 1 12:17:30 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 1 Feb 2013 12:17:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3ODM6?= =?utf-8?q?_Remove_declarations_of_nonexistent_private_variables=2E?= Message-ID: <3YyG4V3z5yzNyM@mail.python.org> http://hg.python.org/cpython/rev/6074530b526f changeset: 81865:6074530b526f branch: 2.7 parent: 81861:79a021beaf58 user: Serhiy Storchaka date: Fri Feb 01 13:13:32 2013 +0200 summary: Issue #1783: Remove declarations of nonexistent private variables. files: Include/sysmodule.h | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Include/sysmodule.h b/Include/sysmodule.h --- a/Include/sysmodule.h +++ b/Include/sysmodule.h @@ -19,9 +19,6 @@ PyAPI_FUNC(void) PySys_WriteStderr(const char *format, ...) Py_GCC_ATTRIBUTE((format(printf, 1, 2))); -PyAPI_DATA(PyObject *) _PySys_TraceFunc, *_PySys_ProfileFunc; -PyAPI_DATA(int) _PySys_CheckInterval; - PyAPI_FUNC(void) PySys_ResetWarnOptions(void); PyAPI_FUNC(void) PySys_AddWarnOption(char *); PyAPI_FUNC(int) PySys_HasWarnOptions(void); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 12:17:31 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 1 Feb 2013 12:17:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE3ODM6?= =?utf-8?q?_Remove_declarations_of_nonexistent_private_variables=2E?= Message-ID: <3YyG4W6ZgczR9Y@mail.python.org> http://hg.python.org/cpython/rev/349419bb6283 changeset: 81866:349419bb6283 branch: 3.2 parent: 81862:e84c5cf92b6f user: Serhiy Storchaka date: Fri Feb 01 13:14:20 2013 +0200 summary: Issue #1783: Remove declarations of nonexistent private variables. files: Include/sysmodule.h | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Include/sysmodule.h b/Include/sysmodule.h --- a/Include/sysmodule.h +++ b/Include/sysmodule.h @@ -20,10 +20,6 @@ PyAPI_FUNC(void) PySys_FormatStdout(const char *format, ...); PyAPI_FUNC(void) PySys_FormatStderr(const char *format, ...); -#ifndef Py_LIMITED_API -PyAPI_DATA(PyObject *) _PySys_TraceFunc, *_PySys_ProfileFunc; -#endif - PyAPI_FUNC(void) PySys_ResetWarnOptions(void); PyAPI_FUNC(void) PySys_AddWarnOption(const wchar_t *); PyAPI_FUNC(void) PySys_AddWarnOptionUnicode(PyObject *); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 12:17:33 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 1 Feb 2013 12:17:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=231783=3A_Remove_declarations_of_nonexistent_private_va?= =?utf-8?q?riables=2E?= Message-ID: <3YyG4Y2NlTzR16@mail.python.org> http://hg.python.org/cpython/rev/9d68f705e25f changeset: 81867:9d68f705e25f branch: 3.3 parent: 81863:d9255c100971 parent: 81866:349419bb6283 user: Serhiy Storchaka date: Fri Feb 01 13:14:47 2013 +0200 summary: Issue #1783: Remove declarations of nonexistent private variables. files: Include/sysmodule.h | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Include/sysmodule.h b/Include/sysmodule.h --- a/Include/sysmodule.h +++ b/Include/sysmodule.h @@ -20,10 +20,6 @@ PyAPI_FUNC(void) PySys_FormatStdout(const char *format, ...); PyAPI_FUNC(void) PySys_FormatStderr(const char *format, ...); -#ifndef Py_LIMITED_API -PyAPI_DATA(PyObject *) _PySys_TraceFunc, *_PySys_ProfileFunc; -#endif - PyAPI_FUNC(void) PySys_ResetWarnOptions(void); PyAPI_FUNC(void) PySys_AddWarnOption(const wchar_t *); PyAPI_FUNC(void) PySys_AddWarnOptionUnicode(PyObject *); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 12:17:34 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 1 Feb 2013 12:17:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=231783=3A_Remove_declarations_of_nonexistent_priv?= =?utf-8?q?ate_variables=2E?= Message-ID: <3YyG4Z4vGhzRB1@mail.python.org> http://hg.python.org/cpython/rev/905b4e3cf6d0 changeset: 81868:905b4e3cf6d0 parent: 81864:1890c63f6153 parent: 81867:9d68f705e25f user: Serhiy Storchaka date: Fri Feb 01 13:15:17 2013 +0200 summary: Issue #1783: Remove declarations of nonexistent private variables. files: Include/sysmodule.h | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Include/sysmodule.h b/Include/sysmodule.h --- a/Include/sysmodule.h +++ b/Include/sysmodule.h @@ -20,10 +20,6 @@ PyAPI_FUNC(void) PySys_FormatStdout(const char *format, ...); PyAPI_FUNC(void) PySys_FormatStderr(const char *format, ...); -#ifndef Py_LIMITED_API -PyAPI_DATA(PyObject *) _PySys_TraceFunc, *_PySys_ProfileFunc; -#endif - PyAPI_FUNC(void) PySys_ResetWarnOptions(void); PyAPI_FUNC(void) PySys_AddWarnOption(const wchar_t *); PyAPI_FUNC(void) PySys_AddWarnOptionUnicode(PyObject *); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 20:07:44 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 20:07:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3MDk4?= =?utf-8?q?=3A_Make_sure_every_module_has_=5F=5Floader=5F=5F_defined=2E?= Message-ID: <3YySW42gXszNgq@mail.python.org> http://hg.python.org/cpython/rev/05747d3bcd9c changeset: 81869:05747d3bcd9c branch: 3.3 parent: 81867:9d68f705e25f user: Brett Cannon date: Fri Feb 01 14:04:12 2013 -0500 summary: Issue #17098: Make sure every module has __loader__ defined. Thanks to Thomas Heller for the bug report. files: Lib/importlib/_bootstrap.py | 8 +- Misc/NEWS | 3 + Modules/signalmodule.c | 3 +- Python/importlib.h | 572 ++++++++++++----------- 4 files changed, 298 insertions(+), 288 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1703,9 +1703,11 @@ else: BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES - for module in (_imp, sys): - if not hasattr(module, '__loader__'): - module.__loader__ = BuiltinImporter + module_type = type(sys) + for module in sys.modules.values(): + if isinstance(module, module_type): + if not hasattr(module, '__loader__'): + module.__loader__ = BuiltinImporter self_module = sys.modules[__name__] for builtin_name in ('_io', '_warnings', 'builtins', 'marshal'): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,6 +12,9 @@ Core and Builtins ----------------- +- Issue #17098: All modules now have __loader__ set even if they pre-exist the + bootstrapping of importlib. + - Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. - Issue #13886: Fix input() to not strip out input bytes that cannot be decoded diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1367,9 +1367,8 @@ void PyOS_InitInterrupts(void) { - PyObject *m = PyInit_signal(); + PyObject *m = PyImport_ImportModule("signal"); if (m) { - _PyImport_FixupBuiltin(m, "signal"); Py_DECREF(m); } } diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 20:07:45 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 20:07:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317098=3A_all_modules_should_have_=5F=5Floader?= =?utf-8?b?X18=?= Message-ID: <3YySW55YCPzSP4@mail.python.org> http://hg.python.org/cpython/rev/1f1a1b3cc416 changeset: 81870:1f1a1b3cc416 parent: 81868:905b4e3cf6d0 parent: 81869:05747d3bcd9c user: Brett Cannon date: Fri Feb 01 14:07:28 2013 -0500 summary: Issue #17098: all modules should have __loader__ files: Lib/importlib/_bootstrap.py | 8 +++++--- Misc/NEWS | 3 +++ Modules/signalmodule.c | 3 +-- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1723,9 +1723,11 @@ else: BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES - for module in (_imp, sys): - if not hasattr(module, '__loader__'): - module.__loader__ = BuiltinImporter + module_type = type(sys) + for module in sys.modules.values(): + if isinstance(module, module_type): + if not hasattr(module, '__loader__'): + module.__loader__ = BuiltinImporter self_module = sys.modules[__name__] for builtin_name in ('_io', '_warnings', 'builtins', 'marshal'): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #17098: All modules now have __loader__ set even if they pre-exist the + bootstrapping of importlib. + - Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. - Issue #13886: Fix input() to not strip out input bytes that cannot be decoded diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1362,9 +1362,8 @@ void PyOS_InitInterrupts(void) { - PyObject *m = PyInit_signal(); + PyObject *m = PyImport_ImportModule("signal"); if (m) { - _PyImport_FixupBuiltin(m, "signal"); Py_DECREF(m); } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 20:40:33 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 20:40:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Fixes_Issue_?= =?utf-8?q?=236972=3A_The_zipfile_module_no_longer_overwrites_files_outsid?= =?utf-8?q?e_of?= Message-ID: <3YyTDx0KCkzSP5@mail.python.org> http://hg.python.org/cpython/rev/0c5fa35c9f12 changeset: 81871:0c5fa35c9f12 branch: 3.2 parent: 81866:349419bb6283 user: Gregory P. Smith date: Fri Feb 01 11:22:43 2013 -0800 summary: Fixes Issue #6972: The zipfile module no longer overwrites files outside of its destination path when extracting malicious zip files. files: Doc/library/zipfile.rst | 17 +++- Lib/test/test_zipfile.py | 86 +++++++++++++++++++++++++-- Lib/zipfile.py | 23 ++++-- Misc/NEWS | 3 + 4 files changed, 106 insertions(+), 23 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -214,6 +214,16 @@ to extract to. *member* can be a filename or a :class:`ZipInfo` object. *pwd* is the password used for encrypted files. + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``?:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.extractall(path=None, members=None, pwd=None) @@ -222,12 +232,9 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. warning:: + .. note:: - Never extract archives from untrusted sources without prior inspection. - It is possible that files are created outside of *path*, e.g. members - that have absolute filenames starting with ``"/"`` or filenames with two - dots ``".."``. + See :meth:`extract` note. .. method:: ZipFile.printdir() diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -29,7 +29,7 @@ SMALL_TEST_DATA = [('_ziptest1', '1q2w3e4r5t'), ('ziptest2dir/_ziptest2', 'qawsedrftg'), - ('/ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), + ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] @@ -409,10 +409,7 @@ writtenfile = zipfp.extract(fpath) # make sure it was written to the right place - if os.path.isabs(fpath): - correctfile = os.path.join(os.getcwd(), fpath[1:]) - else: - correctfile = os.path.join(os.getcwd(), fpath) + correctfile = os.path.join(os.getcwd(), fpath) correctfile = os.path.normpath(correctfile) self.assertEqual(writtenfile, correctfile) @@ -434,10 +431,7 @@ with zipfile.ZipFile(TESTFN2, "r") as zipfp: zipfp.extractall() for fpath, fdata in SMALL_TEST_DATA: - if os.path.isabs(fpath): - outfile = os.path.join(os.getcwd(), fpath[1:]) - else: - outfile = os.path.join(os.getcwd(), fpath) + outfile = os.path.join(os.getcwd(), fpath) with open(outfile, "rb") as f: self.assertEqual(fdata.encode(), f.read()) @@ -447,6 +441,80 @@ # remove the test file subdirectories shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir')) + def check_file(self, filename, content): + self.assertTrue(os.path.isfile(filename)) + with open(filename, 'rb') as f: + self.assertEqual(f.read(), content) + + def test_extract_hackers_arcnames(self): + hacknames = [ + ('../foo/bar', 'foo/bar'), + ('foo/../bar', 'foo/bar'), + ('foo/../../bar', 'foo/bar'), + ('foo/bar/..', 'foo/bar'), + ('./../foo/bar', 'foo/bar'), + ('/foo/bar', 'foo/bar'), + ('/foo/../bar', 'foo/bar'), + ('/foo/../../bar', 'foo/bar'), + ('//foo/bar', 'foo/bar'), + ('../../foo../../ba..r', 'foo../ba..r'), + ] + if os.path.sep == '\\': # Windows. + hacknames.extend([ + (r'..\foo\bar', 'foo/bar'), + (r'..\/foo\/bar', 'foo/bar'), + (r'foo/\..\/bar', 'foo/bar'), + (r'foo\/../\bar', 'foo/bar'), + (r'C:foo/bar', 'foo/bar'), + (r'C:/foo/bar', 'foo/bar'), + (r'C://foo/bar', 'foo/bar'), + (r'C:\foo\bar', 'foo/bar'), + (r'//conky/mountpoint/foo/bar', 'foo/bar'), + (r'\\conky\mountpoint\foo\bar', 'foo/bar'), + (r'///conky/mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\\conky\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//conky//mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\conky\\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//?/C:/foo/bar', 'foo/bar'), + (r'\\?\C:\foo\bar', 'foo/bar'), + (r'C:/../C:/foo/bar', 'C_/foo/bar'), + (r'a:b\ce|f"g?h*i', 'b/c_d_e_f_g_h_i'), + ]) + + for arcname, fixedname in hacknames: + content = b'foobar' + arcname.encode() + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr(arcname, content) + + targetpath = os.path.join('target', 'subdir', 'subsub') + correctfile = os.path.join(targetpath, *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname, targetpath) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree('target') + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall(targetpath) + self.check_file(correctfile, content) + shutil.rmtree('target') + + correctfile = os.path.join(os.getcwd(), *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall() + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + os.remove(TESTFN2) + def test_writestr_compression(self): zipfp = zipfile.ZipFile(TESTFN2, "w") zipfp.writestr("a.txt", "hello world", compress_type=zipfile.ZIP_STORED) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1062,17 +1062,22 @@ """ # build the destination pathname, replacing # forward slashes to platform specific separators. - # Strip trailing path separator, unless it represents the root. - if (targetpath[-1:] in (os.path.sep, os.path.altsep) - and len(os.path.splitdrive(targetpath)[1]) > 1): - targetpath = targetpath[:-1] + arcname = member.filename.replace('/', os.path.sep) - # don't include leading "/" from file name if present - if member.filename[0] == '/': - targetpath = os.path.join(targetpath, member.filename[1:]) - else: - targetpath = os.path.join(targetpath, member.filename) + if os.path.altsep: + arcname = arcname.replace(os.path.altsep, os.path.sep) + # interpret absolute pathname as relative, remove drive letter or + # UNC path, redundant separators, "." and ".." components. + arcname = os.path.splitdrive(arcname)[1] + arcname = os.path.sep.join(x for x in arcname.split(os.path.sep) + if x not in ('', os.path.curdir, os.path.pardir)) + # filter illegal characters on Windows + if os.path.sep == '\\': + illegal = ':<>|"?*' + table = str.maketrans(illegal, '_' * len(illegal)) + arcname = arcname.translate(table) + targetpath = os.path.join(targetpath, arcname) targetpath = os.path.normpath(targetpath) # Create all upper directories if necessary. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -216,6 +216,9 @@ Library ------- +- Issue #6972: The zipfile module no longer overwrites files outside of + its destination path when extracting malicious zip files. + - Issue #4844: ZipFile now raises BadZipFile when opens a ZIP file with an incomplete "End of Central Directory" record. Original patch by Guilherme Polo and Alan McIntyre. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 20:40:34 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 20:40:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Fixes_Issue_=236972=3A_The_zipfile_module_no_longer_overwrites?= =?utf-8?q?_files_outside_of?= Message-ID: <3YyTDy4hrxzSS8@mail.python.org> http://hg.python.org/cpython/rev/483488a1dec5 changeset: 81872:483488a1dec5 branch: 3.3 parent: 81869:05747d3bcd9c parent: 81871:0c5fa35c9f12 user: Gregory P. Smith date: Fri Feb 01 11:31:31 2013 -0800 summary: Fixes Issue #6972: The zipfile module no longer overwrites files outside of its destination path when extracting malicious zip files. files: Doc/library/zipfile.rst | 17 +++- Lib/test/test_zipfile.py | 86 +++++++++++++++++++++++++-- Lib/zipfile.py | 23 ++++-- Misc/NEWS | 3 + 4 files changed, 106 insertions(+), 23 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -242,6 +242,16 @@ to extract to. *member* can be a filename or a :class:`ZipInfo` object. *pwd* is the password used for encrypted files. + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``?:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.extractall(path=None, members=None, pwd=None) @@ -250,12 +260,9 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. warning:: + .. note:: - Never extract archives from untrusted sources without prior inspection. - It is possible that files are created outside of *path*, e.g. members - that have absolute filenames starting with ``"/"`` or filenames with two - dots ``".."``. + See :meth:`extract` note. .. method:: ZipFile.printdir() diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -24,7 +24,7 @@ SMALL_TEST_DATA = [('_ziptest1', '1q2w3e4r5t'), ('ziptest2dir/_ziptest2', 'qawsedrftg'), - ('/ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), + ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] @@ -501,10 +501,7 @@ writtenfile = zipfp.extract(fpath) # make sure it was written to the right place - if os.path.isabs(fpath): - correctfile = os.path.join(os.getcwd(), fpath[1:]) - else: - correctfile = os.path.join(os.getcwd(), fpath) + correctfile = os.path.join(os.getcwd(), fpath) correctfile = os.path.normpath(correctfile) self.assertEqual(writtenfile, correctfile) @@ -526,10 +523,7 @@ with zipfile.ZipFile(TESTFN2, "r") as zipfp: zipfp.extractall() for fpath, fdata in SMALL_TEST_DATA: - if os.path.isabs(fpath): - outfile = os.path.join(os.getcwd(), fpath[1:]) - else: - outfile = os.path.join(os.getcwd(), fpath) + outfile = os.path.join(os.getcwd(), fpath) with open(outfile, "rb") as f: self.assertEqual(fdata.encode(), f.read()) @@ -539,6 +533,80 @@ # remove the test file subdirectories shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir')) + def check_file(self, filename, content): + self.assertTrue(os.path.isfile(filename)) + with open(filename, 'rb') as f: + self.assertEqual(f.read(), content) + + def test_extract_hackers_arcnames(self): + hacknames = [ + ('../foo/bar', 'foo/bar'), + ('foo/../bar', 'foo/bar'), + ('foo/../../bar', 'foo/bar'), + ('foo/bar/..', 'foo/bar'), + ('./../foo/bar', 'foo/bar'), + ('/foo/bar', 'foo/bar'), + ('/foo/../bar', 'foo/bar'), + ('/foo/../../bar', 'foo/bar'), + ('//foo/bar', 'foo/bar'), + ('../../foo../../ba..r', 'foo../ba..r'), + ] + if os.path.sep == '\\': # Windows. + hacknames.extend([ + (r'..\foo\bar', 'foo/bar'), + (r'..\/foo\/bar', 'foo/bar'), + (r'foo/\..\/bar', 'foo/bar'), + (r'foo\/../\bar', 'foo/bar'), + (r'C:foo/bar', 'foo/bar'), + (r'C:/foo/bar', 'foo/bar'), + (r'C://foo/bar', 'foo/bar'), + (r'C:\foo\bar', 'foo/bar'), + (r'//conky/mountpoint/foo/bar', 'foo/bar'), + (r'\\conky\mountpoint\foo\bar', 'foo/bar'), + (r'///conky/mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\\conky\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//conky//mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\conky\\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//?/C:/foo/bar', 'foo/bar'), + (r'\\?\C:\foo\bar', 'foo/bar'), + (r'C:/../C:/foo/bar', 'C_/foo/bar'), + (r'a:b\ce|f"g?h*i', 'b/c_d_e_f_g_h_i'), + ]) + + for arcname, fixedname in hacknames: + content = b'foobar' + arcname.encode() + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr(arcname, content) + + targetpath = os.path.join('target', 'subdir', 'subsub') + correctfile = os.path.join(targetpath, *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname, targetpath) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree('target') + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall(targetpath) + self.check_file(correctfile, content) + shutil.rmtree('target') + + correctfile = os.path.join(os.getcwd(), *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall() + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + os.remove(TESTFN2) + def test_writestr_compression_stored(self): zipfp = zipfile.ZipFile(TESTFN2, "w") zipfp.writestr("a.txt", "hello world", compress_type=zipfile.ZIP_STORED) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1229,17 +1229,22 @@ """ # build the destination pathname, replacing # forward slashes to platform specific separators. - # Strip trailing path separator, unless it represents the root. - if (targetpath[-1:] in (os.path.sep, os.path.altsep) - and len(os.path.splitdrive(targetpath)[1]) > 1): - targetpath = targetpath[:-1] + arcname = member.filename.replace('/', os.path.sep) - # don't include leading "/" from file name if present - if member.filename[0] == '/': - targetpath = os.path.join(targetpath, member.filename[1:]) - else: - targetpath = os.path.join(targetpath, member.filename) + if os.path.altsep: + arcname = arcname.replace(os.path.altsep, os.path.sep) + # interpret absolute pathname as relative, remove drive letter or + # UNC path, redundant separators, "." and ".." components. + arcname = os.path.splitdrive(arcname)[1] + arcname = os.path.sep.join(x for x in arcname.split(os.path.sep) + if x not in ('', os.path.curdir, os.path.pardir)) + # filter illegal characters on Windows + if os.path.sep == '\\': + illegal = ':<>|"?*' + table = str.maketrans(illegal, '_' * len(illegal)) + arcname = arcname.translate(table) + targetpath = os.path.join(targetpath, arcname) targetpath = os.path.normpath(targetpath) # Create all upper directories if necessary. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -167,6 +167,9 @@ Library ------- +- Issue #6972: The zipfile module no longer overwrites files outside of + its destination path when extracting malicious zip files. + - Issue #4844: ZipFile now raises BadZipFile when opens a ZIP file with an incomplete "End of Central Directory" record. Original patch by Guilherme Polo and Alan McIntyre. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 20:40:36 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 20:40:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fixes_Issue_=236972=3A_The_zipfile_module_no_longer_over?= =?utf-8?q?writes_files_outside_of?= Message-ID: <3YyTF01bfrzSP5@mail.python.org> http://hg.python.org/cpython/rev/249e0b47b686 changeset: 81873:249e0b47b686 parent: 81870:1f1a1b3cc416 parent: 81872:483488a1dec5 user: Gregory P. Smith date: Fri Feb 01 11:35:00 2013 -0800 summary: Fixes Issue #6972: The zipfile module no longer overwrites files outside of its destination path when extracting malicious zip files. files: Doc/library/zipfile.rst | 17 +++- Lib/test/test_zipfile.py | 86 +++++++++++++++++++++++++-- Lib/zipfile.py | 23 ++++-- Misc/NEWS | 3 + 4 files changed, 106 insertions(+), 23 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -242,6 +242,16 @@ to extract to. *member* can be a filename or a :class:`ZipInfo` object. *pwd* is the password used for encrypted files. + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``?:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.extractall(path=None, members=None, pwd=None) @@ -250,12 +260,9 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. warning:: + .. note:: - Never extract archives from untrusted sources without prior inspection. - It is possible that files are created outside of *path*, e.g. members - that have absolute filenames starting with ``"/"`` or filenames with two - dots ``".."``. + See :meth:`extract` note. .. method:: ZipFile.printdir() diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -24,7 +24,7 @@ SMALL_TEST_DATA = [('_ziptest1', '1q2w3e4r5t'), ('ziptest2dir/_ziptest2', 'qawsedrftg'), - ('/ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), + ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] @@ -501,10 +501,7 @@ writtenfile = zipfp.extract(fpath) # make sure it was written to the right place - if os.path.isabs(fpath): - correctfile = os.path.join(os.getcwd(), fpath[1:]) - else: - correctfile = os.path.join(os.getcwd(), fpath) + correctfile = os.path.join(os.getcwd(), fpath) correctfile = os.path.normpath(correctfile) self.assertEqual(writtenfile, correctfile) @@ -526,10 +523,7 @@ with zipfile.ZipFile(TESTFN2, "r") as zipfp: zipfp.extractall() for fpath, fdata in SMALL_TEST_DATA: - if os.path.isabs(fpath): - outfile = os.path.join(os.getcwd(), fpath[1:]) - else: - outfile = os.path.join(os.getcwd(), fpath) + outfile = os.path.join(os.getcwd(), fpath) with open(outfile, "rb") as f: self.assertEqual(fdata.encode(), f.read()) @@ -539,6 +533,80 @@ # remove the test file subdirectories shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir')) + def check_file(self, filename, content): + self.assertTrue(os.path.isfile(filename)) + with open(filename, 'rb') as f: + self.assertEqual(f.read(), content) + + def test_extract_hackers_arcnames(self): + hacknames = [ + ('../foo/bar', 'foo/bar'), + ('foo/../bar', 'foo/bar'), + ('foo/../../bar', 'foo/bar'), + ('foo/bar/..', 'foo/bar'), + ('./../foo/bar', 'foo/bar'), + ('/foo/bar', 'foo/bar'), + ('/foo/../bar', 'foo/bar'), + ('/foo/../../bar', 'foo/bar'), + ('//foo/bar', 'foo/bar'), + ('../../foo../../ba..r', 'foo../ba..r'), + ] + if os.path.sep == '\\': # Windows. + hacknames.extend([ + (r'..\foo\bar', 'foo/bar'), + (r'..\/foo\/bar', 'foo/bar'), + (r'foo/\..\/bar', 'foo/bar'), + (r'foo\/../\bar', 'foo/bar'), + (r'C:foo/bar', 'foo/bar'), + (r'C:/foo/bar', 'foo/bar'), + (r'C://foo/bar', 'foo/bar'), + (r'C:\foo\bar', 'foo/bar'), + (r'//conky/mountpoint/foo/bar', 'foo/bar'), + (r'\\conky\mountpoint\foo\bar', 'foo/bar'), + (r'///conky/mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\\conky\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//conky//mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\conky\\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//?/C:/foo/bar', 'foo/bar'), + (r'\\?\C:\foo\bar', 'foo/bar'), + (r'C:/../C:/foo/bar', 'C_/foo/bar'), + (r'a:b\ce|f"g?h*i', 'b/c_d_e_f_g_h_i'), + ]) + + for arcname, fixedname in hacknames: + content = b'foobar' + arcname.encode() + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr(arcname, content) + + targetpath = os.path.join('target', 'subdir', 'subsub') + correctfile = os.path.join(targetpath, *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname, targetpath) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree('target') + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall(targetpath) + self.check_file(correctfile, content) + shutil.rmtree('target') + + correctfile = os.path.join(os.getcwd(), *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall() + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + os.remove(TESTFN2) + def test_writestr_compression_stored(self): zipfp = zipfile.ZipFile(TESTFN2, "w") zipfp.writestr("a.txt", "hello world", compress_type=zipfile.ZIP_STORED) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1229,17 +1229,22 @@ """ # build the destination pathname, replacing # forward slashes to platform specific separators. - # Strip trailing path separator, unless it represents the root. - if (targetpath[-1:] in (os.path.sep, os.path.altsep) - and len(os.path.splitdrive(targetpath)[1]) > 1): - targetpath = targetpath[:-1] + arcname = member.filename.replace('/', os.path.sep) - # don't include leading "/" from file name if present - if member.filename[0] == '/': - targetpath = os.path.join(targetpath, member.filename[1:]) - else: - targetpath = os.path.join(targetpath, member.filename) + if os.path.altsep: + arcname = arcname.replace(os.path.altsep, os.path.sep) + # interpret absolute pathname as relative, remove drive letter or + # UNC path, redundant separators, "." and ".." components. + arcname = os.path.splitdrive(arcname)[1] + arcname = os.path.sep.join(x for x in arcname.split(os.path.sep) + if x not in ('', os.path.curdir, os.path.pardir)) + # filter illegal characters on Windows + if os.path.sep == '\\': + illegal = ':<>|"?*' + table = str.maketrans(illegal, '_' * len(illegal)) + arcname = arcname.translate(table) + targetpath = os.path.join(targetpath, arcname) targetpath = os.path.normpath(targetpath) # Create all upper directories if necessary. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -239,6 +239,9 @@ Library ------- +- Issue #6972: The zipfile module no longer overwrites files outside of + its destination path when extracting malicious zip files. + - Issue #4844: ZipFile now raises BadZipFile when opens a ZIP file with an incomplete "End of Central Directory" record. Original patch by Guilherme Polo and Alan McIntyre. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 20:40:37 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 20:40:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fixes_Issue_?= =?utf-8?q?=236972=3A_The_zipfile_module_no_longer_overwrites_files_outsid?= =?utf-8?q?e_of?= Message-ID: <3YyTF15fKyzSS3@mail.python.org> http://hg.python.org/cpython/rev/4d1948689ee1 changeset: 81874:4d1948689ee1 branch: 2.7 parent: 81865:6074530b526f user: Gregory P. Smith date: Fri Feb 01 11:40:18 2013 -0800 summary: Fixes Issue #6972: The zipfile module no longer overwrites files outside of its destination path when extracting malicious zip files. files: Doc/library/zipfile.rst | 10 +++ Lib/test/test_zipfile.py | 86 +++++++++++++++++++++++++-- Lib/zipfile.py | 23 ++++-- Misc/NEWS | 3 + 4 files changed, 104 insertions(+), 18 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -242,6 +242,16 @@ .. versionadded:: 2.6 + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``?:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.read(name[, pwd]) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -26,7 +26,7 @@ SMALL_TEST_DATA = [('_ziptest1', '1q2w3e4r5t'), ('ziptest2dir/_ziptest2', 'qawsedrftg'), - ('/ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), + ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] @@ -391,10 +391,7 @@ writtenfile = zipfp.extract(fpath) # make sure it was written to the right place - if os.path.isabs(fpath): - correctfile = os.path.join(os.getcwd(), fpath[1:]) - else: - correctfile = os.path.join(os.getcwd(), fpath) + correctfile = os.path.join(os.getcwd(), fpath) correctfile = os.path.normpath(correctfile) self.assertEqual(writtenfile, correctfile) @@ -414,10 +411,7 @@ with zipfile.ZipFile(TESTFN2, "r") as zipfp: zipfp.extractall() for fpath, fdata in SMALL_TEST_DATA: - if os.path.isabs(fpath): - outfile = os.path.join(os.getcwd(), fpath[1:]) - else: - outfile = os.path.join(os.getcwd(), fpath) + outfile = os.path.join(os.getcwd(), fpath) self.assertEqual(fdata, open(outfile, "rb").read()) os.remove(outfile) @@ -425,6 +419,80 @@ # remove the test file subdirectories shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir')) + def check_file(self, filename, content): + self.assertTrue(os.path.isfile(filename)) + with open(filename, 'rb') as f: + self.assertEqual(f.read(), content) + + def test_extract_hackers_arcnames(self): + hacknames = [ + ('../foo/bar', 'foo/bar'), + ('foo/../bar', 'foo/bar'), + ('foo/../../bar', 'foo/bar'), + ('foo/bar/..', 'foo/bar'), + ('./../foo/bar', 'foo/bar'), + ('/foo/bar', 'foo/bar'), + ('/foo/../bar', 'foo/bar'), + ('/foo/../../bar', 'foo/bar'), + ('//foo/bar', 'foo/bar'), + ('../../foo../../ba..r', 'foo../ba..r'), + ] + if os.path.sep == '\\': + hacknames.extend([ + (r'..\foo\bar', 'foo/bar'), + (r'..\/foo\/bar', 'foo/bar'), + (r'foo/\..\/bar', 'foo/bar'), + (r'foo\/../\bar', 'foo/bar'), + (r'C:foo/bar', 'foo/bar'), + (r'C:/foo/bar', 'foo/bar'), + (r'C://foo/bar', 'foo/bar'), + (r'C:\foo\bar', 'foo/bar'), + (r'//conky/mountpoint/foo/bar', 'foo/bar'), + (r'\\conky\mountpoint\foo\bar', 'foo/bar'), + (r'///conky/mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\\conky\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//conky//mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\conky\\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//?/C:/foo/bar', 'foo/bar'), + (r'\\?\C:\foo\bar', 'foo/bar'), + (r'C:/../C:/foo/bar', 'C_/foo/bar'), + (r'a:b\ce|f"g?h*i', 'b/c_d_e_f_g_h_i'), + ]) + + for arcname, fixedname in hacknames: + content = b'foobar' + arcname.encode() + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr(arcname, content) + + targetpath = os.path.join('target', 'subdir', 'subsub') + correctfile = os.path.join(targetpath, *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname, targetpath) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree('target') + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall(targetpath) + self.check_file(correctfile, content) + shutil.rmtree('target') + + correctfile = os.path.join(os.getcwd(), *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall() + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + os.remove(TESTFN2) + def test_writestr_compression(self): zipfp = zipfile.ZipFile(TESTFN2, "w") zipfp.writestr("a.txt", "hello world", compress_type=zipfile.ZIP_STORED) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1040,17 +1040,22 @@ """ # build the destination pathname, replacing # forward slashes to platform specific separators. - # Strip trailing path separator, unless it represents the root. - if (targetpath[-1:] in (os.path.sep, os.path.altsep) - and len(os.path.splitdrive(targetpath)[1]) > 1): - targetpath = targetpath[:-1] + arcname = member.filename.replace('/', os.path.sep) - # don't include leading "/" from file name if present - if member.filename[0] == '/': - targetpath = os.path.join(targetpath, member.filename[1:]) - else: - targetpath = os.path.join(targetpath, member.filename) + if os.path.altsep: + arcname = arcname.replace(os.path.altsep, os.path.sep) + # interpret absolute pathname as relative, remove drive letter or + # UNC path, redundant separators, "." and ".." components. + arcname = os.path.splitdrive(arcname)[1] + arcname = os.path.sep.join(x for x in arcname.split(os.path.sep) + if x not in ('', os.path.curdir, os.path.pardir)) + # filter illegal characters on Windows + if os.path.sep == '\\': + illegal = ':<>|"?*' + table = str.maketrans(illegal, '_' * len(illegal)) + arcname = arcname.translate(table) + targetpath = os.path.join(targetpath, arcname) targetpath = os.path.normpath(targetpath) # Create all upper directories if necessary. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -202,6 +202,9 @@ Library ------- +- Issue #6972: The zipfile module no longer overwrites files outside of + its destination path when extracting malicious zip files. + - Issue #17049: Localized calendar methods now return unicode if a locale includes an encoding and the result string contains month or weekday (was regression from Python 2.6). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:26 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Additional_fix?= =?utf-8?q?_for_Issue_=2312268=3A_The_io_module_file_object_writelines=28?= =?utf-8?q?=29_methods?= Message-ID: <3YyWGy32sqzSP4@mail.python.org> http://hg.python.org/cpython/rev/a5e7b38caee2 changeset: 81875:a5e7b38caee2 branch: 2.7 user: Gregory P. Smith date: Fri Feb 01 13:02:59 2013 -0800 summary: Additional fix for Issue #12268: The io module file object writelines() methods no longer abort early when one of its write system calls is interrupted (EINTR). files: Misc/NEWS | 3 +++ Modules/_io/iobase.c | 5 ++++- Modules/_io/textio.c | 7 +++++-- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -679,6 +679,9 @@ Extension Modules ----------------- +- Issue #12268: The io module file object writelines() methods no longer + abort early when one of its write system calls is interrupted (EINTR). + - Fix the leak of a dict in the time module when used in an embedded interpreter that is repeatedly initialized and shutdown and reinitialized. diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -660,7 +660,10 @@ break; /* Stop Iteration */ } - res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + res = NULL; + do { + res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + } while (res == NULL && _PyIO_trap_eintr()); Py_DECREF(line); if (res == NULL) { Py_DECREF(iter); diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1213,8 +1213,11 @@ Py_DECREF(pending); if (b == NULL) return -1; - ret = PyObject_CallMethodObjArgs(self->buffer, - _PyIO_str_write, b, NULL); + ret = NULL; + do { + ret = PyObject_CallMethodObjArgs(self->buffer, + _PyIO_str_write, b, NULL); + } while (ret == NULL && _PyIO_trap_eintr()); Py_DECREF(b); if (ret == NULL) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:27 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Additional_fix?= =?utf-8?q?_for_Issue_=2312268=3A_The_io_module_file_object_writelines=28?= =?utf-8?q?=29_methods?= Message-ID: <3YyWGz65xZzSSC@mail.python.org> http://hg.python.org/cpython/rev/2fd669aa4abc changeset: 81876:2fd669aa4abc branch: 3.2 parent: 81871:0c5fa35c9f12 user: Gregory P. Smith date: Fri Feb 01 13:03:39 2013 -0800 summary: Additional fix for Issue #12268: The io module file object writelines() methods no longer abort early when one of its write system calls is interrupted (EINTR). files: Misc/NEWS | 3 +++ Modules/_io/iobase.c | 5 ++++- Modules/_io/textio.c | 7 +++++-- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -817,6 +817,9 @@ Extension Modules ----------------- +- Issue #12268: The io module file object writelines() methods no longer + abort early when one of its write system calls is interrupted (EINTR). + - Fix the leak of a dict in the time module when used in an embedded interpreter that is repeatedly initialized and shutdown and reinitialized. diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -674,7 +674,10 @@ break; /* Stop Iteration */ } - res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + res = NULL; + do { + res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + } while (res == NULL && _PyIO_trap_eintr()); Py_DECREF(line); if (res == NULL) { Py_DECREF(iter); diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1249,8 +1249,11 @@ Py_DECREF(pending); if (b == NULL) return -1; - ret = PyObject_CallMethodObjArgs(self->buffer, - _PyIO_str_write, b, NULL); + ret = NULL; + do { + ret = PyObject_CallMethodObjArgs(self->buffer, + _PyIO_str_write, b, NULL); + } while (ret == NULL && _PyIO_trap_eintr()); Py_DECREF(b); if (ret == NULL) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:29 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_better_correct?= =?utf-8?q?ed_news_entry?= Message-ID: <3YyWH11vkczST6@mail.python.org> http://hg.python.org/cpython/rev/81f7bdf7bbb6 changeset: 81877:81f7bdf7bbb6 branch: 3.2 user: Gregory P. Smith date: Fri Feb 01 13:06:44 2013 -0800 summary: better corrected news entry files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -817,8 +817,8 @@ Extension Modules ----------------- -- Issue #12268: The io module file object writelines() methods no longer - abort early when one of its write system calls is interrupted (EINTR). +- Issue #12268: The io module file object write methods no longer abort early + when one of its write system calls is interrupted (EINTR). - Fix the leak of a dict in the time module when used in an embedded interpreter that is repeatedly initialized and shutdown and reinitialized. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:30 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_better_news_en?= =?utf-8?q?try?= Message-ID: <3YyWH24gkwzSTF@mail.python.org> http://hg.python.org/cpython/rev/c8f8708d509a changeset: 81878:c8f8708d509a branch: 2.7 parent: 81875:a5e7b38caee2 user: Gregory P. Smith date: Fri Feb 01 13:07:27 2013 -0800 summary: better news entry files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -679,8 +679,8 @@ Extension Modules ----------------- -- Issue #12268: The io module file object writelines() methods no longer - abort early when one of its write system calls is interrupted (EINTR). +- Issue #12268: The io module file object write methods no longer abort early + when a write system calls is interrupted (EINTR). - Fix the leak of a dict in the time module when used in an embedded interpreter that is repeatedly initialized and shutdown and reinitialized. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:32 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Additional_fix_for_issue_=2312268=3A_The_io_module_file_object?= =?utf-8?q?_write_methods_no?= Message-ID: <3YyWH40K45zSSG@mail.python.org> http://hg.python.org/cpython/rev/30fc620e240e changeset: 81879:30fc620e240e branch: 3.3 parent: 81872:483488a1dec5 parent: 81876:2fd669aa4abc user: Gregory P. Smith date: Fri Feb 01 13:08:23 2013 -0800 summary: Additional fix for issue #12268: The io module file object write methods no longer abort early when a write system call is interrupted (EINTR). files: Misc/NEWS | 6 ++++++ Modules/_io/iobase.c | 5 ++++- Modules/_io/textio.c | 7 +++++-- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -506,6 +506,12 @@ - Issue #15906: Fix a regression in `argparse` caused by the preceding change, when ``action='append'``, ``type='str'`` and ``default=[]``. +Extension Modules +----------------- + +- Issue #12268: The io module file object write methods no longer abort early + when one of its write system calls is interrupted (EINTR). + Tests ----- diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -669,7 +669,10 @@ break; /* Stop Iteration */ } - res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + res = NULL; + do { + res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + } while (res == NULL && _PyIO_trap_eintr()); Py_DECREF(line); if (res == NULL) { Py_DECREF(iter); diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1247,8 +1247,11 @@ Py_DECREF(pending); if (b == NULL) return -1; - ret = PyObject_CallMethodObjArgs(self->buffer, - _PyIO_str_write, b, NULL); + ret = NULL; + do { + ret = PyObject_CallMethodObjArgs(self->buffer, + _PyIO_str_write, b, NULL); + } while (ret == NULL && _PyIO_trap_eintr()); Py_DECREF(b); if (ret == NULL) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:33 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_null_merge?= Message-ID: <3YyWH531HvzST2@mail.python.org> http://hg.python.org/cpython/rev/29e3aa7f2f4b changeset: 81880:29e3aa7f2f4b branch: 3.3 parent: 81879:30fc620e240e parent: 81877:81f7bdf7bbb6 user: Gregory P. Smith date: Fri Feb 01 13:08:51 2013 -0800 summary: null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:12:34 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 1 Feb 2013 22:12:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Additional_fix_for_issue_=2312268=3A_The_io_module_file_?= =?utf-8?q?object_write_methods_no?= Message-ID: <3YyWH65ldVzSTN@mail.python.org> http://hg.python.org/cpython/rev/8f72519fd0e9 changeset: 81881:8f72519fd0e9 parent: 81873:249e0b47b686 parent: 81880:29e3aa7f2f4b user: Gregory P. Smith date: Fri Feb 01 13:10:33 2013 -0800 summary: Additional fix for issue #12268: The io module file object write methods no longer abort early when a write system call is interrupted (EINTR). files: Misc/NEWS | 6 ++++++ Modules/_io/iobase.c | 5 ++++- Modules/_io/textio.c | 7 +++++-- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -715,6 +715,12 @@ `sha3_256`, `sha3_384` and `sha3_512`. As part of the patch some common code was moved from _hashopenssl.c to hashlib.h. +Extension Modules +----------------- + +- Issue #12268: The io module file object write methods no longer abort early + when one of its write system calls is interrupted (EINTR). + Tests ----- diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -669,7 +669,10 @@ break; /* Stop Iteration */ } - res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + res = NULL; + do { + res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + } while (res == NULL && _PyIO_trap_eintr()); Py_DECREF(line); if (res == NULL) { Py_DECREF(iter); diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1247,8 +1247,11 @@ Py_DECREF(pending); if (b == NULL) return -1; - ret = PyObject_CallMethodObjArgs(self->buffer, - _PyIO_str_write, b, NULL); + ret = NULL; + do { + ret = PyObject_CallMethodObjArgs(self->buffer, + _PyIO_str_write, b, NULL); + } while (ret == NULL && _PyIO_trap_eintr()); Py_DECREF(b); if (ret == NULL) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:38:00 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:38:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Add_a_test_for?= =?utf-8?q?_fix_of_issue_=2317098?= Message-ID: <3YyWrS3SZtzSQf@mail.python.org> http://hg.python.org/cpython/rev/4a4688b865ff changeset: 81882:4a4688b865ff branch: 3.3 parent: 81869:05747d3bcd9c user: Brett Cannon date: Fri Feb 01 14:43:59 2013 -0500 summary: Add a test for fix of issue #17098 files: Lib/test/test_importlib/test_api.py | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py --- a/Lib/test/test_importlib/test_api.py +++ b/Lib/test/test_importlib/test_api.py @@ -4,6 +4,7 @@ from importlib import machinery import sys from test import support +import types import unittest @@ -175,12 +176,22 @@ machinery.FrozenImporter)) +class StartupTests(unittest.TestCase): + + def test_everyone_has___loader__(self): + # Issue #17098: all modules should have __loader__ defined. + for name, module in sys.modules.items(): + if isinstance(module, types.ModuleType): + self.assertTrue(hasattr(module, '__loader__'), + '{!r} lacks a __loader__ attribute'.format(name)) + def test_main(): from test.support import run_unittest run_unittest(ImportModuleTests, FindLoaderTests, InvalidateCacheTests, - FrozenImportlibTests) + FrozenImportlibTests, + StartupTests) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:38:01 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:38:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3YyWrT62xszSSQ@mail.python.org> http://hg.python.org/cpython/rev/370882f297a4 changeset: 81883:370882f297a4 parent: 81870:1f1a1b3cc416 parent: 81882:4a4688b865ff user: Brett Cannon date: Fri Feb 01 14:51:43 2013 -0500 summary: merge with 3.3 files: Lib/test/test_importlib/test_api.py | 17 ++++++++++------- 1 files changed, 10 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py --- a/Lib/test/test_importlib/test_api.py +++ b/Lib/test/test_importlib/test_api.py @@ -4,6 +4,7 @@ from importlib import machinery import sys from test import support +import types import unittest @@ -175,13 +176,15 @@ machinery.FrozenImporter)) -def test_main(): - from test.support import run_unittest - run_unittest(ImportModuleTests, - FindLoaderTests, - InvalidateCacheTests, - FrozenImportlibTests) +class StartupTests(unittest.TestCase): + + def test_everyone_has___loader__(self): + # Issue #17098: all modules should have __loader__ defined. + for name, module in sys.modules.items(): + if isinstance(module, types.ModuleType): + self.assertTrue(hasattr(module, '__loader__'), + '{!r} lacks a __loader__ attribute'.format(name)) if __name__ == '__main__': - test_main() + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:38:03 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:38:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3MDk4?= =?utf-8?q?=3A_Be_more_stringent_of_setting_=5F=5Floader=5F=5F_on_early_im?= =?utf-8?q?ported?= Message-ID: <3YyWrW1hC2zSTB@mail.python.org> http://hg.python.org/cpython/rev/19ea454ccdf7 changeset: 81884:19ea454ccdf7 branch: 3.3 parent: 81882:4a4688b865ff user: Brett Cannon date: Fri Feb 01 15:31:49 2013 -0500 summary: Issue #17098: Be more stringent of setting __loader__ on early imported modules. Also made test more rigorous. files: Lib/importlib/_bootstrap.py | 7 +- Lib/test/test_importlib/test_api.py | 6 + Python/importlib.h | 591 ++++++++------- 3 files changed, 310 insertions(+), 294 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1704,10 +1704,13 @@ BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES module_type = type(sys) - for module in sys.modules.values(): + for name, module in sys.modules.items(): if isinstance(module, module_type): if not hasattr(module, '__loader__'): - module.__loader__ = BuiltinImporter + if name in sys.builtin_module_names: + module.__loader__ = BuiltinImporter + elif _imp.is_frozen(name): + module.__loader__ = FrozenImporter self_module = sys.modules[__name__] for builtin_name in ('_io', '_warnings', 'builtins', 'marshal'): diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py --- a/Lib/test/test_importlib/test_api.py +++ b/Lib/test/test_importlib/test_api.py @@ -184,6 +184,12 @@ if isinstance(module, types.ModuleType): self.assertTrue(hasattr(module, '__loader__'), '{!r} lacks a __loader__ attribute'.format(name)) + if name in sys.builtin_module_names: + self.assertEqual(importlib.machinery.BuiltinImporter, + module.__loader__) + elif imp.is_frozen(name): + self.assertEqual(importlib.machinery.FrozenImporter, + module.__loader__) def test_main(): from test.support import run_unittest diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:38:04 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:38:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_w/_3=2E3_more_fixes_thanks_to_issue_=2317098?= Message-ID: <3YyWrX4WnKzSTF@mail.python.org> http://hg.python.org/cpython/rev/306f066e6a33 changeset: 81885:306f066e6a33 parent: 81883:370882f297a4 parent: 81884:19ea454ccdf7 user: Brett Cannon date: Fri Feb 01 16:36:29 2013 -0500 summary: Merge w/ 3.3 more fixes thanks to issue #17098 files: Lib/importlib/_bootstrap.py | 7 +- Lib/test/test_importlib/test_api.py | 9 + Python/importlib.h | 551 ++++++++------- 3 files changed, 296 insertions(+), 271 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1724,10 +1724,13 @@ BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES module_type = type(sys) - for module in sys.modules.values(): + for name, module in sys.modules.items(): if isinstance(module, module_type): if not hasattr(module, '__loader__'): - module.__loader__ = BuiltinImporter + if name in sys.builtin_module_names: + module.__loader__ = BuiltinImporter + elif _imp.is_frozen(name): + module.__loader__ = FrozenImporter self_module = sys.modules[__name__] for builtin_name in ('_io', '_warnings', 'builtins', 'marshal'): diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py --- a/Lib/test/test_importlib/test_api.py +++ b/Lib/test/test_importlib/test_api.py @@ -1,6 +1,7 @@ from . import util import imp import importlib +from importlib import _bootstrap from importlib import machinery import sys from test import support @@ -184,6 +185,14 @@ if isinstance(module, types.ModuleType): self.assertTrue(hasattr(module, '__loader__'), '{!r} lacks a __loader__ attribute'.format(name)) + if name in sys.builtin_module_names: + self.assertIn(module.__loader__, + (importlib.machinery.BuiltinImporter, + importlib._bootstrap.BuiltinImporter)) + elif imp.is_frozen(name): + self.assertIn(module.__loader__, + (importlib.machinery.FrozenImporter, + importlib._bootstrap.FrozenImporter)) if __name__ == '__main__': diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:38:06 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:38:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3YyWrZ1S6bzST6@mail.python.org> http://hg.python.org/cpython/rev/b7a0c91b2174 changeset: 81886:b7a0c91b2174 parent: 81885:306f066e6a33 parent: 81881:8f72519fd0e9 user: Brett Cannon date: Fri Feb 01 16:36:49 2013 -0500 summary: merge files: Doc/library/zipfile.rst | 17 +++- Lib/test/test_zipfile.py | 86 +++++++++++++++++++++++++-- Lib/zipfile.py | 23 ++++-- Misc/NEWS | 9 ++ Modules/_io/iobase.c | 5 +- Modules/_io/textio.c | 7 +- 6 files changed, 121 insertions(+), 26 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -242,6 +242,16 @@ to extract to. *member* can be a filename or a :class:`ZipInfo` object. *pwd* is the password used for encrypted files. + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``?:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.extractall(path=None, members=None, pwd=None) @@ -250,12 +260,9 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. warning:: + .. note:: - Never extract archives from untrusted sources without prior inspection. - It is possible that files are created outside of *path*, e.g. members - that have absolute filenames starting with ``"/"`` or filenames with two - dots ``".."``. + See :meth:`extract` note. .. method:: ZipFile.printdir() diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -24,7 +24,7 @@ SMALL_TEST_DATA = [('_ziptest1', '1q2w3e4r5t'), ('ziptest2dir/_ziptest2', 'qawsedrftg'), - ('/ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), + ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] @@ -501,10 +501,7 @@ writtenfile = zipfp.extract(fpath) # make sure it was written to the right place - if os.path.isabs(fpath): - correctfile = os.path.join(os.getcwd(), fpath[1:]) - else: - correctfile = os.path.join(os.getcwd(), fpath) + correctfile = os.path.join(os.getcwd(), fpath) correctfile = os.path.normpath(correctfile) self.assertEqual(writtenfile, correctfile) @@ -526,10 +523,7 @@ with zipfile.ZipFile(TESTFN2, "r") as zipfp: zipfp.extractall() for fpath, fdata in SMALL_TEST_DATA: - if os.path.isabs(fpath): - outfile = os.path.join(os.getcwd(), fpath[1:]) - else: - outfile = os.path.join(os.getcwd(), fpath) + outfile = os.path.join(os.getcwd(), fpath) with open(outfile, "rb") as f: self.assertEqual(fdata.encode(), f.read()) @@ -539,6 +533,80 @@ # remove the test file subdirectories shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir')) + def check_file(self, filename, content): + self.assertTrue(os.path.isfile(filename)) + with open(filename, 'rb') as f: + self.assertEqual(f.read(), content) + + def test_extract_hackers_arcnames(self): + hacknames = [ + ('../foo/bar', 'foo/bar'), + ('foo/../bar', 'foo/bar'), + ('foo/../../bar', 'foo/bar'), + ('foo/bar/..', 'foo/bar'), + ('./../foo/bar', 'foo/bar'), + ('/foo/bar', 'foo/bar'), + ('/foo/../bar', 'foo/bar'), + ('/foo/../../bar', 'foo/bar'), + ('//foo/bar', 'foo/bar'), + ('../../foo../../ba..r', 'foo../ba..r'), + ] + if os.path.sep == '\\': # Windows. + hacknames.extend([ + (r'..\foo\bar', 'foo/bar'), + (r'..\/foo\/bar', 'foo/bar'), + (r'foo/\..\/bar', 'foo/bar'), + (r'foo\/../\bar', 'foo/bar'), + (r'C:foo/bar', 'foo/bar'), + (r'C:/foo/bar', 'foo/bar'), + (r'C://foo/bar', 'foo/bar'), + (r'C:\foo\bar', 'foo/bar'), + (r'//conky/mountpoint/foo/bar', 'foo/bar'), + (r'\\conky\mountpoint\foo\bar', 'foo/bar'), + (r'///conky/mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\\conky\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//conky//mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\conky\\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//?/C:/foo/bar', 'foo/bar'), + (r'\\?\C:\foo\bar', 'foo/bar'), + (r'C:/../C:/foo/bar', 'C_/foo/bar'), + (r'a:b\ce|f"g?h*i', 'b/c_d_e_f_g_h_i'), + ]) + + for arcname, fixedname in hacknames: + content = b'foobar' + arcname.encode() + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr(arcname, content) + + targetpath = os.path.join('target', 'subdir', 'subsub') + correctfile = os.path.join(targetpath, *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname, targetpath) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree('target') + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall(targetpath) + self.check_file(correctfile, content) + shutil.rmtree('target') + + correctfile = os.path.join(os.getcwd(), *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall() + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + os.remove(TESTFN2) + def test_writestr_compression_stored(self): zipfp = zipfile.ZipFile(TESTFN2, "w") zipfp.writestr("a.txt", "hello world", compress_type=zipfile.ZIP_STORED) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1229,17 +1229,22 @@ """ # build the destination pathname, replacing # forward slashes to platform specific separators. - # Strip trailing path separator, unless it represents the root. - if (targetpath[-1:] in (os.path.sep, os.path.altsep) - and len(os.path.splitdrive(targetpath)[1]) > 1): - targetpath = targetpath[:-1] + arcname = member.filename.replace('/', os.path.sep) - # don't include leading "/" from file name if present - if member.filename[0] == '/': - targetpath = os.path.join(targetpath, member.filename[1:]) - else: - targetpath = os.path.join(targetpath, member.filename) + if os.path.altsep: + arcname = arcname.replace(os.path.altsep, os.path.sep) + # interpret absolute pathname as relative, remove drive letter or + # UNC path, redundant separators, "." and ".." components. + arcname = os.path.splitdrive(arcname)[1] + arcname = os.path.sep.join(x for x in arcname.split(os.path.sep) + if x not in ('', os.path.curdir, os.path.pardir)) + # filter illegal characters on Windows + if os.path.sep == '\\': + illegal = ':<>|"?*' + table = str.maketrans(illegal, '_' * len(illegal)) + arcname = arcname.translate(table) + targetpath = os.path.join(targetpath, arcname) targetpath = os.path.normpath(targetpath) # Create all upper directories if necessary. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -239,6 +239,9 @@ Library ------- +- Issue #6972: The zipfile module no longer overwrites files outside of + its destination path when extracting malicious zip files. + - Issue #4844: ZipFile now raises BadZipFile when opens a ZIP file with an incomplete "End of Central Directory" record. Original patch by Guilherme Polo and Alan McIntyre. @@ -712,6 +715,12 @@ `sha3_256`, `sha3_384` and `sha3_512`. As part of the patch some common code was moved from _hashopenssl.c to hashlib.h. +Extension Modules +----------------- + +- Issue #12268: The io module file object write methods no longer abort early + when one of its write system calls is interrupted (EINTR). + Tests ----- diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -669,7 +669,10 @@ break; /* Stop Iteration */ } - res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + res = NULL; + do { + res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + } while (res == NULL && _PyIO_trap_eintr()); Py_DECREF(line); if (res == NULL) { Py_DECREF(iter); diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1247,8 +1247,11 @@ Py_DECREF(pending); if (b == NULL) return -1; - ret = PyObject_CallMethodObjArgs(self->buffer, - _PyIO_str_write, b, NULL); + ret = NULL; + do { + ret = PyObject_CallMethodObjArgs(self->buffer, + _PyIO_str_write, b, NULL); + } while (ret == NULL && _PyIO_trap_eintr()); Py_DECREF(b); if (ret == NULL) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:38:07 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:38:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_merge?= Message-ID: <3YyWrb5SpdzSSD@mail.python.org> http://hg.python.org/cpython/rev/4d9bcf328e64 changeset: 81887:4d9bcf328e64 branch: 3.3 parent: 81884:19ea454ccdf7 parent: 81880:29e3aa7f2f4b user: Brett Cannon date: Fri Feb 01 16:37:07 2013 -0500 summary: merge files: Doc/library/zipfile.rst | 17 +++- Lib/test/test_zipfile.py | 86 +++++++++++++++++++++++++-- Lib/zipfile.py | 23 ++++-- Misc/NEWS | 9 ++ Modules/_io/iobase.c | 5 +- Modules/_io/textio.c | 7 +- 6 files changed, 121 insertions(+), 26 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -242,6 +242,16 @@ to extract to. *member* can be a filename or a :class:`ZipInfo` object. *pwd* is the password used for encrypted files. + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``?:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.extractall(path=None, members=None, pwd=None) @@ -250,12 +260,9 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. warning:: + .. note:: - Never extract archives from untrusted sources without prior inspection. - It is possible that files are created outside of *path*, e.g. members - that have absolute filenames starting with ``"/"`` or filenames with two - dots ``".."``. + See :meth:`extract` note. .. method:: ZipFile.printdir() diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -24,7 +24,7 @@ SMALL_TEST_DATA = [('_ziptest1', '1q2w3e4r5t'), ('ziptest2dir/_ziptest2', 'qawsedrftg'), - ('/ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), + ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] @@ -501,10 +501,7 @@ writtenfile = zipfp.extract(fpath) # make sure it was written to the right place - if os.path.isabs(fpath): - correctfile = os.path.join(os.getcwd(), fpath[1:]) - else: - correctfile = os.path.join(os.getcwd(), fpath) + correctfile = os.path.join(os.getcwd(), fpath) correctfile = os.path.normpath(correctfile) self.assertEqual(writtenfile, correctfile) @@ -526,10 +523,7 @@ with zipfile.ZipFile(TESTFN2, "r") as zipfp: zipfp.extractall() for fpath, fdata in SMALL_TEST_DATA: - if os.path.isabs(fpath): - outfile = os.path.join(os.getcwd(), fpath[1:]) - else: - outfile = os.path.join(os.getcwd(), fpath) + outfile = os.path.join(os.getcwd(), fpath) with open(outfile, "rb") as f: self.assertEqual(fdata.encode(), f.read()) @@ -539,6 +533,80 @@ # remove the test file subdirectories shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir')) + def check_file(self, filename, content): + self.assertTrue(os.path.isfile(filename)) + with open(filename, 'rb') as f: + self.assertEqual(f.read(), content) + + def test_extract_hackers_arcnames(self): + hacknames = [ + ('../foo/bar', 'foo/bar'), + ('foo/../bar', 'foo/bar'), + ('foo/../../bar', 'foo/bar'), + ('foo/bar/..', 'foo/bar'), + ('./../foo/bar', 'foo/bar'), + ('/foo/bar', 'foo/bar'), + ('/foo/../bar', 'foo/bar'), + ('/foo/../../bar', 'foo/bar'), + ('//foo/bar', 'foo/bar'), + ('../../foo../../ba..r', 'foo../ba..r'), + ] + if os.path.sep == '\\': # Windows. + hacknames.extend([ + (r'..\foo\bar', 'foo/bar'), + (r'..\/foo\/bar', 'foo/bar'), + (r'foo/\..\/bar', 'foo/bar'), + (r'foo\/../\bar', 'foo/bar'), + (r'C:foo/bar', 'foo/bar'), + (r'C:/foo/bar', 'foo/bar'), + (r'C://foo/bar', 'foo/bar'), + (r'C:\foo\bar', 'foo/bar'), + (r'//conky/mountpoint/foo/bar', 'foo/bar'), + (r'\\conky\mountpoint\foo\bar', 'foo/bar'), + (r'///conky/mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\\conky\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//conky//mountpoint/foo/bar', 'conky/mountpoint/foo/bar'), + (r'\\conky\\mountpoint\foo\bar', 'conky/mountpoint/foo/bar'), + (r'//?/C:/foo/bar', 'foo/bar'), + (r'\\?\C:\foo\bar', 'foo/bar'), + (r'C:/../C:/foo/bar', 'C_/foo/bar'), + (r'a:b\ce|f"g?h*i', 'b/c_d_e_f_g_h_i'), + ]) + + for arcname, fixedname in hacknames: + content = b'foobar' + arcname.encode() + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr(arcname, content) + + targetpath = os.path.join('target', 'subdir', 'subsub') + correctfile = os.path.join(targetpath, *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname, targetpath) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree('target') + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall(targetpath) + self.check_file(correctfile, content) + shutil.rmtree('target') + + correctfile = os.path.join(os.getcwd(), *fixedname.split('/')) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + writtenfile = zipfp.extract(arcname) + self.assertEqual(writtenfile, correctfile) + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + with zipfile.ZipFile(TESTFN2, 'r') as zipfp: + zipfp.extractall() + self.check_file(correctfile, content) + shutil.rmtree(fixedname.split('/')[0]) + + os.remove(TESTFN2) + def test_writestr_compression_stored(self): zipfp = zipfile.ZipFile(TESTFN2, "w") zipfp.writestr("a.txt", "hello world", compress_type=zipfile.ZIP_STORED) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1229,17 +1229,22 @@ """ # build the destination pathname, replacing # forward slashes to platform specific separators. - # Strip trailing path separator, unless it represents the root. - if (targetpath[-1:] in (os.path.sep, os.path.altsep) - and len(os.path.splitdrive(targetpath)[1]) > 1): - targetpath = targetpath[:-1] + arcname = member.filename.replace('/', os.path.sep) - # don't include leading "/" from file name if present - if member.filename[0] == '/': - targetpath = os.path.join(targetpath, member.filename[1:]) - else: - targetpath = os.path.join(targetpath, member.filename) + if os.path.altsep: + arcname = arcname.replace(os.path.altsep, os.path.sep) + # interpret absolute pathname as relative, remove drive letter or + # UNC path, redundant separators, "." and ".." components. + arcname = os.path.splitdrive(arcname)[1] + arcname = os.path.sep.join(x for x in arcname.split(os.path.sep) + if x not in ('', os.path.curdir, os.path.pardir)) + # filter illegal characters on Windows + if os.path.sep == '\\': + illegal = ':<>|"?*' + table = str.maketrans(illegal, '_' * len(illegal)) + arcname = arcname.translate(table) + targetpath = os.path.join(targetpath, arcname) targetpath = os.path.normpath(targetpath) # Create all upper directories if necessary. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -167,6 +167,9 @@ Library ------- +- Issue #6972: The zipfile module no longer overwrites files outside of + its destination path when extracting malicious zip files. + - Issue #4844: ZipFile now raises BadZipFile when opens a ZIP file with an incomplete "End of Central Directory" record. Original patch by Guilherme Polo and Alan McIntyre. @@ -503,6 +506,12 @@ - Issue #15906: Fix a regression in `argparse` caused by the preceding change, when ``action='append'``, ``type='str'`` and ``default=[]``. +Extension Modules +----------------- + +- Issue #12268: The io module file object write methods no longer abort early + when one of its write system calls is interrupted (EINTR). + Tests ----- diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -669,7 +669,10 @@ break; /* Stop Iteration */ } - res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + res = NULL; + do { + res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL); + } while (res == NULL && _PyIO_trap_eintr()); Py_DECREF(line); if (res == NULL) { Py_DECREF(iter); diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1247,8 +1247,11 @@ Py_DECREF(pending); if (b == NULL) return -1; - ret = PyObject_CallMethodObjArgs(self->buffer, - _PyIO_str_write, b, NULL); + ret = NULL; + do { + ret = PyObject_CallMethodObjArgs(self->buffer, + _PyIO_str_write, b, NULL); + } while (ret == NULL && _PyIO_trap_eintr()); Py_DECREF(b); if (ret == NULL) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 22:40:01 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 1 Feb 2013 22:40:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_from_3=2E3?= Message-ID: <3YyWtn0qGhzSSD@mail.python.org> http://hg.python.org/cpython/rev/3029319d2d8a changeset: 81888:3029319d2d8a parent: 81886:b7a0c91b2174 parent: 81887:4d9bcf328e64 user: Brett Cannon date: Fri Feb 01 16:39:50 2013 -0500 summary: merge from 3.3 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 23:06:51 2013 From: python-checkins at python.org (ned.deily) Date: Fri, 1 Feb 2013 23:06:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2MjU2?= =?utf-8?q?=3A_OS_X_installer_now_sets_correct_permissions_for_doc_directo?= =?utf-8?b?cnku?= Message-ID: <3YyXTl5HtwzST0@mail.python.org> http://hg.python.org/cpython/rev/d64e0cf5f1a7 changeset: 81889:d64e0cf5f1a7 branch: 2.7 parent: 81878:c8f8708d509a user: Ned Deily date: Fri Feb 01 13:58:00 2013 -0800 summary: Issue #16256: OS X installer now sets correct permissions for doc directory. files: Mac/BuildScript/scripts/postflight.documentation | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Mac/BuildScript/scripts/postflight.documentation b/Mac/BuildScript/scripts/postflight.documentation --- a/Mac/BuildScript/scripts/postflight.documentation +++ b/Mac/BuildScript/scripts/postflight.documentation @@ -16,7 +16,7 @@ # make share/doc link in framework for command line users if [ -d "${SHARE_DIR}" ]; then - mkdir -p "${SHARE_DOCDIR}" + mkdir -m 775 -p "${SHARE_DOCDIR}" # make relative link to html doc directory ln -fhs "${SHARE_DOCDIR_TO_FWK}/${FWK_DOCDIR_SUBPATH}" "${SHARE_DOCDIR}/html" fi diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -789,6 +789,8 @@ - Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. +- Issue #16256: OS X installer now sets correct permissions for doc directory. + - Issue #8767: Restore building with --disable-unicode. Patch by Stefano Taschini. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 23:06:53 2013 From: python-checkins at python.org (ned.deily) Date: Fri, 1 Feb 2013 23:06:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2MjU2?= =?utf-8?q?=3A_OS_X_installer_now_sets_correct_permissions_for_doc_directo?= =?utf-8?b?cnku?= Message-ID: <3YyXTn0rc0zPrk@mail.python.org> http://hg.python.org/cpython/rev/e8a1b5757067 changeset: 81890:e8a1b5757067 branch: 3.2 parent: 81877:81f7bdf7bbb6 user: Ned Deily date: Fri Feb 01 13:59:42 2013 -0800 summary: Issue #16256: OS X installer now sets correct permissions for doc directory. files: Mac/BuildScript/scripts/postflight.documentation | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Mac/BuildScript/scripts/postflight.documentation b/Mac/BuildScript/scripts/postflight.documentation --- a/Mac/BuildScript/scripts/postflight.documentation +++ b/Mac/BuildScript/scripts/postflight.documentation @@ -16,7 +16,7 @@ # make share/doc link in framework for command line users if [ -d "${SHARE_DIR}" ]; then - mkdir -p "${SHARE_DOCDIR}" + mkdir -m 775 -p "${SHARE_DOCDIR}" # make relative link to html doc directory ln -fhs "${SHARE_DOCDIR_TO_FWK}/${FWK_DOCDIR_SUBPATH}" "${SHARE_DOCDIR}/html" fi diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -958,6 +958,8 @@ - Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. +- Issue #16256: OS X installer now sets correct permissions for doc directory. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 23:06:54 2013 From: python-checkins at python.org (ned.deily) Date: Fri, 1 Feb 2013 23:06:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2316256=3A_merge_from_3=2E2?= Message-ID: <3YyXTp3YLdzSTZ@mail.python.org> http://hg.python.org/cpython/rev/1db5ed6a2dc2 changeset: 81891:1db5ed6a2dc2 branch: 3.3 parent: 81887:4d9bcf328e64 parent: 81890:e8a1b5757067 user: Ned Deily date: Fri Feb 01 14:05:26 2013 -0800 summary: Issue #16256: merge from 3.2 files: Mac/BuildScript/scripts/postflight.documentation | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Mac/BuildScript/scripts/postflight.documentation b/Mac/BuildScript/scripts/postflight.documentation --- a/Mac/BuildScript/scripts/postflight.documentation +++ b/Mac/BuildScript/scripts/postflight.documentation @@ -16,7 +16,7 @@ # make share/doc link in framework for command line users if [ -d "${SHARE_DIR}" ]; then - mkdir -p "${SHARE_DOCDIR}" + mkdir -m 775 -p "${SHARE_DOCDIR}" # make relative link to html doc directory ln -fhs "${SHARE_DOCDIR_TO_FWK}/${FWK_DOCDIR_SUBPATH}" "${SHARE_DOCDIR}/html" fi diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1313,6 +1313,8 @@ - Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. +- Issue #16256: OS X installer now sets correct permissions for doc directory. + - Issue #15431: Add _freeze_importlib project to regenerate importlib.h on Windows. Patch by Kristj?n Valur J?nsson. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 1 23:06:55 2013 From: python-checkins at python.org (ned.deily) Date: Fri, 1 Feb 2013 23:06:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316256=3A_merge_from_3=2E3?= Message-ID: <3YyXTq70CFzSSF@mail.python.org> http://hg.python.org/cpython/rev/bc2c40e84b58 changeset: 81892:bc2c40e84b58 parent: 81888:3029319d2d8a parent: 81891:1db5ed6a2dc2 user: Ned Deily date: Fri Feb 01 14:06:24 2013 -0800 summary: Issue #16256: merge from 3.3 files: Mac/BuildScript/scripts/postflight.documentation | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Mac/BuildScript/scripts/postflight.documentation b/Mac/BuildScript/scripts/postflight.documentation --- a/Mac/BuildScript/scripts/postflight.documentation +++ b/Mac/BuildScript/scripts/postflight.documentation @@ -16,7 +16,7 @@ # make share/doc link in framework for command line users if [ -d "${SHARE_DIR}" ]; then - mkdir -p "${SHARE_DOCDIR}" + mkdir -m 775 -p "${SHARE_DOCDIR}" # make relative link to html doc directory ln -fhs "${SHARE_DOCDIR_TO_FWK}/${FWK_DOCDIR_SUBPATH}" "${SHARE_DOCDIR}/html" fi diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1558,6 +1558,8 @@ - Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. +- Issue #16256: OS X installer now sets correct permissions for doc directory. + - Issue #15431: Add _freeze_importlib project to regenerate importlib.h on Windows. Patch by Kristj?n Valur J?nsson. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 01:15:55 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 01:15:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Silence_a_-Wfo?= =?utf-8?q?rmat-extra-argument_warning_when_compiling=2E?= Message-ID: <3YybLg59DDzM75@mail.python.org> http://hg.python.org/cpython/rev/59acde449f8d changeset: 81893:59acde449f8d branch: 2.7 parent: 81889:d64e0cf5f1a7 user: Gregory P. Smith date: Fri Feb 01 16:13:27 2013 -0800 summary: Silence a -Wformat-extra-argument warning when compiling. files: Objects/weakrefobject.c | 22 +++++++++++++++------- 1 files changed, 15 insertions(+), 7 deletions(-) diff --git a/Objects/weakrefobject.c b/Objects/weakrefobject.c --- a/Objects/weakrefobject.c +++ b/Objects/weakrefobject.c @@ -167,13 +167,21 @@ PyErr_Clear(); else if (PyString_Check(nameobj)) name = PyString_AS_STRING(nameobj); - PyOS_snprintf(buffer, sizeof(buffer), - name ? "" - : "", - self, - Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name, - PyWeakref_GET_OBJECT(self), - name); + if (name != NULL) { + PyOS_snprintf(buffer, sizeof(buffer), + "", + self, + Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name, + PyWeakref_GET_OBJECT(self), + name); + } + else { + PyOS_snprintf(buffer, sizeof(buffer), + "", + self, + Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name, + PyWeakref_GET_OBJECT(self)); + } Py_XDECREF(nameobj); } return PyString_FromString(buffer); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 01:15:57 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 01:15:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Silence_a_-Wfo?= =?utf-8?q?rmat-extra-argument_warning_when_compiling=2E?= Message-ID: <3YybLj0lw2zM75@mail.python.org> http://hg.python.org/cpython/rev/11047da3e5f6 changeset: 81894:11047da3e5f6 branch: 3.2 parent: 81890:e8a1b5757067 user: Gregory P. Smith date: Fri Feb 01 16:14:00 2013 -0800 summary: Silence a -Wformat-extra-argument warning when compiling. files: Objects/weakrefobject.c | 22 +++++++++++++++------- 1 files changed, 15 insertions(+), 7 deletions(-) diff --git a/Objects/weakrefobject.c b/Objects/weakrefobject.c --- a/Objects/weakrefobject.c +++ b/Objects/weakrefobject.c @@ -167,13 +167,21 @@ PyErr_Clear(); else if (PyUnicode_Check(nameobj)) name = _PyUnicode_AsString(nameobj); - PyOS_snprintf(buffer, sizeof(buffer), - name ? "" - : "", - self, - Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name, - PyWeakref_GET_OBJECT(self), - name); + if (name != NULL) { + PyOS_snprintf(buffer, sizeof(buffer), + "", + self, + Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name, + PyWeakref_GET_OBJECT(self), + name); + } + else { + PyOS_snprintf(buffer, sizeof(buffer), + "", + self, + Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name, + PyWeakref_GET_OBJECT(self)); + } Py_XDECREF(nameobj); } return PyUnicode_FromString(buffer); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 01:15:58 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 01:15:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_null_merge_from_3=2E2_=283=2E3=27s_code_already_refactored=29?= Message-ID: <3YybLk3X9jzQ0F@mail.python.org> http://hg.python.org/cpython/rev/6a4eb999d4b2 changeset: 81895:6a4eb999d4b2 branch: 3.3 parent: 81891:1db5ed6a2dc2 parent: 81894:11047da3e5f6 user: Gregory P. Smith date: Fri Feb 01 16:15:01 2013 -0800 summary: null merge from 3.2 (3.3's code already refactored) files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 01:15:59 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 01:15:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_null_merge_from_3=2E3?= Message-ID: <3YybLl6BzpzQ1C@mail.python.org> http://hg.python.org/cpython/rev/90f78d138dc9 changeset: 81896:90f78d138dc9 parent: 81892:bc2c40e84b58 parent: 81895:6a4eb999d4b2 user: Gregory P. Smith date: Fri Feb 01 16:15:45 2013 -0800 summary: null merge from 3.3 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 02:10:27 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 02:10:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSW4gdGhlIF9oYXNo?= =?utf-8?q?lib_module=2C_only_initialize_the_static_data_for_OpenSSL=27s?= Message-ID: <3YycYb0b18zPx0@mail.python.org> http://hg.python.org/cpython/rev/09fc7f466bd4 changeset: 81897:09fc7f466bd4 branch: 2.7 parent: 81893:59acde449f8d user: Gregory P. Smith date: Fri Feb 01 17:00:14 2013 -0800 summary: In the _hashlib module, only initialize the static data for OpenSSL's constructors once, to avoid memory leaks when finalizing and re-initializing the Python interpreter. files: Modules/_hashopenssl.c | 13 ++++++++----- 1 files changed, 8 insertions(+), 5 deletions(-) diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -67,7 +67,7 @@ #define DEFINE_CONSTS_FOR_NEW(Name) \ - static PyObject *CONST_ ## Name ## _name_obj; \ + static PyObject *CONST_ ## Name ## _name_obj = NULL; \ static EVP_MD_CTX CONST_new_ ## Name ## _ctx; \ static EVP_MD_CTX *CONST_new_ ## Name ## _ctx_p = NULL; @@ -525,12 +525,15 @@ " hash object; optionally initialized with a string") \ } -/* used in the init function to setup a constructor */ +/* used in the init function to setup a constructor: initialize OpenSSL + constructor constants if they haven't been initialized already. */ #define INIT_CONSTRUCTOR_CONSTANTS(NAME) do { \ + if (CONST_ ## NAME ## _name_obj == NULL) { \ CONST_ ## NAME ## _name_obj = PyString_FromString(#NAME); \ - if (EVP_get_digestbyname(#NAME)) { \ - CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ - EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + if (EVP_get_digestbyname(#NAME)) { \ + CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ + EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + } \ } \ } while (0); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 02:10:28 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 02:10:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSW4gdGhlIF9oYXNo?= =?utf-8?q?lib_module=2C_only_initialize_the_static_data_for_OpenSSL=27s?= Message-ID: <3YycYc398YzRC3@mail.python.org> http://hg.python.org/cpython/rev/b6792067aafa changeset: 81898:b6792067aafa branch: 3.2 parent: 81894:11047da3e5f6 user: Gregory P. Smith date: Fri Feb 01 17:05:29 2013 -0800 summary: In the _hashlib module, only initialize the static data for OpenSSL's constructors once, to avoid memory leaks when finalizing and re-initializing the Python interpreter. files: Modules/_hashopenssl.c | 15 +++++++++------ 1 files changed, 9 insertions(+), 6 deletions(-) diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -70,7 +70,7 @@ #define DEFINE_CONSTS_FOR_NEW(Name) \ - static PyObject *CONST_ ## Name ## _name_obj; \ + static PyObject *CONST_ ## Name ## _name_obj = NULL; \ static EVP_MD_CTX CONST_new_ ## Name ## _ctx; \ static EVP_MD_CTX *CONST_new_ ## Name ## _ctx_p = NULL; @@ -587,12 +587,15 @@ " hash object; optionally initialized with a string") \ } -/* used in the init function to setup a constructor */ +/* used in the init function to setup a constructor: initialize OpenSSL + constructor constants if they haven't been initialized already. */ #define INIT_CONSTRUCTOR_CONSTANTS(NAME) do { \ - CONST_ ## NAME ## _name_obj = PyUnicode_FromString(#NAME); \ - if (EVP_get_digestbyname(#NAME)) { \ - CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ - EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + if (CONST_ ## NAME ## _name_obj == NULL) { \ + CONST_ ## NAME ## _name_obj = PyUnicode_FromString(#NAME); \ + if (EVP_get_digestbyname(#NAME)) { \ + CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ + EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + } \ } \ } while (0); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 02:10:29 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 02:10:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_In_the_=5Fhashlib_module=2C_only_initialize_the_static_data_fo?= =?utf-8?q?r_OpenSSL=27s?= Message-ID: <3YycYd5xjgzRGB@mail.python.org> http://hg.python.org/cpython/rev/80499534179b changeset: 81899:80499534179b branch: 3.3 parent: 81895:6a4eb999d4b2 parent: 81898:b6792067aafa user: Gregory P. Smith date: Fri Feb 01 17:07:39 2013 -0800 summary: In the _hashlib module, only initialize the static data for OpenSSL's constructors once, to avoid memory leaks when finalizing and re-initializing the Python interpreter. files: Modules/_hashopenssl.c | 15 +++++++++------ 1 files changed, 9 insertions(+), 6 deletions(-) diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -70,7 +70,7 @@ #define DEFINE_CONSTS_FOR_NEW(Name) \ - static PyObject *CONST_ ## Name ## _name_obj; \ + static PyObject *CONST_ ## Name ## _name_obj = NULL; \ static EVP_MD_CTX CONST_new_ ## Name ## _ctx; \ static EVP_MD_CTX *CONST_new_ ## Name ## _ctx_p = NULL; @@ -585,12 +585,15 @@ " hash object; optionally initialized with a string") \ } -/* used in the init function to setup a constructor */ +/* used in the init function to setup a constructor: initialize OpenSSL + constructor constants if they haven't been initialized already. */ #define INIT_CONSTRUCTOR_CONSTANTS(NAME) do { \ - CONST_ ## NAME ## _name_obj = PyUnicode_FromString(#NAME); \ - if (EVP_get_digestbyname(#NAME)) { \ - CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ - EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + if (CONST_ ## NAME ## _name_obj == NULL) { \ + CONST_ ## NAME ## _name_obj = PyUnicode_FromString(#NAME); \ + if (EVP_get_digestbyname(#NAME)) { \ + CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ + EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + } \ } \ } while (0); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 2 02:10:31 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sat, 2 Feb 2013 02:10:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_In_the_=5Fhashlib_module=2C_only_initialize_the_static_d?= =?utf-8?q?ata_for_OpenSSL=27s?= Message-ID: <3YycYg1PD4zRC0@mail.python.org> http://hg.python.org/cpython/rev/0cc290da01b2 changeset: 81900:0cc290da01b2 parent: 81896:90f78d138dc9 parent: 81899:80499534179b user: Gregory P. Smith date: Fri Feb 01 17:07:56 2013 -0800 summary: In the _hashlib module, only initialize the static data for OpenSSL's constructors once, to avoid memory leaks when finalizing and re-initializing the Python interpreter. files: Modules/_hashopenssl.c | 15 +++++++++------ 1 files changed, 9 insertions(+), 6 deletions(-) diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -48,7 +48,7 @@ #define DEFINE_CONSTS_FOR_NEW(Name) \ - static PyObject *CONST_ ## Name ## _name_obj; \ + static PyObject *CONST_ ## Name ## _name_obj = NULL; \ static EVP_MD_CTX CONST_new_ ## Name ## _ctx; \ static EVP_MD_CTX *CONST_new_ ## Name ## _ctx_p = NULL; @@ -563,12 +563,15 @@ " hash object; optionally initialized with a string") \ } -/* used in the init function to setup a constructor */ +/* used in the init function to setup a constructor: initialize OpenSSL + constructor constants if they haven't been initialized already. */ #define INIT_CONSTRUCTOR_CONSTANTS(NAME) do { \ - CONST_ ## NAME ## _name_obj = PyUnicode_FromString(#NAME); \ - if (EVP_get_digestbyname(#NAME)) { \ - CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ - EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + if (CONST_ ## NAME ## _name_obj == NULL) { \ + CONST_ ## NAME ## _name_obj = PyUnicode_FromString(#NAME); \ + if (EVP_get_digestbyname(#NAME)) { \ + CONST_new_ ## NAME ## _ctx_p = &CONST_new_ ## NAME ## _ctx; \ + EVP_DigestInit(CONST_new_ ## NAME ## _ctx_p, EVP_get_digestbyname(#NAME)); \ + } \ } \ } while (0); -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Feb 2 06:03:26 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 02 Feb 2013 06:03:26 +0100 Subject: [Python-checkins] Daily reference leaks (0cc290da01b2): sum=2 Message-ID: results for 0cc290da01b2 on branch "default" -------------------------------------------- test_support leaked [0, -1, 1] references, sum=0 test_support leaked [0, -1, 3] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog_CTvvs', '-x'] From python-checkins at python.org Sat Feb 2 08:18:15 2013 From: python-checkins at python.org (ned.deily) Date: Sat, 2 Feb 2013 08:18:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE1NTg3?= =?utf-8?q?=3A_Enable_Tk_high-resolution_text_rendering_on_Macs_with?= Message-ID: <3Yymjz0CvJzPrk@mail.python.org> http://hg.python.org/cpython/rev/2274f3196a44 changeset: 81901:2274f3196a44 branch: 2.7 parent: 81897:09fc7f466bd4 user: Ned Deily date: Fri Feb 01 23:10:56 2013 -0800 summary: Issue #15587: Enable Tk high-resolution text rendering on Macs with Retina displays. Applies to Tkinter apps, such as IDLE, on OS X framework builds linked with Cocoa Tk 8.5+. Suggested by Kevin Walzer files: Mac/IDLE/Info.plist.in | 2 ++ Mac/Resources/app/Info.plist.in | 2 ++ Misc/NEWS | 4 ++++ 3 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Mac/IDLE/Info.plist.in b/Mac/IDLE/Info.plist.in --- a/Mac/IDLE/Info.plist.in +++ b/Mac/IDLE/Info.plist.in @@ -51,6 +51,8 @@ %VERSION% CFBundleVersion %VERSION% + NSHighResolutionCapable + 45 ", 1) + self.parser.Parse(b"12345 ", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "3", "", "4", "", "5", ""], @@ -400,7 +411,7 @@ parser = expat.ParserCreate() parser.StartElementHandler = self.StartElementHandler try: - parser.Parse("", 1) + parser.Parse(b"", 1) self.fail() except RuntimeError as e: self.assertEqual(e.args[0], 'a', @@ -436,7 +447,7 @@ self.expected_list = [('s', 0, 1, 0), ('s', 5, 2, 1), ('s', 11, 3, 2), ('e', 15, 3, 6), ('e', 17, 4, 1), ('e', 22, 5, 0)] - xml = '\n \n \n \n' + xml = b'\n \n \n \n' self.parser.Parse(xml, 1) @@ -457,7 +468,7 @@ parser = expat.ParserCreate() parser.CharacterDataHandler = handler - self.assertRaises(Exception, parser.Parse, xml) + self.assertRaises(Exception, parser.Parse, xml.encode('iso8859')) class ChardataBufferTest(unittest.TestCase): """ @@ -480,8 +491,8 @@ self.assertRaises(ValueError, f, 0) def test_unchanged_size(self): - xml1 = ("%s" % ('a' * 512)) - xml2 = 'a'*512 + '' + xml1 = b"" + b'a' * 512 + xml2 = b'a'*512 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_size = 512 @@ -503,9 +514,9 @@ def test_disabling_buffer(self): - xml1 = "%s" % ('a' * 512) - xml2 = ('b' * 1024) - xml3 = "%s" % ('c' * 1024) + xml1 = b"" + b'a' * 512 + xml2 = b'b' * 1024 + xml3 = b'c' * 1024 + b''; parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -532,16 +543,11 @@ parser.Parse(xml3, 1) self.assertEqual(self.n, 12) - - - def make_document(self, bytes): - return ("" + bytes * 'a' + '') - def counting_handler(self, text): self.n += 1 def small_buffer_test(self, buffer_len): - xml = "%s" % ('a' * buffer_len) + xml = b"" + b'a' * buffer_len + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_size = 1024 @@ -552,8 +558,8 @@ return self.n def test_change_size_1(self): - xml1 = "%s" % ('a' * 1024) - xml2 = "aaa%s" % ('a' * 1025) + xml1 = b"" + b'a' * 1024 + xml2 = b'aaa' + b'a' * 1025 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -568,8 +574,8 @@ self.assertEqual(self.n, 2) def test_change_size_2(self): - xml1 = "a%s" % ('a' * 1023) - xml2 = "aaa%s" % ('a' * 1025) + xml1 = b"a" + b'a' * 1023 + xml2 = b'aaa' + b'a' * 1025 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -585,7 +591,7 @@ class MalformedInputTest(unittest.TestCase): def test1(self): - xml = "\0\r\n" + xml = b"\0\r\n" parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -594,7 +600,8 @@ self.assertEqual(str(e), 'unclosed token: line 2, column 0') def test2(self): - xml = "\r\n" + #?\xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) + xml = b"\r\n" parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -609,7 +616,7 @@ errors.messages[errors.codes[errors.XML_ERROR_SYNTAX]]) def test_expaterror(self): - xml = '<' + xml = b'<' parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -638,7 +645,7 @@ parser.UseForeignDTD(True) parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity - parser.Parse("") + parser.Parse(b"") self.assertEqual(handler_call_args, [(None, None)]) # test UseForeignDTD() is equal to UseForeignDTD(True) @@ -648,7 +655,7 @@ parser.UseForeignDTD() parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity - parser.Parse("") + parser.Parse(b"") self.assertEqual(handler_call_args, [(None, None)]) def test_ignore_use_foreign_dtd(self): @@ -667,7 +674,7 @@ parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity parser.Parse( - "") + b"") self.assertEqual(handler_call_args, [("bar", "baz")]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -212,6 +212,10 @@ Library ------- +- Issue #17089: Expat parser now correctly works with string input not only when + an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and + strings larger than 2 GiB. + - Issue #16903: Popen.communicate() on Unix now accepts strings when universal_newlines is true as on Windows. diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -777,17 +777,52 @@ "Parse(data[, isfinal])\n\ Parse XML data. `isfinal' should be true at end of input."); +#define MAX_CHUNK_SIZE (1 << 20) + static PyObject * xmlparse_Parse(xmlparseobject *self, PyObject *args) { - char *s; - int slen; + PyObject *data; int isFinal = 0; + const char *s; + Py_ssize_t slen; + Py_buffer view; + int rc; - if (!PyArg_ParseTuple(args, "s#|i:Parse", &s, &slen, &isFinal)) + if (!PyArg_ParseTuple(args, "O|i:Parse", &data, &isFinal)) return NULL; - return get_parse_result(self, XML_Parse(self->itself, s, slen, isFinal)); + if (PyUnicode_Check(data)) { + PyObject *bytes; + bytes = PyUnicode_AsUTF8String(data); + if (bytes == NULL) + return NULL; + view.buf = NULL; + s = PyBytes_AS_STRING(bytes); + slen = PyBytes_GET_SIZE(bytes); + /* Explicitly set UTF-8 encoding. Return code ignored. */ + (void)XML_SetEncoding(self->itself, "utf-8"); + } + else { + if (PyObject_GetBuffer(data, &view, PyBUF_SIMPLE) < 0) + return NULL; + s = view.buf; + slen = view.len; + } + + while (slen > MAX_CHUNK_SIZE) { + rc = XML_Parse(self->itself, s, MAX_CHUNK_SIZE, 0); + if (!rc) + goto done; + s += MAX_CHUNK_SIZE; + slen -= MAX_CHUNK_SIZE; + } + rc = XML_Parse(self->itself, s, slen, isFinal); + +done: + if (view.buf != NULL) + PyBuffer_Release(&view); + return get_parse_result(self, rc); } /* File reading copied from cPickle */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Feb 4 17:32:56 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 4 Feb 2013 17:32:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2317089=3A_Expat_parser_now_correctly_works_with_string?= =?utf-8?q?_input_not_only_when?= Message-ID: <3Z0Dx43G8PzSdw@mail.python.org> http://hg.python.org/cpython/rev/6c27b0e09c43 changeset: 82008:6c27b0e09c43 branch: 3.3 parent: 82004:b414b2dfd3d3 parent: 82007:3cc2a2de36e3 user: Serhiy Storchaka date: Mon Feb 04 18:28:01 2013 +0200 summary: Issue #17089: Expat parser now correctly works with string input not only when an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and strings larger than 2 GiB. files: Lib/test/test_pyexpat.py | 79 +++++++++++++++------------ Misc/NEWS | 4 + Modules/pyexpat.c | 40 ++++++++++++- 3 files changed, 83 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -52,6 +52,7 @@ &external_entity; &skipped_entity; +\xb5 ''' @@ -195,13 +196,13 @@ "End element: 'sub2'", "External entity ref: (None, 'entity.file', None)", ('Skipped entity', ('skipped_entity', 0)), + "Character data: '\xb5'", "End element: 'root'", ] for operation, expected_operation in zip(operations, expected_operations): self.assertEqual(operation, expected_operation) - def test_unicode(self): - # Try the parse again, this time producing Unicode output + def test_parse_bytes(self): out = self.Outputter() parser = expat.ParserCreate(namespace_separator='!') self._hookup_callbacks(parser, out) @@ -213,6 +214,16 @@ # Issue #6697. self.assertRaises(AttributeError, getattr, parser, '\uD800') + def test_parse_str(self): + out = self.Outputter() + parser = expat.ParserCreate(namespace_separator='!') + self._hookup_callbacks(parser, out) + + parser.Parse(data.decode('iso-8859-1'), 1) + + operations = out.out + self._verify_parse_output(operations) + def test_parse_file(self): # Try parsing a file out = self.Outputter() @@ -269,7 +280,7 @@ L.append(name) p.StartElementHandler = collector p.EndElementHandler = collector - p.Parse(" ", 1) + p.Parse(b" ", 1) tag = L[0] self.assertEqual(len(L), 6) for entry in L: @@ -285,7 +296,7 @@ def ExternalEntityRefHandler(self, context, base, sysId, pubId): external_parser = self.parser.ExternalEntityParserCreate("") - self.parser_result = external_parser.Parse("", 1) + self.parser_result = external_parser.Parse(b"", 1) return 1 parser = expat.ParserCreate(namespace_separator='!') @@ -336,7 +347,7 @@ def test_buffering_enabled(self): # Make sure buffering is turned on self.assertTrue(self.parser.buffer_text) - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ['123'], "buffered text not properly collapsed") @@ -344,39 +355,39 @@ # XXX This test exposes more detail of Expat's text chunking than we # XXX like, but it tests what we need to concisely. self.setHandlers(["StartElementHandler"]) - self.parser.Parse("12\n34\n5", 1) + self.parser.Parse(b"12\n34\n5", 1) self.assertEqual(self.stuff, ["", "1", "", "2", "\n", "3", "", "4\n5"], "buffering control not reacting as expected") def test2(self): - self.parser.Parse("1<2> \n 3", 1) + self.parser.Parse(b"1<2> \n 3", 1) self.assertEqual(self.stuff, ["1<2> \n 3"], "buffered text not properly collapsed") def test3(self): self.setHandlers(["StartElementHandler"]) - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ["", "1", "", "2", "", "3"], "buffered text not properly split") def test4(self): self.setHandlers(["StartElementHandler", "EndElementHandler"]) self.parser.CharacterDataHandler = None - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ["", "", "", "", "", ""]) def test5(self): self.setHandlers(["StartElementHandler", "EndElementHandler"]) - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "3", ""]) def test6(self): self.setHandlers(["CommentHandler", "EndElementHandler", "StartElementHandler"]) - self.parser.Parse("12345 ", 1) + self.parser.Parse(b"12345 ", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "345", ""], "buffered text not properly split") @@ -384,7 +395,7 @@ def test7(self): self.setHandlers(["CommentHandler", "EndElementHandler", "StartElementHandler"]) - self.parser.Parse("12345 ", 1) + self.parser.Parse(b"12345 ", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "3", "", "4", "", "5", ""], @@ -400,7 +411,7 @@ parser = expat.ParserCreate() parser.StartElementHandler = self.StartElementHandler try: - parser.Parse("", 1) + parser.Parse(b"", 1) self.fail() except RuntimeError as e: self.assertEqual(e.args[0], 'a', @@ -436,7 +447,7 @@ self.expected_list = [('s', 0, 1, 0), ('s', 5, 2, 1), ('s', 11, 3, 2), ('e', 15, 3, 6), ('e', 17, 4, 1), ('e', 22, 5, 0)] - xml = '\n \n \n \n' + xml = b'\n \n \n \n' self.parser.Parse(xml, 1) @@ -457,7 +468,7 @@ parser = expat.ParserCreate() parser.CharacterDataHandler = handler - self.assertRaises(Exception, parser.Parse, xml) + self.assertRaises(Exception, parser.Parse, xml.encode('iso8859')) class ChardataBufferTest(unittest.TestCase): """ @@ -480,8 +491,8 @@ self.assertRaises(ValueError, f, 0) def test_unchanged_size(self): - xml1 = ("%s" % ('a' * 512)) - xml2 = 'a'*512 + '' + xml1 = b"" + b'a' * 512 + xml2 = b'a'*512 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_size = 512 @@ -503,9 +514,9 @@ def test_disabling_buffer(self): - xml1 = "%s" % ('a' * 512) - xml2 = ('b' * 1024) - xml3 = "%s" % ('c' * 1024) + xml1 = b"" + b'a' * 512 + xml2 = b'b' * 1024 + xml3 = b'c' * 1024 + b''; parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -532,16 +543,11 @@ parser.Parse(xml3, 1) self.assertEqual(self.n, 12) - - - def make_document(self, bytes): - return ("" + bytes * 'a' + '') - def counting_handler(self, text): self.n += 1 def small_buffer_test(self, buffer_len): - xml = "%s" % ('a' * buffer_len) + xml = b"" + b'a' * buffer_len + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_size = 1024 @@ -552,8 +558,8 @@ return self.n def test_change_size_1(self): - xml1 = "%s" % ('a' * 1024) - xml2 = "aaa%s" % ('a' * 1025) + xml1 = b"" + b'a' * 1024 + xml2 = b'aaa' + b'a' * 1025 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -568,8 +574,8 @@ self.assertEqual(self.n, 2) def test_change_size_2(self): - xml1 = "a%s" % ('a' * 1023) - xml2 = "aaa%s" % ('a' * 1025) + xml1 = b"a" + b'a' * 1023 + xml2 = b'aaa' + b'a' * 1025 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -585,7 +591,7 @@ class MalformedInputTest(unittest.TestCase): def test1(self): - xml = "\0\r\n" + xml = b"\0\r\n" parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -594,7 +600,8 @@ self.assertEqual(str(e), 'unclosed token: line 2, column 0') def test2(self): - xml = "\r\n" + #?\xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) + xml = b"\r\n" parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -609,7 +616,7 @@ errors.messages[errors.codes[errors.XML_ERROR_SYNTAX]]) def test_expaterror(self): - xml = '<' + xml = b'<' parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -638,7 +645,7 @@ parser.UseForeignDTD(True) parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity - parser.Parse("") + parser.Parse(b"") self.assertEqual(handler_call_args, [(None, None)]) # test UseForeignDTD() is equal to UseForeignDTD(True) @@ -648,7 +655,7 @@ parser.UseForeignDTD() parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity - parser.Parse("") + parser.Parse(b"") self.assertEqual(handler_call_args, [(None, None)]) def test_ignore_use_foreign_dtd(self): @@ -667,7 +674,7 @@ parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity parser.Parse( - "") + b"") self.assertEqual(handler_call_args, [("bar", "baz")]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,10 @@ Library ------- +- Issue #17089: Expat parser now correctly works with string input not only when + an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and + strings larger than 2 GiB. + - Issue #6083: Fix multiple segmentation faults occured when PyArg_ParseTuple parses nested mutating sequence. diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -778,17 +778,49 @@ "Parse(data[, isfinal])\n\ Parse XML data. `isfinal' should be true at end of input."); +#define MAX_CHUNK_SIZE (1 << 20) + static PyObject * xmlparse_Parse(xmlparseobject *self, PyObject *args) { - char *s; - int slen; + PyObject *data; int isFinal = 0; + const char *s; + Py_ssize_t slen; + Py_buffer view; + int rc; - if (!PyArg_ParseTuple(args, "s#|i:Parse", &s, &slen, &isFinal)) + if (!PyArg_ParseTuple(args, "O|i:Parse", &data, &isFinal)) return NULL; - return get_parse_result(self, XML_Parse(self->itself, s, slen, isFinal)); + if (PyUnicode_Check(data)) { + view.buf = NULL; + s = PyUnicode_AsUTF8AndSize(data, &slen); + if (s == NULL) + return NULL; + /* Explicitly set UTF-8 encoding. Return code ignored. */ + (void)XML_SetEncoding(self->itself, "utf-8"); + } + else { + if (PyObject_GetBuffer(data, &view, PyBUF_SIMPLE) < 0) + return NULL; + s = view.buf; + slen = view.len; + } + + while (slen > MAX_CHUNK_SIZE) { + rc = XML_Parse(self->itself, s, MAX_CHUNK_SIZE, 0); + if (!rc) + goto done; + s += MAX_CHUNK_SIZE; + slen -= MAX_CHUNK_SIZE; + } + rc = XML_Parse(self->itself, s, slen, isFinal); + +done: + if (view.buf != NULL) + PyBuffer_Release(&view); + return get_parse_result(self, rc); } /* File reading copied from cPickle */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Feb 4 17:32:58 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 4 Feb 2013 17:32:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317089=3A_Expat_parser_now_correctly_works_with_?= =?utf-8?q?string_input_not_only_when?= Message-ID: <3Z0Dx60FBqzSdr@mail.python.org> http://hg.python.org/cpython/rev/c4e6e560e6f5 changeset: 82009:c4e6e560e6f5 parent: 82005:a80abb179ba1 parent: 82008:6c27b0e09c43 user: Serhiy Storchaka date: Mon Feb 04 18:29:47 2013 +0200 summary: Issue #17089: Expat parser now correctly works with string input not only when an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and strings larger than 2 GiB. files: Lib/test/test_pyexpat.py | 79 +++++++++++++++------------ Misc/NEWS | 4 + Modules/pyexpat.c | 40 ++++++++++++- 3 files changed, 83 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -52,6 +52,7 @@ &external_entity; &skipped_entity; +\xb5 ''' @@ -195,13 +196,13 @@ "End element: 'sub2'", "External entity ref: (None, 'entity.file', None)", ('Skipped entity', ('skipped_entity', 0)), + "Character data: '\xb5'", "End element: 'root'", ] for operation, expected_operation in zip(operations, expected_operations): self.assertEqual(operation, expected_operation) - def test_unicode(self): - # Try the parse again, this time producing Unicode output + def test_parse_bytes(self): out = self.Outputter() parser = expat.ParserCreate(namespace_separator='!') self._hookup_callbacks(parser, out) @@ -213,6 +214,16 @@ # Issue #6697. self.assertRaises(AttributeError, getattr, parser, '\uD800') + def test_parse_str(self): + out = self.Outputter() + parser = expat.ParserCreate(namespace_separator='!') + self._hookup_callbacks(parser, out) + + parser.Parse(data.decode('iso-8859-1'), 1) + + operations = out.out + self._verify_parse_output(operations) + def test_parse_file(self): # Try parsing a file out = self.Outputter() @@ -269,7 +280,7 @@ L.append(name) p.StartElementHandler = collector p.EndElementHandler = collector - p.Parse(" ", 1) + p.Parse(b" ", 1) tag = L[0] self.assertEqual(len(L), 6) for entry in L: @@ -285,7 +296,7 @@ def ExternalEntityRefHandler(self, context, base, sysId, pubId): external_parser = self.parser.ExternalEntityParserCreate("") - self.parser_result = external_parser.Parse("", 1) + self.parser_result = external_parser.Parse(b"", 1) return 1 parser = expat.ParserCreate(namespace_separator='!') @@ -336,7 +347,7 @@ def test_buffering_enabled(self): # Make sure buffering is turned on self.assertTrue(self.parser.buffer_text) - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ['123'], "buffered text not properly collapsed") @@ -344,39 +355,39 @@ # XXX This test exposes more detail of Expat's text chunking than we # XXX like, but it tests what we need to concisely. self.setHandlers(["StartElementHandler"]) - self.parser.Parse("12\n34\n5", 1) + self.parser.Parse(b"12\n34\n5", 1) self.assertEqual(self.stuff, ["", "1", "", "2", "\n", "3", "", "4\n5"], "buffering control not reacting as expected") def test2(self): - self.parser.Parse("1<2> \n 3", 1) + self.parser.Parse(b"1<2> \n 3", 1) self.assertEqual(self.stuff, ["1<2> \n 3"], "buffered text not properly collapsed") def test3(self): self.setHandlers(["StartElementHandler"]) - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ["", "1", "", "2", "", "3"], "buffered text not properly split") def test4(self): self.setHandlers(["StartElementHandler", "EndElementHandler"]) self.parser.CharacterDataHandler = None - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ["", "", "", "", "", ""]) def test5(self): self.setHandlers(["StartElementHandler", "EndElementHandler"]) - self.parser.Parse("123", 1) + self.parser.Parse(b"123", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "3", ""]) def test6(self): self.setHandlers(["CommentHandler", "EndElementHandler", "StartElementHandler"]) - self.parser.Parse("12345 ", 1) + self.parser.Parse(b"12345 ", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "345", ""], "buffered text not properly split") @@ -384,7 +395,7 @@ def test7(self): self.setHandlers(["CommentHandler", "EndElementHandler", "StartElementHandler"]) - self.parser.Parse("12345 ", 1) + self.parser.Parse(b"12345 ", 1) self.assertEqual(self.stuff, ["", "1", "", "", "2", "", "", "3", "", "4", "", "5", ""], @@ -400,7 +411,7 @@ parser = expat.ParserCreate() parser.StartElementHandler = self.StartElementHandler try: - parser.Parse("", 1) + parser.Parse(b"", 1) self.fail() except RuntimeError as e: self.assertEqual(e.args[0], 'a', @@ -436,7 +447,7 @@ self.expected_list = [('s', 0, 1, 0), ('s', 5, 2, 1), ('s', 11, 3, 2), ('e', 15, 3, 6), ('e', 17, 4, 1), ('e', 22, 5, 0)] - xml = '\n \n \n \n' + xml = b'\n \n \n \n' self.parser.Parse(xml, 1) @@ -457,7 +468,7 @@ parser = expat.ParserCreate() parser.CharacterDataHandler = handler - self.assertRaises(Exception, parser.Parse, xml) + self.assertRaises(Exception, parser.Parse, xml.encode('iso8859')) class ChardataBufferTest(unittest.TestCase): """ @@ -480,8 +491,8 @@ self.assertRaises(ValueError, f, 0) def test_unchanged_size(self): - xml1 = ("%s" % ('a' * 512)) - xml2 = 'a'*512 + '' + xml1 = b"" + b'a' * 512 + xml2 = b'a'*512 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_size = 512 @@ -503,9 +514,9 @@ def test_disabling_buffer(self): - xml1 = "%s" % ('a' * 512) - xml2 = ('b' * 1024) - xml3 = "%s" % ('c' * 1024) + xml1 = b"" + b'a' * 512 + xml2 = b'b' * 1024 + xml3 = b'c' * 1024 + b''; parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -532,16 +543,11 @@ parser.Parse(xml3, 1) self.assertEqual(self.n, 12) - - - def make_document(self, bytes): - return ("" + bytes * 'a' + '') - def counting_handler(self, text): self.n += 1 def small_buffer_test(self, buffer_len): - xml = "%s" % ('a' * buffer_len) + xml = b"" + b'a' * buffer_len + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_size = 1024 @@ -552,8 +558,8 @@ return self.n def test_change_size_1(self): - xml1 = "%s" % ('a' * 1024) - xml2 = "aaa%s" % ('a' * 1025) + xml1 = b"" + b'a' * 1024 + xml2 = b'aaa' + b'a' * 1025 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -568,8 +574,8 @@ self.assertEqual(self.n, 2) def test_change_size_2(self): - xml1 = "a%s" % ('a' * 1023) - xml2 = "aaa%s" % ('a' * 1025) + xml1 = b"a" + b'a' * 1023 + xml2 = b'aaa' + b'a' * 1025 + b'' parser = expat.ParserCreate() parser.CharacterDataHandler = self.counting_handler parser.buffer_text = 1 @@ -585,7 +591,7 @@ class MalformedInputTest(unittest.TestCase): def test1(self): - xml = "\0\r\n" + xml = b"\0\r\n" parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -594,7 +600,8 @@ self.assertEqual(str(e), 'unclosed token: line 2, column 0') def test2(self): - xml = "\r\n" + #?\xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) + xml = b"\r\n" parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -609,7 +616,7 @@ errors.messages[errors.codes[errors.XML_ERROR_SYNTAX]]) def test_expaterror(self): - xml = '<' + xml = b'<' parser = expat.ParserCreate() try: parser.Parse(xml, True) @@ -638,7 +645,7 @@ parser.UseForeignDTD(True) parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity - parser.Parse("") + parser.Parse(b"") self.assertEqual(handler_call_args, [(None, None)]) # test UseForeignDTD() is equal to UseForeignDTD(True) @@ -648,7 +655,7 @@ parser.UseForeignDTD() parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity - parser.Parse("") + parser.Parse(b"") self.assertEqual(handler_call_args, [(None, None)]) def test_ignore_use_foreign_dtd(self): @@ -667,7 +674,7 @@ parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS) parser.ExternalEntityRefHandler = resolve_entity parser.Parse( - "") + b"") self.assertEqual(handler_call_args, [("bar", "baz")]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,10 @@ Library ------- +- Issue #17089: Expat parser now correctly works with string input not only when + an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and + strings larger than 2 GiB. + - Issue #6083: Fix multiple segmentation faults occured when PyArg_ParseTuple parses nested mutating sequence. diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -778,17 +778,49 @@ "Parse(data[, isfinal])\n\ Parse XML data. `isfinal' should be true at end of input."); +#define MAX_CHUNK_SIZE (1 << 20) + static PyObject * xmlparse_Parse(xmlparseobject *self, PyObject *args) { - char *s; - int slen; + PyObject *data; int isFinal = 0; + const char *s; + Py_ssize_t slen; + Py_buffer view; + int rc; - if (!PyArg_ParseTuple(args, "s#|i:Parse", &s, &slen, &isFinal)) + if (!PyArg_ParseTuple(args, "O|i:Parse", &data, &isFinal)) return NULL; - return get_parse_result(self, XML_Parse(self->itself, s, slen, isFinal)); + if (PyUnicode_Check(data)) { + view.buf = NULL; + s = PyUnicode_AsUTF8AndSize(data, &slen); + if (s == NULL) + return NULL; + /* Explicitly set UTF-8 encoding. Return code ignored. */ + (void)XML_SetEncoding(self->itself, "utf-8"); + } + else { + if (PyObject_GetBuffer(data, &view, PyBUF_SIMPLE) < 0) + return NULL; + s = view.buf; + slen = view.len; + } + + while (slen > MAX_CHUNK_SIZE) { + rc = XML_Parse(self->itself, s, MAX_CHUNK_SIZE, 0); + if (!rc) + goto done; + s += MAX_CHUNK_SIZE; + slen -= MAX_CHUNK_SIZE; + } + rc = XML_Parse(self->itself, s, slen, isFinal); + +done: + if (view.buf != NULL) + PyBuffer_Release(&view); + return get_parse_result(self, rc); } /* File reading copied from cPickle */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Feb 4 21:25:23 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 4 Feb 2013 21:25:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE2ODExOiBGaXgg?= =?utf-8?q?folding_of_headers_with_no_value_in_provisional_policies=2E?= Message-ID: <3Z0L5H6HyBzSj3@mail.python.org> http://hg.python.org/cpython/rev/e64b74227198 changeset: 82010:e64b74227198 branch: 3.3 parent: 82008:6c27b0e09c43 user: R David Murray date: Mon Feb 04 15:22:53 2013 -0500 summary: #16811: Fix folding of headers with no value in provisional policies. files: Lib/email/policy.py | 2 +- Lib/test/test_email/test_inversion.py | 45 +++++++++++++++ 2 files changed, 46 insertions(+), 1 deletions(-) diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -173,7 +173,7 @@ lines = value.splitlines() refold = (self.refold_source == 'all' or self.refold_source == 'long' and - (len(lines[0])+len(name)+2 > maxlen or + (lines and len(lines[0])+len(name)+2 > maxlen or any(len(x) > maxlen for x in lines[1:]))) if refold or refold_binary and _has_surrogates(value): return self.header_factory(name, ''.join(lines)).fold(policy=self) diff --git a/Lib/test/test_email/test_inversion.py b/Lib/test/test_email/test_inversion.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test_inversion.py @@ -0,0 +1,45 @@ +"""Test the parser and generator are inverses. + +Note that this is only strictly true if we are parsing RFC valid messages and +producing RFC valid messages. +""" + +import io +import unittest +from email import policy, message_from_bytes +from email.generator import BytesGenerator +from test.test_email import TestEmailBase, parameterize + +# This is like textwrap.dedent for bytes, except that it uses \r\n for the line +# separators on the rebuilt string. +def dedent(bstr): + lines = bstr.splitlines() + if not lines[0].strip(): + raise ValueError("First line must contain text") + stripamt = len(lines[0]) - len(lines[0].lstrip()) + return b'\r\n'.join( + [x[stripamt:] if len(x)>=stripamt else b'' + for x in lines]) + + + at parameterize +class TestInversion(TestEmailBase, unittest.TestCase): + + def msg_as_input(self, msg): + m = message_from_bytes(msg, policy=policy.SMTP) + b = io.BytesIO() + g = BytesGenerator(b) + g.flatten(m) + self.assertEqual(b.getvalue(), msg) + + # XXX: spaces are not preserved correctly here yet in the general case. + msg_params = { + 'header_with_one_space_body': (dedent(b"""\ + From: abc at xyz.com + X-Status:\x20 + Subject: test + + foo + """),), + + } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Feb 4 21:25:25 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 4 Feb 2013 21:25:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_=2316811=3A_Fix_folding_of_headers_with_no_value_i?= =?utf-8?q?n_provisional_policies=2E?= Message-ID: <3Z0L5K1t1RzSjb@mail.python.org> http://hg.python.org/cpython/rev/fe7f3e2e49ce changeset: 82011:fe7f3e2e49ce parent: 82009:c4e6e560e6f5 parent: 82010:e64b74227198 user: R David Murray date: Mon Feb 04 15:25:06 2013 -0500 summary: Merge #16811: Fix folding of headers with no value in provisional policies. files: Lib/email/policy.py | 2 +- Lib/test/test_email/test_inversion.py | 45 +++++++++++++++ 2 files changed, 46 insertions(+), 1 deletions(-) diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -173,7 +173,7 @@ lines = value.splitlines() refold = (self.refold_source == 'all' or self.refold_source == 'long' and - (len(lines[0])+len(name)+2 > maxlen or + (lines and len(lines[0])+len(name)+2 > maxlen or any(len(x) > maxlen for x in lines[1:]))) if refold or refold_binary and _has_surrogates(value): return self.header_factory(name, ''.join(lines)).fold(policy=self) diff --git a/Lib/test/test_email/test_inversion.py b/Lib/test/test_email/test_inversion.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test_inversion.py @@ -0,0 +1,45 @@ +"""Test the parser and generator are inverses. + +Note that this is only strictly true if we are parsing RFC valid messages and +producing RFC valid messages. +""" + +import io +import unittest +from email import policy, message_from_bytes +from email.generator import BytesGenerator +from test.test_email import TestEmailBase, parameterize + +# This is like textwrap.dedent for bytes, except that it uses \r\n for the line +# separators on the rebuilt string. +def dedent(bstr): + lines = bstr.splitlines() + if not lines[0].strip(): + raise ValueError("First line must contain text") + stripamt = len(lines[0]) - len(lines[0].lstrip()) + return b'\r\n'.join( + [x[stripamt:] if len(x)>=stripamt else b'' + for x in lines]) + + + at parameterize +class TestInversion(TestEmailBase, unittest.TestCase): + + def msg_as_input(self, msg): + m = message_from_bytes(msg, policy=policy.SMTP) + b = io.BytesIO() + g = BytesGenerator(b) + g.flatten(m) + self.assertEqual(b.getvalue(), msg) + + # XXX: spaces are not preserved correctly here yet in the general case. + msg_params = { + 'header_with_one_space_body': (dedent(b"""\ + From: abc at xyz.com + X-Status:\x20 + Subject: test + + foo + """),), + + } -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Feb 5 06:01:49 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 05 Feb 2013 06:01:49 +0100 Subject: [Python-checkins] Daily reference leaks (fe7f3e2e49ce): sum=0 Message-ID: results for fe7f3e2e49ce on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogu36eg3', '-x'] From python-checkins at python.org Tue Feb 5 07:30:56 2013 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 5 Feb 2013 07:30:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_variable_access_clea?= =?utf-8?q?n-ups_for_deque=2Erotate=28=29=2E?= Message-ID: <3Z0bX04w3mzSsy@mail.python.org> http://hg.python.org/cpython/rev/22cac8350d68 changeset: 82012:22cac8350d68 user: Raymond Hettinger date: Tue Feb 05 01:30:46 2013 -0500 summary: Minor variable access clean-ups for deque.rotate(). files: Modules/_collectionsmodule.c | 26 ++++++++++++------------ 1 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -413,7 +413,7 @@ static int _deque_rotate(dequeobject *deque, Py_ssize_t n) { - Py_ssize_t i, m, len=deque->len, halflen=len>>1; + Py_ssize_t m, len=deque->len, halflen=len>>1; block *prevblock; if (len <= 1) @@ -425,13 +425,13 @@ else if (n < -halflen) n += len; } - assert(deque->len > 1); + assert(len > 1); assert(-halflen <= n && n <= halflen); deque->state++; - for (i=0 ; i 0) { if (deque->leftindex == 0) { - block *b = newblock(NULL, deque->leftblock, deque->len); + block *b = newblock(NULL, deque->leftblock, len); if (b == NULL) return -1; assert(deque->leftblock->leftlink == NULL); @@ -441,18 +441,18 @@ } assert(deque->leftindex > 0); - m = n - i; + m = n; if (m > deque->rightindex + 1) m = deque->rightindex + 1; if (m > deque->leftindex) m = deque->leftindex; - assert (m > 0 && m <= deque->len); + assert (m > 0 && m <= len); memcpy(&deque->leftblock->data[deque->leftindex - m], - &deque->rightblock->data[deque->rightindex - m + 1], + &deque->rightblock->data[deque->rightindex + 1 - m], m * sizeof(PyObject *)); deque->rightindex -= m; deque->leftindex -= m; - i += m; + n -= m; if (deque->rightindex == -1) { assert(deque->rightblock != NULL); @@ -464,9 +464,9 @@ deque->rightindex = BLOCKLEN - 1; } } - for (i=0 ; i>n ; ) { + while (n < 0) { if (deque->rightindex == BLOCKLEN - 1) { - block *b = newblock(deque->rightblock, NULL, deque->len); + block *b = newblock(deque->rightblock, NULL, len); if (b == NULL) return -1; assert(deque->rightblock->rightlink == NULL); @@ -476,18 +476,18 @@ } assert (deque->rightindex < BLOCKLEN - 1); - m = i - n; + m = -n; if (m > BLOCKLEN - deque->leftindex) m = BLOCKLEN - deque->leftindex; if (m > BLOCKLEN - 1 - deque->rightindex) m = BLOCKLEN - 1 - deque->rightindex; - assert (m > 0 && m <= deque->len); + assert (m > 0 && m <= len); memcpy(&deque->rightblock->data[deque->rightindex + 1], &deque->leftblock->data[deque->leftindex], m * sizeof(PyObject *)); deque->leftindex += m; deque->rightindex += m; - i -= m; + n += m; if (deque->leftindex == BLOCKLEN) { assert(deque->leftblock != deque->rightblock); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 08:26:21 2013 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 5 Feb 2013 08:26:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE3MDc2OiBNYWtl?= =?utf-8?q?_copying_of_xattrs_more_permissive_of_missing_FS_support?= Message-ID: <3Z0clx0wn8zSqD@mail.python.org> http://hg.python.org/cpython/rev/47c65639390d changeset: 82013:47c65639390d branch: 3.3 parent: 82010:e64b74227198 user: Hynek Schlawack date: Tue Feb 05 08:22:44 2013 +0100 summary: #17076: Make copying of xattrs more permissive of missing FS support Patch by Thomas Wouters. files: Lib/shutil.py | 8 +++++++- Lib/test/test_shutil.py | 11 +++++++++++ Misc/NEWS | 3 +++ 3 files changed, 21 insertions(+), 1 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -142,7 +142,13 @@ """ - for name in os.listxattr(src, follow_symlinks=follow_symlinks): + try: + names = os.listxattr(src, follow_symlinks=follow_symlinks) + except OSError as e: + if e.errno not in (errno.ENOTSUP, errno.ENODATA): + raise + return + for name in names: try: value = os.getxattr(src, name, follow_symlinks=follow_symlinks) os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -449,6 +449,17 @@ self.assertIn('user.bar', os.listxattr(dst)) finally: os.setxattr = orig_setxattr + # the source filesystem not supporting xattrs should be ok, too. + def _raise_on_src(fname, *, follow_symlinks=True): + if fname == src: + raise OSError(errno.ENOTSUP, 'Operation not supported') + return orig_listxattr(fname, follow_symlinks=follow_symlinks) + try: + orig_listxattr = os.listxattr + os.listxattr = _raise_on_src + shutil._copyxattr(src, dst) + finally: + os.listxattr = orig_listxattr # test that shutil.copystat copies xattrs src = os.path.join(tmp_dir, 'the_original') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,9 @@ Library ------- +- Issue #17076: Make copying of xattrs more permissive of missing FS support. + Patch by Thomas Wouters. + - Issue #17089: Expat parser now correctly works with string input not only when an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and strings larger than 2 GiB. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 08:26:22 2013 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 5 Feb 2013 08:26:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=2317076=3A_Make_copying_of_xattrs_more_permissive_of_mi?= =?utf-8?q?ssing_FS_support?= Message-ID: <3Z0cly3yXQzStS@mail.python.org> http://hg.python.org/cpython/rev/7ccdbd1cd213 changeset: 82014:7ccdbd1cd213 parent: 82012:22cac8350d68 parent: 82013:47c65639390d user: Hynek Schlawack date: Tue Feb 05 08:25:24 2013 +0100 summary: #17076: Make copying of xattrs more permissive of missing FS support Patch by Thomas Wouters. files: Lib/shutil.py | 8 +++++++- Lib/test/test_shutil.py | 11 +++++++++++ Misc/NEWS | 3 +++ 3 files changed, 21 insertions(+), 1 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -140,7 +140,13 @@ """ - for name in os.listxattr(src, follow_symlinks=follow_symlinks): + try: + names = os.listxattr(src, follow_symlinks=follow_symlinks) + except OSError as e: + if e.errno not in (errno.ENOTSUP, errno.ENODATA): + raise + return + for name in names: try: value = os.getxattr(src, name, follow_symlinks=follow_symlinks) os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -450,6 +450,17 @@ self.assertIn('user.bar', os.listxattr(dst)) finally: os.setxattr = orig_setxattr + # the source filesystem not supporting xattrs should be ok, too. + def _raise_on_src(fname, *, follow_symlinks=True): + if fname == src: + raise OSError(errno.ENOTSUP, 'Operation not supported') + return orig_listxattr(fname, follow_symlinks=follow_symlinks) + try: + orig_listxattr = os.listxattr + os.listxattr = _raise_on_src + shutil._copyxattr(src, dst) + finally: + os.listxattr = orig_listxattr # test that shutil.copystat copies xattrs src = os.path.join(tmp_dir, 'the_original') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,9 @@ Library ------- +- Issue #17076: Make copying of xattrs more permissive of missing FS support. + Patch by Thomas Wouters. + - Issue #17089: Expat parser now correctly works with string input not only when an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and strings larger than 2 GiB. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 16:14:16 2013 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 5 Feb 2013 16:14:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogdG9rZW4ubWFpbiBp?= =?utf-8?q?s_now_token=2E=5Fmain?= Message-ID: <3Z0q7r57qYzSgV@mail.python.org> http://hg.python.org/cpython/rev/c2278cb6cd44 changeset: 82015:c2278cb6cd44 branch: 3.3 parent: 82013:47c65639390d user: Benjamin Peterson date: Tue Feb 05 10:11:13 2013 -0500 summary: token.main is now token._main files: Lib/symbol.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/symbol.py b/Lib/symbol.py --- a/Lib/symbol.py +++ b/Lib/symbol.py @@ -104,7 +104,7 @@ import token if len(sys.argv) == 1: sys.argv = sys.argv + ["Include/graminit.h", "Lib/symbol.py"] - token.main() + token._main() if __name__ == "__main__": main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 16:14:18 2013 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 5 Feb 2013 16:14:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_update_symbol?= =?utf-8?q?=2Epy_for_yield_from_grammar_changes_=28closes_=2317132=29?= Message-ID: <3Z0q7t10pDzSxQ@mail.python.org> http://hg.python.org/cpython/rev/23850c3899e8 changeset: 82016:23850c3899e8 branch: 3.3 user: Benjamin Peterson date: Tue Feb 05 10:12:14 2013 -0500 summary: update symbol.py for yield from grammar changes (closes #17132) files: Lib/symbol.py | 1 + Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/symbol.py b/Lib/symbol.py --- a/Lib/symbol.py +++ b/Lib/symbol.py @@ -91,6 +91,7 @@ comp_if = 334 encoding_decl = 335 yield_expr = 336 +yield_arg = 337 #--end constants-- sym_name = {} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,8 @@ Library ------- +- Issue #17132: Update symbol for "yield from" grammar changes. + - Issue #17076: Make copying of xattrs more permissive of missing FS support. Patch by Thomas Wouters. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 16:14:19 2013 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 5 Feb 2013 16:14:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMTcxMzIp?= Message-ID: <3Z0q7v3g0RzSxR@mail.python.org> http://hg.python.org/cpython/rev/0fcd975765a7 changeset: 82017:0fcd975765a7 parent: 82014:7ccdbd1cd213 parent: 82016:23850c3899e8 user: Benjamin Peterson date: Tue Feb 05 10:12:31 2013 -0500 summary: merge 3.3 (#17132) files: Lib/symbol.py | 3 ++- Misc/NEWS | 2 ++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/symbol.py b/Lib/symbol.py --- a/Lib/symbol.py +++ b/Lib/symbol.py @@ -91,6 +91,7 @@ comp_if = 334 encoding_decl = 335 yield_expr = 336 +yield_arg = 337 #--end constants-- sym_name = {} @@ -104,7 +105,7 @@ import token if len(sys.argv) == 1: sys.argv = sys.argv + ["Include/graminit.h", "Lib/symbol.py"] - token.main() + token._main() if __name__ == "__main__": main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,8 @@ Library ------- +- Issue #17132: Update symbol for "yield from" grammar changes. + - Issue #17076: Make copying of xattrs more permissive of missing FS support. Patch by Thomas Wouters. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 16:14:20 2013 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 5 Feb 2013 16:14:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_remain_symbol=2Emain_to_sy?= =?utf-8?q?mbol=2E=5Fmain_mirroring_token=2Epy?= Message-ID: <3Z0q7w65jSzSxh@mail.python.org> http://hg.python.org/cpython/rev/704a38a1d048 changeset: 82018:704a38a1d048 user: Benjamin Peterson date: Tue Feb 05 10:13:22 2013 -0500 summary: remain symbol.main to symbol._main mirroring token.py files: Lib/symbol.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/symbol.py b/Lib/symbol.py --- a/Lib/symbol.py +++ b/Lib/symbol.py @@ -100,7 +100,7 @@ sym_name[_value] = _name -def main(): +def _main(): import sys import token if len(sys.argv) == 1: @@ -108,4 +108,4 @@ token._main() if __name__ == "__main__": - main() + _main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 16:18:58 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 5 Feb 2013 16:18:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_News_item_for_?= =?utf-8?q?issue_=2316811_fix=2E?= Message-ID: <3Z0qFG6ShRzSxH@mail.python.org> http://hg.python.org/cpython/rev/4553dfcafac7 changeset: 82019:4553dfcafac7 branch: 3.3 parent: 82016:23850c3899e8 user: R David Murray date: Tue Feb 05 10:17:09 2013 -0500 summary: News item for issue #16811 fix. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,9 @@ Library ------- +- Issue #16811: Fix folding of headers with no value in the provisional email + policies. + - Issue #17132: Update symbol for "yield from" grammar changes. - Issue #17076: Make copying of xattrs more permissive of missing FS support. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 16:19:00 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 5 Feb 2013 16:19:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_News_item_for_issue_=2316811_fix=2E?= Message-ID: <3Z0qFJ28PCzSvF@mail.python.org> http://hg.python.org/cpython/rev/68be406e76e1 changeset: 82020:68be406e76e1 parent: 82018:704a38a1d048 parent: 82019:4553dfcafac7 user: R David Murray date: Tue Feb 05 10:18:46 2013 -0500 summary: Merge: News item for issue #16811 fix. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,9 @@ Library ------- +- Issue #16811: Fix folding of headers with no value in the provisional email + policies. + - Issue #17132: Update symbol for "yield from" grammar changes. - Issue #17076: Make copying of xattrs more permissive of missing FS support. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 17:35:03 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 5 Feb 2013 17:35:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogIzE2OTQ4OiBGaXgg?= =?utf-8?q?quopri_encoding_of_non-latin1_character_sets=2E?= Message-ID: <3Z0rx32gX0zSXS@mail.python.org> http://hg.python.org/cpython/rev/801cb3918212 changeset: 82021:801cb3918212 branch: 3.2 parent: 82007:3cc2a2de36e3 user: R David Murray date: Tue Feb 05 10:49:49 2013 -0500 summary: #16948: Fix quopri encoding of non-latin1 character sets. files: Lib/email/charset.py | 13 +++++++++++++ Lib/email/test/test_email.py | 21 +++++++++++++++++++++ Misc/NEWS | 4 ++++ 3 files changed, 38 insertions(+), 0 deletions(-) diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -392,6 +392,19 @@ string = string.encode(self.output_charset) return email.base64mime.body_encode(string) elif self.body_encoding is QP: + # quopromime.body_encode takes a string, but operates on it as if + # it were a list of byte codes. For a (minimal) history on why + # this is so, see changeset 0cf700464177. To correctly encode a + # character set, then, we must turn it into pseudo bytes via the + # latin1 charset, which will encode any byte as a single code point + # between 0 and 255, which is what body_encode is expecting. + # + # Note that this clause doesn't handle the case of a _payload that + # is already bytes. It never did, and the semantics of _payload + # being bytes has never been nailed down, so fixing that is a + # longer term TODO. + if isinstance(string, str): + string = string.encode(self.output_charset).decode('latin1') return email.quoprimime.body_encode(string) else: if isinstance(string, str): diff --git a/Lib/email/test/test_email.py b/Lib/email/test/test_email.py --- a/Lib/email/test/test_email.py +++ b/Lib/email/test/test_email.py @@ -670,6 +670,27 @@ msg = MIMEText('?', _charset='euc-jp') eq(msg['content-transfer-encoding'], '7bit') + def test_qp_encode_latin1(self): + msg = MIMEText('\xe1\xf6\n', 'text', 'ISO-8859-1') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + =E1=F6 + """)) + + def test_qp_encode_non_latin1(self): + # Issue 16948 + msg = MIMEText('\u017c\n', 'text', 'ISO-8859-2') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-2" + Content-Transfer-Encoding: quoted-printable + + =BF + """)) + # Test long header wrapping class TestLongHeaders(TestEmailBase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -212,6 +212,10 @@ Library ------- + +- Issue #16948: Fix quoted printable body encoding for non-latin1 character + sets in the email package. + - Issue #17089: Expat parser now correctly works with string input not only when an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and strings larger than 2 GiB. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 17:35:04 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 5 Feb 2013 17:35:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Merge=3A_=2316948=3A_Fix_quopri_encoding_of_non-latin1_charact?= =?utf-8?q?er_sets=2E?= Message-ID: <3Z0rx45Q3mzSWM@mail.python.org> http://hg.python.org/cpython/rev/e644058e8e7b changeset: 82022:e644058e8e7b branch: 3.3 parent: 82019:4553dfcafac7 parent: 82021:801cb3918212 user: R David Murray date: Tue Feb 05 10:55:27 2013 -0500 summary: Merge: #16948: Fix quopri encoding of non-latin1 character sets. files: Lib/email/charset.py | 13 +++++++++++ Lib/test/test_email/test_email.py | 21 +++++++++++++++++++ Misc/NEWS | 3 ++ 3 files changed, 37 insertions(+), 0 deletions(-) diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -392,6 +392,19 @@ string = string.encode(self.output_charset) return email.base64mime.body_encode(string) elif self.body_encoding is QP: + # quopromime.body_encode takes a string, but operates on it as if + # it were a list of byte codes. For a (minimal) history on why + # this is so, see changeset 0cf700464177. To correctly encode a + # character set, then, we must turn it into pseudo bytes via the + # latin1 charset, which will encode any byte as a single code point + # between 0 and 255, which is what body_encode is expecting. + # + # Note that this clause doesn't handle the case of a _payload that + # is already bytes. It never did, and the semantics of _payload + # being bytes has never been nailed down, so fixing that is a + # longer term TODO. + if isinstance(string, str): + string = string.encode(self.output_charset).decode('latin1') return email.quoprimime.body_encode(string) else: if isinstance(string, str): diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -677,6 +677,27 @@ msg = MIMEText('?', _charset='euc-jp') eq(msg['content-transfer-encoding'], '7bit') + def test_qp_encode_latin1(self): + msg = MIMEText('\xe1\xf6\n', 'text', 'ISO-8859-1') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + =E1=F6 + """)) + + def test_qp_encode_non_latin1(self): + # Issue 16948 + msg = MIMEText('\u017c\n', 'text', 'ISO-8859-2') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-2" + Content-Transfer-Encoding: quoted-printable + + =BF + """)) + # Test long header wrapping class TestLongHeaders(TestEmailBase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,9 @@ Library ------- +- Issue #16948: Fix quoted printable body encoding for non-latin1 character + sets in the email package. + - Issue #16811: Fix folding of headers with no value in the provisional email policies. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 17:35:06 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 5 Feb 2013 17:35:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2316948=3A_Fix_quopri_encoding_of_non-latin1_c?= =?utf-8?q?haracter_sets=2E?= Message-ID: <3Z0rx614FxzSXK@mail.python.org> http://hg.python.org/cpython/rev/009dc81e8bc9 changeset: 82023:009dc81e8bc9 parent: 82020:68be406e76e1 parent: 82022:e644058e8e7b user: R David Murray date: Tue Feb 05 11:34:39 2013 -0500 summary: Merge: #16948: Fix quopri encoding of non-latin1 character sets. files: Lib/email/charset.py | 13 +++++++++++ Lib/test/test_email/test_email.py | 21 +++++++++++++++++++ Misc/NEWS | 3 ++ 3 files changed, 37 insertions(+), 0 deletions(-) diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -392,6 +392,19 @@ string = string.encode(self.output_charset) return email.base64mime.body_encode(string) elif self.body_encoding is QP: + # quopromime.body_encode takes a string, but operates on it as if + # it were a list of byte codes. For a (minimal) history on why + # this is so, see changeset 0cf700464177. To correctly encode a + # character set, then, we must turn it into pseudo bytes via the + # latin1 charset, which will encode any byte as a single code point + # between 0 and 255, which is what body_encode is expecting. + # + # Note that this clause doesn't handle the case of a _payload that + # is already bytes. It never did, and the semantics of _payload + # being bytes has never been nailed down, so fixing that is a + # longer term TODO. + if isinstance(string, str): + string = string.encode(self.output_charset).decode('latin1') return email.quoprimime.body_encode(string) else: if isinstance(string, str): diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -677,6 +677,27 @@ msg = MIMEText('?', _charset='euc-jp') eq(msg['content-transfer-encoding'], '7bit') + def test_qp_encode_latin1(self): + msg = MIMEText('\xe1\xf6\n', 'text', 'ISO-8859-1') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + =E1=F6 + """)) + + def test_qp_encode_non_latin1(self): + # Issue 16948 + msg = MIMEText('\u017c\n', 'text', 'ISO-8859-2') + self.assertEqual(str(msg), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/text; charset="iso-8859-2" + Content-Transfer-Encoding: quoted-printable + + =BF + """)) + # Test long header wrapping class TestLongHeaders(TestEmailBase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,9 @@ Library ------- +- Issue #16948: Fix quoted printable body encoding for non-latin1 character + sets in the email package. + - Issue #16811: Fix folding of headers with no value in the provisional email policies. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 19:43:00 2013 From: python-checkins at python.org (charles-francois.natali) Date: Tue, 5 Feb 2013 19:43:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315359=3A_Add_CAN?= =?utf-8?q?=5FBCM_protocol_support_to_the_socket_module=2E_Patch_by_Brian?= Message-ID: <3Z0vmh5zGjzSXk@mail.python.org> http://hg.python.org/cpython/rev/f714af60508d changeset: 82024:f714af60508d user: Charles-Fran?ois Natali date: Tue Feb 05 19:42:01 2013 +0100 summary: Issue #15359: Add CAN_BCM protocol support to the socket module. Patch by Brian Thorne. files: Doc/library/socket.rst | 32 +++++++- Lib/test/test_socket.py | 106 +++++++++++++++++++++++++-- Misc/NEWS | 3 + Modules/socketmodule.c | 17 ++++ Modules/socketmodule.h | 4 + configure | 2 +- configure.ac | 2 +- pyconfig.h.in | 3 + 8 files changed, 151 insertions(+), 18 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -107,8 +107,8 @@ .. versionadded:: 3.3 -- Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`) - support specific representations. +- Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`, + :const:`AF_CAN`) support specific representations. .. XXX document them! @@ -257,6 +257,16 @@ .. versionadded:: 3.3 +.. data:: CAN_BCM + CAN_BCM_* + + CAN_BCM, in the CAN protocol family, is the broadcast manager (BCM) protocol. + Broadcast manager constants, documented in the Linux documentation, are also + defined in the socket module. + + Availability: Linux >= 2.6.25. + + .. versionadded:: 3.4 .. data:: AF_RDS PF_RDS @@ -452,13 +462,16 @@ :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The socket type should be :const:`SOCK_STREAM` (the default), :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` - constants. The protocol number is usually zero and may be omitted in that - case or :const:`CAN_RAW` in case the address family is :const:`AF_CAN`. + constants. The protocol number is usually zero and may be omitted or in the + case where the address family is :const:`AF_CAN` the protocol should be one + of :const:`CAN_RAW` or :const:`CAN_BCM`. .. versionchanged:: 3.3 The AF_CAN family was added. The AF_RDS family was added. + .. versionchanged:: 3.4 + The CAN_BCM protocol was added. .. function:: socketpair([family[, type[, proto]]]) @@ -1331,7 +1344,16 @@ s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF) The last example shows how to use the socket interface to communicate to a CAN -network. This example might require special priviledge:: +network using the raw socket protocol. To use CAN with the broadcast +manager protocol instead, open a socket with:: + + socket.socket(socket.AF_CAN, socket.SOCK_DGRAM, socket.CAN_BCM) + +After binding (:const:`CAN_RAW`) or connecting (:const:`CAN_BCM`) the socket, you +can use the :method:`socket.send`, and the :method:`socket.recv` operations (and +their counterparts) on the socket object as usual. + +This example might require special priviledge:: import socket import struct diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -121,6 +121,36 @@ interface = 'vcan0' bufsize = 128 + """The CAN frame structure is defined in : + + struct can_frame { + canid_t can_id; /* 32 bit CAN_ID + EFF/RTR/ERR flags */ + __u8 can_dlc; /* data length code: 0 .. 8 */ + __u8 data[8] __attribute__((aligned(8))); + }; + """ + can_frame_fmt = "=IB3x8s" + can_frame_size = struct.calcsize(can_frame_fmt) + + """The Broadcast Management Command frame structure is defined + in : + + struct bcm_msg_head { + __u32 opcode; + __u32 flags; + __u32 count; + struct timeval ival1, ival2; + canid_t can_id; + __u32 nframes; + struct can_frame frames[0]; + } + + `bcm_msg_head` must be 8 bytes aligned because of the `frames` member (see + `struct can_frame` definition). Must use native not standard types for packing. + """ + bcm_cmd_msg_fmt = "@3I4l2I" + bcm_cmd_msg_fmt += "x" * (struct.calcsize(bcm_cmd_msg_fmt) % 8) + def setUp(self): self.s = socket.socket(socket.PF_CAN, socket.SOCK_RAW, socket.CAN_RAW) self.addCleanup(self.s.close) @@ -1291,10 +1321,35 @@ socket.PF_CAN socket.CAN_RAW + @unittest.skipUnless(hasattr(socket, "CAN_BCM"), + 'socket.CAN_BCM required for this test.') + def testBCMConstants(self): + socket.CAN_BCM + + # opcodes + socket.CAN_BCM_TX_SETUP # create (cyclic) transmission task + socket.CAN_BCM_TX_DELETE # remove (cyclic) transmission task + socket.CAN_BCM_TX_READ # read properties of (cyclic) transmission task + socket.CAN_BCM_TX_SEND # send one CAN frame + socket.CAN_BCM_RX_SETUP # create RX content filter subscription + socket.CAN_BCM_RX_DELETE # remove RX content filter subscription + socket.CAN_BCM_RX_READ # read properties of RX content filter subscription + socket.CAN_BCM_TX_STATUS # reply to TX_READ request + socket.CAN_BCM_TX_EXPIRED # notification on performed transmissions (count=0) + socket.CAN_BCM_RX_STATUS # reply to RX_READ request + socket.CAN_BCM_RX_TIMEOUT # cyclic message is absent + socket.CAN_BCM_RX_CHANGED # updated CAN frame (detected content change) + def testCreateSocket(self): with socket.socket(socket.PF_CAN, socket.SOCK_RAW, socket.CAN_RAW) as s: pass + @unittest.skipUnless(hasattr(socket, "CAN_BCM"), + 'socket.CAN_BCM required for this test.') + def testCreateBCMSocket(self): + with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_BCM) as s: + pass + def testBindAny(self): with socket.socket(socket.PF_CAN, socket.SOCK_RAW, socket.CAN_RAW) as s: s.bind(('', )) @@ -1327,19 +1382,8 @@ @unittest.skipUnless(HAVE_SOCKET_CAN, 'SocketCan required for this test.') - at unittest.skipUnless(thread, 'Threading required for this test.') class CANTest(ThreadedCANSocketTest): - """The CAN frame structure is defined in : - - struct can_frame { - canid_t can_id; /* 32 bit CAN_ID + EFF/RTR/ERR flags */ - __u8 can_dlc; /* data length code: 0 .. 8 */ - __u8 data[8] __attribute__((aligned(8))); - }; - """ - can_frame_fmt = "=IB3x8s" - def __init__(self, methodName='runTest'): ThreadedCANSocketTest.__init__(self, methodName=methodName) @@ -1388,6 +1432,46 @@ self.cf2 = self.build_can_frame(0x12, b'\x99\x22\x33') self.cli.send(self.cf2) + @unittest.skipUnless(hasattr(socket, "CAN_BCM"), + 'socket.CAN_BCM required for this test.') + def _testBCM(self): + cf, addr = self.cli.recvfrom(self.bufsize) + self.assertEqual(self.cf, cf) + can_id, can_dlc, data = self.dissect_can_frame(cf) + self.assertEqual(self.can_id, can_id) + self.assertEqual(self.data, data) + + @unittest.skipUnless(hasattr(socket, "CAN_BCM"), + 'socket.CAN_BCM required for this test.') + def testBCM(self): + bcm = socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_BCM) + self.addCleanup(bcm.close) + bcm.connect((self.interface,)) + self.can_id = 0x123 + self.data = bytes([0xc0, 0xff, 0xee]) + self.cf = self.build_can_frame(self.can_id, self.data) + opcode = socket.CAN_BCM_TX_SEND + flags = 0 + count = 0 + ival1_seconds = ival1_usec = ival2_seconds = ival2_usec = 0 + bcm_can_id = 0x0222 + nframes = 1 + assert len(self.cf) == 16 + header = struct.pack(self.bcm_cmd_msg_fmt, + opcode, + flags, + count, + ival1_seconds, + ival1_usec, + ival2_seconds, + ival2_usec, + bcm_can_id, + nframes, + ) + header_plus_frame = header + self.cf + bytes_sent = bcm.send(header_plus_frame) + self.assertEqual(bytes_sent, len(header_plus_frame)) + @unittest.skipUnless(HAVE_SOCKET_RDS, 'RDS sockets required for this test.') class BasicRDSTest(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,9 @@ Library ------- +- Issue #15359: Add CAN_BCM protocol support to the socket module. Patch by + Brian Thorne. + - Issue #16948: Fix quoted printable body encoding for non-latin1 character sets in the email package. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1598,6 +1598,8 @@ case AF_CAN: switch (s->sock_proto) { case CAN_RAW: + /* fall-through */ + case CAN_BCM: { struct sockaddr_can *addr; PyObject *interfaceName; @@ -6031,6 +6033,21 @@ PyModule_AddIntConstant(m, "CAN_RAW_LOOPBACK", CAN_RAW_LOOPBACK); PyModule_AddIntConstant(m, "CAN_RAW_RECV_OWN_MSGS", CAN_RAW_RECV_OWN_MSGS); #endif +#ifdef HAVE_LINUX_CAN_BCM_H + PyModule_AddIntConstant(m, "CAN_BCM", CAN_BCM); + PyModule_AddIntConstant(m, "CAN_BCM_TX_SETUP", TX_SETUP); + PyModule_AddIntConstant(m, "CAN_BCM_TX_DELETE", TX_DELETE); + PyModule_AddIntConstant(m, "CAN_BCM_TX_READ", TX_READ); + PyModule_AddIntConstant(m, "CAN_BCM_TX_SEND", TX_SEND); + PyModule_AddIntConstant(m, "CAN_BCM_RX_SETUP", RX_SETUP); + PyModule_AddIntConstant(m, "CAN_BCM_RX_DELETE", RX_DELETE); + PyModule_AddIntConstant(m, "CAN_BCM_RX_READ", RX_READ); + PyModule_AddIntConstant(m, "CAN_BCM_TX_STATUS", TX_STATUS); + PyModule_AddIntConstant(m, "CAN_BCM_TX_EXPIRED", TX_EXPIRED); + PyModule_AddIntConstant(m, "CAN_BCM_RX_STATUS", RX_STATUS); + PyModule_AddIntConstant(m, "CAN_BCM_RX_TIMEOUT", RX_TIMEOUT); + PyModule_AddIntConstant(m, "CAN_BCM_RX_CHANGED", RX_CHANGED); +#endif #ifdef SOL_RDS PyModule_AddIntConstant(m, "SOL_RDS", SOL_RDS); #endif diff --git a/Modules/socketmodule.h b/Modules/socketmodule.h --- a/Modules/socketmodule.h +++ b/Modules/socketmodule.h @@ -80,6 +80,10 @@ #include #endif +#ifdef HAVE_LINUX_CAN_BCM_H +#include +#endif + #ifdef HAVE_SYS_SYS_DOMAIN_H #include #endif diff --git a/configure b/configure --- a/configure +++ b/configure @@ -7224,7 +7224,7 @@ # On Linux, can.h and can/raw.h require sys/socket.h -for ac_header in linux/can.h linux/can/raw.h +for ac_header in linux/can.h linux/can/raw.h linux/can/bcm.h do : as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh` ac_fn_c_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" " diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -1568,7 +1568,7 @@ ]) # On Linux, can.h and can/raw.h require sys/socket.h -AC_CHECK_HEADERS(linux/can.h linux/can/raw.h,,,[ +AC_CHECK_HEADERS(linux/can.h linux/can/raw.h linux/can/bcm.h,,,[ #ifdef HAVE_SYS_SOCKET_H #include #endif diff --git a/pyconfig.h.in b/pyconfig.h.in --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -501,6 +501,9 @@ /* Define to 1 if you have the `linkat' function. */ #undef HAVE_LINKAT +/* Define to 1 if you have the header file. */ +#undef HAVE_LINUX_CAN_BCM_H + /* Define to 1 if you have the header file. */ #undef HAVE_LINUX_CAN_H -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 21:14:35 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 5 Feb 2013 21:14:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2317122=3A_Fix_and_?= =?utf-8?q?cleanup_test=5Ffunctools=2Epy=2E?= Message-ID: <3Z0xpM3lMczSS3@mail.python.org> http://hg.python.org/cpython/rev/2a369c32f1f1 changeset: 82025:2a369c32f1f1 user: Serhiy Storchaka date: Tue Feb 05 22:12:59 2013 +0200 summary: Issue #17122: Fix and cleanup test_functools.py. files: Lib/test/test_functools.py | 71 +++++++------------------ 1 files changed, 21 insertions(+), 50 deletions(-) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -8,30 +8,9 @@ import functools -original_functools = functools py_functools = support.import_fresh_module('functools', blocked=['_functools']) c_functools = support.import_fresh_module('functools', fresh=['_functools']) -class BaseTest(unittest.TestCase): - - """Base class required for testing C and Py implementations.""" - - def setUp(self): - - # The module must be explicitly set so that the proper - # interaction between the c module and the python module - # can be controlled. - self.partial = self.module.partial - super(BaseTest, self).setUp() - -class BaseTestC(BaseTest): - module = c_functools - -class BaseTestPy(BaseTest): - module = py_functools - -PythonPartial = py_functools.partial - def capture(*args, **kw): """capture all positional and keyword arguments""" return args, kw @@ -40,9 +19,7 @@ """ return the signature of a partial object """ return (part.func, part.args, part.keywords, part.__dict__) -class TestPartial(object): - - partial = functools.partial +class TestPartial: def test_basic_examples(self): p = self.partial(capture, 1, 2, a=10, b=20) @@ -161,12 +138,17 @@ join = self.partial(''.join) self.assertEqual(join(data), '0123456789') + at unittest.skipUnless(c_functools, 'requires the C _functools module') +class TestPartialC(TestPartial, unittest.TestCase): + if c_functools: + partial = c_functools.partial + def test_repr(self): args = (object(), object()) args_repr = ', '.join(repr(a) for a in args) kwargs = {'a': object(), 'b': object()} kwargs_repr = ', '.join("%s=%r" % (k, v) for k, v in kwargs.items()) - if self.partial is functools.partial: + if self.partial is c_functools.partial: name = 'functools.partial' else: name = self.partial.__name__ @@ -193,8 +175,6 @@ f_copy = pickle.loads(pickle.dumps(f)) self.assertEqual(signature(f), signature(f_copy)) -class TestPartialC(BaseTestC, TestPartial): - # Issue 6083: Reference counting bug def test_setstate_refcount(self): class BadSequence: @@ -214,27 +194,17 @@ "new style getargs format but argument is not a tuple", f.__setstate__, BadSequence()) -class TestPartialPy(BaseTestPy, TestPartial): +class TestPartialPy(TestPartial, unittest.TestCase): + partial = staticmethod(py_functools.partial) - def test_pickle(self): - raise unittest.SkipTest("Python implementation of partial isn't picklable") - - def test_repr(self): - raise unittest.SkipTest("Python implementation of partial uses own repr") - -class TestPartialCSubclass(TestPartialC): - +if c_functools: class PartialSubclass(c_functools.partial): pass - partial = staticmethod(PartialSubclass) - -class TestPartialPySubclass(TestPartialPy): - - class PartialSubclass(c_functools.partial): - pass - - partial = staticmethod(PartialSubclass) + at unittest.skipUnless(c_functools, 'requires the C _functools module') +class TestPartialCSubclass(TestPartialC): + if c_functools: + partial = PartialSubclass class TestUpdateWrapper(unittest.TestCase): @@ -482,7 +452,7 @@ d = {"one": 1, "two": 2, "three": 3} self.assertEqual(self.func(add, d), "".join(d.keys())) -class TestCmpToKey(object): +class TestCmpToKey: def test_cmp_to_key(self): def cmp1(x, y): @@ -513,7 +483,7 @@ with self.assertRaises(TypeError): key = self.cmp_to_key() # too few args with self.assertRaises(TypeError): - key = self.module.cmp_to_key(cmp1, None) # too many args + key = self.cmp_to_key(cmp1, None) # too many args key = self.cmp_to_key(cmp1) with self.assertRaises(TypeError): key() # too few args @@ -564,10 +534,12 @@ self.assertRaises(TypeError, hash, k) self.assertNotIsInstance(k, collections.Hashable) -class TestCmpToKeyC(BaseTestC, TestCmpToKey): - cmp_to_key = c_functools.cmp_to_key + at unittest.skipUnless(c_functools, 'requires the C _functools module') +class TestCmpToKeyC(TestCmpToKey, unittest.TestCase): + if c_functools: + cmp_to_key = c_functools.cmp_to_key -class TestCmpToKeyPy(BaseTestPy, TestCmpToKey): +class TestCmpToKeyPy(TestCmpToKey, unittest.TestCase): cmp_to_key = staticmethod(py_functools.cmp_to_key) class TestTotalOrdering(unittest.TestCase): @@ -842,7 +814,6 @@ TestPartialC, TestPartialPy, TestPartialCSubclass, - TestPartialPySubclass, TestUpdateWrapper, TestTotalOrdering, TestCmpToKeyC, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 21:24:58 2013 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 5 Feb 2013 21:24:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2317107=3A_Test_cli?= =?utf-8?q?ent-side_SNI_support_in_urllib=2Erequest_thanks_to_the_new?= Message-ID: <3Z0y2L2J31zSZx@mail.python.org> http://hg.python.org/cpython/rev/f74a12e23aaa changeset: 82026:f74a12e23aaa user: Antoine Pitrou date: Tue Feb 05 21:20:51 2013 +0100 summary: Issue #17107: Test client-side SNI support in urllib.request thanks to the new server-side SNI support in the ssl module. Initial patch by Daniel Black. files: Lib/test/ssl_servers.py | 8 +++-- Lib/test/test_httplib.py | 2 +- Lib/test/test_ssl.py | 2 +- Lib/test/test_urllib2_localnet.py | 28 ++++++++++++++++-- Lib/test/test_urllib2net.py | 22 -------------- Misc/NEWS | 4 ++ 6 files changed, 36 insertions(+), 30 deletions(-) diff --git a/Lib/test/ssl_servers.py b/Lib/test/ssl_servers.py --- a/Lib/test/ssl_servers.py +++ b/Lib/test/ssl_servers.py @@ -147,9 +147,11 @@ self.server.shutdown() -def make_https_server(case, certfile=CERTFILE, host=HOST, handler_class=None): - # we assume the certfile contains both private key and certificate - context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) +def make_https_server(case, *, context=None, certfile=CERTFILE, + host=HOST, handler_class=None): + if context is None: + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + # We assume the certfile contains both private key and certificate context.load_cert_chain(certfile) server = HTTPSServerThread(context, host, handler_class) flag = threading.Event() diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -703,7 +703,7 @@ def make_server(self, certfile): from test.ssl_servers import make_https_server - return make_https_server(self, certfile) + return make_https_server(self, certfile=certfile) def test_attributes(self): # simple test to check it's storing the timeout diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1637,7 +1637,7 @@ def test_socketserver(self): """Using a SocketServer to create and manage SSL connections.""" - server = make_https_server(self, CERTFILE) + server = make_https_server(self, certfile=CERTFILE) # try to connect if support.verbose: sys.stdout.write('\n') diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -9,7 +9,10 @@ import hashlib from test import support threading = support.import_module('threading') - +try: + import ssl +except ImportError: + ssl = None here = os.path.dirname(__file__) # Self-signed cert file for 'localhost' @@ -17,6 +20,7 @@ # Self-signed cert file for 'fakehostname' CERT_fakehostname = os.path.join(here, 'keycert2.pem') + # Loopback http server infrastructure class LoopbackHttpServer(http.server.HTTPServer): @@ -353,12 +357,15 @@ def setUp(self): super(TestUrlopen, self).setUp() # Ignore proxies for localhost tests. + self.old_environ = os.environ.copy() os.environ['NO_PROXY'] = '*' self.server = None def tearDown(self): if self.server is not None: self.server.stop() + os.environ.clear() + os.environ.update(self.old_environ) super(TestUrlopen, self).tearDown() def urlopen(self, url, data=None, **kwargs): @@ -386,14 +393,14 @@ handler.port = port return handler - def start_https_server(self, responses=None, certfile=CERT_localhost): + def start_https_server(self, responses=None, **kwargs): if not hasattr(urllib.request, 'HTTPSHandler'): self.skipTest('ssl support required') from test.ssl_servers import make_https_server if responses is None: responses = [(200, [], b"we care a bit")] handler = GetRequestHandler(responses) - server = make_https_server(self, certfile=certfile, handler_class=handler) + server = make_https_server(self, handler_class=handler, **kwargs) handler.port = server.port return handler @@ -483,6 +490,21 @@ self.urlopen("https://localhost:%s/bizarre" % handler.port, cadefault=True) + def test_https_sni(self): + if ssl is None: + self.skipTest("ssl module required") + if not ssl.HAS_SNI: + self.skipTest("SNI support required in OpenSSL") + sni_name = None + def cb_sni(ssl_sock, server_name, initial_context): + nonlocal sni_name + sni_name = server_name + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.set_servername_callback(cb_sni) + handler = self.start_https_server(context=context, certfile=CERT_localhost) + self.urlopen("https://localhost:%s" % handler.port) + self.assertEqual(sni_name, "localhost") + def test_sending_headers(self): handler = self.start_server() req = urllib.request.Request("http://localhost:%s/" % handler.port, diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -330,31 +330,9 @@ self.assertEqual(u.fp.fp.raw._sock.gettimeout(), 60) - at unittest.skipUnless(ssl, "requires SSL support") -class HTTPSTests(unittest.TestCase): - - def test_sni(self): - self.skipTest("test disabled - test server needed") - # Checks that Server Name Indication works, if supported by the - # OpenSSL linked to. - # The ssl module itself doesn't have server-side support for SNI, - # so we rely on a third-party test site. - expect_sni = ssl.HAS_SNI - with support.transient_internet("XXX"): - u = urllib.request.urlopen("XXX") - contents = u.readall() - if expect_sni: - self.assertIn(b"Great", contents) - self.assertNotIn(b"Unfortunately", contents) - else: - self.assertNotIn(b"Great", contents) - self.assertIn(b"Unfortunately", contents) - - def test_main(): support.requires("network") support.run_unittest(AuthTests, - HTTPSTests, OtherNetworkTests, CloseSocketTest, TimeoutTest, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -754,6 +754,10 @@ Tests ----- +- Issue #17107: Test client-side SNI support in urllib.request thanks to + the new server-side SNI support in the ssl module. Initial patch by + Daniel Black. + - Issue #17041: Fix testing when Python is configured with the --without-doc-strings. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Feb 5 23:15:10 2013 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 5 Feb 2013 23:15:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Get_rid_of_add=5Fconnector=28?= =?utf-8?q?=29=2E_The_code_will_follow_suit_soon=2E?= Message-ID: <3Z10TV4JvRzQNp@mail.python.org> http://hg.python.org/peps/rev/abfbb4ee96a6 changeset: 4717:abfbb4ee96a6 user: Guido van Rossum date: Tue Feb 05 14:15:06 2013 -0800 summary: Get rid of add_connector(). The code will follow suit soon. files: pep-3156.txt | 7 ------- 1 files changed, 0 insertions(+), 7 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -399,13 +399,6 @@ - ``remove_writer(fd)``. This is to ``add_writer()`` as ``remove_reader()`` is to ``add_reader()``. -- ``add_connector(fd, callback, *args)``. Like ``add_writer()`` but - meant to wait for ``connect()`` operations, which on some platforms - require different handling (e.g. ``WSAPoll()`` on Windows). - -- ``remove_connector(fd)``. This is to ``remove_writer()`` as - ``add_connector()`` is to ``add_writer()``. - TBD: What about multiple callbacks per fd? The current semantics is that ``add_reader()/add_writer()`` replace a previously registered callback. Change this to raise an exception if a callback is already -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Wed Feb 6 05:59:32 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 06 Feb 2013 05:59:32 +0100 Subject: [Python-checkins] Daily reference leaks (f74a12e23aaa): sum=0 Message-ID: results for f74a12e23aaa on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogSBxIDr', '-x'] From python-checkins at python.org Wed Feb 6 09:38:54 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 6 Feb 2013 09:38:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2NzIz?= =?utf-8?q?=3A_httplib=2EHTTPResponse_no_longer_marked_closed_when_the_con?= =?utf-8?q?nection?= Message-ID: <3Z1GKB4qCDzSPv@mail.python.org> http://hg.python.org/cpython/rev/6cc5bbfcf04e changeset: 82027:6cc5bbfcf04e branch: 3.2 parent: 82021:801cb3918212 user: Serhiy Storchaka date: Wed Feb 06 10:31:57 2013 +0200 summary: Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. files: Lib/http/client.py | 34 +++++++++++++++------------ Lib/test/test_httplib.py | 18 ++++++++++++++ Misc/NEWS | 2 + 3 files changed, 39 insertions(+), 15 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -324,7 +324,7 @@ # empty version will cause next test to fail. version = "" if not version.startswith("HTTP/"): - self.close() + self._close_conn() raise BadStatusLine(line) # The status code is a three-digit number @@ -446,22 +446,25 @@ # otherwise, assume it will close return True + def _close_conn(self): + fp = self.fp + self.fp = None + fp.close() + def close(self): + super().close() # set "closed" flag if self.fp: - self.fp.close() - self.fp = None + self._close_conn() # These implementations are for the benefit of io.BufferedReader. # XXX This class should probably be revised to act more like # the "raw stream" that BufferedReader expects. - @property - def closed(self): - return self.isclosed() - def flush(self): - self.fp.flush() + super().flush() + if self.fp: + self.fp.flush() def readable(self): return True @@ -469,6 +472,7 @@ # End of "raw stream" methods def isclosed(self): + """True if the connection is closed.""" # NOTE: it is possible that we will not ever call self.close(). This # case occurs when will_close is TRUE, length is None, and we # read up to the last byte, but NOT past it. @@ -482,7 +486,7 @@ return b"" if self._method == "HEAD": - self.close() + self._close_conn() return b"" if self.chunked: @@ -496,10 +500,10 @@ try: s = self._safe_read(self.length) except IncompleteRead: - self.close() + self._close_conn() raise self.length = 0 - self.close() # we read everything + self._close_conn() # we read everything return s if self.length is not None: @@ -514,11 +518,11 @@ if not s: # Ideally, we would raise IncompleteRead if the content-length # wasn't satisfied, but it might break compatibility. - self.close() + self._close_conn() elif self.length is not None: self.length -= len(s) if not self.length: - self.close() + self._close_conn() return s @@ -539,7 +543,7 @@ except ValueError: # close the connection as protocol synchronisation is # probably lost - self.close() + self._close_conn() raise IncompleteRead(b''.join(value)) if chunk_left == 0: break @@ -576,7 +580,7 @@ break # we read everything; close the "file" - self.close() + self._close_conn() return b''.join(value) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -164,6 +164,9 @@ resp.begin() self.assertEqual(resp.read(), b"Text") self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) body = "HTTP/1.1 400.100 Not Ok\r\n\r\nText" sock = FakeSocket(body) @@ -185,6 +188,9 @@ self.assertFalse(resp.isclosed()) self.assertEqual(resp.read(2), b'xt') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_reads_no_content_length(self): # when no length is present, the socket should be gracefully closed when @@ -198,6 +204,9 @@ self.assertEqual(resp.read(2), b'xt') self.assertEqual(resp.read(1), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_reads_incomplete_body(self): # if the server shuts down the connection before the whole @@ -211,6 +220,9 @@ self.assertEqual(resp.read(2), b'xt') self.assertEqual(resp.read(1), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_host_port(self): # Check invalid host_port @@ -355,6 +367,9 @@ self.assertEqual(resp.status, 200) self.assertEqual(resp.reason, 'OK') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_negative_content_length(self): sock = FakeSocket( @@ -430,6 +445,9 @@ resp.begin() self.assertEqual(resp.read(), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) class OfflineTest(TestCase): def test_responses(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -212,6 +212,8 @@ Library ------- +- Issue #16723: httplib.HTTPResponse no longer marked closed when the connection + is automatically closed. - Issue #16948: Fix quoted printable body encoding for non-latin1 character sets in the email package. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 09:38:56 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 6 Feb 2013 09:38:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2316723=3A_httplib=2EHTTPResponse_no_longer_marked_clos?= =?utf-8?q?ed_when_the_connection?= Message-ID: <3Z1GKD20FzzSZL@mail.python.org> http://hg.python.org/cpython/rev/0461ed77ee4e changeset: 82028:0461ed77ee4e branch: 3.3 parent: 82022:e644058e8e7b parent: 82027:6cc5bbfcf04e user: Serhiy Storchaka date: Wed Feb 06 10:35:40 2013 +0200 summary: Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. files: Lib/http/client.py | 38 +++++++++++++++------------ Lib/test/test_httplib.py | 24 +++++++++++++++++ Misc/NEWS | 3 ++ 3 files changed, 48 insertions(+), 17 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -332,7 +332,7 @@ # empty version will cause next test to fail. version = "" if not version.startswith("HTTP/"): - self.close() + self._close_conn() raise BadStatusLine(line) # The status code is a three-digit number @@ -454,22 +454,25 @@ # otherwise, assume it will close return True + def _close_conn(self): + fp = self.fp + self.fp = None + fp.close() + def close(self): + super().close() # set "closed" flag if self.fp: - self.fp.close() - self.fp = None + self._close_conn() # These implementations are for the benefit of io.BufferedReader. # XXX This class should probably be revised to act more like # the "raw stream" that BufferedReader expects. - @property - def closed(self): - return self.isclosed() - def flush(self): - self.fp.flush() + super().flush() + if self.fp: + self.fp.flush() def readable(self): return True @@ -477,6 +480,7 @@ # End of "raw stream" methods def isclosed(self): + """True if the connection is closed.""" # NOTE: it is possible that we will not ever call self.close(). This # case occurs when will_close is TRUE, length is None, and we # read up to the last byte, but NOT past it. @@ -490,7 +494,7 @@ return b"" if self._method == "HEAD": - self.close() + self._close_conn() return b"" if amt is not None: @@ -510,10 +514,10 @@ try: s = self._safe_read(self.length) except IncompleteRead: - self.close() + self._close_conn() raise self.length = 0 - self.close() # we read everything + self._close_conn() # we read everything return s def readinto(self, b): @@ -521,7 +525,7 @@ return 0 if self._method == "HEAD": - self.close() + self._close_conn() return 0 if self.chunked: @@ -539,11 +543,11 @@ if not n: # Ideally, we would raise IncompleteRead if the content-length # wasn't satisfied, but it might break compatibility. - self.close() + self._close_conn() elif self.length is not None: self.length -= n if not self.length: - self.close() + self._close_conn() return n def _read_next_chunk_size(self): @@ -559,7 +563,7 @@ except ValueError: # close the connection as protocol synchronisation is # probably lost - self.close() + self._close_conn() raise def _read_and_discard_trailer(self): @@ -597,7 +601,7 @@ self._read_and_discard_trailer() # we read everything; close the "file" - self.close() + self._close_conn() return b''.join(value) @@ -638,7 +642,7 @@ self._read_and_discard_trailer() # we read everything; close the "file" - self.close() + self._close_conn() return total_bytes diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -164,6 +164,9 @@ resp.begin() self.assertEqual(resp.read(), b"Text") self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) body = "HTTP/1.1 400.100 Not Ok\r\n\r\nText" sock = FakeSocket(body) @@ -185,6 +188,9 @@ self.assertFalse(resp.isclosed()) self.assertEqual(resp.read(2), b'xt') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_readintos(self): # if we have a length, the system knows when to close itself @@ -202,6 +208,9 @@ self.assertEqual(n, 2) self.assertEqual(bytes(b), b'xt') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_reads_no_content_length(self): # when no length is present, the socket should be gracefully closed when @@ -215,6 +224,9 @@ self.assertEqual(resp.read(2), b'xt') self.assertEqual(resp.read(1), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_readintos_no_content_length(self): # when no length is present, the socket should be gracefully closed when @@ -266,6 +278,9 @@ n = resp.readinto(b) self.assertEqual(n, 0) self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_host_port(self): # Check invalid host_port @@ -493,6 +508,9 @@ self.assertEqual(resp.status, 200) self.assertEqual(resp.reason, 'OK') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_readinto_chunked_head(self): chunked_start = ( @@ -513,6 +531,9 @@ self.assertEqual(resp.status, 200) self.assertEqual(resp.reason, 'OK') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_negative_content_length(self): sock = FakeSocket( @@ -588,6 +609,9 @@ resp.begin() self.assertEqual(resp.read(), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) class OfflineTest(TestCase): def test_responses(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,9 @@ Library ------- +- Issue #16723: httplib.HTTPResponse no longer marked closed when the connection + is automatically closed. + - Issue #16948: Fix quoted printable body encoding for non-latin1 character sets in the email package. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 09:38:57 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 6 Feb 2013 09:38:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316723=3A_httplib=2EHTTPResponse_no_longer_marke?= =?utf-8?q?d_closed_when_the_connection?= Message-ID: <3Z1GKF60SPzSZl@mail.python.org> http://hg.python.org/cpython/rev/5f8c68281d18 changeset: 82029:5f8c68281d18 parent: 82026:f74a12e23aaa parent: 82028:0461ed77ee4e user: Serhiy Storchaka date: Wed Feb 06 10:37:19 2013 +0200 summary: Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. files: Lib/http/client.py | 38 +++++++++++++++------------ Lib/test/test_httplib.py | 24 +++++++++++++++++ Misc/NEWS | 3 ++ 3 files changed, 48 insertions(+), 17 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -332,7 +332,7 @@ # empty version will cause next test to fail. version = "" if not version.startswith("HTTP/"): - self.close() + self._close_conn() raise BadStatusLine(line) # The status code is a three-digit number @@ -454,22 +454,25 @@ # otherwise, assume it will close return True + def _close_conn(self): + fp = self.fp + self.fp = None + fp.close() + def close(self): + super().close() # set "closed" flag if self.fp: - self.fp.close() - self.fp = None + self._close_conn() # These implementations are for the benefit of io.BufferedReader. # XXX This class should probably be revised to act more like # the "raw stream" that BufferedReader expects. - @property - def closed(self): - return self.isclosed() - def flush(self): - self.fp.flush() + super().flush() + if self.fp: + self.fp.flush() def readable(self): return True @@ -477,6 +480,7 @@ # End of "raw stream" methods def isclosed(self): + """True if the connection is closed.""" # NOTE: it is possible that we will not ever call self.close(). This # case occurs when will_close is TRUE, length is None, and we # read up to the last byte, but NOT past it. @@ -490,7 +494,7 @@ return b"" if self._method == "HEAD": - self.close() + self._close_conn() return b"" if amt is not None: @@ -510,10 +514,10 @@ try: s = self._safe_read(self.length) except IncompleteRead: - self.close() + self._close_conn() raise self.length = 0 - self.close() # we read everything + self._close_conn() # we read everything return s def readinto(self, b): @@ -521,7 +525,7 @@ return 0 if self._method == "HEAD": - self.close() + self._close_conn() return 0 if self.chunked: @@ -539,11 +543,11 @@ if not n: # Ideally, we would raise IncompleteRead if the content-length # wasn't satisfied, but it might break compatibility. - self.close() + self._close_conn() elif self.length is not None: self.length -= n if not self.length: - self.close() + self._close_conn() return n def _read_next_chunk_size(self): @@ -559,7 +563,7 @@ except ValueError: # close the connection as protocol synchronisation is # probably lost - self.close() + self._close_conn() raise def _read_and_discard_trailer(self): @@ -597,7 +601,7 @@ self._read_and_discard_trailer() # we read everything; close the "file" - self.close() + self._close_conn() return b''.join(value) @@ -638,7 +642,7 @@ self._read_and_discard_trailer() # we read everything; close the "file" - self.close() + self._close_conn() return total_bytes diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -166,6 +166,9 @@ resp.begin() self.assertEqual(resp.read(), b"Text") self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) body = "HTTP/1.1 400.100 Not Ok\r\n\r\nText" sock = FakeSocket(body) @@ -187,6 +190,9 @@ self.assertFalse(resp.isclosed()) self.assertEqual(resp.read(2), b'xt') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_readintos(self): # if we have a length, the system knows when to close itself @@ -204,6 +210,9 @@ self.assertEqual(n, 2) self.assertEqual(bytes(b), b'xt') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_reads_no_content_length(self): # when no length is present, the socket should be gracefully closed when @@ -217,6 +226,9 @@ self.assertEqual(resp.read(2), b'xt') self.assertEqual(resp.read(1), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_partial_readintos_no_content_length(self): # when no length is present, the socket should be gracefully closed when @@ -268,6 +280,9 @@ n = resp.readinto(b) self.assertEqual(n, 0) self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_host_port(self): # Check invalid host_port @@ -495,6 +510,9 @@ self.assertEqual(resp.status, 200) self.assertEqual(resp.reason, 'OK') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_readinto_chunked_head(self): chunked_start = ( @@ -515,6 +533,9 @@ self.assertEqual(resp.status, 200) self.assertEqual(resp.reason, 'OK') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_negative_content_length(self): sock = FakeSocket( @@ -590,6 +611,9 @@ resp.begin() self.assertEqual(resp.read(), b'') self.assertTrue(resp.isclosed()) + self.assertFalse(resp.closed) + resp.close() + self.assertTrue(resp.closed) def test_delayed_ack_opt(self): # Test that Nagle/delayed_ack optimistaion works correctly. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,9 @@ Library ------- +- Issue #16723: httplib.HTTPResponse no longer marked closed when the connection + is automatically closed. + - Issue #15359: Add CAN_BCM protocol support to the socket module. Patch by Brian Thorne. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 16:06:26 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 6 Feb 2013 16:06:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogIzE3MTQyOiBmaXgg?= =?utf-8?q?apparent_copy_and_paste_error_in_test=5Fall=2E?= Message-ID: <3Z1QwL6FrTzSXK@mail.python.org> http://hg.python.org/cpython/rev/1fc87fa05333 changeset: 82030:1fc87fa05333 branch: 3.2 parent: 82027:6cc5bbfcf04e user: R David Murray date: Wed Feb 06 09:56:19 2013 -0500 summary: #17142: fix apparent copy and paste error in test_all. files: Lib/test/test_builtin.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -164,7 +164,7 @@ self.assertEqual(any([None, None, None]), False) self.assertEqual(any([None, 4, None]), True) self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6]) - self.assertRaises(RuntimeError, all, TestFailingIter()) + self.assertRaises(RuntimeError, any, TestFailingIter()) self.assertRaises(TypeError, any, 10) # Non-iterable self.assertRaises(TypeError, any) # No args self.assertRaises(TypeError, any, [2, 4, 6], []) # Too many args -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 16:06:28 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 6 Feb 2013 16:06:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Merge=3A_=2317142=3A_fix_apparent_copy_and_paste_error_in_test?= =?utf-8?b?X2FsbC4=?= Message-ID: <3Z1QwN1qn3zSXB@mail.python.org> http://hg.python.org/cpython/rev/4db932a303b4 changeset: 82031:4db932a303b4 branch: 3.3 parent: 82028:0461ed77ee4e parent: 82030:1fc87fa05333 user: R David Murray date: Wed Feb 06 09:57:51 2013 -0500 summary: Merge: #17142: fix apparent copy and paste error in test_all. files: Lib/test/test_builtin.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -189,7 +189,7 @@ self.assertEqual(any([None, None, None]), False) self.assertEqual(any([None, 4, None]), True) self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6]) - self.assertRaises(RuntimeError, all, TestFailingIter()) + self.assertRaises(RuntimeError, any, TestFailingIter()) self.assertRaises(TypeError, any, 10) # Non-iterable self.assertRaises(TypeError, any) # No args self.assertRaises(TypeError, any, [2, 4, 6], []) # Too many args -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 16:06:29 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 6 Feb 2013 16:06:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2317142=3A_fix_apparent_copy_and_paste_error_i?= =?utf-8?q?n_test=5Fall=2E?= Message-ID: <3Z1QwP4V1TzSWF@mail.python.org> http://hg.python.org/cpython/rev/acdb0da0df2b changeset: 82032:acdb0da0df2b parent: 82029:5f8c68281d18 parent: 82031:4db932a303b4 user: R David Murray date: Wed Feb 06 10:05:56 2013 -0500 summary: Merge: #17142: fix apparent copy and paste error in test_all. files: Lib/test/test_builtin.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -189,7 +189,7 @@ self.assertEqual(any([None, None, None]), False) self.assertEqual(any([None, 4, None]), True) self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6]) - self.assertRaises(RuntimeError, all, TestFailingIter()) + self.assertRaises(RuntimeError, any, TestFailingIter()) self.assertRaises(TypeError, any, 10) # Non-iterable self.assertRaises(TypeError, any) # No args self.assertRaises(TypeError, any, [2, 4, 6], []) # Too many args -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 16:06:31 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 6 Feb 2013 16:06:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE3MTQyOiBmaXgg?= =?utf-8?q?apparent_copy_and_paste_error_in_test=5Fall=2E?= Message-ID: <3Z1QwR1KK3zQXF@mail.python.org> http://hg.python.org/cpython/rev/d0cfabed2ef3 changeset: 82033:d0cfabed2ef3 branch: 2.7 parent: 82006:20f0c5398e97 user: R David Murray date: Wed Feb 06 10:06:10 2013 -0500 summary: #17142: fix apparent copy and paste error in test_all. files: Lib/test/test_builtin.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -119,7 +119,7 @@ self.assertEqual(any([None, None, None]), False) self.assertEqual(any([None, 4, None]), True) self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6]) - self.assertRaises(RuntimeError, all, TestFailingIter()) + self.assertRaises(RuntimeError, any, TestFailingIter()) self.assertRaises(TypeError, any, 10) # Non-iterable self.assertRaises(TypeError, any) # No args self.assertRaises(TypeError, any, [2, 4, 6], []) # Too many args -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Feb 6 23:37:42 2013 From: python-checkins at python.org (guido.van.rossum) Date: Wed, 6 Feb 2013 23:37:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Link_to_Tulip_repo=2E?= Message-ID: <3Z1cx23FHkzSRJ@mail.python.org> http://hg.python.org/peps/rev/ba64738031da changeset: 4718:ba64738031da user: Guido van Rossum date: Wed Feb 06 14:37:40 2013 -0800 summary: Link to Tulip repo. files: pep-3156.txt | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -17,7 +17,8 @@ PEP 3153. The proposal includes a pluggable event loop API, transport and protocol abstractions similar to those in Twisted, and a higher-level scheduler based on ``yield from`` (PEP 380). A reference -implementation is in the works under the code name tulip. +implementation is in the works under the code name Tulip (the Tulip +repo is linked from the References section at the end). Introduction @@ -1027,6 +1028,8 @@ - PEP 3153, while rejected, has a good write-up explaining the need to separate transports and protocols. +- Tulip repo: http://code.google.com/p/tulip/ + - Nick Coghlan wrote a nice blog post with some background, thoughts about different approaches to async I/O, gevent, and how to use futures with constructs like ``while``, ``for`` and ``with``: -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Thu Feb 7 06:01:44 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 07 Feb 2013 06:01:44 +0100 Subject: [Python-checkins] Daily reference leaks (acdb0da0df2b): sum=1 Message-ID: results for acdb0da0df2b on branch "default" -------------------------------------------- test_concurrent_futures leaked [0, -2, 3] memory blocks, sum=1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog2jEKDV', '-x'] From python-checkins at python.org Thu Feb 7 06:57:30 2013 From: python-checkins at python.org (raymond.hettinger) Date: Thu, 7 Feb 2013 06:57:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_tweaks_to_varnames?= =?utf-8?q?=2C_declarations=2C_and_comments=2E?= Message-ID: <3Z1phV3DCXzRV1@mail.python.org> http://hg.python.org/cpython/rev/2f3669aedc9a changeset: 82034:2f3669aedc9a parent: 82032:acdb0da0df2b user: Raymond Hettinger date: Thu Feb 07 00:57:19 2013 -0500 summary: Minor tweaks to varnames, declarations, and comments. files: Modules/_collectionsmodule.c | 15 +++++++-------- 1 files changed, 7 insertions(+), 8 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -9,7 +9,7 @@ /* The block length may be set to any number over 1. Larger numbers * reduce the number of calls to the memory allocator but take more - * memory. Ideally, BLOCKLEN should be set to a multiple of the + * memory. Ideally, (BLOCKLEN+2) should be set to a multiple of the * length of a cache line. */ @@ -71,7 +71,7 @@ return NULL; } if (numfreeblocks) { - numfreeblocks -= 1; + numfreeblocks--; b = freeblocks[numfreeblocks]; } else { b = PyMem_Malloc(sizeof(block)); @@ -414,7 +414,6 @@ _deque_rotate(dequeobject *deque, Py_ssize_t n) { Py_ssize_t m, len=deque->len, halflen=len>>1; - block *prevblock; if (len <= 1) return 0; @@ -455,8 +454,8 @@ n -= m; if (deque->rightindex == -1) { + block *prevblock = deque->rightblock->leftlink; assert(deque->rightblock != NULL); - prevblock = deque->rightblock->leftlink; assert(deque->leftblock != deque->rightblock); freeblock(deque->rightblock); prevblock->rightlink = NULL; @@ -490,12 +489,12 @@ n += m; if (deque->leftindex == BLOCKLEN) { + block *nextblock = deque->leftblock->rightlink; assert(deque->leftblock != deque->rightblock); - prevblock = deque->leftblock->rightlink; freeblock(deque->leftblock); - assert(prevblock != NULL); - prevblock->leftlink = NULL; - deque->leftblock = prevblock; + assert(nextblock != NULL); + nextblock->leftlink = NULL; + deque->leftblock = nextblock; deque->leftindex = 0; } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 09:49:54 2013 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 7 Feb 2013 09:49:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Fix_Issue17069?= =?utf-8?q?=3A_Document_getcode_method_in_urllib=2Erequest=2Erst?= Message-ID: <3Z1tWQ1FnPzMRY@mail.python.org> http://hg.python.org/cpython/rev/fae8e212e870 changeset: 82035:fae8e212e870 branch: 3.2 parent: 82030:1fc87fa05333 user: Senthil Kumaran date: Thu Feb 07 00:47:01 2013 -0800 summary: Fix Issue17069: Document getcode method in urllib.request.rst files: Doc/library/urllib.request.rst | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -57,16 +57,24 @@ If neither *cafile* nor *capath* is specified, an HTTPS request will not do any verification of the server's certificate. - This function returns a file-like object that works as a :term:`context manager`, - with two additional methods from the :mod:`urllib.response` module + For http and https urls, this function returns a + :class:`http.client.HTTPResponse` object which has the following + :ref:`httpresponse-objects` methods. - * :meth:`geturl` --- return the URL of the resource retrieved, + For ftp, file, data urls and requests are explicity handled by legacy + :class:`URLopener` and :class:`FancyURLopener` class, this function returns + an :class:`urllib.response.addinfourl` object which can work as + :term:`context manager` and has methods such as + + * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, commonly used to determine if a redirect was followed - * :meth:`info` --- return the meta-information of the page, such as headers, + * :meth:`~urllib.response.addinfourl.info` --- return the meta-information of the page, such as headers, in the form of an :func:`email.message_from_string` instance (see `Quick Reference to HTTP Headers `_) + * :meth:`~urllib.response.addinfourl.getcode` -- return the HTTP status code of the response. + Raises :exc:`URLError` on errors. Note that ``None`` may be returned if no handler handles the request (though -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 09:49:55 2013 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 7 Feb 2013 09:49:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Fix_Issue17069=3A_Document_getcode_method_in_urllib=2Erequest?= =?utf-8?q?=2Erst?= Message-ID: <3Z1tWR3zRpzSZQ@mail.python.org> http://hg.python.org/cpython/rev/e15d2ad42d93 changeset: 82036:e15d2ad42d93 branch: 3.3 parent: 82031:4db932a303b4 parent: 82035:fae8e212e870 user: Senthil Kumaran date: Thu Feb 07 00:49:12 2013 -0800 summary: Fix Issue17069: Document getcode method in urllib.request.rst files: Doc/library/urllib.request.rst | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -63,16 +63,24 @@ an HTTPS request will not do any verification of the server's certificate. - This function returns a file-like object that works as a :term:`context manager`, - with two additional methods from the :mod:`urllib.response` module + For http and https urls, this function returns a + :class:`http.client.HTTPResponse` object which has the following + :ref:`httpresponse-objects` methods. - * :meth:`geturl` --- return the URL of the resource retrieved, + For ftp, file, data urls and requests are explicity handled by legacy + :class:`URLopener` and :class:`FancyURLopener` class, this function returns + an :class:`urllib.response.addinfourl` object which can work as + :term:`context manager` and has methods such as + + * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, commonly used to determine if a redirect was followed - * :meth:`info` --- return the meta-information of the page, such as headers, + * :meth:`~urllib.response.addinfourl.info` --- return the meta-information of the page, such as headers, in the form of an :func:`email.message_from_string` instance (see `Quick Reference to HTTP Headers `_) + * :meth:`~urllib.response.addinfourl.getcode` -- return the HTTP status code of the response. + Raises :exc:`URLError` on errors. Note that ``None`` may be returned if no handler handles the request (though -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 09:49:56 2013 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 7 Feb 2013 09:49:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fix_Issue17069=3A_Document_getcode_method_in_urllib=2Ere?= =?utf-8?q?quest=2Erst?= Message-ID: <3Z1tWS6yhMzSXB@mail.python.org> http://hg.python.org/cpython/rev/b79df3e8a9a0 changeset: 82037:b79df3e8a9a0 parent: 82034:2f3669aedc9a parent: 82036:e15d2ad42d93 user: Senthil Kumaran date: Thu Feb 07 00:50:02 2013 -0800 summary: Fix Issue17069: Document getcode method in urllib.request.rst files: Doc/library/urllib.request.rst | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -63,16 +63,24 @@ an HTTPS request will not do any verification of the server's certificate. - This function returns a file-like object that works as a :term:`context manager`, - with two additional methods from the :mod:`urllib.response` module + For http and https urls, this function returns a + :class:`http.client.HTTPResponse` object which has the following + :ref:`httpresponse-objects` methods. - * :meth:`geturl` --- return the URL of the resource retrieved, + For ftp, file, data urls and requests are explicity handled by legacy + :class:`URLopener` and :class:`FancyURLopener` class, this function returns + an :class:`urllib.response.addinfourl` object which can work as + :term:`context manager` and has methods such as + + * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, commonly used to determine if a redirect was followed - * :meth:`info` --- return the meta-information of the page, such as headers, + * :meth:`~urllib.response.addinfourl.info` --- return the meta-information of the page, such as headers, in the form of an :func:`email.message_from_string` instance (see `Quick Reference to HTTP Headers `_) + * :meth:`~urllib.response.addinfourl.getcode` -- return the HTTP status code of the response. + Raises :exc:`URLError` on errors. Note that ``None`` may be returned if no handler handles the request (though -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 09:49:58 2013 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 7 Feb 2013 09:49:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_Issue17069?= =?utf-8?q?=3A_Document_getcode_method_in_urllib=2Erequest=2Erst?= Message-ID: <3Z1tWV2WsdzSbr@mail.python.org> http://hg.python.org/cpython/rev/5630f0aff6ac changeset: 82038:5630f0aff6ac branch: 2.7 parent: 82033:d0cfabed2ef3 user: Senthil Kumaran date: Thu Feb 07 00:51:34 2013 -0800 summary: Fix Issue17069: Document getcode method in urllib.request.rst files: Doc/library/urllib2.rst | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst --- a/Doc/library/urllib2.rst +++ b/Doc/library/urllib2.rst @@ -52,6 +52,8 @@ in the form of an :class:`mimetools.Message` instance (see `Quick Reference to HTTP Headers `_) + * :meth:`getcode` --- return the HTTP status code of the response. + Raises :exc:`URLError` on errors. Note that ``None`` may be returned if no handler handles the request (though the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:02:59 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:02:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogRml4IHRlc3RfZnJv?= =?utf-8?q?m=5Fdll*_in_test=5Freturnfuncptrs=2Epy=2E?= Message-ID: <3Z207R5LTfzMYT@mail.python.org> http://hg.python.org/cpython/rev/8fb98fb758e8 changeset: 82039:8fb98fb758e8 branch: 2.7 user: Serhiy Storchaka date: Thu Feb 07 14:57:53 2013 +0200 summary: Fix test_from_dll* in test_returnfuncptrs.py. files: Lib/ctypes/test/test_returnfuncptrs.py | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Lib/ctypes/test/test_returnfuncptrs.py b/Lib/ctypes/test/test_returnfuncptrs.py --- a/Lib/ctypes/test/test_returnfuncptrs.py +++ b/Lib/ctypes/test/test_returnfuncptrs.py @@ -32,31 +32,30 @@ self.assertRaises(ArgumentError, strchr, "abcdef", 3) self.assertRaises(TypeError, strchr, "abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') def test_from_dll(self): dll = CDLL(_ctypes_test.__file__) # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("strchr", dll)) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("my_strchr", dll)) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") # Issue 6083: Reference counting bug - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') def test_from_dll_refcount(self): class BadSequence(tuple): def __getitem__(self, key): if key == 0: - return "strchr" + return "my_strchr" if key == 1: return CDLL(_ctypes_test.__file__) raise IndexError # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(BadSequence(("strchr", CDLL(_ctypes_test.__file__)))) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)( + BadSequence(("my_strchr", CDLL(_ctypes_test.__file__)))) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:03:01 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:03:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogRml4IHRlc3RfZnJv?= =?utf-8?q?m=5Fdll*_in_test=5Freturnfuncptrs=2Epy=2E?= Message-ID: <3Z207T0rjPzRNk@mail.python.org> http://hg.python.org/cpython/rev/ec70abe8c886 changeset: 82040:ec70abe8c886 branch: 3.2 parent: 82035:fae8e212e870 user: Serhiy Storchaka date: Thu Feb 07 14:58:44 2013 +0200 summary: Fix test_from_dll* in test_returnfuncptrs.py. files: Lib/ctypes/test/test_returnfuncptrs.py | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Lib/ctypes/test/test_returnfuncptrs.py b/Lib/ctypes/test/test_returnfuncptrs.py --- a/Lib/ctypes/test/test_returnfuncptrs.py +++ b/Lib/ctypes/test/test_returnfuncptrs.py @@ -34,31 +34,30 @@ self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') def test_from_dll(self): dll = CDLL(_ctypes_test.__file__) # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("strchr", dll)) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("my_strchr", dll)) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') # Issue 6083: Reference counting bug def test_from_dll_refcount(self): class BadSequence(tuple): def __getitem__(self, key): if key == 0: - return "strchr" + return "my_strchr" if key == 1: return CDLL(_ctypes_test.__file__) raise IndexError # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(BadSequence(("strchr", CDLL(_ctypes_test.__file__)))) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)( + BadSequence(("my_strchr", CDLL(_ctypes_test.__file__)))) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:03:02 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:03:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Fix_test=5Ffrom=5Fdll*_in_test=5Freturnfuncptrs=2Epy=2E?= Message-ID: <3Z207V3dljzSbS@mail.python.org> http://hg.python.org/cpython/rev/e49cc1585966 changeset: 82041:e49cc1585966 branch: 3.3 parent: 82036:e15d2ad42d93 parent: 82040:ec70abe8c886 user: Serhiy Storchaka date: Thu Feb 07 14:59:25 2013 +0200 summary: Fix test_from_dll* in test_returnfuncptrs.py. files: Lib/ctypes/test/test_returnfuncptrs.py | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Lib/ctypes/test/test_returnfuncptrs.py b/Lib/ctypes/test/test_returnfuncptrs.py --- a/Lib/ctypes/test/test_returnfuncptrs.py +++ b/Lib/ctypes/test/test_returnfuncptrs.py @@ -34,31 +34,30 @@ self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') def test_from_dll(self): dll = CDLL(_ctypes_test.__file__) # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("strchr", dll)) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("my_strchr", dll)) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') # Issue 6083: Reference counting bug def test_from_dll_refcount(self): class BadSequence(tuple): def __getitem__(self, key): if key == 0: - return "strchr" + return "my_strchr" if key == 1: return CDLL(_ctypes_test.__file__) raise IndexError # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(BadSequence(("strchr", CDLL(_ctypes_test.__file__)))) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)( + BadSequence(("my_strchr", CDLL(_ctypes_test.__file__)))) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:03:03 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:03:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fix_test=5Ffrom=5Fdll*_in_test=5Freturnfuncptrs=2Epy=2E?= Message-ID: <3Z207W6XNNzSc3@mail.python.org> http://hg.python.org/cpython/rev/3236ebe7dd82 changeset: 82042:3236ebe7dd82 parent: 82037:b79df3e8a9a0 parent: 82041:e49cc1585966 user: Serhiy Storchaka date: Thu Feb 07 15:00:02 2013 +0200 summary: Fix test_from_dll* in test_returnfuncptrs.py. files: Lib/ctypes/test/test_returnfuncptrs.py | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Lib/ctypes/test/test_returnfuncptrs.py b/Lib/ctypes/test/test_returnfuncptrs.py --- a/Lib/ctypes/test/test_returnfuncptrs.py +++ b/Lib/ctypes/test/test_returnfuncptrs.py @@ -34,31 +34,30 @@ self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') def test_from_dll(self): dll = CDLL(_ctypes_test.__file__) # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("strchr", dll)) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(("my_strchr", dll)) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) self.assertRaises(TypeError, strchr, b"abcdef") - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') # Issue 6083: Reference counting bug def test_from_dll_refcount(self): class BadSequence(tuple): def __getitem__(self, key): if key == 0: - return "strchr" + return "my_strchr" if key == 1: return CDLL(_ctypes_test.__file__) raise IndexError # _CFuncPtr instances are now callable with a tuple argument # which denotes a function name and a dll: - strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)(BadSequence(("strchr", CDLL(_ctypes_test.__file__)))) + strchr = CFUNCTYPE(c_char_p, c_char_p, c_char)( + BadSequence(("my_strchr", CDLL(_ctypes_test.__file__)))) self.assertTrue(strchr(b"abcdef", b"b"), "bcdef") self.assertEqual(strchr(b"abcdef", b"x"), None) self.assertRaises(ArgumentError, strchr, b"abcdef", 3.0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:27:15 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:27:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE3MTE0?= =?utf-8?q?=3A_IDLE=C2=A0now_uses_non-strict_config_parser=2E?= Message-ID: <3Z20gR0cdSzSMW@mail.python.org> http://hg.python.org/cpython/rev/cf98766f464e changeset: 82043:cf98766f464e branch: 3.2 parent: 82040:ec70abe8c886 user: Serhiy Storchaka date: Thu Feb 07 15:24:36 2013 +0200 summary: Issue #17114: IDLE?now uses non-strict config parser. files: Lib/idlelib/configHandler.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -37,7 +37,7 @@ cfgFile - string, fully specified configuration file name """ self.file=cfgFile - ConfigParser.__init__(self,defaults=cfgDefaults) + ConfigParser.__init__(self, defaults=cfgDefaults, strict=False) def Get(self, section, option, type=None, default=None, raw=False): """ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -212,6 +212,8 @@ Library ------- +- Issue #17114: IDLE?now uses non-strict config parser. + - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:27:16 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:27:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2317114=3A_IDLE=C2=A0now_uses_non-strict_config_parser?= =?utf-8?q?=2E?= Message-ID: <3Z20gS3cJvzSVb@mail.python.org> http://hg.python.org/cpython/rev/c2ed79fbb9c6 changeset: 82044:c2ed79fbb9c6 branch: 3.3 parent: 82041:e49cc1585966 parent: 82043:cf98766f464e user: Serhiy Storchaka date: Thu Feb 07 15:25:09 2013 +0200 summary: Issue #17114: IDLE?now uses non-strict config parser. files: Lib/idlelib/configHandler.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -37,7 +37,7 @@ cfgFile - string, fully specified configuration file name """ self.file=cfgFile - ConfigParser.__init__(self,defaults=cfgDefaults) + ConfigParser.__init__(self, defaults=cfgDefaults, strict=False) def Get(self, section, option, type=None, default=None, raw=False): """ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -163,6 +163,8 @@ Library ------- +- Issue #17114: IDLE?now uses non-strict config parser. + - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:27:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:27:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317114=3A_IDLE=C2=A0now_uses_non-strict_config_p?= =?utf-8?q?arser=2E?= Message-ID: <3Z20gT6mQ2zSbS@mail.python.org> http://hg.python.org/cpython/rev/877fae8d6f5b changeset: 82045:877fae8d6f5b parent: 82042:3236ebe7dd82 parent: 82044:c2ed79fbb9c6 user: Serhiy Storchaka date: Thu Feb 07 15:25:33 2013 +0200 summary: Issue #17114: IDLE?now uses non-strict config parser. files: Lib/idlelib/configHandler.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -37,7 +37,7 @@ cfgFile - string, fully specified configuration file name """ self.file=cfgFile - ConfigParser.__init__(self,defaults=cfgDefaults) + ConfigParser.__init__(self, defaults=cfgDefaults, strict=False) def Get(self, section, option, type=None, default=None, raw=False): """ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,8 @@ Library ------- +- Issue #17114: IDLE?now uses non-strict config parser. + - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:42:49 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:42:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3MTE4?= =?utf-8?q?=3A_Add_new_tests_for_testing_Python-Tcl_interaction=2E?= Message-ID: <3Z211P4N6rzSXk@mail.python.org> http://hg.python.org/cpython/rev/f7cc6fbd7ae1 changeset: 82046:f7cc6fbd7ae1 branch: 2.7 parent: 82039:8fb98fb758e8 user: Serhiy Storchaka date: Thu Feb 07 15:37:53 2013 +0200 summary: Issue #17118: Add new tests for testing Python-Tcl interaction. files: Lib/test/test_tcl.py | 22 ++++++++++++++++++++++ 1 files changed, 22 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -1,6 +1,7 @@ #!/usr/bin/env python import unittest +import sys import os from test import test_support @@ -151,6 +152,27 @@ # exit code must be zero self.assertEqual(f.close(), None) + def test_passing_values(self): + def passValue(value): + return self.interp.call('set', '_', value) + self.assertEqual(passValue(True), True) + self.assertEqual(passValue(False), False) + self.assertEqual(passValue('string'), 'string') + self.assertEqual(passValue('string\u20ac'), 'string\u20ac') + self.assertEqual(passValue(u'string'), u'string') + self.assertEqual(passValue(u'string\u20ac'), u'string\u20ac') + for i in (0, 1, -1, int(2**31-1), int(-2**31)): + self.assertEqual(passValue(i), i) + for f in (0.0, 1.0, -1.0, 1/3, + sys.float_info.min, sys.float_info.max, + -sys.float_info.min, -sys.float_info.max): + self.assertEqual(passValue(f), f) + for f in float('nan'), float('inf'), -float('inf'): + if f != f: # NaN + self.assertNotEqual(passValue(f), f) + else: + self.assertEqual(passValue(f), f) + self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) def test_main(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:42:51 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:42:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE3MTE4?= =?utf-8?q?=3A_Add_new_tests_for_testing_Python-Tcl_interaction=2E?= Message-ID: <3Z211R05B2zSbY@mail.python.org> http://hg.python.org/cpython/rev/148e6ebfe854 changeset: 82047:148e6ebfe854 branch: 3.2 parent: 82043:cf98766f464e user: Serhiy Storchaka date: Thu Feb 07 15:40:03 2013 +0200 summary: Issue #17118: Add new tests for testing Python-Tcl interaction. files: Lib/test/test_tcl.py | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -151,6 +151,26 @@ # exit code must be zero self.assertEqual(f.close(), None) + def test_passing_values(self): + def passValue(value): + return self.interp.call('set', '_', value) + + self.assertEqual(passValue(True), True) + self.assertEqual(passValue(False), False) + self.assertEqual(passValue('string'), 'string') + self.assertEqual(passValue('string\u20ac'), 'string\u20ac') + for i in (0, 1, -1, 2**31-1, -2**31): + self.assertEqual(passValue(i), i) + for f in (0.0, 1.0, -1.0, 1/3, + sys.float_info.min, sys.float_info.max, + -sys.float_info.min, -sys.float_info.max): + self.assertEqual(passValue(f), f) + for f in float('nan'), float('inf'), -float('inf'): + if f != f: # NaN + self.assertNotEqual(passValue(f), f) + else: + self.assertEqual(passValue(f), f) + self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) def test_main(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:42:52 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:42:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2317118=3A_Add_new_tests_for_testing_Python-Tcl_interac?= =?utf-8?q?tion=2E?= Message-ID: <3Z211S2gG7zScd@mail.python.org> http://hg.python.org/cpython/rev/452344620c97 changeset: 82048:452344620c97 branch: 3.3 parent: 82044:c2ed79fbb9c6 parent: 82047:148e6ebfe854 user: Serhiy Storchaka date: Thu Feb 07 15:40:26 2013 +0200 summary: Issue #17118: Add new tests for testing Python-Tcl interaction. files: Lib/test/test_tcl.py | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -151,6 +151,26 @@ # exit code must be zero self.assertEqual(f.close(), None) + def test_passing_values(self): + def passValue(value): + return self.interp.call('set', '_', value) + + self.assertEqual(passValue(True), True) + self.assertEqual(passValue(False), False) + self.assertEqual(passValue('string'), 'string') + self.assertEqual(passValue('string\u20ac'), 'string\u20ac') + for i in (0, 1, -1, 2**31-1, -2**31): + self.assertEqual(passValue(i), i) + for f in (0.0, 1.0, -1.0, 1/3, + sys.float_info.min, sys.float_info.max, + -sys.float_info.min, -sys.float_info.max): + self.assertEqual(passValue(f), f) + for f in float('nan'), float('inf'), -float('inf'): + if f != f: # NaN + self.assertNotEqual(passValue(f), f) + else: + self.assertEqual(passValue(f), f) + self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) def test_main(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 14:42:53 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 14:42:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317118=3A_Add_new_tests_for_testing_Python-Tcl_i?= =?utf-8?q?nteraction=2E?= Message-ID: <3Z211T5JzFzScM@mail.python.org> http://hg.python.org/cpython/rev/f0d603948cff changeset: 82049:f0d603948cff parent: 82045:877fae8d6f5b parent: 82048:452344620c97 user: Serhiy Storchaka date: Thu Feb 07 15:40:48 2013 +0200 summary: Issue #17118: Add new tests for testing Python-Tcl interaction. files: Lib/test/test_tcl.py | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -151,6 +151,26 @@ # exit code must be zero self.assertEqual(f.close(), None) + def test_passing_values(self): + def passValue(value): + return self.interp.call('set', '_', value) + + self.assertEqual(passValue(True), True) + self.assertEqual(passValue(False), False) + self.assertEqual(passValue('string'), 'string') + self.assertEqual(passValue('string\u20ac'), 'string\u20ac') + for i in (0, 1, -1, 2**31-1, -2**31): + self.assertEqual(passValue(i), i) + for f in (0.0, 1.0, -1.0, 1/3, + sys.float_info.min, sys.float_info.max, + -sys.float_info.min, -sys.float_info.max): + self.assertEqual(passValue(f), f) + for f in float('nan'), float('inf'), -float('inf'): + if f != f: # NaN + self.assertNotEqual(passValue(f), f) + else: + self.assertEqual(passValue(f), f) + self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) def test_main(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 15:30:48 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 15:30:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3MDQz?= =?utf-8?q?=3A_The_unicode-internal_decoder_no_longer_read_past_the_end_of?= Message-ID: <3Z224m2NzqzSZ4@mail.python.org> http://hg.python.org/cpython/rev/498b54e0e856 changeset: 82050:498b54e0e856 branch: 2.7 parent: 82046:f7cc6fbd7ae1 user: Serhiy Storchaka date: Thu Feb 07 16:23:11 2013 +0200 summary: Issue #17043: The unicode-internal decoder no longer read past the end of input buffer. files: Misc/NEWS | 3 + Objects/unicodeobject.c | 51 +++++++++++++--------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,9 @@ Core and Builtins ----------------- +- Issue #17043: The unicode-internal decoder no longer read past the end of + input buffer. + - Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. - Issue #10156: In the interpreter's initialization phase, unicode globals diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3376,37 +3376,34 @@ end = s + size; while (s < end) { + if (end-s < Py_UNICODE_SIZE) { + endinpos = end-starts; + reason = "truncated input"; + goto error; + } memcpy(p, s, sizeof(Py_UNICODE)); +#ifdef Py_UNICODE_WIDE /* We have to sanity check the raw data, otherwise doom looms for some malformed UCS-4 data. */ - if ( -#ifdef Py_UNICODE_WIDE - *p > unimax || *p < 0 || + if (*p > unimax || *p < 0) { + endinpos = s - starts + Py_UNICODE_SIZE; + reason = "illegal code point (> 0x10FFFF)"; + goto error; + } #endif - end-s < Py_UNICODE_SIZE - ) - { - startinpos = s - starts; - if (end-s < Py_UNICODE_SIZE) { - endinpos = end-starts; - reason = "truncated input"; - } - else { - endinpos = s - starts + Py_UNICODE_SIZE; - reason = "illegal code point (> 0x10FFFF)"; - } - outpos = p - PyUnicode_AS_UNICODE(v); - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "unicode_internal", reason, - starts, size, &startinpos, &endinpos, &exc, &s, - &v, &outpos, &p)) { - goto onError; - } - } - else { - p++; - s += Py_UNICODE_SIZE; + p++; + s += Py_UNICODE_SIZE; + continue; + + error: + startinpos = s - starts; + outpos = p - PyUnicode_AS_UNICODE(v); + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "unicode_internal", reason, + starts, size, &startinpos, &endinpos, &exc, &s, + &v, &outpos, &p)) { + goto onError; } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 15:30:49 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 15:30:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE3MDQz?= =?utf-8?q?=3A_The_unicode-internal_decoder_no_longer_read_past_the_end_of?= Message-ID: <3Z224n683XzSZV@mail.python.org> http://hg.python.org/cpython/rev/0f1c2e2b6bc2 changeset: 82051:0f1c2e2b6bc2 branch: 3.2 parent: 82047:148e6ebfe854 user: Serhiy Storchaka date: Thu Feb 07 16:23:21 2013 +0200 summary: Issue #17043: The unicode-internal decoder no longer read past the end of input buffer. files: Misc/NEWS | 3 + Objects/unicodeobject.c | 51 +++++++++++++--------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #17043: The unicode-internal decoder no longer read past the end of + input buffer. + - Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. - Issue #10156: In the interpreter's initialization phase, unicode globals diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4392,37 +4392,34 @@ end = s + size; while (s < end) { + if (end-s < Py_UNICODE_SIZE) { + endinpos = end-starts; + reason = "truncated input"; + goto error; + } memcpy(p, s, sizeof(Py_UNICODE)); +#ifdef Py_UNICODE_WIDE /* We have to sanity check the raw data, otherwise doom looms for some malformed UCS-4 data. */ - if ( -#ifdef Py_UNICODE_WIDE - *p > unimax || *p < 0 || + if (*p > unimax || *p < 0) { + endinpos = s - starts + Py_UNICODE_SIZE; + reason = "illegal code point (> 0x10FFFF)"; + goto error; + } #endif - end-s < Py_UNICODE_SIZE - ) - { - startinpos = s - starts; - if (end-s < Py_UNICODE_SIZE) { - endinpos = end-starts; - reason = "truncated input"; - } - else { - endinpos = s - starts + Py_UNICODE_SIZE; - reason = "illegal code point (> 0x10FFFF)"; - } - outpos = p - PyUnicode_AS_UNICODE(v); - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "unicode_internal", reason, - &starts, &end, &startinpos, &endinpos, &exc, &s, - &v, &outpos, &p)) { - goto onError; - } - } - else { - p++; - s += Py_UNICODE_SIZE; + p++; + s += Py_UNICODE_SIZE; + continue; + + error: + startinpos = s - starts; + outpos = p - PyUnicode_AS_UNICODE(v); + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "unicode_internal", reason, + &starts, &end, &startinpos, &endinpos, &exc, &s, + &v, &outpos, &p)) { + goto onError; } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 15:30:51 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 15:30:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2317043=3A_The_unicode-internal_decoder_no_longer_read_?= =?utf-8?q?past_the_end_of?= Message-ID: <3Z224q2dLhzScS@mail.python.org> http://hg.python.org/cpython/rev/fec2976c8503 changeset: 82052:fec2976c8503 branch: 3.3 parent: 82048:452344620c97 parent: 82051:0f1c2e2b6bc2 user: Serhiy Storchaka date: Thu Feb 07 16:25:25 2013 +0200 summary: Issue #17043: The unicode-internal decoder no longer read past the end of input buffer. files: Misc/NEWS | 3 + Objects/unicodeobject.c | 50 +++++++++++++--------------- 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,6 +12,9 @@ Core and Builtins ----------------- +- Issue #17043: The unicode-internal decoder no longer read past the end of + input buffer. + - Issue #17098: All modules now have __loader__ set even if they pre-exist the bootstrapping of importlib. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6103,6 +6103,11 @@ while (s < end) { Py_UNICODE uch; Py_UCS4 ch; + if (end - s < Py_UNICODE_SIZE) { + endinpos = end-starts; + reason = "truncated input"; + goto error; + } /* We copy the raw representation one byte at a time because the pointer may be unaligned (see test_codeccallbacks). */ ((char *) &uch)[0] = s[0]; @@ -6112,37 +6117,18 @@ ((char *) &uch)[3] = s[3]; #endif ch = uch; - +#ifdef Py_UNICODE_WIDE /* We have to sanity check the raw data, otherwise doom looms for some malformed UCS-4 data. */ - if ( -#ifdef Py_UNICODE_WIDE - ch > 0x10ffff || -#endif - end-s < Py_UNICODE_SIZE - ) - { - startinpos = s - starts; - if (end-s < Py_UNICODE_SIZE) { - endinpos = end-starts; - reason = "truncated input"; - } - else { - endinpos = s - starts + Py_UNICODE_SIZE; - reason = "illegal code point (> 0x10FFFF)"; - } - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "unicode_internal", reason, - &starts, &end, &startinpos, &endinpos, &exc, &s, - &v, &outpos)) - goto onError; - continue; - } - + if (ch > 0x10ffff) { + endinpos = s - starts + Py_UNICODE_SIZE; + reason = "illegal code point (> 0x10FFFF)"; + goto error; + } +#endif s += Py_UNICODE_SIZE; #ifndef Py_UNICODE_WIDE - if (Py_UNICODE_IS_HIGH_SURROGATE(ch) && s < end) + if (Py_UNICODE_IS_HIGH_SURROGATE(ch) && end - s >= Py_UNICODE_SIZE) { Py_UNICODE uch2; ((char *) &uch2)[0] = s[0]; @@ -6157,6 +6143,16 @@ if (unicode_putchar(&v, &outpos, ch) < 0) goto onError; + continue; + + error: + startinpos = s - starts; + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "unicode_internal", reason, + &starts, &end, &startinpos, &endinpos, &exc, &s, + &v, &outpos)) + goto onError; } if (unicode_resize(&v, outpos) < 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 15:30:52 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 15:30:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317043=3A_The_unicode-internal_decoder_no_longer?= =?utf-8?q?_read_past_the_end_of?= Message-ID: <3Z224r5gpDzScr@mail.python.org> http://hg.python.org/cpython/rev/eb0370d4686c changeset: 82053:eb0370d4686c parent: 82049:f0d603948cff parent: 82052:fec2976c8503 user: Serhiy Storchaka date: Thu Feb 07 16:26:55 2013 +0200 summary: Issue #17043: The unicode-internal decoder no longer read past the end of input buffer. files: Misc/NEWS | 3 + Objects/unicodeobject.c | 50 +++++++++++++--------------- 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #17043: The unicode-internal decoder no longer read past the end of + input buffer. + - Issue #17098: All modules now have __loader__ set even if they pre-exist the bootstrapping of importlib. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5976,6 +5976,11 @@ while (s < end) { Py_UNICODE uch; Py_UCS4 ch; + if (end - s < Py_UNICODE_SIZE) { + endinpos = end-starts; + reason = "truncated input"; + goto error; + } /* We copy the raw representation one byte at a time because the pointer may be unaligned (see test_codeccallbacks). */ ((char *) &uch)[0] = s[0]; @@ -5985,37 +5990,18 @@ ((char *) &uch)[3] = s[3]; #endif ch = uch; - +#ifdef Py_UNICODE_WIDE /* We have to sanity check the raw data, otherwise doom looms for some malformed UCS-4 data. */ - if ( -#ifdef Py_UNICODE_WIDE - ch > 0x10ffff || -#endif - end-s < Py_UNICODE_SIZE - ) - { - startinpos = s - starts; - if (end-s < Py_UNICODE_SIZE) { - endinpos = end-starts; - reason = "truncated input"; - } - else { - endinpos = s - starts + Py_UNICODE_SIZE; - reason = "illegal code point (> 0x10FFFF)"; - } - if (unicode_decode_call_errorhandler_writer( - errors, &errorHandler, - "unicode_internal", reason, - &starts, &end, &startinpos, &endinpos, &exc, &s, - &writer)) - goto onError; - continue; - } - + if (ch > 0x10ffff) { + endinpos = s - starts + Py_UNICODE_SIZE; + reason = "illegal code point (> 0x10FFFF)"; + goto error; + } +#endif s += Py_UNICODE_SIZE; #ifndef Py_UNICODE_WIDE - if (Py_UNICODE_IS_HIGH_SURROGATE(ch) && s < end) + if (Py_UNICODE_IS_HIGH_SURROGATE(ch) && end - s >= Py_UNICODE_SIZE) { Py_UNICODE uch2; ((char *) &uch2)[0] = s[0]; @@ -6032,6 +6018,16 @@ goto onError; PyUnicode_WRITE(writer.kind, writer.data, writer.pos, ch); writer.pos++; + continue; + + error: + startinpos = s - starts; + if (unicode_decode_call_errorhandler_writer( + errors, &errorHandler, + "unicode_internal", reason, + &starts, &end, &startinpos, &endinpos, &exc, &s, + &writer)) + goto onError; } Py_XDECREF(errorHandler); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 16:08:28 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 16:08:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3MDcz?= =?utf-8?q?=3A_Fix_some_integer_overflows_in_sqlite3_module=2E?= Message-ID: <3Z22wD508JzSZq@mail.python.org> http://hg.python.org/cpython/rev/649937bb8f1c changeset: 82054:649937bb8f1c branch: 2.7 parent: 82050:498b54e0e856 user: Serhiy Storchaka date: Thu Feb 07 16:59:34 2013 +0200 summary: Issue #17073: Fix some integer overflows in sqlite3 module. files: Lib/sqlite3/test/hooks.py | 19 ++++ Lib/sqlite3/test/userfunctions.py | 60 ++++++++++-- Misc/NEWS | 2 + Modules/_sqlite/connection.c | 84 ++++++++++-------- Modules/_sqlite/cursor.c | 20 +--- Modules/_sqlite/statement.c | 23 ++-- Modules/_sqlite/util.c | 66 ++++++++++++++ Modules/_sqlite/util.h | 4 + 8 files changed, 203 insertions(+), 75 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -76,6 +76,25 @@ except sqlite.OperationalError, e: self.assertEqual(e.args[0].lower(), "no such collation sequence: mycoll") + def CheckCollationReturnsLargeInteger(self): + def mycoll(x, y): + # reverse order + return -((x > y) - (x < y)) * 2**32 + con = sqlite.connect(":memory:") + con.create_collation("mycoll", mycoll) + sql = """ + select x from ( + select 'a' as x + union + select 'b' as x + union + select 'c' as x + ) order by x collate mycoll + """ + result = con.execute(sql).fetchall() + self.assertEqual(result, [('c',), ('b',), ('a',)], + msg="the expected order was not returned") + def CheckCollationRegisterTwice(self): """ Register two different collation functions under the same name. diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py --- a/Lib/sqlite3/test/userfunctions.py +++ b/Lib/sqlite3/test/userfunctions.py @@ -374,14 +374,15 @@ val = cur.fetchone()[0] self.assertEqual(val, 60) -def authorizer_cb(action, arg1, arg2, dbname, source): - if action != sqlite.SQLITE_SELECT: - return sqlite.SQLITE_DENY - if arg2 == 'c2' or arg1 == 't2': - return sqlite.SQLITE_DENY - return sqlite.SQLITE_OK +class AuthorizerTests(unittest.TestCase): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return sqlite.SQLITE_DENY + if arg2 == 'c2' or arg1 == 't2': + return sqlite.SQLITE_DENY + return sqlite.SQLITE_OK -class AuthorizerTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") self.con.executescript(""" @@ -394,12 +395,12 @@ # For our security test: self.con.execute("select c2 from t2") - self.con.set_authorizer(authorizer_cb) + self.con.set_authorizer(self.authorizer_cb) def tearDown(self): pass - def CheckTableAccess(self): + def test_table_access(self): try: self.con.execute("select * from t2") except sqlite.DatabaseError, e: @@ -408,7 +409,7 @@ return self.fail("should have raised an exception due to missing privileges") - def CheckColumnAccess(self): + def test_column_access(self): try: self.con.execute("select c2 from t1") except sqlite.DatabaseError, e: @@ -417,11 +418,46 @@ return self.fail("should have raised an exception due to missing privileges") +class AuthorizerRaiseExceptionTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + raise ValueError + if arg2 == 'c2' or arg1 == 't2': + raise ValueError + return sqlite.SQLITE_OK + +class AuthorizerIllegalTypeTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 0.0 + if arg2 == 'c2' or arg1 == 't2': + return 0.0 + return sqlite.SQLITE_OK + +class AuthorizerLargeIntegerTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 2**32 + if arg2 == 'c2' or arg1 == 't2': + return 2**32 + return sqlite.SQLITE_OK + + def suite(): function_suite = unittest.makeSuite(FunctionTests, "Check") aggregate_suite = unittest.makeSuite(AggregateTests, "Check") - authorizer_suite = unittest.makeSuite(AuthorizerTests, "Check") - return unittest.TestSuite((function_suite, aggregate_suite, authorizer_suite)) + authorizer_suite = unittest.makeSuite(AuthorizerTests) + return unittest.TestSuite(( + function_suite, + aggregate_suite, + authorizer_suite, + unittest.makeSuite(AuthorizerRaiseExceptionTests), + unittest.makeSuite(AuthorizerIllegalTypeTests), + unittest.makeSuite(AuthorizerLargeIntegerTests), + )) def test(): runner = unittest.TextTestRunner() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -202,6 +202,8 @@ Library ------- +- Issue #17073: Fix some integer overflows in sqlite3 module. + - Issue #6083: Fix multiple segmentation faults occured when PyArg_ParseTuple parses nested mutating sequence. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -538,39 +538,40 @@ } } -void _pysqlite_set_result(sqlite3_context* context, PyObject* py_val) +static int +_pysqlite_set_result(sqlite3_context* context, PyObject* py_val) { - const char* buffer; - Py_ssize_t buflen; - PyObject* stringval; - - if ((!py_val) || PyErr_Occurred()) { - sqlite3_result_null(context); - } else if (py_val == Py_None) { + if (py_val == Py_None) { sqlite3_result_null(context); } else if (PyInt_Check(py_val)) { sqlite3_result_int64(context, (sqlite_int64)PyInt_AsLong(py_val)); } else if (PyLong_Check(py_val)) { - sqlite3_result_int64(context, PyLong_AsLongLong(py_val)); + sqlite_int64 value = _pysqlite_long_as_int64(py_val); + if (value == -1 && PyErr_Occurred()) + return -1; + sqlite3_result_int64(context, value); } else if (PyFloat_Check(py_val)) { sqlite3_result_double(context, PyFloat_AsDouble(py_val)); } else if (PyBuffer_Check(py_val)) { + const char* buffer; + Py_ssize_t buflen; if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) { PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - } else { - sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); + return -1; } + sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); } else if (PyString_Check(py_val)) { sqlite3_result_text(context, PyString_AsString(py_val), -1, SQLITE_TRANSIENT); } else if (PyUnicode_Check(py_val)) { - stringval = PyUnicode_AsUTF8String(py_val); - if (stringval) { - sqlite3_result_text(context, PyString_AsString(stringval), -1, SQLITE_TRANSIENT); - Py_DECREF(stringval); - } + PyObject * stringval = PyUnicode_AsUTF8String(py_val); + if (!stringval) + return -1; + sqlite3_result_text(context, PyString_AsString(stringval), -1, SQLITE_TRANSIENT); + Py_DECREF(stringval); } else { - /* TODO: raise error */ + return -1; } + return 0; } PyObject* _pysqlite_build_py_params(sqlite3_context *context, int argc, sqlite3_value** argv) @@ -580,7 +581,6 @@ sqlite3_value* cur_value; PyObject* cur_py_value; const char* val_str; - sqlite_int64 val_int; Py_ssize_t buflen; void* raw_buffer; @@ -593,11 +593,7 @@ cur_value = argv[i]; switch (sqlite3_value_type(argv[i])) { case SQLITE_INTEGER: - val_int = sqlite3_value_int64(cur_value); - if(val_int < LONG_MIN || val_int > LONG_MAX) - cur_py_value = PyLong_FromLongLong(val_int); - else - cur_py_value = PyInt_FromLong((long)val_int); + cur_py_value = _pysqlite_long_from_int64(sqlite3_value_int64(cur_value)); break; case SQLITE_FLOAT: cur_py_value = PyFloat_FromDouble(sqlite3_value_double(cur_value)); @@ -648,6 +644,7 @@ PyObject* args; PyObject* py_func; PyObject* py_retval = NULL; + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -663,10 +660,12 @@ Py_DECREF(args); } + ok = 0; if (py_retval) { - _pysqlite_set_result(context, py_retval); + ok = _pysqlite_set_result(context, py_retval) == 0; Py_DECREF(py_retval); - } else { + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { @@ -746,8 +745,9 @@ void _pysqlite_final_callback(sqlite3_context* context) { - PyObject* function_result = NULL; + PyObject* function_result; PyObject** aggregate_instance; + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -764,21 +764,23 @@ } function_result = PyObject_CallMethod(*aggregate_instance, "finalize", ""); - if (!function_result) { + Py_DECREF(*aggregate_instance); + + ok = 0; + if (function_result) { + ok = _pysqlite_set_result(context, function_result) == 0; + Py_DECREF(function_result); + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); } _sqlite3_result_error(context, "user-defined aggregate's 'finalize' method raised error", -1); - } else { - _pysqlite_set_result(context, function_result); } error: - Py_XDECREF(*aggregate_instance); - Py_XDECREF(function_result); - #ifdef WITH_THREAD PyGILState_Release(threadstate); #endif @@ -935,7 +937,9 @@ rc = SQLITE_DENY; } else { if (PyInt_Check(ret)) { - rc = (int)PyInt_AsLong(ret); + rc = _PyInt_AsInt(ret); + if (rc == -1 && PyErr_Occurred()) + rc = SQLITE_DENY; } else { rc = SQLITE_DENY; } @@ -967,7 +971,7 @@ } /* abort query if error occurred */ - rc = 1; + rc = 1; } else { rc = (int)PyObject_IsTrue(ret); Py_DECREF(ret); @@ -1337,6 +1341,7 @@ PyGILState_STATE gilstate; #endif PyObject* retval = NULL; + long longval; int result = 0; #ifdef WITH_THREAD gilstate = PyGILState_Ensure(); @@ -1360,10 +1365,17 @@ goto finally; } - result = PyInt_AsLong(retval); - if (PyErr_Occurred()) { + longval = PyLong_AsLongAndOverflow(retval, &result); + if (longval == -1 && PyErr_Occurred()) { + PyErr_Clear(); result = 0; } + else if (!result) { + if (longval > 0) + result = 1; + else if (longval < 0) + result = -1; + } finally: Py_XDECREF(string1); diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -26,14 +26,6 @@ #include "util.h" #include "sqlitecompat.h" -/* used to decide wether to call PyInt_FromLong or PyLong_FromLongLong */ -#ifndef INT32_MIN -#define INT32_MIN (-2147483647 - 1) -#endif -#ifndef INT32_MAX -#define INT32_MAX 2147483647 -#endif - PyObject* pysqlite_cursor_iternext(pysqlite_Cursor* self); static char* errmsg_fetch_across_rollback = "Cursor needed to be reset because of commit/rollback and can no longer be fetched from."; @@ -307,7 +299,6 @@ PyObject* row; PyObject* item = NULL; int coltype; - PY_LONG_LONG intval; PyObject* converter; PyObject* converted; Py_ssize_t nbytes; @@ -366,12 +357,7 @@ Py_INCREF(Py_None); converted = Py_None; } else if (coltype == SQLITE_INTEGER) { - intval = sqlite3_column_int64(self->statement->st, i); - if (intval < INT32_MIN || intval > INT32_MAX) { - converted = PyLong_FromLongLong(intval); - } else { - converted = PyInt_FromLong((long)intval); - } + converted = _pysqlite_long_from_int64(sqlite3_column_int64(self->statement->st, i)); } else if (coltype == SQLITE_FLOAT) { converted = PyFloat_FromDouble(sqlite3_column_double(self->statement->st, i)); } else if (coltype == SQLITE_TEXT) { @@ -466,7 +452,6 @@ PyObject* func_args; PyObject* result; int numcols; - PY_LONG_LONG lastrowid; int statement_type; PyObject* descriptor; PyObject* second_argument = NULL; @@ -747,10 +732,11 @@ Py_DECREF(self->lastrowid); if (!multiple && statement_type == STATEMENT_INSERT) { + sqlite3_int64 lastrowid; Py_BEGIN_ALLOW_THREADS lastrowid = sqlite3_last_insert_rowid(self->connection->db); Py_END_ALLOW_THREADS - self->lastrowid = PyInt_FromLong((long)lastrowid); + self->lastrowid = _pysqlite_long_from_int64(lastrowid); } else { Py_INCREF(Py_None); self->lastrowid = Py_None; diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -26,6 +26,7 @@ #include "connection.h" #include "microprotocols.h" #include "prepare_protocol.h" +#include "util.h" #include "sqlitecompat.h" /* prototypes */ @@ -101,8 +102,6 @@ int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter, int allow_8bit_chars) { int rc = SQLITE_OK; - long longval; - PY_LONG_LONG longlongval; const char* buffer; char* string; Py_ssize_t buflen; @@ -153,15 +152,19 @@ } switch (paramtype) { - case TYPE_INT: - longval = PyInt_AsLong(parameter); - rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longval); + case TYPE_INT: { + long longval = PyInt_AsLong(parameter); + rc = sqlite3_bind_int64(self->st, pos, longval); break; - case TYPE_LONG: - longlongval = PyLong_AsLongLong(parameter); - /* in the overflow error case, longlongval is -1, and an exception is set */ - rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longlongval); + } + case TYPE_LONG: { + sqlite_int64 value = _pysqlite_long_as_int64(parameter); + if (value == -1 && PyErr_Occurred()) + rc = -1; + else + rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)value); break; + } case TYPE_FLOAT: rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); break; @@ -198,7 +201,7 @@ return 1; } - if (PyInt_CheckExact(obj) || PyLong_CheckExact(obj) + if (PyInt_CheckExact(obj) || PyLong_CheckExact(obj) || PyFloat_CheckExact(obj) || PyString_CheckExact(obj) || PyUnicode_CheckExact(obj) || PyBuffer_Check(obj)) { return 0; diff --git a/Modules/_sqlite/util.c b/Modules/_sqlite/util.c --- a/Modules/_sqlite/util.c +++ b/Modules/_sqlite/util.c @@ -104,3 +104,69 @@ return errorcode; } +#ifdef WORDS_BIGENDIAN +# define IS_LITTLE_ENDIAN 0 +#else +# define IS_LITTLE_ENDIAN 1 +#endif + +PyObject * +_pysqlite_long_from_int64(sqlite3_int64 value) +{ +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG < 8 + if (value > PY_LLONG_MAX || value < PY_LLONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +# if SIZEOF_LONG < SIZEOF_LONG_LONG + if (value > LONG_MAX || value < LONG_MIN) + return PyLong_FromLongLong(value); +# endif +#else +# if SIZEOF_LONG < 8 + if (value > LONG_MAX || value < LONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +#endif + return PyInt_FromLong(value); +} + +sqlite3_int64 +_pysqlite_long_as_int64(PyObject * py_val) +{ + int overflow; +#ifdef HAVE_LONG_LONG + PY_LONG_LONG value = PyLong_AsLongLongAndOverflow(py_val, &overflow); +#else + long value = PyLong_AsLongAndOverflow(py_val, &overflow); +#endif + if (value == -1 && PyErr_Occurred()) + return -1; + if (!overflow) { +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG > 8 + if (-0x8000000000000000LL <= value && value <= 0x7FFFFFFFFFFFFFFFLL) +# endif +#else +# if SIZEOF_LONG > 8 + if (-0x8000000000000000L <= value && value <= 0x7FFFFFFFFFFFFFFFL) +# endif +#endif + return value; + } + else if (sizeof(value) < sizeof(sqlite3_int64)) { + sqlite3_int64 int64val; + if (_PyLong_AsByteArray((PyLongObject *)py_val, + (unsigned char *)&int64val, sizeof(int64val), + IS_LITTLE_ENDIAN, 1 /* signed */) >= 0) { + return int64val; + } + } + PyErr_SetString(PyExc_OverflowError, + "Python int too large to convert to SQLite INTEGER"); + return -1; +} diff --git a/Modules/_sqlite/util.h b/Modules/_sqlite/util.h --- a/Modules/_sqlite/util.h +++ b/Modules/_sqlite/util.h @@ -35,4 +35,8 @@ * Returns the error code (0 means no error occurred). */ int _pysqlite_seterror(sqlite3* db, sqlite3_stmt* st); + +PyObject * _pysqlite_long_from_int64(sqlite3_int64 value); +sqlite3_int64 _pysqlite_long_as_int64(PyObject * value); + #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 16:08:30 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 16:08:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE3MDcz?= =?utf-8?q?=3A_Fix_some_integer_overflows_in_sqlite3_module=2E?= Message-ID: <3Z22wG3940zSbb@mail.python.org> http://hg.python.org/cpython/rev/55a89352e220 changeset: 82055:55a89352e220 branch: 3.2 parent: 82051:0f1c2e2b6bc2 user: Serhiy Storchaka date: Thu Feb 07 17:01:47 2013 +0200 summary: Issue #17073: Fix some integer overflows in sqlite3 module. files: Lib/sqlite3/test/hooks.py | 19 ++++ Lib/sqlite3/test/userfunctions.py | 60 ++++++++++++--- Misc/NEWS | 2 + Modules/_sqlite/connection.c | 73 +++++++++++------- Modules/_sqlite/cursor.c | 20 +---- Modules/_sqlite/statement.c | 13 ++- Modules/_sqlite/util.c | 66 +++++++++++++++++ Modules/_sqlite/util.h | 4 + 8 files changed, 196 insertions(+), 61 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -76,6 +76,25 @@ except sqlite.OperationalError as e: self.assertEqual(e.args[0].lower(), "no such collation sequence: mycoll") + def CheckCollationReturnsLargeInteger(self): + def mycoll(x, y): + # reverse order + return -((x > y) - (x < y)) * 2**32 + con = sqlite.connect(":memory:") + con.create_collation("mycoll", mycoll) + sql = """ + select x from ( + select 'a' as x + union + select 'b' as x + union + select 'c' as x + ) order by x collate mycoll + """ + result = con.execute(sql).fetchall() + self.assertEqual(result, [('c',), ('b',), ('a',)], + msg="the expected order was not returned") + def CheckCollationRegisterTwice(self): """ Register two different collation functions under the same name. diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py --- a/Lib/sqlite3/test/userfunctions.py +++ b/Lib/sqlite3/test/userfunctions.py @@ -375,14 +375,15 @@ val = cur.fetchone()[0] self.assertEqual(val, 60) -def authorizer_cb(action, arg1, arg2, dbname, source): - if action != sqlite.SQLITE_SELECT: - return sqlite.SQLITE_DENY - if arg2 == 'c2' or arg1 == 't2': - return sqlite.SQLITE_DENY - return sqlite.SQLITE_OK +class AuthorizerTests(unittest.TestCase): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return sqlite.SQLITE_DENY + if arg2 == 'c2' or arg1 == 't2': + return sqlite.SQLITE_DENY + return sqlite.SQLITE_OK -class AuthorizerTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") self.con.executescript(""" @@ -395,12 +396,12 @@ # For our security test: self.con.execute("select c2 from t2") - self.con.set_authorizer(authorizer_cb) + self.con.set_authorizer(self.authorizer_cb) def tearDown(self): pass - def CheckTableAccess(self): + def test_table_access(self): try: self.con.execute("select * from t2") except sqlite.DatabaseError as e: @@ -409,7 +410,7 @@ return self.fail("should have raised an exception due to missing privileges") - def CheckColumnAccess(self): + def test_column_access(self): try: self.con.execute("select c2 from t1") except sqlite.DatabaseError as e: @@ -418,11 +419,46 @@ return self.fail("should have raised an exception due to missing privileges") +class AuthorizerRaiseExceptionTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + raise ValueError + if arg2 == 'c2' or arg1 == 't2': + raise ValueError + return sqlite.SQLITE_OK + +class AuthorizerIllegalTypeTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 0.0 + if arg2 == 'c2' or arg1 == 't2': + return 0.0 + return sqlite.SQLITE_OK + +class AuthorizerLargeIntegerTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 2**32 + if arg2 == 'c2' or arg1 == 't2': + return 2**32 + return sqlite.SQLITE_OK + + def suite(): function_suite = unittest.makeSuite(FunctionTests, "Check") aggregate_suite = unittest.makeSuite(AggregateTests, "Check") - authorizer_suite = unittest.makeSuite(AuthorizerTests, "Check") - return unittest.TestSuite((function_suite, aggregate_suite, authorizer_suite)) + authorizer_suite = unittest.makeSuite(AuthorizerTests) + return unittest.TestSuite(( + function_suite, + aggregate_suite, + authorizer_suite, + unittest.makeSuite(AuthorizerRaiseExceptionTests), + unittest.makeSuite(AuthorizerIllegalTypeTests), + unittest.makeSuite(AuthorizerLargeIntegerTests), + )) def test(): runner = unittest.TextTestRunner() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -215,6 +215,8 @@ Library ------- +- Issue #17073: Fix some integer overflows in sqlite3 module. + - Issue #17114: IDLE?now uses non-strict config parser. - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -482,32 +482,35 @@ } } -void _pysqlite_set_result(sqlite3_context* context, PyObject* py_val) +static int +_pysqlite_set_result(sqlite3_context* context, PyObject* py_val) { - const char* buffer; - Py_ssize_t buflen; - - if ((!py_val) || PyErr_Occurred()) { - sqlite3_result_null(context); - } else if (py_val == Py_None) { + if (py_val == Py_None) { sqlite3_result_null(context); } else if (PyLong_Check(py_val)) { - sqlite3_result_int64(context, PyLong_AsLongLong(py_val)); + sqlite_int64 value = _pysqlite_long_as_int64(py_val); + if (value == -1 && PyErr_Occurred()) + return -1; + sqlite3_result_int64(context, value); } else if (PyFloat_Check(py_val)) { sqlite3_result_double(context, PyFloat_AsDouble(py_val)); } else if (PyUnicode_Check(py_val)) { - char *str = _PyUnicode_AsString(py_val); - if (str != NULL) - sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT); + const char *str = _PyUnicode_AsString(py_val); + if (str == NULL) + return -1; + sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT); } else if (PyObject_CheckBuffer(py_val)) { + const char* buffer; + Py_ssize_t buflen; if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) { PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - } else { - sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); + return -1; } + sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); } else { - /* TODO: raise error */ + return -1; } + return 0; } PyObject* _pysqlite_build_py_params(sqlite3_context *context, int argc, sqlite3_value** argv) @@ -528,7 +531,7 @@ cur_value = argv[i]; switch (sqlite3_value_type(argv[i])) { case SQLITE_INTEGER: - cur_py_value = PyLong_FromLongLong(sqlite3_value_int64(cur_value)); + cur_py_value = _pysqlite_long_from_int64(sqlite3_value_int64(cur_value)); break; case SQLITE_FLOAT: cur_py_value = PyFloat_FromDouble(sqlite3_value_double(cur_value)); @@ -571,6 +574,7 @@ PyObject* args; PyObject* py_func; PyObject* py_retval = NULL; + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -586,10 +590,12 @@ Py_DECREF(args); } + ok = 0; if (py_retval) { - _pysqlite_set_result(context, py_retval); + ok = _pysqlite_set_result(context, py_retval) == 0; Py_DECREF(py_retval); - } else { + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { @@ -669,9 +675,10 @@ void _pysqlite_final_callback(sqlite3_context* context) { - PyObject* function_result = NULL; + PyObject* function_result; PyObject** aggregate_instance; PyObject* aggregate_class; + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -690,21 +697,23 @@ } function_result = PyObject_CallMethod(*aggregate_instance, "finalize", ""); - if (!function_result) { + Py_DECREF(*aggregate_instance); + + ok = 0; + if (function_result) { + ok = _pysqlite_set_result(context, function_result) == 0; + Py_DECREF(function_result); + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); } _sqlite3_result_error(context, "user-defined aggregate's 'finalize' method raised error", -1); - } else { - _pysqlite_set_result(context, function_result); } error: - Py_XDECREF(*aggregate_instance); - Py_XDECREF(function_result); - #ifdef WITH_THREAD PyGILState_Release(threadstate); #endif @@ -861,7 +870,9 @@ rc = SQLITE_DENY; } else { if (PyLong_Check(ret)) { - rc = (int)PyLong_AsLong(ret); + rc = _PyLong_AsInt(ret); + if (rc == -1 && PyErr_Occurred()) + rc = SQLITE_DENY; } else { rc = SQLITE_DENY; } @@ -1266,6 +1277,7 @@ PyGILState_STATE gilstate; #endif PyObject* retval = NULL; + long longval; int result = 0; #ifdef WITH_THREAD gilstate = PyGILState_Ensure(); @@ -1289,10 +1301,17 @@ goto finally; } - result = PyLong_AsLong(retval); - if (PyErr_Occurred()) { + longval = PyLong_AsLongAndOverflow(retval, &result); + if (longval == -1 && PyErr_Occurred()) { + PyErr_Clear(); result = 0; } + else if (!result) { + if (longval > 0) + result = 1; + else if (longval < 0) + result = -1; + } finally: Py_XDECREF(string1); diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -26,14 +26,6 @@ #include "util.h" #include "sqlitecompat.h" -/* used to decide wether to call PyLong_FromLong or PyLong_FromLongLong */ -#ifndef INT32_MIN -#define INT32_MIN (-2147483647 - 1) -#endif -#ifndef INT32_MAX -#define INT32_MAX 2147483647 -#endif - PyObject* pysqlite_cursor_iternext(pysqlite_Cursor* self); static char* errmsg_fetch_across_rollback = "Cursor needed to be reset because of commit/rollback and can no longer be fetched from."; @@ -285,7 +277,6 @@ PyObject* row; PyObject* item = NULL; int coltype; - PY_LONG_LONG intval; PyObject* converter; PyObject* converted; Py_ssize_t nbytes; @@ -345,12 +336,7 @@ Py_INCREF(Py_None); converted = Py_None; } else if (coltype == SQLITE_INTEGER) { - intval = sqlite3_column_int64(self->statement->st, i); - if (intval < INT32_MIN || intval > INT32_MAX) { - converted = PyLong_FromLongLong(intval); - } else { - converted = PyLong_FromLong((long)intval); - } + converted = _pysqlite_long_from_int64(sqlite3_column_int64(self->statement->st, i)); } else if (coltype == SQLITE_FLOAT) { converted = PyFloat_FromDouble(sqlite3_column_double(self->statement->st, i)); } else if (coltype == SQLITE_TEXT) { @@ -456,7 +442,6 @@ PyObject* func_args; PyObject* result; int numcols; - PY_LONG_LONG lastrowid; int statement_type; PyObject* descriptor; PyObject* second_argument = NULL; @@ -731,10 +716,11 @@ Py_DECREF(self->lastrowid); if (!multiple && statement_type == STATEMENT_INSERT) { + sqlite3_int64 lastrowid; Py_BEGIN_ALLOW_THREADS lastrowid = sqlite3_last_insert_rowid(self->connection->db); Py_END_ALLOW_THREADS - self->lastrowid = PyLong_FromLong((long)lastrowid); + self->lastrowid = _pysqlite_long_from_int64(lastrowid); } else { Py_INCREF(Py_None); self->lastrowid = Py_None; diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -26,6 +26,7 @@ #include "connection.h" #include "microprotocols.h" #include "prepare_protocol.h" +#include "util.h" #include "sqlitecompat.h" /* prototypes */ @@ -90,7 +91,6 @@ int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter, int allow_8bit_chars) { int rc = SQLITE_OK; - PY_LONG_LONG longlongval; const char* buffer; char* string; Py_ssize_t buflen; @@ -120,11 +120,14 @@ } switch (paramtype) { - case TYPE_LONG: - /* in the overflow error case, longval/longlongval is -1, and an exception is set */ - longlongval = PyLong_AsLongLong(parameter); - rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longlongval); + case TYPE_LONG: { + sqlite_int64 value = _pysqlite_long_as_int64(parameter); + if (value == -1 && PyErr_Occurred()) + rc = -1; + else + rc = sqlite3_bind_int64(self->st, pos, value); break; + } case TYPE_FLOAT: rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); break; diff --git a/Modules/_sqlite/util.c b/Modules/_sqlite/util.c --- a/Modules/_sqlite/util.c +++ b/Modules/_sqlite/util.c @@ -104,3 +104,69 @@ return errorcode; } +#ifdef WORDS_BIGENDIAN +# define IS_LITTLE_ENDIAN 0 +#else +# define IS_LITTLE_ENDIAN 1 +#endif + +PyObject * +_pysqlite_long_from_int64(sqlite3_int64 value) +{ +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG < 8 + if (value > PY_LLONG_MAX || value < PY_LLONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +# if SIZEOF_LONG < SIZEOF_LONG_LONG + if (value > LONG_MAX || value < LONG_MIN) + return PyLong_FromLongLong(value); +# endif +#else +# if SIZEOF_LONG < 8 + if (value > LONG_MAX || value < LONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +#endif + return PyLong_FromLong(value); +} + +sqlite3_int64 +_pysqlite_long_as_int64(PyObject * py_val) +{ + int overflow; +#ifdef HAVE_LONG_LONG + PY_LONG_LONG value = PyLong_AsLongLongAndOverflow(py_val, &overflow); +#else + long value = PyLong_AsLongAndOverflow(py_val, &overflow); +#endif + if (value == -1 && PyErr_Occurred()) + return -1; + if (!overflow) { +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG > 8 + if (-0x8000000000000000LL <= value && value <= 0x7FFFFFFFFFFFFFFFLL) +# endif +#else +# if SIZEOF_LONG > 8 + if (-0x8000000000000000L <= value && value <= 0x7FFFFFFFFFFFFFFFL) +# endif +#endif + return value; + } + else if (sizeof(value) < sizeof(sqlite3_int64)) { + sqlite3_int64 int64val; + if (_PyLong_AsByteArray((PyLongObject *)py_val, + (unsigned char *)&int64val, sizeof(int64val), + IS_LITTLE_ENDIAN, 1 /* signed */) >= 0) { + return int64val; + } + } + PyErr_SetString(PyExc_OverflowError, + "Python int too large to convert to SQLite INTEGER"); + return -1; +} diff --git a/Modules/_sqlite/util.h b/Modules/_sqlite/util.h --- a/Modules/_sqlite/util.h +++ b/Modules/_sqlite/util.h @@ -35,4 +35,8 @@ * Returns the error code (0 means no error occurred). */ int _pysqlite_seterror(sqlite3* db, sqlite3_stmt* st); + +PyObject * _pysqlite_long_from_int64(sqlite3_int64 value); +sqlite3_int64 _pysqlite_long_as_int64(PyObject * value); + #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 16:08:32 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 16:08:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2317073=3A_Fix_some_integer_overflows_in_sqlite3_module?= =?utf-8?q?=2E?= Message-ID: <3Z22wJ1TwZzSc3@mail.python.org> http://hg.python.org/cpython/rev/c5fb8bc56def changeset: 82056:c5fb8bc56def branch: 3.3 parent: 82052:fec2976c8503 parent: 82055:55a89352e220 user: Serhiy Storchaka date: Thu Feb 07 17:03:46 2013 +0200 summary: Issue #17073: Fix some integer overflows in sqlite3 module. files: Lib/sqlite3/test/hooks.py | 19 ++++ Lib/sqlite3/test/userfunctions.py | 60 ++++++++++++--- Misc/NEWS | 2 + Modules/_sqlite/connection.c | 73 +++++++++++------- Modules/_sqlite/cursor.c | 20 +---- Modules/_sqlite/statement.c | 13 ++- Modules/_sqlite/util.c | 66 +++++++++++++++++ Modules/_sqlite/util.h | 4 + 8 files changed, 196 insertions(+), 61 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -76,6 +76,25 @@ except sqlite.OperationalError as e: self.assertEqual(e.args[0].lower(), "no such collation sequence: mycoll") + def CheckCollationReturnsLargeInteger(self): + def mycoll(x, y): + # reverse order + return -((x > y) - (x < y)) * 2**32 + con = sqlite.connect(":memory:") + con.create_collation("mycoll", mycoll) + sql = """ + select x from ( + select 'a' as x + union + select 'b' as x + union + select 'c' as x + ) order by x collate mycoll + """ + result = con.execute(sql).fetchall() + self.assertEqual(result, [('c',), ('b',), ('a',)], + msg="the expected order was not returned") + def CheckCollationRegisterTwice(self): """ Register two different collation functions under the same name. diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py --- a/Lib/sqlite3/test/userfunctions.py +++ b/Lib/sqlite3/test/userfunctions.py @@ -375,14 +375,15 @@ val = cur.fetchone()[0] self.assertEqual(val, 60) -def authorizer_cb(action, arg1, arg2, dbname, source): - if action != sqlite.SQLITE_SELECT: - return sqlite.SQLITE_DENY - if arg2 == 'c2' or arg1 == 't2': - return sqlite.SQLITE_DENY - return sqlite.SQLITE_OK +class AuthorizerTests(unittest.TestCase): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return sqlite.SQLITE_DENY + if arg2 == 'c2' or arg1 == 't2': + return sqlite.SQLITE_DENY + return sqlite.SQLITE_OK -class AuthorizerTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") self.con.executescript(""" @@ -395,12 +396,12 @@ # For our security test: self.con.execute("select c2 from t2") - self.con.set_authorizer(authorizer_cb) + self.con.set_authorizer(self.authorizer_cb) def tearDown(self): pass - def CheckTableAccess(self): + def test_table_access(self): try: self.con.execute("select * from t2") except sqlite.DatabaseError as e: @@ -409,7 +410,7 @@ return self.fail("should have raised an exception due to missing privileges") - def CheckColumnAccess(self): + def test_column_access(self): try: self.con.execute("select c2 from t1") except sqlite.DatabaseError as e: @@ -418,11 +419,46 @@ return self.fail("should have raised an exception due to missing privileges") +class AuthorizerRaiseExceptionTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + raise ValueError + if arg2 == 'c2' or arg1 == 't2': + raise ValueError + return sqlite.SQLITE_OK + +class AuthorizerIllegalTypeTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 0.0 + if arg2 == 'c2' or arg1 == 't2': + return 0.0 + return sqlite.SQLITE_OK + +class AuthorizerLargeIntegerTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 2**32 + if arg2 == 'c2' or arg1 == 't2': + return 2**32 + return sqlite.SQLITE_OK + + def suite(): function_suite = unittest.makeSuite(FunctionTests, "Check") aggregate_suite = unittest.makeSuite(AggregateTests, "Check") - authorizer_suite = unittest.makeSuite(AuthorizerTests, "Check") - return unittest.TestSuite((function_suite, aggregate_suite, authorizer_suite)) + authorizer_suite = unittest.makeSuite(AuthorizerTests) + return unittest.TestSuite(( + function_suite, + aggregate_suite, + authorizer_suite, + unittest.makeSuite(AuthorizerRaiseExceptionTests), + unittest.makeSuite(AuthorizerIllegalTypeTests), + unittest.makeSuite(AuthorizerLargeIntegerTests), + )) def test(): runner = unittest.TextTestRunner() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -166,6 +166,8 @@ Library ------- +- Issue #17073: Fix some integer overflows in sqlite3 module. + - Issue #17114: IDLE?now uses non-strict config parser. - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -482,32 +482,35 @@ } } -void _pysqlite_set_result(sqlite3_context* context, PyObject* py_val) +static int +_pysqlite_set_result(sqlite3_context* context, PyObject* py_val) { - const char* buffer; - Py_ssize_t buflen; - - if ((!py_val) || PyErr_Occurred()) { - sqlite3_result_null(context); - } else if (py_val == Py_None) { + if (py_val == Py_None) { sqlite3_result_null(context); } else if (PyLong_Check(py_val)) { - sqlite3_result_int64(context, PyLong_AsLongLong(py_val)); + sqlite_int64 value = _pysqlite_long_as_int64(py_val); + if (value == -1 && PyErr_Occurred()) + return -1; + sqlite3_result_int64(context, value); } else if (PyFloat_Check(py_val)) { sqlite3_result_double(context, PyFloat_AsDouble(py_val)); } else if (PyUnicode_Check(py_val)) { - char *str = _PyUnicode_AsString(py_val); - if (str != NULL) - sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT); + const char *str = _PyUnicode_AsString(py_val); + if (str == NULL) + return -1; + sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT); } else if (PyObject_CheckBuffer(py_val)) { + const char* buffer; + Py_ssize_t buflen; if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) { PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - } else { - sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); + return -1; } + sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); } else { - /* TODO: raise error */ + return -1; } + return 0; } PyObject* _pysqlite_build_py_params(sqlite3_context *context, int argc, sqlite3_value** argv) @@ -528,7 +531,7 @@ cur_value = argv[i]; switch (sqlite3_value_type(argv[i])) { case SQLITE_INTEGER: - cur_py_value = PyLong_FromLongLong(sqlite3_value_int64(cur_value)); + cur_py_value = _pysqlite_long_from_int64(sqlite3_value_int64(cur_value)); break; case SQLITE_FLOAT: cur_py_value = PyFloat_FromDouble(sqlite3_value_double(cur_value)); @@ -571,6 +574,7 @@ PyObject* args; PyObject* py_func; PyObject* py_retval = NULL; + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -586,10 +590,12 @@ Py_DECREF(args); } + ok = 0; if (py_retval) { - _pysqlite_set_result(context, py_retval); + ok = _pysqlite_set_result(context, py_retval) == 0; Py_DECREF(py_retval); - } else { + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { @@ -669,9 +675,10 @@ void _pysqlite_final_callback(sqlite3_context* context) { - PyObject* function_result = NULL; + PyObject* function_result; PyObject** aggregate_instance; _Py_IDENTIFIER(finalize); + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -688,21 +695,23 @@ } function_result = _PyObject_CallMethodId(*aggregate_instance, &PyId_finalize, ""); - if (!function_result) { + Py_DECREF(*aggregate_instance); + + ok = 0; + if (function_result) { + ok = _pysqlite_set_result(context, function_result) == 0; + Py_DECREF(function_result); + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); } _sqlite3_result_error(context, "user-defined aggregate's 'finalize' method raised error", -1); - } else { - _pysqlite_set_result(context, function_result); } error: - Py_XDECREF(*aggregate_instance); - Py_XDECREF(function_result); - #ifdef WITH_THREAD PyGILState_Release(threadstate); #endif @@ -859,7 +868,9 @@ rc = SQLITE_DENY; } else { if (PyLong_Check(ret)) { - rc = (int)PyLong_AsLong(ret); + rc = _PyLong_AsInt(ret); + if (rc == -1 && PyErr_Occurred()) + rc = SQLITE_DENY; } else { rc = SQLITE_DENY; } @@ -1327,6 +1338,7 @@ PyGILState_STATE gilstate; #endif PyObject* retval = NULL; + long longval; int result = 0; #ifdef WITH_THREAD gilstate = PyGILState_Ensure(); @@ -1350,10 +1362,17 @@ goto finally; } - result = PyLong_AsLong(retval); - if (PyErr_Occurred()) { + longval = PyLong_AsLongAndOverflow(retval, &result); + if (longval == -1 && PyErr_Occurred()) { + PyErr_Clear(); result = 0; } + else if (!result) { + if (longval > 0) + result = 1; + else if (longval < 0) + result = -1; + } finally: Py_XDECREF(string1); diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -26,14 +26,6 @@ #include "util.h" #include "sqlitecompat.h" -/* used to decide wether to call PyLong_FromLong or PyLong_FromLongLong */ -#ifndef INT32_MIN -#define INT32_MIN (-2147483647 - 1) -#endif -#ifndef INT32_MAX -#define INT32_MAX 2147483647 -#endif - PyObject* pysqlite_cursor_iternext(pysqlite_Cursor* self); static char* errmsg_fetch_across_rollback = "Cursor needed to be reset because of commit/rollback and can no longer be fetched from."; @@ -279,7 +271,6 @@ PyObject* row; PyObject* item = NULL; int coltype; - PY_LONG_LONG intval; PyObject* converter; PyObject* converted; Py_ssize_t nbytes; @@ -339,12 +330,7 @@ Py_INCREF(Py_None); converted = Py_None; } else if (coltype == SQLITE_INTEGER) { - intval = sqlite3_column_int64(self->statement->st, i); - if (intval < INT32_MIN || intval > INT32_MAX) { - converted = PyLong_FromLongLong(intval); - } else { - converted = PyLong_FromLong((long)intval); - } + converted = _pysqlite_long_from_int64(sqlite3_column_int64(self->statement->st, i)); } else if (coltype == SQLITE_FLOAT) { converted = PyFloat_FromDouble(sqlite3_column_double(self->statement->st, i)); } else if (coltype == SQLITE_TEXT) { @@ -446,7 +432,6 @@ PyObject* func_args; PyObject* result; int numcols; - PY_LONG_LONG lastrowid; int statement_type; PyObject* descriptor; PyObject* second_argument = NULL; @@ -716,10 +701,11 @@ Py_DECREF(self->lastrowid); if (!multiple && statement_type == STATEMENT_INSERT) { + sqlite3_int64 lastrowid; Py_BEGIN_ALLOW_THREADS lastrowid = sqlite3_last_insert_rowid(self->connection->db); Py_END_ALLOW_THREADS - self->lastrowid = PyLong_FromLong((long)lastrowid); + self->lastrowid = _pysqlite_long_from_int64(lastrowid); } else { Py_INCREF(Py_None); self->lastrowid = Py_None; diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -26,6 +26,7 @@ #include "connection.h" #include "microprotocols.h" #include "prepare_protocol.h" +#include "util.h" #include "sqlitecompat.h" /* prototypes */ @@ -90,7 +91,6 @@ int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter) { int rc = SQLITE_OK; - PY_LONG_LONG longlongval; const char* buffer; char* string; Py_ssize_t buflen; @@ -120,11 +120,14 @@ } switch (paramtype) { - case TYPE_LONG: - /* in the overflow error case, longval/longlongval is -1, and an exception is set */ - longlongval = PyLong_AsLongLong(parameter); - rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longlongval); + case TYPE_LONG: { + sqlite_int64 value = _pysqlite_long_as_int64(parameter); + if (value == -1 && PyErr_Occurred()) + rc = -1; + else + rc = sqlite3_bind_int64(self->st, pos, value); break; + } case TYPE_FLOAT: rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); break; diff --git a/Modules/_sqlite/util.c b/Modules/_sqlite/util.c --- a/Modules/_sqlite/util.c +++ b/Modules/_sqlite/util.c @@ -104,3 +104,69 @@ return errorcode; } +#ifdef WORDS_BIGENDIAN +# define IS_LITTLE_ENDIAN 0 +#else +# define IS_LITTLE_ENDIAN 1 +#endif + +PyObject * +_pysqlite_long_from_int64(sqlite3_int64 value) +{ +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG < 8 + if (value > PY_LLONG_MAX || value < PY_LLONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +# if SIZEOF_LONG < SIZEOF_LONG_LONG + if (value > LONG_MAX || value < LONG_MIN) + return PyLong_FromLongLong(value); +# endif +#else +# if SIZEOF_LONG < 8 + if (value > LONG_MAX || value < LONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +#endif + return PyLong_FromLong(value); +} + +sqlite3_int64 +_pysqlite_long_as_int64(PyObject * py_val) +{ + int overflow; +#ifdef HAVE_LONG_LONG + PY_LONG_LONG value = PyLong_AsLongLongAndOverflow(py_val, &overflow); +#else + long value = PyLong_AsLongAndOverflow(py_val, &overflow); +#endif + if (value == -1 && PyErr_Occurred()) + return -1; + if (!overflow) { +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG > 8 + if (-0x8000000000000000LL <= value && value <= 0x7FFFFFFFFFFFFFFFLL) +# endif +#else +# if SIZEOF_LONG > 8 + if (-0x8000000000000000L <= value && value <= 0x7FFFFFFFFFFFFFFFL) +# endif +#endif + return value; + } + else if (sizeof(value) < sizeof(sqlite3_int64)) { + sqlite3_int64 int64val; + if (_PyLong_AsByteArray((PyLongObject *)py_val, + (unsigned char *)&int64val, sizeof(int64val), + IS_LITTLE_ENDIAN, 1 /* signed */) >= 0) { + return int64val; + } + } + PyErr_SetString(PyExc_OverflowError, + "Python int too large to convert to SQLite INTEGER"); + return -1; +} diff --git a/Modules/_sqlite/util.h b/Modules/_sqlite/util.h --- a/Modules/_sqlite/util.h +++ b/Modules/_sqlite/util.h @@ -35,4 +35,8 @@ * Returns the error code (0 means no error occurred). */ int _pysqlite_seterror(sqlite3* db, sqlite3_stmt* st); + +PyObject * _pysqlite_long_from_int64(sqlite3_int64 value); +sqlite3_int64 _pysqlite_long_as_int64(PyObject * value); + #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 16:08:33 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 7 Feb 2013 16:08:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317073=3A_Fix_some_integer_overflows_in_sqlite3_?= =?utf-8?q?module=2E?= Message-ID: <3Z22wK6ppQzSc3@mail.python.org> http://hg.python.org/cpython/rev/b8a6bc70fc08 changeset: 82057:b8a6bc70fc08 parent: 82053:eb0370d4686c parent: 82056:c5fb8bc56def user: Serhiy Storchaka date: Thu Feb 07 17:05:32 2013 +0200 summary: Issue #17073: Fix some integer overflows in sqlite3 module. files: Lib/sqlite3/test/hooks.py | 19 ++++ Lib/sqlite3/test/userfunctions.py | 60 ++++++++++++--- Misc/NEWS | 2 + Modules/_sqlite/connection.c | 73 +++++++++++------- Modules/_sqlite/cursor.c | 20 +---- Modules/_sqlite/statement.c | 13 ++- Modules/_sqlite/util.c | 66 +++++++++++++++++ Modules/_sqlite/util.h | 4 + 8 files changed, 196 insertions(+), 61 deletions(-) diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -76,6 +76,25 @@ except sqlite.OperationalError as e: self.assertEqual(e.args[0].lower(), "no such collation sequence: mycoll") + def CheckCollationReturnsLargeInteger(self): + def mycoll(x, y): + # reverse order + return -((x > y) - (x < y)) * 2**32 + con = sqlite.connect(":memory:") + con.create_collation("mycoll", mycoll) + sql = """ + select x from ( + select 'a' as x + union + select 'b' as x + union + select 'c' as x + ) order by x collate mycoll + """ + result = con.execute(sql).fetchall() + self.assertEqual(result, [('c',), ('b',), ('a',)], + msg="the expected order was not returned") + def CheckCollationRegisterTwice(self): """ Register two different collation functions under the same name. diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py --- a/Lib/sqlite3/test/userfunctions.py +++ b/Lib/sqlite3/test/userfunctions.py @@ -375,14 +375,15 @@ val = cur.fetchone()[0] self.assertEqual(val, 60) -def authorizer_cb(action, arg1, arg2, dbname, source): - if action != sqlite.SQLITE_SELECT: - return sqlite.SQLITE_DENY - if arg2 == 'c2' or arg1 == 't2': - return sqlite.SQLITE_DENY - return sqlite.SQLITE_OK +class AuthorizerTests(unittest.TestCase): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return sqlite.SQLITE_DENY + if arg2 == 'c2' or arg1 == 't2': + return sqlite.SQLITE_DENY + return sqlite.SQLITE_OK -class AuthorizerTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") self.con.executescript(""" @@ -395,12 +396,12 @@ # For our security test: self.con.execute("select c2 from t2") - self.con.set_authorizer(authorizer_cb) + self.con.set_authorizer(self.authorizer_cb) def tearDown(self): pass - def CheckTableAccess(self): + def test_table_access(self): try: self.con.execute("select * from t2") except sqlite.DatabaseError as e: @@ -409,7 +410,7 @@ return self.fail("should have raised an exception due to missing privileges") - def CheckColumnAccess(self): + def test_column_access(self): try: self.con.execute("select c2 from t1") except sqlite.DatabaseError as e: @@ -418,11 +419,46 @@ return self.fail("should have raised an exception due to missing privileges") +class AuthorizerRaiseExceptionTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + raise ValueError + if arg2 == 'c2' or arg1 == 't2': + raise ValueError + return sqlite.SQLITE_OK + +class AuthorizerIllegalTypeTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 0.0 + if arg2 == 'c2' or arg1 == 't2': + return 0.0 + return sqlite.SQLITE_OK + +class AuthorizerLargeIntegerTests(AuthorizerTests): + @staticmethod + def authorizer_cb(action, arg1, arg2, dbname, source): + if action != sqlite.SQLITE_SELECT: + return 2**32 + if arg2 == 'c2' or arg1 == 't2': + return 2**32 + return sqlite.SQLITE_OK + + def suite(): function_suite = unittest.makeSuite(FunctionTests, "Check") aggregate_suite = unittest.makeSuite(AggregateTests, "Check") - authorizer_suite = unittest.makeSuite(AuthorizerTests, "Check") - return unittest.TestSuite((function_suite, aggregate_suite, authorizer_suite)) + authorizer_suite = unittest.makeSuite(AuthorizerTests) + return unittest.TestSuite(( + function_suite, + aggregate_suite, + authorizer_suite, + unittest.makeSuite(AuthorizerRaiseExceptionTests), + unittest.makeSuite(AuthorizerIllegalTypeTests), + unittest.makeSuite(AuthorizerLargeIntegerTests), + )) def test(): runner = unittest.TextTestRunner() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -238,6 +238,8 @@ Library ------- +- Issue #17073: Fix some integer overflows in sqlite3 module. + - Issue #17114: IDLE?now uses non-strict config parser. - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -482,32 +482,35 @@ } } -void _pysqlite_set_result(sqlite3_context* context, PyObject* py_val) +static int +_pysqlite_set_result(sqlite3_context* context, PyObject* py_val) { - const char* buffer; - Py_ssize_t buflen; - - if ((!py_val) || PyErr_Occurred()) { - sqlite3_result_null(context); - } else if (py_val == Py_None) { + if (py_val == Py_None) { sqlite3_result_null(context); } else if (PyLong_Check(py_val)) { - sqlite3_result_int64(context, PyLong_AsLongLong(py_val)); + sqlite_int64 value = _pysqlite_long_as_int64(py_val); + if (value == -1 && PyErr_Occurred()) + return -1; + sqlite3_result_int64(context, value); } else if (PyFloat_Check(py_val)) { sqlite3_result_double(context, PyFloat_AsDouble(py_val)); } else if (PyUnicode_Check(py_val)) { - char *str = _PyUnicode_AsString(py_val); - if (str != NULL) - sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT); + const char *str = _PyUnicode_AsString(py_val); + if (str == NULL) + return -1; + sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT); } else if (PyObject_CheckBuffer(py_val)) { + const char* buffer; + Py_ssize_t buflen; if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) { PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer"); - } else { - sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); + return -1; } + sqlite3_result_blob(context, buffer, buflen, SQLITE_TRANSIENT); } else { - /* TODO: raise error */ + return -1; } + return 0; } PyObject* _pysqlite_build_py_params(sqlite3_context *context, int argc, sqlite3_value** argv) @@ -528,7 +531,7 @@ cur_value = argv[i]; switch (sqlite3_value_type(argv[i])) { case SQLITE_INTEGER: - cur_py_value = PyLong_FromLongLong(sqlite3_value_int64(cur_value)); + cur_py_value = _pysqlite_long_from_int64(sqlite3_value_int64(cur_value)); break; case SQLITE_FLOAT: cur_py_value = PyFloat_FromDouble(sqlite3_value_double(cur_value)); @@ -571,6 +574,7 @@ PyObject* args; PyObject* py_func; PyObject* py_retval = NULL; + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -586,10 +590,12 @@ Py_DECREF(args); } + ok = 0; if (py_retval) { - _pysqlite_set_result(context, py_retval); + ok = _pysqlite_set_result(context, py_retval) == 0; Py_DECREF(py_retval); - } else { + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { @@ -669,9 +675,10 @@ void _pysqlite_final_callback(sqlite3_context* context) { - PyObject* function_result = NULL; + PyObject* function_result; PyObject** aggregate_instance; _Py_IDENTIFIER(finalize); + int ok; #ifdef WITH_THREAD PyGILState_STATE threadstate; @@ -688,21 +695,23 @@ } function_result = _PyObject_CallMethodId(*aggregate_instance, &PyId_finalize, ""); - if (!function_result) { + Py_DECREF(*aggregate_instance); + + ok = 0; + if (function_result) { + ok = _pysqlite_set_result(context, function_result) == 0; + Py_DECREF(function_result); + } + if (!ok) { if (_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); } _sqlite3_result_error(context, "user-defined aggregate's 'finalize' method raised error", -1); - } else { - _pysqlite_set_result(context, function_result); } error: - Py_XDECREF(*aggregate_instance); - Py_XDECREF(function_result); - #ifdef WITH_THREAD PyGILState_Release(threadstate); #endif @@ -859,7 +868,9 @@ rc = SQLITE_DENY; } else { if (PyLong_Check(ret)) { - rc = (int)PyLong_AsLong(ret); + rc = _PyLong_AsInt(ret); + if (rc == -1 && PyErr_Occurred()) + rc = SQLITE_DENY; } else { rc = SQLITE_DENY; } @@ -1327,6 +1338,7 @@ PyGILState_STATE gilstate; #endif PyObject* retval = NULL; + long longval; int result = 0; #ifdef WITH_THREAD gilstate = PyGILState_Ensure(); @@ -1350,10 +1362,17 @@ goto finally; } - result = PyLong_AsLong(retval); - if (PyErr_Occurred()) { + longval = PyLong_AsLongAndOverflow(retval, &result); + if (longval == -1 && PyErr_Occurred()) { + PyErr_Clear(); result = 0; } + else if (!result) { + if (longval > 0) + result = 1; + else if (longval < 0) + result = -1; + } finally: Py_XDECREF(string1); diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -26,14 +26,6 @@ #include "util.h" #include "sqlitecompat.h" -/* used to decide wether to call PyLong_FromLong or PyLong_FromLongLong */ -#ifndef INT32_MIN -#define INT32_MIN (-2147483647 - 1) -#endif -#ifndef INT32_MAX -#define INT32_MAX 2147483647 -#endif - PyObject* pysqlite_cursor_iternext(pysqlite_Cursor* self); static char* errmsg_fetch_across_rollback = "Cursor needed to be reset because of commit/rollback and can no longer be fetched from."; @@ -279,7 +271,6 @@ PyObject* row; PyObject* item = NULL; int coltype; - PY_LONG_LONG intval; PyObject* converter; PyObject* converted; Py_ssize_t nbytes; @@ -339,12 +330,7 @@ Py_INCREF(Py_None); converted = Py_None; } else if (coltype == SQLITE_INTEGER) { - intval = sqlite3_column_int64(self->statement->st, i); - if (intval < INT32_MIN || intval > INT32_MAX) { - converted = PyLong_FromLongLong(intval); - } else { - converted = PyLong_FromLong((long)intval); - } + converted = _pysqlite_long_from_int64(sqlite3_column_int64(self->statement->st, i)); } else if (coltype == SQLITE_FLOAT) { converted = PyFloat_FromDouble(sqlite3_column_double(self->statement->st, i)); } else if (coltype == SQLITE_TEXT) { @@ -446,7 +432,6 @@ PyObject* func_args; PyObject* result; int numcols; - PY_LONG_LONG lastrowid; int statement_type; PyObject* descriptor; PyObject* second_argument = NULL; @@ -716,10 +701,11 @@ Py_DECREF(self->lastrowid); if (!multiple && statement_type == STATEMENT_INSERT) { + sqlite3_int64 lastrowid; Py_BEGIN_ALLOW_THREADS lastrowid = sqlite3_last_insert_rowid(self->connection->db); Py_END_ALLOW_THREADS - self->lastrowid = PyLong_FromLong((long)lastrowid); + self->lastrowid = _pysqlite_long_from_int64(lastrowid); } else { Py_INCREF(Py_None); self->lastrowid = Py_None; diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -26,6 +26,7 @@ #include "connection.h" #include "microprotocols.h" #include "prepare_protocol.h" +#include "util.h" #include "sqlitecompat.h" /* prototypes */ @@ -90,7 +91,6 @@ int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter) { int rc = SQLITE_OK; - PY_LONG_LONG longlongval; const char* buffer; char* string; Py_ssize_t buflen; @@ -120,11 +120,14 @@ } switch (paramtype) { - case TYPE_LONG: - /* in the overflow error case, longval/longlongval is -1, and an exception is set */ - longlongval = PyLong_AsLongLong(parameter); - rc = sqlite3_bind_int64(self->st, pos, (sqlite_int64)longlongval); + case TYPE_LONG: { + sqlite_int64 value = _pysqlite_long_as_int64(parameter); + if (value == -1 && PyErr_Occurred()) + rc = -1; + else + rc = sqlite3_bind_int64(self->st, pos, value); break; + } case TYPE_FLOAT: rc = sqlite3_bind_double(self->st, pos, PyFloat_AsDouble(parameter)); break; diff --git a/Modules/_sqlite/util.c b/Modules/_sqlite/util.c --- a/Modules/_sqlite/util.c +++ b/Modules/_sqlite/util.c @@ -104,3 +104,69 @@ return errorcode; } +#ifdef WORDS_BIGENDIAN +# define IS_LITTLE_ENDIAN 0 +#else +# define IS_LITTLE_ENDIAN 1 +#endif + +PyObject * +_pysqlite_long_from_int64(sqlite3_int64 value) +{ +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG < 8 + if (value > PY_LLONG_MAX || value < PY_LLONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +# if SIZEOF_LONG < SIZEOF_LONG_LONG + if (value > LONG_MAX || value < LONG_MIN) + return PyLong_FromLongLong(value); +# endif +#else +# if SIZEOF_LONG < 8 + if (value > LONG_MAX || value < LONG_MIN) { + return _PyLong_FromByteArray(&value, sizeof(value), + IS_LITTLE_ENDIAN, 1 /* signed */); + } +# endif +#endif + return PyLong_FromLong(value); +} + +sqlite3_int64 +_pysqlite_long_as_int64(PyObject * py_val) +{ + int overflow; +#ifdef HAVE_LONG_LONG + PY_LONG_LONG value = PyLong_AsLongLongAndOverflow(py_val, &overflow); +#else + long value = PyLong_AsLongAndOverflow(py_val, &overflow); +#endif + if (value == -1 && PyErr_Occurred()) + return -1; + if (!overflow) { +#ifdef HAVE_LONG_LONG +# if SIZEOF_LONG_LONG > 8 + if (-0x8000000000000000LL <= value && value <= 0x7FFFFFFFFFFFFFFFLL) +# endif +#else +# if SIZEOF_LONG > 8 + if (-0x8000000000000000L <= value && value <= 0x7FFFFFFFFFFFFFFFL) +# endif +#endif + return value; + } + else if (sizeof(value) < sizeof(sqlite3_int64)) { + sqlite3_int64 int64val; + if (_PyLong_AsByteArray((PyLongObject *)py_val, + (unsigned char *)&int64val, sizeof(int64val), + IS_LITTLE_ENDIAN, 1 /* signed */) >= 0) { + return int64val; + } + } + PyErr_SetString(PyExc_OverflowError, + "Python int too large to convert to SQLite INTEGER"); + return -1; +} diff --git a/Modules/_sqlite/util.h b/Modules/_sqlite/util.h --- a/Modules/_sqlite/util.h +++ b/Modules/_sqlite/util.h @@ -35,4 +35,8 @@ * Returns the error code (0 means no error occurred). */ int _pysqlite_seterror(sqlite3* db, sqlite3_stmt* st); + +PyObject * _pysqlite_long_from_int64(sqlite3_int64 value); +sqlite3_int64 _pysqlite_long_as_int64(PyObject * value); + #endif -- Repository URL: http://hg.python.org/cpython From tjreedy at udel.edu Thu Feb 7 19:35:32 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 07 Feb 2013 13:35:32 -0500 Subject: [Python-checkins] cpython (3.2): Fix Issue17069: Document getcode method in urllib.request.rst In-Reply-To: <3Z1tWQ1FnPzMRY@mail.python.org> References: <3Z1tWQ1FnPzMRY@mail.python.org> Message-ID: <5113F3F4.7000200@udel.edu> 3 suggested changes: 1. On 2/7/2013 3:49 AM, senthil.kumaran wrote: > + For ftp, file, data urls and requests are explicity handled by legacy > + :class:`URLopener` and :class:`FancyURLopener` class, this function returns The first part up to "class," is not a proper English clause. Perhaps you meant ('that' added): "For ftp, file, data urls and requests that are explicity handled ..." or (to make clear that 'that explicitly' does not apply to ftp,file,data ('and' added): "For ftp, file, and data urls and requests that are explicity handled " or perhaps clearest ('are' deleted, 'that' not added) "For ftp, file, and data urls and requests explicity handled ..." 2. it seems that 'class' should be 'classes'. 3. + ... this function returns + an :class:`urllib.response.addinfourl` object Since 'url' is generally pronounced 'you-are-el' rather than like 'earl' (I checked Merriam-Webster online), that should be 'a' rather than 'an' url... . https://owl.english.purdue.edu/owl/resource/591/01/ From python-checkins at python.org Thu Feb 7 23:18:39 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 7 Feb 2013 23:18:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3MTM3?= =?utf-8?q?=3A_When_an_Unicode_string_is_resized=2C_the_internal_wide_char?= =?utf-8?q?acter?= Message-ID: <3Z2DSb6WcKzSd8@mail.python.org> http://hg.python.org/cpython/rev/3b316ea5aa82 changeset: 82058:3b316ea5aa82 branch: 3.3 parent: 82056:c5fb8bc56def user: Victor Stinner date: Thu Feb 07 23:12:46 2013 +0100 summary: Issue #17137: When an Unicode string is resized, the internal wide character string (wstr) format is now cleared. files: Lib/test/test_unicode.py | 15 +++++++++++++++ Misc/NEWS | 3 +++ Objects/unicodeobject.c | 4 ++++ 3 files changed, 22 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2167,6 +2167,21 @@ self.assertEqual(args[0], text) self.assertEqual(len(args), 1) + def test_resize(self): + for length in range(1, 100, 7): + # generate a fresh string (refcount=1) + text = 'a' * length + 'b' + + # fill wstr internal field + abc = text.encode('unicode_internal') + self.assertEqual(abc.decode('unicode_internal'), text) + + # resize text: wstr field must be cleared and then recomputed + text += 'c' + abcdef = text.encode('unicode_internal') + self.assertNotEqual(abc, abcdef) + self.assertEqual(abcdef.decode('unicode_internal'), text) + class StringModuleTest(unittest.TestCase): def test_formatter_parser(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,6 +12,9 @@ Core and Builtins ----------------- +- Issue #17137: When an Unicode string is resized, the internal wide character + string (wstr) format is now cleared. + - Issue #17043: The unicode-internal decoder no longer read past the end of input buffer. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -702,6 +702,10 @@ if (!PyUnicode_IS_ASCII(unicode)) _PyUnicode_WSTR_LENGTH(unicode) = length; } + else if (_PyUnicode_HAS_WSTR_MEMORY(unicode)) { + PyObject_DEL(_PyUnicode_WSTR(unicode)); + _PyUnicode_WSTR(unicode) = NULL; + } PyUnicode_WRITE(PyUnicode_KIND(unicode), PyUnicode_DATA(unicode), length, 0); assert(_PyUnicode_CheckConsistency(unicode, 0)); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Feb 7 23:18:41 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 7 Feb 2013 23:18:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2317137=3A_When_an_Unicode_stri?= =?utf-8?q?ng_is_resized=2C_the_internal_wide?= Message-ID: <3Z2DSd2KQczScx@mail.python.org> http://hg.python.org/cpython/rev/c10a3ddba483 changeset: 82059:c10a3ddba483 parent: 82057:b8a6bc70fc08 parent: 82058:3b316ea5aa82 user: Victor Stinner date: Thu Feb 07 23:17:34 2013 +0100 summary: (Merge 3.3) Issue #17137: When an Unicode string is resized, the internal wide character string (wstr) format is now cleared. files: Lib/test/test_unicode.py | 15 +++++++++++++++ Misc/NEWS | 3 +++ Objects/unicodeobject.c | 4 ++++ 3 files changed, 22 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2191,6 +2191,21 @@ self.assertEqual(args[0], text) self.assertEqual(len(args), 1) + def test_resize(self): + for length in range(1, 100, 7): + # generate a fresh string (refcount=1) + text = 'a' * length + 'b' + + # fill wstr internal field + abc = text.encode('unicode_internal') + self.assertEqual(abc.decode('unicode_internal'), text) + + # resize text: wstr field must be cleared and then recomputed + text += 'c' + abcdef = text.encode('unicode_internal') + self.assertNotEqual(abc, abcdef) + self.assertEqual(abcdef.decode('unicode_internal'), text) + class StringModuleTest(unittest.TestCase): def test_formatter_parser(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #17137: When an Unicode string is resized, the internal wide character + string (wstr) format is now cleared. + - Issue #17043: The unicode-internal decoder no longer read past the end of input buffer. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -717,6 +717,10 @@ if (!PyUnicode_IS_ASCII(unicode)) _PyUnicode_WSTR_LENGTH(unicode) = length; } + else if (_PyUnicode_HAS_WSTR_MEMORY(unicode)) { + PyObject_DEL(_PyUnicode_WSTR(unicode)); + _PyUnicode_WSTR(unicode) = NULL; + } #ifdef Py_DEBUG unicode_fill_invalid(unicode, old_length); #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 01:17:01 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Fri, 8 Feb 2013 01:17:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_self_to_experts_for_J?= =?utf-8?q?ava/JVM=2E?= Message-ID: <3Z2H596B43zPXn@mail.python.org> http://hg.python.org/devguide/rev/75d1706be67c changeset: 596:75d1706be67c user: Frank Wierzbicki date: Thu Feb 07 16:16:41 2013 -0800 summary: Add self to experts for Java/JVM. files: experts.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -285,6 +285,7 @@ OS2/EMX aimacintyre Solaris/OpenIndiana jcea Windows tim.golden, brian.curtin +JVM/Java frank.wierzbicki =================== =========== -- Repository URL: http://hg.python.org/devguide From solipsis at pitrou.net Fri Feb 8 06:04:50 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 08 Feb 2013 06:04:50 +0100 Subject: [Python-checkins] Daily reference leaks (c10a3ddba483): sum=1 Message-ID: results for c10a3ddba483 on branch "default" -------------------------------------------- test_concurrent_futures leaked [0, -2, 3] memory blocks, sum=1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogsIQ_Ms', '-x'] From python-checkins at python.org Fri Feb 8 06:43:51 2013 From: python-checkins at python.org (senthil.kumaran) Date: Fri, 8 Feb 2013 06:43:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Addressing_the?= =?utf-8?q?_review_comment_made_by_Terry_Reedy?= Message-ID: <3Z2QLH3XtSzSdq@mail.python.org> http://hg.python.org/cpython/rev/28229bdb1571 changeset: 82060:28229bdb1571 branch: 3.2 parent: 82055:55a89352e220 user: Senthil Kumaran date: Thu Feb 07 21:43:21 2013 -0800 summary: Addressing the review comment made by Terry Reedy files: Doc/library/urllib.request.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -61,9 +61,9 @@ :class:`http.client.HTTPResponse` object which has the following :ref:`httpresponse-objects` methods. - For ftp, file, data urls and requests are explicity handled by legacy - :class:`URLopener` and :class:`FancyURLopener` class, this function returns - an :class:`urllib.response.addinfourl` object which can work as + For ftp, file, and data urls and requests explicity handled by legacy + :class:`URLopener` and :class:`FancyURLopener` classes, this function + returns a :class:`urllib.response.addinfourl` object which can work as :term:`context manager` and has methods such as * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 06:43:52 2013 From: python-checkins at python.org (senthil.kumaran) Date: Fri, 8 Feb 2013 06:43:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Addressing_the_review_comment_made_by_Terry_Reedy?= Message-ID: <3Z2QLJ6PbZzSdw@mail.python.org> http://hg.python.org/cpython/rev/3942c20bebdb changeset: 82061:3942c20bebdb branch: 3.3 parent: 82058:3b316ea5aa82 parent: 82060:28229bdb1571 user: Senthil Kumaran date: Thu Feb 07 21:44:42 2013 -0800 summary: Addressing the review comment made by Terry Reedy files: Doc/library/urllib.request.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -67,9 +67,9 @@ :class:`http.client.HTTPResponse` object which has the following :ref:`httpresponse-objects` methods. - For ftp, file, data urls and requests are explicity handled by legacy - :class:`URLopener` and :class:`FancyURLopener` class, this function returns - an :class:`urllib.response.addinfourl` object which can work as + For ftp, file, and data urls and requests explicity handled by legacy + :class:`URLopener` and :class:`FancyURLopener` classes, this function + returns a :class:`urllib.response.addinfourl` object which can work as :term:`context manager` and has methods such as * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 06:43:54 2013 From: python-checkins at python.org (senthil.kumaran) Date: Fri, 8 Feb 2013 06:43:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Addressing_the_review_comment_made_by_Terry_Reedy?= Message-ID: <3Z2QLL1x28zSdq@mail.python.org> http://hg.python.org/cpython/rev/771a0317da83 changeset: 82062:771a0317da83 parent: 82059:c10a3ddba483 parent: 82061:3942c20bebdb user: Senthil Kumaran date: Thu Feb 07 21:45:08 2013 -0800 summary: Addressing the review comment made by Terry Reedy files: Doc/library/urllib.request.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -67,9 +67,9 @@ :class:`http.client.HTTPResponse` object which has the following :ref:`httpresponse-objects` methods. - For ftp, file, data urls and requests are explicity handled by legacy - :class:`URLopener` and :class:`FancyURLopener` class, this function returns - an :class:`urllib.response.addinfourl` object which can work as + For ftp, file, and data urls and requests explicity handled by legacy + :class:`URLopener` and :class:`FancyURLopener` classes, this function + returns a :class:`urllib.response.addinfourl` object which can work as :term:`context manager` and has methods such as * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, -- Repository URL: http://hg.python.org/cpython From senthil at uthcode.com Fri Feb 8 06:46:40 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Thu, 7 Feb 2013 21:46:40 -0800 Subject: [Python-checkins] cpython (3.2): Fix Issue17069: Document getcode method in urllib.request.rst In-Reply-To: <5113F3F4.7000200@udel.edu> References: <3Z1tWQ1FnPzMRY@mail.python.org> <5113F3F4.7000200@udel.edu> Message-ID: On Thu, Feb 7, 2013 at 10:35 AM, Terry Reedy wrote: > 3 suggested changes: Thanks for the review comments. I have made the changes. > 1. On 2/7/2013 3:49 AM, senthil.kumaran wrote: > The first part up to "class," is not a proper English clause. Perhaps you > meant ('that' added): Yes, I had intended to. Looks like I missed it. Ended up using your third suggestion. > Since 'url' is generally pronounced 'you-are-el' rather than like 'earl' (I > checked Merriam-Webster online), that should be 'a' rather than 'an' url... > . > https://owl.english.purdue.edu/owl/resource/591/01/ This is important to know. Thanks for sharing. -- Senthil From python-checkins at python.org Fri Feb 8 07:18:35 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 8 Feb 2013 07:18:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzY5NzI6?= =?utf-8?q?_fix_the_documentation_mis_applied_patch=2E?= Message-ID: <3Z2R6M4QS4zSgH@mail.python.org> http://hg.python.org/cpython/rev/d73fb6b06891 changeset: 82063:d73fb6b06891 branch: 2.7 parent: 82054:649937bb8f1c user: Gregory P. Smith date: Thu Feb 07 22:11:03 2013 -0800 summary: Issue #6972: fix the documentation mis applied patch. files: Doc/library/zipfile.rst | 23 +++++++++++++---------- 1 files changed, 13 insertions(+), 10 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -213,6 +213,16 @@ .. versionadded:: 2.6 + .. note:: + + If a member filename is an absolute path, a drive/UNC sharepoint and + leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes + ``foo/bar`` on Unix, and ``C:\foo\bar`` becomes ``foo\bar`` on Windows. + And all ``".."`` components in a member filename will be removed, e.g.: + ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal + characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) + replaced by underscore (``_``). + .. method:: ZipFile.extractall([path[, members[, pwd]]]) @@ -227,6 +237,9 @@ It is possible that files are created outside of *path*, e.g. members that have absolute filenames starting with ``"/"`` or filenames with two dots ``".."``. + + .. versionchanged:: 2.7.4 + The zipfile module attempts to prevent that. See :meth:`extract` note. .. versionadded:: 2.6 @@ -242,16 +255,6 @@ .. versionadded:: 2.6 - .. note:: - - If a member filename is an absolute path, a drive/UNC sharepoint and - leading (back)slashes will be stripped, e.g.: ``///foo/bar`` becomes - ``foo/bar`` on Unix, and ``C:\foo\bar`` becomes ``foo\bar`` on Windows. - And all ``".."`` components in a member filename will be removed, e.g.: - ``../../foo../../ba..r`` becomes ``foo../ba..r``. On Windows illegal - characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) - replaced by underscore (``_``). - .. method:: ZipFile.read(name[, pwd]) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 07:18:36 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 8 Feb 2013 07:18:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzY5NzI6?= =?utf-8?q?_keep_the_warning_about_untrusted_extraction_and_mention?= Message-ID: <3Z2R6N734jzRRP@mail.python.org> http://hg.python.org/cpython/rev/1c2d41850147 changeset: 82064:1c2d41850147 branch: 3.2 parent: 82060:28229bdb1571 user: Gregory P. Smith date: Thu Feb 07 22:15:04 2013 -0800 summary: Issue #6972: keep the warning about untrusted extraction and mention the version it was improved in. files: Doc/library/zipfile.rst | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -232,9 +232,15 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. note:: + .. warning:: - See :meth:`extract` note. + Never extract archives from untrusted sources without prior inspection. + It is possible that files are created outside of *path*, e.g. members + that have absolute filenames starting with ``"/"`` or filenames with two + dots ``".."``. + + .. versionchanged:: 3.2.4 + The zipfile module attempts to prevent that. See :meth:`extract` note. .. method:: ZipFile.printdir() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 07:18:38 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 8 Feb 2013 07:18:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=236972=3A_keep_the_warning_about_untrusted_extraction_a?= =?utf-8?q?nd_mention?= Message-ID: <3Z2R6Q2SK9zSgY@mail.python.org> http://hg.python.org/cpython/rev/5fbca37de9b1 changeset: 82065:5fbca37de9b1 branch: 3.3 parent: 82061:3942c20bebdb parent: 82064:1c2d41850147 user: Gregory P. Smith date: Thu Feb 07 22:15:51 2013 -0800 summary: Issue #6972: keep the warning about untrusted extraction and mention the version it was improved in. files: Doc/library/zipfile.rst | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -260,9 +260,15 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. note:: + .. warning:: - See :meth:`extract` note. + Never extract archives from untrusted sources without prior inspection. + It is possible that files are created outside of *path*, e.g. members + that have absolute filenames starting with ``"/"`` or filenames with two + dots ``".."``. + + .. versionchanged:: 3.3.1 + The zipfile module attempts to prevent that. See :meth:`extract` note. .. method:: ZipFile.printdir() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 07:18:39 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 8 Feb 2013 07:18:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=236972=3A_keep_the_warning_about_untrusted_extrac?= =?utf-8?q?tion_and_mention?= Message-ID: <3Z2R6R52mLzSfR@mail.python.org> http://hg.python.org/cpython/rev/f5e3f2f0fe79 changeset: 82066:f5e3f2f0fe79 parent: 82062:771a0317da83 parent: 82065:5fbca37de9b1 user: Gregory P. Smith date: Thu Feb 07 22:17:21 2013 -0800 summary: Issue #6972: keep the warning about untrusted extraction and mention the version it was improved in. files: Doc/library/zipfile.rst | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -260,8 +260,12 @@ be a subset of the list returned by :meth:`namelist`. *pwd* is the password used for encrypted files. - .. note:: + .. warning:: + Never extract archives from untrusted sources without prior inspection. + It is possible that files are created outside of *path*, e.g. members + that have absolute filenames starting with ``"/"`` or filenames with two + dots ``".."``. This module attempts to prevent that. See :meth:`extract` note. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 07:18:41 2013 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 8 Feb 2013 07:18:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_whitespace_fix?= Message-ID: <3Z2R6T1R3kzSgT@mail.python.org> http://hg.python.org/cpython/rev/b370cf4d8cc1 changeset: 82067:b370cf4d8cc1 branch: 2.7 parent: 82063:d73fb6b06891 user: Gregory P. Smith date: Thu Feb 07 22:18:21 2013 -0800 summary: whitespace fix files: Doc/library/zipfile.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -237,7 +237,7 @@ It is possible that files are created outside of *path*, e.g. members that have absolute filenames starting with ``"/"`` or filenames with two dots ``".."``. - + .. versionchanged:: 2.7.4 The zipfile module attempts to prevent that. See :meth:`extract` note. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 10:28:37 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 8 Feb 2013 10:28:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_accidental?= =?utf-8?q?_non-breakable_space_=28U+00A0=29=2E?= Message-ID: <3Z2WKd0mnJzRX8@mail.python.org> http://hg.python.org/cpython/rev/532c59c52cb3 changeset: 82068:532c59c52cb3 branch: 2.7 user: Serhiy Storchaka date: Fri Feb 08 11:21:32 2013 +0200 summary: Fix accidental non-breakable space (U+00A0). files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -239,7 +239,7 @@ - Issue #15505: `unittest.installHandler` no longer assumes SIGINT handler is set to a callable object. -- Issue #17051: Fix a memory leak in os.path.isdir()?on Windows. Patch by +- Issue #17051: Fix a memory leak in os.path.isdir() on Windows. Patch by Robert Xiao. - Issue #9290: In IDLE the sys.std* streams now implement io.TextIOBase -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 10:28:38 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 8 Feb 2013 10:28:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Fix_accidental?= =?utf-8?q?_non-breakable_spaces_=28U+00A0=29=2E?= Message-ID: <3Z2WKf3V4gzRc1@mail.python.org> http://hg.python.org/cpython/rev/83bf8191e4d1 changeset: 82069:83bf8191e4d1 branch: 3.2 parent: 82064:1c2d41850147 user: Serhiy Storchaka date: Fri Feb 08 11:22:05 2013 +0200 summary: Fix accidental non-breakable spaces (U+00A0). files: Lib/test/test_pyexpat.py | 2 +- Misc/NEWS | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -600,7 +600,7 @@ self.assertEqual(str(e), 'unclosed token: line 2, column 0') def test2(self): - #?\xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) + # \xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) xml = b"\r\n" parser = expat.ParserCreate() try: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -217,7 +217,7 @@ - Issue #17073: Fix some integer overflows in sqlite3 module. -- Issue #17114: IDLE?now uses non-strict config parser. +- Issue #17114: IDLE now uses non-strict config parser. - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 10:28:39 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 8 Feb 2013 10:28:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Fix_accidental_non-breakable_spaces_=28U+00A0=29=2E?= Message-ID: <3Z2WKg6Z4tzSbw@mail.python.org> http://hg.python.org/cpython/rev/ae92c8759c43 changeset: 82070:ae92c8759c43 branch: 3.3 parent: 82065:5fbca37de9b1 parent: 82069:83bf8191e4d1 user: Serhiy Storchaka date: Fri Feb 08 11:24:16 2013 +0200 summary: Fix accidental non-breakable spaces (U+00A0). files: Lib/test/test_lzma.py | 2 +- Lib/test/test_pyexpat.py | 2 +- Misc/NEWS | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -671,7 +671,7 @@ def test_read_truncated(self): # Drop stream footer: CRC (4 bytes), index size (4 bytes), - # flags?(2 bytes) and magic number (2 bytes). + # flags (2 bytes) and magic number (2 bytes). truncated = COMPRESSED_XZ[:-12] with LZMAFile(BytesIO(truncated)) as f: self.assertRaises(EOFError, f.read) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -600,7 +600,7 @@ self.assertEqual(str(e), 'unclosed token: line 2, column 0') def test2(self): - #?\xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) + # \xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) xml = b"\r\n" parser = expat.ParserCreate() try: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -171,7 +171,7 @@ - Issue #17073: Fix some integer overflows in sqlite3 module. -- Issue #17114: IDLE?now uses non-strict config parser. +- Issue #17114: IDLE now uses non-strict config parser. - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 10:28:41 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 8 Feb 2013 10:28:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fix_accidental_non-breakable_spaces_=28U+00A0=29=2E?= Message-ID: <3Z2WKj2Ck8zSbq@mail.python.org> http://hg.python.org/cpython/rev/5c4b00581198 changeset: 82071:5c4b00581198 parent: 82066:f5e3f2f0fe79 parent: 82070:ae92c8759c43 user: Serhiy Storchaka date: Fri Feb 08 11:24:55 2013 +0200 summary: Fix accidental non-breakable spaces (U+00A0). files: Lib/test/test_lzma.py | 2 +- Lib/test/test_pyexpat.py | 2 +- Misc/NEWS | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -671,7 +671,7 @@ def test_read_truncated(self): # Drop stream footer: CRC (4 bytes), index size (4 bytes), - # flags?(2 bytes) and magic number (2 bytes). + # flags (2 bytes) and magic number (2 bytes). truncated = COMPRESSED_XZ[:-12] with LZMAFile(BytesIO(truncated)) as f: self.assertRaises(EOFError, f.read) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -600,7 +600,7 @@ self.assertEqual(str(e), 'unclosed token: line 2, column 0') def test2(self): - #?\xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) + # \xc2\x85 is UTF-8 encoded U+0085 (NEXT LINE) xml = b"\r\n" parser = expat.ParserCreate() try: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -243,7 +243,7 @@ - Issue #17073: Fix some integer overflows in sqlite3 module. -- Issue #17114: IDLE?now uses non-strict config parser. +- Issue #17114: IDLE now uses non-strict config parser. - Issue #16723: httplib.HTTPResponse no longer marked closed when the connection is automatically closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Feb 8 17:56:20 2013 From: python-checkins at python.org (daniel.holth) Date: Fri, 8 Feb 2013 17:56:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_wheel_rationales?= Message-ID: <3Z2jGD2R3SzRpt@mail.python.org> http://hg.python.org/peps/rev/fb66173832fb changeset: 4719:fb66173832fb user: Daniel Holth date: Fri Feb 08 11:56:08 2013 -0500 summary: wheel rationales files: pep-0427.txt | 15 ++++++++++++++- 1 files changed, 14 insertions(+), 1 deletions(-) diff --git a/pep-0427.txt b/pep-0427.txt --- a/pep-0427.txt +++ b/pep-0427.txt @@ -235,7 +235,7 @@ ------------------ Wheel files include an extended RECORD that enables digital -signatures. PEP 376's RECORD is altered to include +signatures. PEP 376's RECORD is altered to include a secure hash ``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64 encoding with no trailing = characters) as the second column instead of an md5sum. All possible entries are hashed, including any @@ -316,6 +316,19 @@ the signature, or individual files can be verified without having to download the whole archive. +Why does wheel allow JWS signatures? + The JOSE specifications including JWS are designed to be easy to + implement, a feature that is also one of wheel's primary design goals. + +Why does wheel also allow S/MIME signatures? + S/MIME signatures are allowed for users who need or want to use an + existing public key infrastructure with wheel. + + Signed packages are only a basic building block in a secured package + update system. Wheel only provides the building block. A complete + system would provide for key distribution and trust and would specify + which signature format was required. + Appendix ======== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Feb 9 04:32:43 2013 From: python-checkins at python.org (daniel.holth) Date: Sat, 9 Feb 2013 04:32:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP-427_=28wheel=29_edits?= Message-ID: <3Z2zNW4DZ5zPpg@mail.python.org> http://hg.python.org/peps/rev/d9d42d5c1b41 changeset: 4720:d9d42d5c1b41 user: Daniel Holth date: Fri Feb 08 22:32:34 2013 -0500 summary: PEP-427 (wheel) edits files: pep-0427.txt | 69 ++++++++++++++++++++------------------- 1 files changed, 36 insertions(+), 33 deletions(-) diff --git a/pep-0427.txt b/pep-0427.txt --- a/pep-0427.txt +++ b/pep-0427.txt @@ -17,13 +17,13 @@ This PEP describes a built-package format for Python called "wheel". -A wheel is a ZIP-format archive with a specially formatted file name -and the ``.whl`` extension. It contains a single distribution nearly -as it would be installed according to PEP 376 with a particular -installation scheme. A wheel file may be installed by simply -unpacking into site-packages with the standard 'unzip' tool, while -preserving enough information to spread its contents out onto their -final paths at any later time. +A wheel is a ZIP-format archive with a specially formatted file name and +the ``.whl`` extension. It contains a single distribution nearly as it +would be installed according to PEP 376 with a particular installation +scheme. Although a specialized installer is recommended, a wheel file +may be installed by simply unpacking into site-packages with the standard +'unzip' tool while preserving enough information to spread its contents +out onto their final paths at any later time. Note @@ -147,7 +147,7 @@ File contents ''''''''''''' -The conents of a wheel file, where {distribution} is replaced with the +The contents of a wheel file, where {distribution} is replaced with the name of the package, e.g. ``beaglevote`` and {version} is replaced with its version, e.g. ``1.0.0``, consist of: @@ -163,8 +163,8 @@ ``b'#!python'`` in order to enjoy script wrapper generation and ``#!python`` rewriting at install time. They may have any or no extension. -#. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.3 - (PEP 426) or greater format metadata. +#. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.2 + (PEP 345) or greater format metadata. #. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the archive itself:: @@ -195,20 +195,21 @@ #. Wheel .dist-info directories include at a minimum METADATA, WHEEL, and RECORD. -#. METADATA is the PEP 426 metadata (Metadata version 1.3 or greater) -#. WHEEL is the wheel metadata, specific to a build of the package. +#. METADATA is the package metadata, the same format as PKG-INFO as + found at the root of sdists. +#. WHEEL is the wheel metadata specific to a build of the package. #. RECORD is a list of (almost) all the files in the wheel and their secure hashes. Unlike PEP 376, every file except RECORD, which cannot contain a hash of itself, must include its hash. The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted, as signed wheel files rely on the strong hashes in RECORD to validate the integrity of the archive. -#. INSTALLER and REQUESTED are not included in the archive. +#. PEP 376's INSTALLER and REQUESTED are not included in the archive. #. RECORD.jws is used for digital signatures. It is not mentioned in RECORD. #. RECORD.p7s is allowed as a courtesy to anyone who would prefer to - use s/mime signatures to secure their wheel files. It is not - mentioned in RECORD and it is ignored by the official tools. + use S/MIME signatures to secure their wheel files. It is not + mentioned in RECORD. #. During extraction, wheel installers verify all the hashes in RECORD against the file contents. Apart from RECORD and its signatures, installation will fail if any file in the archive is not both @@ -239,29 +240,31 @@ ``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64 encoding with no trailing = characters) as the second column instead of an md5sum. All possible entries are hashed, including any -generated files such as .pyc files, but not RECORD. For example:: +generated files such as .pyc files, but not RECORD which cannot contain its +own hash. For example:: file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144 distribution-1.0.dist-info/RECORD,, The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in RECORD at all since they can only be added after RECORD is generated. -Every other file in the archive must have a correct hash in RECORD, +Every other file in the archive must have a correct hash in RECORD or the installation will fail. If JSON web signatures are used, one or more JSON Web Signature JSON -Serialization (JWS-JS) signatures may be stored in a file RECORD.jws -adjacent to RECORD. JWS is used to sign RECORD by including the SHA-256 -hash of RECORD as the JWS payload:: +Serialization (JWS-JS) signatures is stored in a file RECORD.jws adjacent +to RECORD. JWS is used to sign RECORD by including the SHA-256 hash of +RECORD as the signature's JSON payload:: { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" } -If RECORD.p7s is used, it must contain a PKCS#7 format signature of -RECORD. +If RECORD.p7s is used, it must contain a detached S/MIME format signature +of RECORD. -A wheel installer may assume that the signature has already been checked -against RECORD, and only must verify the hashes in RECORD against the -extracted file contents. +A wheel installer is not required to understand digital signatures but +MUST verify the hashes in RECORD against the extracted file contents. +When the installer checks file hashes against RECORD, a separate signature +checker only needs to establish that RECORD matches the signature. See @@ -313,21 +316,21 @@ Attached signatures are more convenient than detached signatures because they travel with the archive. Since only the individual files are signed, the archive can be recompressed without invalidating - the signature, or individual files can be verified without having + the signature or individual files can be verified without having to download the whole archive. Why does wheel allow JWS signatures? - The JOSE specifications including JWS are designed to be easy to - implement, a feature that is also one of wheel's primary design goals. + The JOSE specifications of which JWS is a part are designed to be easy + to implement, a feature that is also one of wheel's primary design + goals. JWS yields a useful, concise pure-Python implementation. Why does wheel also allow S/MIME signatures? - S/MIME signatures are allowed for users who need or want to use an + S/MIME signatures are allowed for users who need or want to use existing public key infrastructure with wheel. - Signed packages are only a basic building block in a secured package - update system. Wheel only provides the building block. A complete - system would provide for key distribution and trust and would specify - which signature format was required. + Signed packages are only a basic building block in a secure package + update system and many kinds of attacks are possible even when + packages are signed. Wheel only provides the building block. Appendix ======== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Feb 9 05:12:20 2013 From: python-checkins at python.org (daniel.holth) Date: Sat, 9 Feb 2013 05:12:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP-0426_=28Metadata_1=2E3=29?= =?utf-8?q?_edits?= Message-ID: <3Z30GD5LQhzRQn@mail.python.org> http://hg.python.org/peps/rev/25daa0625d1d changeset: 4721:25daa0625d1d user: Daniel Holth date: Fri Feb 08 23:12:01 2013 -0500 summary: PEP-0426 (Metadata 1.3) edits files: pep-0426.txt | 65 ++++++++++++++++++++++++--------------- 1 files changed, 40 insertions(+), 25 deletions(-) diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -553,11 +553,10 @@ N.N[.N]+[{a|b|c|rc}N][.postN][.devN] -Version numbers which do not comply with this scheme are an error. Projects -which wish to use non-compliant version numbers must restrict themselves -to metadata v1.1 (PEP 314) or earlier, as those versions do not mandate -a particular versioning scheme. - +Version numbers which do not comply with this scheme are an +error. Projects which wish to use non-compliant version numbers may +be heuristically normalized to this scheme and are less likely to sort +correctly. Suffixes and ordering --------------------- @@ -610,29 +609,44 @@ 1.0.post456 1.1.dev1 +Recommended subset +------------------ + +The PEP authors recommend using a subset of the allowed version scheme, +similar to http://semver.org/ but without hyphenated versions. + +* Version numbers are always three positive digits ``X.Y.Z`` (Major.Minor.Patch) +* The patch version is incremented for backwards-compatible bug fixes. +* The minor version is incremented for backwards-compatible API additions. + When the minor version is incremented the patch version resets to 0. +* The major version is incremented for backwards-incompatible API changes. + When the major version is incremented the minor and patch versions + reset to 0. +* Pre-release versions ending in ``a``, ``b``, and ``c`` may be used. +* Dev- and post-release versions are discouraged. Increment the patch number + instead of issuing a post-release. + +When the major version is 0, the API is not considered stable, may change at +any time, and the rules about when to increment the minor and patch version +numbers are relaxed. Ordering across different metadata versions ------------------------------------------- -After making a release with a given metadata version, it is assumed that -projects will not revert to an older metadata version. - -Accordingly, and to allow projects with non-compliant version schemes -to more easily migrate to the version scheme defined in this PEP, -releases should be sorted by their declared metadata version *before* -being sorted by the distribution version. - -Software that processes distribution metadata should account for the fact -that metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not +Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version numbering or sorting scheme. This PEP does not mandate any particular approach to handling such versions, but acknowledges that the de facto standard for sorting such versions is the scheme used by the ``pkg_resources`` component of ``setuptools``. For metadata v1.2 (PEP 345), the recommended sort order is defined in -PEP 386 (It is expected that projects where the defined PEP 386 sort -order is incorrect will skip straight from metadata v1.1 to metadata v1.3). +PEP 386. +The best way for a publisher to get predictable ordering is to excuse +non-compliant versions from sorting by hiding them on PyPI or by removing +them from any private index that is being used. Otherwise a client +may be restricted to using exact versions to get the correct or latest +version of your project. Version specifiers ================== @@ -647,17 +661,18 @@ `Version scheme`_. Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` -and ``!=``. +``!=``, and ``~>``. -When no comparison operator is provided, it is equivalent to using the -following pair of version clauses:: +When no comparison operator is provided, it is equivalent to using ``==``. - >= V, < V+1 +The ``~>`` operator, "equal or greater in the last digit" is equivalent +to a pair of version clauses:: -where ``V+1`` is the "next version" after ``V``, as determined by -incrementing the last numeric component in ``V`` (for example, if -``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then -``V+1 == 1.1``). + ~> 2.3.3 + +is equivalent to:: + + >= 2.3.3, < 2.4.0 The comma (",") is equivalent to a logical **and** operator. -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Sat Feb 9 06:04:27 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 09 Feb 2013 06:04:27 +0100 Subject: [Python-checkins] Daily reference leaks (5c4b00581198): sum=0 Message-ID: results for 5c4b00581198 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogsLT3ad', '-x'] From python-checkins at python.org Sat Feb 9 08:05:57 2013 From: python-checkins at python.org (ned.deily) Date: Sat, 9 Feb 2013 08:05:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3MTYx?= =?utf-8?q?=3A_make_install_now_also_installs_a_python2_and_python_man_pag?= =?utf-8?q?e=2E?= Message-ID: <3Z346Y5f39zNBG@mail.python.org> http://hg.python.org/cpython/rev/29826cb3f12e changeset: 82072:29826cb3f12e branch: 2.7 parent: 82068:532c59c52cb3 user: Ned Deily date: Fri Feb 08 22:51:52 2013 -0800 summary: Issue #17161: make install now also installs a python2 and python man page. files: Makefile.pre.in | 16 ++++++++++++---- Misc/NEWS | 2 ++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -806,7 +806,8 @@ install: @FRAMEWORKINSTALLFIRST@ altinstall bininstall maninstall @FRAMEWORKINSTALLLAST@ # Install almost everything without disturbing previous versions -altinstall: @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall libainstall \ +altinstall: @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall \ + libainstall altmaninstall \ sharedinstall oldsharedinstall @FRAMEWORKALTINSTALLLAST@ # Install shared libraries enabled by Setup @@ -876,8 +877,8 @@ else true; \ fi -# Install the manual page -maninstall: +# Install the versioned manual page +altmaninstall: @for i in $(MANDIR) $(MANDIR)/man1; \ do \ if test ! -d $(DESTDIR)$$i; then \ @@ -889,6 +890,13 @@ $(INSTALL_DATA) $(srcdir)/Misc/python.man \ $(DESTDIR)$(MANDIR)/man1/python$(VERSION).1 +# Install the unversioned manual pages +maninstall: altmaninstall + -rm -f $(DESTDIR)$(MANDIR)/man1/python2.1 + (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python$(VERSION).1 python2.1) + -rm -f $(DESTDIR)$(MANDIR)/man1/python.1 + (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python2.1 python.1) + # Install the library PLATDIR= plat-$(MACHDEP) EXTRAPLATDIR= @EXTRAPLATDIR@ @@ -1326,7 +1334,7 @@ .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure .PHONY: frameworkinstallmaclib frameworkinstallapps frameworkinstallunixtools .PHONY: frameworkaltinstallunixtools recheck autoconf clean clobber distclean -.PHONY: smelly funny patchcheck +.PHONY: smelly funny patchcheck altmaninstall .PHONY: gdbhooks # IF YOU PUT ANYTHING HERE IT WILL GO AWAY diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -830,6 +830,8 @@ Retina displays. Applies to Tkinter apps, such as IDLE, on OS X framework builds linked with Cocoa Tk 8.5. +- Issue #17161: make install now also installs a python2 and python man page. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 9 08:05:59 2013 From: python-checkins at python.org (ned.deily) Date: Sat, 9 Feb 2013 08:05:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE3MTYx?= =?utf-8?q?=3A_make_install_now_also_installs_a_python3_man_page=2E?= Message-ID: <3Z346b1KpPzPSh@mail.python.org> http://hg.python.org/cpython/rev/b0d9b273c029 changeset: 82073:b0d9b273c029 branch: 3.2 parent: 82069:83bf8191e4d1 user: Ned Deily date: Fri Feb 08 22:53:51 2013 -0800 summary: Issue #17161: make install now also installs a python3 man page. files: Makefile.pre.in | 15 ++++++++++----- Misc/NEWS | 2 ++ 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -830,10 +830,10 @@ -$(TESTPYTHON) $(TESTPROG) $(MEMTESTOPTS) $(TESTPYTHON) $(TESTPROG) $(MEMTESTOPTS) -install: altinstall bininstall +install: altinstall bininstall maninstall altinstall: @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall libainstall \ - sharedinstall oldsharedinstall maninstall @FRAMEWORKALTINSTALLLAST@ + sharedinstall oldsharedinstall altmaninstall @FRAMEWORKALTINSTALLLAST@ # Install shared libraries enabled by Setup DESTDIRS= $(exec_prefix) $(LIBDIR) $(BINLIBDEST) $(DESTSHARED) @@ -912,8 +912,8 @@ -rm -f $(DESTDIR)$(BINDIR)/2to3 (cd $(DESTDIR)$(BINDIR); $(LN) -s 2to3-$(VERSION) 2to3) -# Install the manual page -maninstall: +# Install the versioned manual page +altmaninstall: @for i in $(MANDIR) $(MANDIR)/man1; \ do \ if test ! -d $(DESTDIR)$$i; then \ @@ -925,6 +925,11 @@ $(INSTALL_DATA) $(srcdir)/Misc/python.man \ $(DESTDIR)$(MANDIR)/man1/python$(VERSION).1 +# Install the unversioned manual page +maninstall: altmaninstall + -rm -f $(DESTDIR)$(MANDIR)/man1/python3.1 + (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python$(VERSION).1 python3.1) + # Install the library PLATDIR= plat-$(MACHDEP) EXTRAPLATDIR= @EXTRAPLATDIR@ @@ -1360,7 +1365,7 @@ .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure .PHONY: frameworkinstallmaclib frameworkinstallapps frameworkinstallunixtools .PHONY: frameworkaltinstallunixtools recheck autoconf clean clobber distclean -.PHONY: smelly funny patchcheck +.PHONY: smelly funny patchcheck altmaninstall .PHONY: gdbhooks # IF YOU PUT ANYTHING HERE IT WILL GO AWAY diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -993,6 +993,8 @@ Retina displays. Applies to Tkinter apps, such as IDLE, on OS X framework builds linked with Cocoa Tk 8.5. +- Issue #17161: make install now also installs a python3 man page. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 9 08:06:00 2013 From: python-checkins at python.org (ned.deily) Date: Sat, 9 Feb 2013 08:06:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2317161=3A_make_install_now_also_installs_a_python3_man?= =?utf-8?q?_page=2E?= Message-ID: <3Z346c4KbCzSTw@mail.python.org> http://hg.python.org/cpython/rev/9828c4ffb401 changeset: 82074:9828c4ffb401 branch: 3.3 parent: 82070:ae92c8759c43 parent: 82073:b0d9b273c029 user: Ned Deily date: Fri Feb 08 23:02:09 2013 -0800 summary: Issue #17161: make install now also installs a python3 man page. files: Makefile.pre.in | 15 ++++++++++----- Misc/NEWS | 2 ++ 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -886,10 +886,10 @@ $(TESTRUNNER) $(QUICKTESTOPTS) -install: altinstall bininstall +install: altinstall bininstall maninstall altinstall: @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall libainstall \ - sharedinstall oldsharedinstall maninstall @FRAMEWORKALTINSTALLLAST@ + sharedinstall oldsharedinstall altmaninstall @FRAMEWORKALTINSTALLLAST@ # Install shared libraries enabled by Setup DESTDIRS= $(exec_prefix) $(LIBDIR) $(BINLIBDEST) $(DESTSHARED) @@ -970,8 +970,8 @@ -rm -f $(DESTDIR)$(BINDIR)/pyvenv (cd $(DESTDIR)$(BINDIR); $(LN) -s pyvenv-$(VERSION) pyvenv) -# Install the manual page -maninstall: +# Install the versioned manual page +altmaninstall: @for i in $(MANDIR) $(MANDIR)/man1; \ do \ if test ! -d $(DESTDIR)$$i; then \ @@ -983,6 +983,11 @@ $(INSTALL_DATA) $(srcdir)/Misc/python.man \ $(DESTDIR)$(MANDIR)/man1/python$(VERSION).1 +# Install the unversioned manual page +maninstall: altmaninstall + -rm -f $(DESTDIR)$(MANDIR)/man1/python3.1 + (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python$(VERSION).1 python3.1) + # Install the library PLATDIR= plat-$(MACHDEP) EXTRAPLATDIR= @EXTRAPLATDIR@ @@ -1452,7 +1457,7 @@ .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure .PHONY: frameworkinstallmaclib frameworkinstallapps frameworkinstallunixtools .PHONY: frameworkaltinstallunixtools recheck autoconf clean clobber distclean -.PHONY: smelly funny patchcheck touch +.PHONY: smelly funny patchcheck touch altmaninstall .PHONY: gdbhooks # IF YOU PUT ANYTHING HERE IT WILL GO AWAY diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -663,6 +663,8 @@ Retina displays. Applies to Tkinter apps, such as IDLE, on OS X framework builds linked with Cocoa Tk 8.5. +- Issue #17161: make install now also installs a python3 man page. + Tools/Demos ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 9 08:06:01 2013 From: python-checkins at python.org (ned.deily) Date: Sat, 9 Feb 2013 08:06:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317161=3A_merge_from_3=2E3?= Message-ID: <3Z346d70lVzSfs@mail.python.org> http://hg.python.org/cpython/rev/5e874b2a0469 changeset: 82075:5e874b2a0469 parent: 82071:5c4b00581198 parent: 82074:9828c4ffb401 user: Ned Deily date: Fri Feb 08 23:05:10 2013 -0800 summary: Issue #17161: merge from 3.3 files: Makefile.pre.in | 15 ++++++++++----- Misc/NEWS | 2 ++ 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -886,10 +886,10 @@ $(TESTRUNNER) $(QUICKTESTOPTS) -install: altinstall bininstall +install: altinstall bininstall maninstall altinstall: @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall libainstall \ - sharedinstall oldsharedinstall maninstall @FRAMEWORKALTINSTALLLAST@ + sharedinstall oldsharedinstall altmaninstall @FRAMEWORKALTINSTALLLAST@ # Install shared libraries enabled by Setup DESTDIRS= $(exec_prefix) $(LIBDIR) $(BINLIBDEST) $(DESTSHARED) @@ -970,8 +970,8 @@ -rm -f $(DESTDIR)$(BINDIR)/pyvenv (cd $(DESTDIR)$(BINDIR); $(LN) -s pyvenv-$(VERSION) pyvenv) -# Install the manual page -maninstall: +# Install the versioned manual page +altmaninstall: @for i in $(MANDIR) $(MANDIR)/man1; \ do \ if test ! -d $(DESTDIR)$$i; then \ @@ -983,6 +983,11 @@ $(INSTALL_DATA) $(srcdir)/Misc/python.man \ $(DESTDIR)$(MANDIR)/man1/python$(VERSION).1 +# Install the unversioned manual page +maninstall: altmaninstall + -rm -f $(DESTDIR)$(MANDIR)/man1/python3.1 + (cd $(DESTDIR)$(MANDIR)/man1; $(LN) -s python$(VERSION).1 python3.1) + # Install the library PLATDIR= plat-$(MACHDEP) EXTRAPLATDIR= @EXTRAPLATDIR@ @@ -1455,7 +1460,7 @@ .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure .PHONY: frameworkinstallmaclib frameworkinstallapps frameworkinstallunixtools .PHONY: frameworkaltinstallunixtools recheck autoconf clean clobber distclean -.PHONY: smelly funny patchcheck touch +.PHONY: smelly funny patchcheck touch altmaninstall .PHONY: gdbhooks # IF YOU PUT ANYTHING HERE IT WILL GO AWAY diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -901,6 +901,8 @@ Retina displays. Applies to Tkinter apps, such as IDLE, on OS X framework builds linked with Cocoa Tk 8.5. +- Issue #17161: make install now also installs a python3 man page. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 9 10:17:45 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 9 Feb 2013 10:17:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2Njg2?= =?utf-8?q?=3A_Fixed_a_lot_of_bugs_in_audioop_module=2E?= Message-ID: <3Z372d3Dj8zRXP@mail.python.org> http://hg.python.org/cpython/rev/6add6ac6a802 changeset: 82076:6add6ac6a802 branch: 2.7 parent: 82072:29826cb3f12e user: Serhiy Storchaka date: Sat Feb 09 11:10:30 2013 +0200 summary: Issue #16686: Fixed a lot of bugs in audioop module. * avgpp() and maxpp() no more crash on empty and 1-samples input fragment. They now work when peak-peak values are greater INT_MAX. * ratecv() no more crashes on empty input fragment. * Fixed an integer overflow in ratecv(). * Fixed an integer overflow in add() and bias() for 32-bit samples. * reverse(), lin2lin() and ratecv() no more lose precision for 32-bit samples. * max() and rms() no more returns negative result for 32-bit sample -0x80000000. * minmax() now returns correct max value for 32-bit sample -0x80000000. * avg(), mul(), tomono() and tostereo() now round negative result down and can return 32-bit sample -0x80000000. * add() now can return 32-bit sample -0x80000000. files: Doc/library/audioop.rst | 6 +- Lib/test/test_audioop.py | 405 ++++++++++++++++++-------- Misc/NEWS | 6 + Modules/audioop.c | 320 +++++++++++---------- 4 files changed, 457 insertions(+), 280 deletions(-) diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -38,7 +38,7 @@ Return a fragment which is the addition of the two samples passed as parameters. *width* is the sample width in bytes, either ``1``, ``2`` or ``4``. Both - fragments should have the same length. + fragments should have the same length. Samples are truncated in case of overflow. .. function:: adpcm2lin(adpcmfragment, width, state) @@ -71,7 +71,7 @@ .. function:: bias(fragment, width, bias) Return a fragment that is the original fragment with a bias added to each - sample. + sample. Samples wrap around in case of overflow. .. function:: cross(fragment, width) @@ -181,7 +181,7 @@ .. function:: mul(fragment, width, factor) Return a fragment that has all samples in the original fragment multiplied by - the floating-point value *factor*. Overflow is silently ignored. + the floating-point value *factor*. Samples are truncated in case of overflow. .. function:: ratecv(fragment, width, nchannels, inrate, outrate, state[, weightA[, weightB]]) diff --git a/Lib/test/test_audioop.py b/Lib/test/test_audioop.py --- a/Lib/test/test_audioop.py +++ b/Lib/test/test_audioop.py @@ -1,25 +1,33 @@ import audioop +import sys import unittest +import struct from test.test_support import run_unittest -endian = 'big' if audioop.getsample('\0\1', 2, 0) == 1 else 'little' -def gendata1(): - return '\0\1\2' +formats = { + 1: 'b', + 2: 'h', + 4: 'i', +} -def gendata2(): - if endian == 'big': - return '\0\0\0\1\0\2' - else: - return '\0\0\1\0\2\0' +def pack(width, data): + return struct.pack('=%d%s' % (len(data), formats[width]), *data) -def gendata4(): - if endian == 'big': - return '\0\0\0\0\0\0\0\1\0\0\0\2' - else: - return '\0\0\0\0\1\0\0\0\2\0\0\0' +packs = { + 1: lambda *data: pack(1, data), + 2: lambda *data: pack(2, data), + 4: lambda *data: pack(4, data), +} +maxvalues = {w: (1 << (8 * w - 1)) - 1 for w in (1, 2, 4)} +minvalues = {w: -1 << (8 * w - 1) for w in (1, 2, 4)} -data = [gendata1(), gendata2(), gendata4()] +datas = { + 1: b'\x00\x12\x45\xbb\x7f\x80\xff', + 2: packs[2](0, 0x1234, 0x4567, -0x4567, 0x7fff, -0x8000, -1), + 4: packs[4](0, 0x12345678, 0x456789ab, -0x456789ab, + 0x7fffffff, -0x80000000, -1), +} INVALID_DATA = [ (b'abc', 0), @@ -31,164 +39,315 @@ class TestAudioop(unittest.TestCase): def test_max(self): - self.assertEqual(audioop.max(data[0], 1), 2) - self.assertEqual(audioop.max(data[1], 2), 2) - self.assertEqual(audioop.max(data[2], 4), 2) + for w in 1, 2, 4: + self.assertEqual(audioop.max(b'', w), 0) + p = packs[w] + self.assertEqual(audioop.max(p(5), w), 5) + self.assertEqual(audioop.max(p(5, -8, -1), w), 8) + self.assertEqual(audioop.max(p(maxvalues[w]), w), maxvalues[w]) + self.assertEqual(audioop.max(p(minvalues[w]), w), -minvalues[w]) + self.assertEqual(audioop.max(datas[w], w), -minvalues[w]) def test_minmax(self): - self.assertEqual(audioop.minmax(data[0], 1), (0, 2)) - self.assertEqual(audioop.minmax(data[1], 2), (0, 2)) - self.assertEqual(audioop.minmax(data[2], 4), (0, 2)) + for w in 1, 2, 4: + self.assertEqual(audioop.minmax(b'', w), + (0x7fffffff, -0x80000000)) + p = packs[w] + self.assertEqual(audioop.minmax(p(5), w), (5, 5)) + self.assertEqual(audioop.minmax(p(5, -8, -1), w), (-8, 5)) + self.assertEqual(audioop.minmax(p(maxvalues[w]), w), + (maxvalues[w], maxvalues[w])) + self.assertEqual(audioop.minmax(p(minvalues[w]), w), + (minvalues[w], minvalues[w])) + self.assertEqual(audioop.minmax(datas[w], w), + (minvalues[w], maxvalues[w])) def test_maxpp(self): - self.assertEqual(audioop.maxpp(data[0], 1), 0) - self.assertEqual(audioop.maxpp(data[1], 2), 0) - self.assertEqual(audioop.maxpp(data[2], 4), 0) + for w in 1, 2, 4: + self.assertEqual(audioop.maxpp(b'', w), 0) + self.assertEqual(audioop.maxpp(packs[w](*range(100)), w), 0) + self.assertEqual(audioop.maxpp(packs[w](9, 10, 5, 5, 0, 1), w), 10) + self.assertEqual(audioop.maxpp(datas[w], w), + maxvalues[w] - minvalues[w]) def test_avg(self): - self.assertEqual(audioop.avg(data[0], 1), 1) - self.assertEqual(audioop.avg(data[1], 2), 1) - self.assertEqual(audioop.avg(data[2], 4), 1) + for w in 1, 2, 4: + self.assertEqual(audioop.avg(b'', w), 0) + p = packs[w] + self.assertEqual(audioop.avg(p(5), w), 5) + self .assertEqual(audioop.avg(p(5, 8), w), 6) + self.assertEqual(audioop.avg(p(5, -8), w), -2) + self.assertEqual(audioop.avg(p(maxvalues[w], maxvalues[w]), w), + maxvalues[w]) + self.assertEqual(audioop.avg(p(minvalues[w], minvalues[w]), w), + minvalues[w]) + self.assertEqual(audioop.avg(packs[4](0x50000000, 0x70000000), 4), + 0x60000000) + self.assertEqual(audioop.avg(packs[4](-0x50000000, -0x70000000), 4), + -0x60000000) def test_avgpp(self): - self.assertEqual(audioop.avgpp(data[0], 1), 0) - self.assertEqual(audioop.avgpp(data[1], 2), 0) - self.assertEqual(audioop.avgpp(data[2], 4), 0) + for w in 1, 2, 4: + self.assertEqual(audioop.avgpp(b'', w), 0) + self.assertEqual(audioop.avgpp(packs[w](*range(100)), w), 0) + self.assertEqual(audioop.avgpp(packs[w](9, 10, 5, 5, 0, 1), w), 10) + self.assertEqual(audioop.avgpp(datas[1], 1), 196) + self.assertEqual(audioop.avgpp(datas[2], 2), 50534) + self.assertEqual(audioop.avgpp(datas[4], 4), 3311897002) def test_rms(self): - self.assertEqual(audioop.rms(data[0], 1), 1) - self.assertEqual(audioop.rms(data[1], 2), 1) - self.assertEqual(audioop.rms(data[2], 4), 1) + for w in 1, 2, 4: + self.assertEqual(audioop.rms(b'', w), 0) + p = packs[w] + self.assertEqual(audioop.rms(p(*range(100)), w), 57) + self.assertAlmostEqual(audioop.rms(p(maxvalues[w]) * 5, w), + maxvalues[w], delta=1) + self.assertAlmostEqual(audioop.rms(p(minvalues[w]) * 5, w), + -minvalues[w], delta=1) + self.assertEqual(audioop.rms(datas[1], 1), 77) + self.assertEqual(audioop.rms(datas[2], 2), 20001) + self.assertEqual(audioop.rms(datas[4], 4), 1310854152) def test_cross(self): - self.assertEqual(audioop.cross(data[0], 1), 0) - self.assertEqual(audioop.cross(data[1], 2), 0) - self.assertEqual(audioop.cross(data[2], 4), 0) + for w in 1, 2, 4: + self.assertEqual(audioop.cross(b'', w), -1) + p = packs[w] + self.assertEqual(audioop.cross(p(0, 1, 2), w), 0) + self.assertEqual(audioop.cross(p(1, 2, -3, -4), w), 1) + self.assertEqual(audioop.cross(p(-1, -2, 3, 4), w), 1) + self.assertEqual(audioop.cross(p(0, minvalues[w]), w), 1) + self.assertEqual(audioop.cross(p(minvalues[w], maxvalues[w]), w), 1) def test_add(self): - data2 = [] - for d in data: - str = '' - for s in d: - str = str + chr(ord(s)*2) - data2.append(str) - self.assertEqual(audioop.add(data[0], data[0], 1), data2[0]) - self.assertEqual(audioop.add(data[1], data[1], 2), data2[1]) - self.assertEqual(audioop.add(data[2], data[2], 4), data2[2]) + for w in 1, 2, 4: + self.assertEqual(audioop.add(b'', b'', w), b'') + self.assertEqual(audioop.add(datas[w], b'\0' * len(datas[w]), w), + datas[w]) + self.assertEqual(audioop.add(datas[1], datas[1], 1), + b'\x00\x24\x7f\x80\x7f\x80\xfe') + self.assertEqual(audioop.add(datas[2], datas[2], 2), + packs[2](0, 0x2468, 0x7fff, -0x8000, 0x7fff, -0x8000, -2)) + self.assertEqual(audioop.add(datas[4], datas[4], 4), + packs[4](0, 0x2468acf0, 0x7fffffff, -0x80000000, + 0x7fffffff, -0x80000000, -2)) def test_bias(self): - # Note: this test assumes that avg() works - d1 = audioop.bias(data[0], 1, 100) - d2 = audioop.bias(data[1], 2, 100) - d4 = audioop.bias(data[2], 4, 100) - self.assertEqual(audioop.avg(d1, 1), 101) - self.assertEqual(audioop.avg(d2, 2), 101) - self.assertEqual(audioop.avg(d4, 4), 101) + for w in 1, 2, 4: + for bias in 0, 1, -1, 127, -128, 0x7fffffff, -0x80000000: + self.assertEqual(audioop.bias(b'', w, bias), b'') + self.assertEqual(audioop.bias(datas[1], 1, 1), + b'\x01\x13\x46\xbc\x80\x81\x00') + self.assertEqual(audioop.bias(datas[1], 1, -1), + b'\xff\x11\x44\xba\x7e\x7f\xfe') + self.assertEqual(audioop.bias(datas[1], 1, 0x7fffffff), + b'\xff\x11\x44\xba\x7e\x7f\xfe') + self.assertEqual(audioop.bias(datas[1], 1, -0x80000000), + datas[1]) + self.assertEqual(audioop.bias(datas[2], 2, 1), + packs[2](1, 0x1235, 0x4568, -0x4566, -0x8000, -0x7fff, 0)) + self.assertEqual(audioop.bias(datas[2], 2, -1), + packs[2](-1, 0x1233, 0x4566, -0x4568, 0x7ffe, 0x7fff, -2)) + self.assertEqual(audioop.bias(datas[2], 2, 0x7fffffff), + packs[2](-1, 0x1233, 0x4566, -0x4568, 0x7ffe, 0x7fff, -2)) + self.assertEqual(audioop.bias(datas[2], 2, -0x80000000), + datas[2]) + self.assertEqual(audioop.bias(datas[4], 4, 1), + packs[4](1, 0x12345679, 0x456789ac, -0x456789aa, + -0x80000000, -0x7fffffff, 0)) + self.assertEqual(audioop.bias(datas[4], 4, -1), + packs[4](-1, 0x12345677, 0x456789aa, -0x456789ac, + 0x7ffffffe, 0x7fffffff, -2)) + self.assertEqual(audioop.bias(datas[4], 4, 0x7fffffff), + packs[4](0x7fffffff, -0x6dcba989, -0x3a987656, 0x3a987654, + -2, -1, 0x7ffffffe)) + self.assertEqual(audioop.bias(datas[4], 4, -0x80000000), + packs[4](-0x80000000, -0x6dcba988, -0x3a987655, 0x3a987655, + -1, 0, 0x7fffffff)) def test_lin2lin(self): - # too simple: we test only the size - for d1 in data: - for d2 in data: - got = len(d1)//3 - wtd = len(d2)//3 - self.assertEqual(len(audioop.lin2lin(d1, got, wtd)), len(d2)) + for w in 1, 2, 4: + self.assertEqual(audioop.lin2lin(datas[w], w, w), datas[w]) + + self.assertEqual(audioop.lin2lin(datas[1], 1, 2), + packs[2](0, 0x1200, 0x4500, -0x4500, 0x7f00, -0x8000, -0x100)) + self.assertEqual(audioop.lin2lin(datas[1], 1, 4), + packs[4](0, 0x12000000, 0x45000000, -0x45000000, + 0x7f000000, -0x80000000, -0x1000000)) + self.assertEqual(audioop.lin2lin(datas[2], 2, 1), + b'\x00\x12\x45\xba\x7f\x80\xff') + self.assertEqual(audioop.lin2lin(datas[2], 2, 4), + packs[4](0, 0x12340000, 0x45670000, -0x45670000, + 0x7fff0000, -0x80000000, -0x10000)) + self.assertEqual(audioop.lin2lin(datas[4], 4, 1), + b'\x00\x12\x45\xba\x7f\x80\xff') + self.assertEqual(audioop.lin2lin(datas[4], 4, 2), + packs[2](0, 0x1234, 0x4567, -0x4568, 0x7fff, -0x8000, -1)) def test_adpcm2lin(self): + self.assertEqual(audioop.adpcm2lin(b'\x07\x7f\x7f', 1, None), + (b'\x00\x00\x00\xff\x00\xff', (-179, 40))) + self.assertEqual(audioop.adpcm2lin(b'\x07\x7f\x7f', 2, None), + (packs[2](0, 0xb, 0x29, -0x16, 0x72, -0xb3), (-179, 40))) + self.assertEqual(audioop.adpcm2lin(b'\x07\x7f\x7f', 4, None), + (packs[4](0, 0xb0000, 0x290000, -0x160000, 0x720000, + -0xb30000), (-179, 40))) + # Very cursory test - self.assertEqual(audioop.adpcm2lin(b'\0\0', 1, None), (b'\0' * 4, (0,0))) - self.assertEqual(audioop.adpcm2lin(b'\0\0', 2, None), (b'\0' * 8, (0,0))) - self.assertEqual(audioop.adpcm2lin(b'\0\0', 4, None), (b'\0' * 16, (0,0))) + for w in 1, 2, 4: + self.assertEqual(audioop.adpcm2lin(b'\0' * 5, w, None), + (b'\0' * w * 10, (0, 0))) def test_lin2adpcm(self): + self.assertEqual(audioop.lin2adpcm(datas[1], 1, None), + (b'\x07\x7f\x7f', (-221, 39))) + self.assertEqual(audioop.lin2adpcm(datas[2], 2, None), + (b'\x07\x7f\x7f', (31, 39))) + self.assertEqual(audioop.lin2adpcm(datas[4], 4, None), + (b'\x07\x7f\x7f', (31, 39))) + # Very cursory test - self.assertEqual(audioop.lin2adpcm('\0\0\0\0', 1, None), ('\0\0', (0,0))) + for w in 1, 2, 4: + self.assertEqual(audioop.lin2adpcm(b'\0' * w * 10, w, None), + (b'\0' * 5, (0, 0))) def test_lin2alaw(self): - self.assertEqual(audioop.lin2alaw(data[0], 1), '\xd5\xc5\xf5') - self.assertEqual(audioop.lin2alaw(data[1], 2), '\xd5\xd5\xd5') - self.assertEqual(audioop.lin2alaw(data[2], 4), '\xd5\xd5\xd5') + self.assertEqual(audioop.lin2alaw(datas[1], 1), + b'\xd5\x87\xa4\x24\xaa\x2a\x5a') + self.assertEqual(audioop.lin2alaw(datas[2], 2), + b'\xd5\x87\xa4\x24\xaa\x2a\x55') + self.assertEqual(audioop.lin2alaw(datas[4], 4), + b'\xd5\x87\xa4\x24\xaa\x2a\x55') def test_alaw2lin(self): - # Cursory - d = audioop.lin2alaw(data[0], 1) - self.assertEqual(audioop.alaw2lin(d, 1), data[0]) - if endian == 'big': - self.assertEqual(audioop.alaw2lin(d, 2), - b'\x00\x08\x01\x08\x02\x10') - self.assertEqual(audioop.alaw2lin(d, 4), - b'\x00\x08\x00\x00\x01\x08\x00\x00\x02\x10\x00\x00') - else: - self.assertEqual(audioop.alaw2lin(d, 2), - b'\x08\x00\x08\x01\x10\x02') - self.assertEqual(audioop.alaw2lin(d, 4), - b'\x00\x00\x08\x00\x00\x00\x08\x01\x00\x00\x10\x02') + encoded = b'\x00\x03\x24\x2a\x51\x54\x55\x58\x6b\x71\x7f'\ + b'\x80\x83\xa4\xaa\xd1\xd4\xd5\xd8\xeb\xf1\xff' + src = [-688, -720, -2240, -4032, -9, -3, -1, -27, -244, -82, -106, + 688, 720, 2240, 4032, 9, 3, 1, 27, 244, 82, 106] + for w in 1, 2, 4: + self.assertEqual(audioop.alaw2lin(encoded, w), + packs[w](*(x << (w * 8) >> 13 for x in src))) + + encoded = ''.join(chr(x) for x in xrange(256)) + for w in 2, 4: + decoded = audioop.alaw2lin(encoded, w) + self.assertEqual(audioop.lin2alaw(decoded, w), encoded) def test_lin2ulaw(self): - self.assertEqual(audioop.lin2ulaw(data[0], 1), '\xff\xe7\xdb') - self.assertEqual(audioop.lin2ulaw(data[1], 2), '\xff\xff\xff') - self.assertEqual(audioop.lin2ulaw(data[2], 4), '\xff\xff\xff') + self.assertEqual(audioop.lin2ulaw(datas[1], 1), + b'\xff\xad\x8e\x0e\x80\x00\x67') + self.assertEqual(audioop.lin2ulaw(datas[2], 2), + b'\xff\xad\x8e\x0e\x80\x00\x7e') + self.assertEqual(audioop.lin2ulaw(datas[4], 4), + b'\xff\xad\x8e\x0e\x80\x00\x7e') def test_ulaw2lin(self): - # Cursory - d = audioop.lin2ulaw(data[0], 1) - self.assertEqual(audioop.ulaw2lin(d, 1), data[0]) - if endian == 'big': - self.assertEqual(audioop.ulaw2lin(d, 2), - b'\x00\x00\x01\x04\x02\x0c') - self.assertEqual(audioop.ulaw2lin(d, 4), - b'\x00\x00\x00\x00\x01\x04\x00\x00\x02\x0c\x00\x00') - else: - self.assertEqual(audioop.ulaw2lin(d, 2), - b'\x00\x00\x04\x01\x0c\x02') - self.assertEqual(audioop.ulaw2lin(d, 4), - b'\x00\x00\x00\x00\x00\x00\x04\x01\x00\x00\x0c\x02') + encoded = b'\x00\x0e\x28\x3f\x57\x6a\x76\x7c\x7e\x7f'\ + b'\x80\x8e\xa8\xbf\xd7\xea\xf6\xfc\xfe\xff' + src = [-8031, -4447, -1471, -495, -163, -53, -18, -6, -2, 0, + 8031, 4447, 1471, 495, 163, 53, 18, 6, 2, 0] + for w in 1, 2, 4: + self.assertEqual(audioop.ulaw2lin(encoded, w), + packs[w](*(x << (w * 8) >> 14 for x in src))) + + # Current u-law implementation has two codes fo 0: 0x7f and 0xff. + encoded = ''.join(chr(x) for x in range(127) + range(128, 256)) + for w in 2, 4: + decoded = audioop.ulaw2lin(encoded, w) + self.assertEqual(audioop.lin2ulaw(decoded, w), encoded) def test_mul(self): - data2 = [] - for d in data: - str = '' - for s in d: - str = str + chr(ord(s)*2) - data2.append(str) - self.assertEqual(audioop.mul(data[0], 1, 2), data2[0]) - self.assertEqual(audioop.mul(data[1],2, 2), data2[1]) - self.assertEqual(audioop.mul(data[2], 4, 2), data2[2]) + for w in 1, 2, 4: + self.assertEqual(audioop.mul(b'', w, 2), b'') + self.assertEqual(audioop.mul(datas[w], w, 0), + b'\0' * len(datas[w])) + self.assertEqual(audioop.mul(datas[w], w, 1), + datas[w]) + self.assertEqual(audioop.mul(datas[1], 1, 2), + b'\x00\x24\x7f\x80\x7f\x80\xfe') + self.assertEqual(audioop.mul(datas[2], 2, 2), + packs[2](0, 0x2468, 0x7fff, -0x8000, 0x7fff, -0x8000, -2)) + self.assertEqual(audioop.mul(datas[4], 4, 2), + packs[4](0, 0x2468acf0, 0x7fffffff, -0x80000000, + 0x7fffffff, -0x80000000, -2)) def test_ratecv(self): + for w in 1, 2, 4: + self.assertEqual(audioop.ratecv(b'', w, 1, 8000, 8000, None), + (b'', (-1, ((0, 0),)))) + self.assertEqual(audioop.ratecv(b'', w, 5, 8000, 8000, None), + (b'', (-1, ((0, 0),) * 5))) + self.assertEqual(audioop.ratecv(b'', w, 1, 8000, 16000, None), + (b'', (-2, ((0, 0),)))) + self.assertEqual(audioop.ratecv(datas[w], w, 1, 8000, 8000, None)[0], + datas[w]) state = None - d1, state = audioop.ratecv(data[0], 1, 1, 8000, 16000, state) - d2, state = audioop.ratecv(data[0], 1, 1, 8000, 16000, state) - self.assertEqual(d1 + d2, '\000\000\001\001\002\001\000\000\001\001\002') + d1, state = audioop.ratecv(b'\x00\x01\x02', 1, 1, 8000, 16000, state) + d2, state = audioop.ratecv(b'\x00\x01\x02', 1, 1, 8000, 16000, state) + self.assertEqual(d1 + d2, b'\000\000\001\001\002\001\000\000\001\001\002') + + for w in 1, 2, 4: + d0, state0 = audioop.ratecv(datas[w], w, 1, 8000, 16000, None) + d, state = b'', None + for i in range(0, len(datas[w]), w): + d1, state = audioop.ratecv(datas[w][i:i + w], w, 1, + 8000, 16000, state) + d += d1 + self.assertEqual(d, d0) + self.assertEqual(state, state0) def test_reverse(self): - self.assertEqual(audioop.reverse(data[0], 1), '\2\1\0') + for w in 1, 2, 4: + self.assertEqual(audioop.reverse(b'', w), b'') + self.assertEqual(audioop.reverse(packs[w](0, 1, 2), w), + packs[w](2, 1, 0)) def test_tomono(self): - data2 = '' - for d in data[0]: - data2 = data2 + d + d - self.assertEqual(audioop.tomono(data2, 1, 0.5, 0.5), data[0]) + for w in 1, 2, 4: + data1 = datas[w] + data2 = bytearray(2 * len(data1)) + for k in range(w): + data2[k::2*w] = data1[k::w] + self.assertEqual(audioop.tomono(str(data2), w, 1, 0), data1) + self.assertEqual(audioop.tomono(str(data2), w, 0, 1), b'\0' * len(data1)) + for k in range(w): + data2[k+w::2*w] = data1[k::w] + self.assertEqual(audioop.tomono(str(data2), w, 0.5, 0.5), data1) def test_tostereo(self): - data2 = '' - for d in data[0]: - data2 = data2 + d + d - self.assertEqual(audioop.tostereo(data[0], 1, 1, 1), data2) + for w in 1, 2, 4: + data1 = datas[w] + data2 = bytearray(2 * len(data1)) + for k in range(w): + data2[k::2*w] = data1[k::w] + self.assertEqual(audioop.tostereo(data1, w, 1, 0), data2) + self.assertEqual(audioop.tostereo(data1, w, 0, 0), b'\0' * len(data2)) + for k in range(w): + data2[k+w::2*w] = data1[k::w] + self.assertEqual(audioop.tostereo(data1, w, 1, 1), data2) def test_findfactor(self): - self.assertEqual(audioop.findfactor(data[1], data[1]), 1.0) + self.assertEqual(audioop.findfactor(datas[2], datas[2]), 1.0) + self.assertEqual(audioop.findfactor(b'\0' * len(datas[2]), datas[2]), + 0.0) def test_findfit(self): - self.assertEqual(audioop.findfit(data[1], data[1]), (0, 1.0)) + self.assertEqual(audioop.findfit(datas[2], datas[2]), (0, 1.0)) + self.assertEqual(audioop.findfit(datas[2], packs[2](1, 2, 0)), + (1, 8038.8)) + self.assertEqual(audioop.findfit(datas[2][:-2] * 5 + datas[2], datas[2]), + (30, 1.0)) def test_findmax(self): - self.assertEqual(audioop.findmax(data[1], 1), 2) + self.assertEqual(audioop.findmax(datas[2], 1), 5) def test_getsample(self): - for i in range(3): - self.assertEqual(audioop.getsample(data[0], 1, i), i) - self.assertEqual(audioop.getsample(data[1], 2, i), i) - self.assertEqual(audioop.getsample(data[2], 4, i), i) + for w in 1, 2, 4: + data = packs[w](0, 1, -1, maxvalues[w], minvalues[w]) + self.assertEqual(audioop.getsample(data, w, 0), 0) + self.assertEqual(audioop.getsample(data, w, 1), 1) + self.assertEqual(audioop.getsample(data, w, 2), -1) + self.assertEqual(audioop.getsample(data, w, 3), maxvalues[w]) + self.assertEqual(audioop.getsample(data, w, 4), minvalues[w]) def test_negativelen(self): # from issue 3306, previously it segfaulted @@ -220,9 +379,9 @@ self.assertRaises(audioop.error, audioop.lin2adpcm, data, size, state) def test_wrongsize(self): - data = b'abc' + data = b'abcdefgh' state = None - for size in (-1, 3, 5): + for size in (-1, 0, 3, 5, 1024): self.assertRaises(audioop.error, audioop.ulaw2lin, data, size) self.assertRaises(audioop.error, audioop.alaw2lin, data, size) self.assertRaises(audioop.error, audioop.adpcm2lin, data, size, state) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -202,6 +202,12 @@ Library ------- +- Issue #16686: Fixed a lot of bugs in audioop module. Fixed crashes in + avgpp(), maxpp() and ratecv(). Fixed an integer overflow in add(), bias(), + and ratecv(). reverse(), lin2lin() and ratecv() no more lose precision for + 32-bit samples. max() and rms() no more returns a negative result and + various other functions now work correctly with 32-bit sample -0x80000000. + - Issue #17073: Fix some integer overflows in sqlite3 module. - Issue #6083: Fix multiple segmentation faults occured when PyArg_ParseTuple diff --git a/Modules/audioop.c b/Modules/audioop.c --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -24,6 +24,21 @@ #endif #endif +static const int maxvals[] = {0, 0x7F, 0x7FFF, 0x7FFFFF, 0x7FFFFFFF}; +static const int minvals[] = {0, -0x80, -0x8000, -0x800000, -0x80000000}; +static const unsigned int masks[] = {0, 0xFF, 0xFFFF, 0xFFFFFF, 0xFFFFFFFF}; + +static int +fbound(double val, double minval, double maxval) +{ + if (val > maxval) + val = maxval; + else if (val < minval + 1) + val = minval; + return val; +} + + /* Code shamelessly stolen from sox, 12.17.7, g711.c ** (c) Craig Reese, Joe Campbell and Jeff Poskanzer 1989 */ @@ -345,7 +360,7 @@ signed char *cp; int len, size, val = 0; int i; - int max = 0; + unsigned int absval, max = 0; if ( !PyArg_ParseTuple(args, "s#i:max", &cp, &len, &size) ) return 0; @@ -355,10 +370,14 @@ if ( size == 1 ) val = (int)*CHARP(cp, i); else if ( size == 2 ) val = (int)*SHORTP(cp, i); else if ( size == 4 ) val = (int)*LONGP(cp, i); - if ( val < 0 ) val = (-val); - if ( val > max ) max = val; + if (val < 0) absval = (-val); + else absval = val; + if (absval > max) max = absval; } - return PyInt_FromLong(max); + if (max <= INT_MAX) + return PyInt_FromLong(max); + else + return PyLong_FromUnsignedLong(max); } static PyObject * @@ -367,7 +386,7 @@ signed char *cp; int len, size, val = 0; int i; - int min = 0x7fffffff, max = -0x7fffffff; + int min = 0x7fffffff, max = -0x80000000; if (!PyArg_ParseTuple(args, "s#i:minmax", &cp, &len, &size)) return NULL; @@ -404,7 +423,7 @@ if ( len == 0 ) val = 0; else - val = (int)(avg / (double)(len/size)); + val = (int)floor(avg / (double)(len/size)); return PyInt_FromLong(val); } @@ -414,6 +433,7 @@ signed char *cp; int len, size, val = 0; int i; + unsigned int res; double sum_squares = 0.0; if ( !PyArg_ParseTuple(args, "s#i:rms", &cp, &len, &size) ) @@ -427,10 +447,13 @@ sum_squares += (double)val*(double)val; } if ( len == 0 ) - val = 0; + res = 0; else - val = (int)sqrt(sum_squares / (double)(len/size)); - return PyInt_FromLong(val); + res = (unsigned int)sqrt(sum_squares / (double)(len/size)); + if (res <= INT_MAX) + return PyInt_FromLong(res); + else + return PyLong_FromUnsignedLong(res); } static double _sum2(short *a, short *b, int len) @@ -620,52 +643,49 @@ int len, size, val = 0, prevval = 0, prevextremevalid = 0, prevextreme = 0; int i; - double avg = 0.0; - int diff, prevdiff, extremediff, nextreme = 0; + double sum = 0.0; + unsigned int avg; + int diff, prevdiff, nextreme = 0; if ( !PyArg_ParseTuple(args, "s#i:avgpp", &cp, &len, &size) ) return 0; if (!audioop_check_parameters(len, size)) return NULL; - /* Compute first delta value ahead. Also automatically makes us - ** skip the first extreme value - */ + if (len <= size*2) + return PyInt_FromLong(0); if ( size == 1 ) prevval = (int)*CHARP(cp, 0); else if ( size == 2 ) prevval = (int)*SHORTP(cp, 0); else if ( size == 4 ) prevval = (int)*LONGP(cp, 0); - if ( size == 1 ) val = (int)*CHARP(cp, size); - else if ( size == 2 ) val = (int)*SHORTP(cp, size); - else if ( size == 4 ) val = (int)*LONGP(cp, size); - prevdiff = val - prevval; - + prevdiff = 17; /* Anything != 0, 1 */ for ( i=size; i max ) - max = extremediff; + if (val != prevval) { + diff = val < prevval; + if (prevdiff == !diff) { + /* Derivative changed sign. Compute difference to + ** last extreme value and remember. + */ + if (prevextremevalid) { + if (prevval < prevextreme) + extremediff = (unsigned int)prevextreme - + (unsigned int)prevval; + else + extremediff = (unsigned int)prevval - + (unsigned int)prevextreme; + if ( extremediff > max ) + max = extremediff; + } + prevextremevalid = 1; + prevextreme = prevval; } - prevextremevalid = 1; - prevextreme = prevval; + prevval = val; + prevdiff = diff; } - prevval = val; - if ( diff != 0 ) - prevdiff = diff; } - return PyInt_FromLong(max); + if (max <= INT_MAX) + return PyInt_FromLong(max); + else + return PyLong_FromUnsignedLong(max); } static PyObject * @@ -749,7 +771,7 @@ { signed char *cp, *ncp; int len, size, val = 0; - double factor, fval, maxval; + double factor, fval, maxval, minval; PyObject *rv; int i; @@ -758,13 +780,8 @@ if (!audioop_check_parameters(len, size)) return NULL; - if ( size == 1 ) maxval = (double) 0x7f; - else if ( size == 2 ) maxval = (double) 0x7fff; - else if ( size == 4 ) maxval = (double) 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = (double) maxvals[size]; + minval = (double) minvals[size]; rv = PyString_FromStringAndSize(NULL, len); if ( rv == 0 ) @@ -777,9 +794,7 @@ else if ( size == 2 ) val = (int)*SHORTP(cp, i); else if ( size == 4 ) val = (int)*LONGP(cp, i); fval = (double)val*factor; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val = (int)fval; + val = (int)floor(fbound(fval, minval, maxval)); if ( size == 1 ) *CHARP(ncp, i) = (signed char)val; else if ( size == 2 ) *SHORTP(ncp, i) = (short)val; else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)val; @@ -792,7 +807,7 @@ { signed char *cp, *ncp; int len, size, val1 = 0, val2 = 0; - double fac1, fac2, fval, maxval; + double fac1, fac2, fval, maxval, minval; PyObject *rv; int i; @@ -806,13 +821,8 @@ return NULL; } - if ( size == 1 ) maxval = (double) 0x7f; - else if ( size == 2 ) maxval = (double) 0x7fff; - else if ( size == 4 ) maxval = (double) 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = (double) maxvals[size]; + minval = (double) minvals[size]; rv = PyString_FromStringAndSize(NULL, len/2); if ( rv == 0 ) @@ -828,9 +838,7 @@ else if ( size == 2 ) val2 = (int)*SHORTP(cp, i+2); else if ( size == 4 ) val2 = (int)*LONGP(cp, i+4); fval = (double)val1*fac1 + (double)val2*fac2; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val1 = (int)fval; + val1 = (int)floor(fbound(fval, minval, maxval)); if ( size == 1 ) *CHARP(ncp, i/2) = (signed char)val1; else if ( size == 2 ) *SHORTP(ncp, i/2) = (short)val1; else if ( size == 4 ) *LONGP(ncp, i/2)= (Py_Int32)val1; @@ -843,7 +851,7 @@ { signed char *cp, *ncp; int len, size, val1, val2, val = 0; - double fac1, fac2, fval, maxval; + double fac1, fac2, fval, maxval, minval; PyObject *rv; int i; @@ -853,13 +861,8 @@ if (!audioop_check_parameters(len, size)) return NULL; - if ( size == 1 ) maxval = (double) 0x7f; - else if ( size == 2 ) maxval = (double) 0x7fff; - else if ( size == 4 ) maxval = (double) 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = (double) maxvals[size]; + minval = (double) minvals[size]; if (len > INT_MAX/2) { PyErr_SetString(PyExc_MemoryError, @@ -879,14 +882,10 @@ else if ( size == 4 ) val = (int)*LONGP(cp, i); fval = (double)val*fac1; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val1 = (int)fval; + val1 = (int)floor(fbound(fval, minval, maxval)); fval = (double)val*fac2; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val2 = (int)fval; + val2 = (int)floor(fbound(fval, minval, maxval)); if ( size == 1 ) *CHARP(ncp, i*2) = (signed char)val1; else if ( size == 2 ) *SHORTP(ncp, i*2) = (short)val1; @@ -903,7 +902,7 @@ audioop_add(PyObject *self, PyObject *args) { signed char *cp1, *cp2, *ncp; - int len1, len2, size, val1 = 0, val2 = 0, maxval, newval; + int len1, len2, size, val1 = 0, val2 = 0, minval, maxval, newval; PyObject *rv; int i; @@ -917,13 +916,8 @@ return 0; } - if ( size == 1 ) maxval = 0x7f; - else if ( size == 2 ) maxval = 0x7fff; - else if ( size == 4 ) maxval = 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = maxvals[size]; + minval = minvals[size]; rv = PyString_FromStringAndSize(NULL, len1); if ( rv == 0 ) @@ -939,12 +933,19 @@ else if ( size == 2 ) val2 = (int)*SHORTP(cp2, i); else if ( size == 4 ) val2 = (int)*LONGP(cp2, i); - newval = val1 + val2; - /* truncate in case of overflow */ - if (newval > maxval) newval = maxval; - else if (newval < -maxval) newval = -maxval; - else if (size == 4 && (newval^val1) < 0 && (newval^val2) < 0) - newval = val1 > 0 ? maxval : - maxval; + if (size < 4) { + newval = val1 + val2; + /* truncate in case of overflow */ + if (newval > maxval) + newval = maxval; + else if (newval < minval) + newval = minval; + } + else { + double fval = (double)val1 + (double)val2; + /* truncate in case of overflow */ + newval = (int)floor(fbound(fval, minval, maxval)); + } if ( size == 1 ) *CHARP(ncp, i) = (signed char)newval; else if ( size == 2 ) *SHORTP(ncp, i) = (short)newval; @@ -957,7 +958,8 @@ audioop_bias(PyObject *self, PyObject *args) { signed char *cp, *ncp; - int len, size, val = 0; + int len, size; + unsigned int val = 0, mask; PyObject *rv; int i; int bias; @@ -974,15 +976,20 @@ return 0; ncp = (signed char *)PyString_AsString(rv); + mask = masks[size]; for ( i=0; i < len; i += size ) { - if ( size == 1 ) val = (int)*CHARP(cp, i); - else if ( size == 2 ) val = (int)*SHORTP(cp, i); - else if ( size == 4 ) val = (int)*LONGP(cp, i); + if ( size == 1 ) val = (unsigned int)(unsigned char)*CHARP(cp, i); + else if ( size == 2 ) val = (unsigned int)(unsigned short)*SHORTP(cp, i); + else if ( size == 4 ) val = (unsigned int)(Py_UInt32)*LONGP(cp, i); - if ( size == 1 ) *CHARP(ncp, i) = (signed char)(val+bias); - else if ( size == 2 ) *SHORTP(ncp, i) = (short)(val+bias); - else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)(val+bias); + val += (unsigned int)bias; + /* wrap around in case of overflow */ + val &= mask; + + if ( size == 1 ) *CHARP(ncp, i) = (signed char)(unsigned char)val; + else if ( size == 2 ) *SHORTP(ncp, i) = (short)(unsigned short)val; + else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)(Py_UInt32)val; } return rv; } @@ -1009,15 +1016,15 @@ ncp = (unsigned char *)PyString_AsString(rv); for ( i=0; i < len; i += size ) { - if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 8; - else if ( size == 2 ) val = (int)*SHORTP(cp, i); - else if ( size == 4 ) val = ((int)*LONGP(cp, i)) >> 16; + if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 24; + else if ( size == 2 ) val = ((int)*SHORTP(cp, i)) << 16; + else if ( size == 4 ) val = (int)*LONGP(cp, i); j = len - i - size; - if ( size == 1 ) *CHARP(ncp, j) = (signed char)(val >> 8); - else if ( size == 2 ) *SHORTP(ncp, j) = (short)(val); - else if ( size == 4 ) *LONGP(ncp, j) = (Py_Int32)(val<<16); + if ( size == 1 ) *CHARP(ncp, j) = (signed char)(val >> 24); + else if ( size == 2 ) *SHORTP(ncp, j) = (short)(val >> 16); + else if ( size == 4 ) *LONGP(ncp, j) = (Py_Int32)val; } return rv; } @@ -1051,13 +1058,13 @@ ncp = (unsigned char *)PyString_AsString(rv); for ( i=0, j=0; i < len; i += size, j += size2 ) { - if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 8; - else if ( size == 2 ) val = (int)*SHORTP(cp, i); - else if ( size == 4 ) val = ((int)*LONGP(cp, i)) >> 16; + if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 24; + else if ( size == 2 ) val = ((int)*SHORTP(cp, i)) << 16; + else if ( size == 4 ) val = (int)*LONGP(cp, i); - if ( size2 == 1 ) *CHARP(ncp, j) = (signed char)(val >> 8); - else if ( size2 == 2 ) *SHORTP(ncp, j) = (short)(val); - else if ( size2 == 4 ) *LONGP(ncp, j) = (Py_Int32)(val<<16); + if ( size2 == 1 ) *CHARP(ncp, j) = (signed char)(val >> 24); + else if ( size2 == 2 ) *SHORTP(ncp, j) = (short)(val >> 16); + else if ( size2 == 4 ) *LONGP(ncp, j) = (Py_Int32)val; } return rv; } @@ -1120,6 +1127,10 @@ d = gcd(inrate, outrate); inrate /= d; outrate /= d; + /* divide weightA and weightB by their greatest common divisor */ + d = gcd(weightA, weightB); + weightA /= d; + weightA /= d; if ((size_t)nchannels > PY_SIZE_MAX/sizeof(int)) { PyErr_SetString(PyExc_MemoryError, @@ -1159,7 +1170,9 @@ } /* str <- Space for the output buffer. */ - { + if (len == 0) + str = PyString_FromStringAndSize(NULL, 0); + else { /* There are len input frames, so we need (mathematically) ceiling(len*outrate/inrate) output frames, and each frame requires bytes_per_frame bytes. Computing this @@ -1174,12 +1187,11 @@ else str = PyString_FromStringAndSize(NULL, q * outrate * bytes_per_frame); - - if (str == NULL) { - PyErr_SetString(PyExc_MemoryError, - "not enough memory for output buffer"); - goto exit; - } + } + if (str == NULL) { + PyErr_SetString(PyExc_MemoryError, + "not enough memory for output buffer"); + goto exit; } ncp = PyString_AsString(str); @@ -1214,32 +1226,32 @@ for (chan = 0; chan < nchannels; chan++) { prev_i[chan] = cur_i[chan]; if (size == 1) - cur_i[chan] = ((int)*CHARP(cp, 0)) << 8; + cur_i[chan] = ((int)*CHARP(cp, 0)) << 24; else if (size == 2) - cur_i[chan] = (int)*SHORTP(cp, 0); + cur_i[chan] = ((int)*SHORTP(cp, 0)) << 16; else if (size == 4) - cur_i[chan] = ((int)*LONGP(cp, 0)) >> 16; + cur_i[chan] = (int)*LONGP(cp, 0); cp += size; /* implements a simple digital filter */ - cur_i[chan] = - (weightA * cur_i[chan] + - weightB * prev_i[chan]) / - (weightA + weightB); + cur_i[chan] = (int)( + ((double)weightA * (double)cur_i[chan] + + (double)weightB * (double)prev_i[chan]) / + ((double)weightA + (double)weightB)); } len--; d += outrate; } while (d >= 0) { for (chan = 0; chan < nchannels; chan++) { - cur_o = (prev_i[chan] * d + - cur_i[chan] * (outrate - d)) / - outrate; + cur_o = (int)(((double)prev_i[chan] * (double)d + + (double)cur_i[chan] * (double)(outrate - d)) / + (double)outrate); if (size == 1) - *CHARP(ncp, 0) = (signed char)(cur_o >> 8); + *CHARP(ncp, 0) = (signed char)(cur_o >> 24); else if (size == 2) - *SHORTP(ncp, 0) = (short)(cur_o); + *SHORTP(ncp, 0) = (short)(cur_o >> 16); else if (size == 4) - *LONGP(ncp, 0) = (Py_Int32)(cur_o<<16); + *LONGP(ncp, 0) = (Py_Int32)(cur_o); ncp += size; } d -= inrate; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Feb 9 10:17:47 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 9 Feb 2013 10:17:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2Njg2?= =?utf-8?q?=3A_Fixed_a_lot_of_bugs_in_audioop_module=2E?= Message-ID: <3Z372g38PDzScm@mail.python.org> http://hg.python.org/cpython/rev/104b17f8316b changeset: 82077:104b17f8316b branch: 3.2 parent: 82073:b0d9b273c029 user: Serhiy Storchaka date: Sat Feb 09 11:10:53 2013 +0200 summary: Issue #16686: Fixed a lot of bugs in audioop module. * avgpp() and maxpp() no more crash on empty and 1-samples input fragment. They now work when peak-peak values are greater INT_MAX. * ratecv() no more crashes on empty input fragment. * Fixed an integer overflow in ratecv(). * Fixed an integer overflow in add() and bias() for 32-bit samples. * reverse(), lin2lin() and ratecv() no more lose precision for 32-bit samples. * max() and rms() no more returns negative result for 32-bit sample -0x80000000. * minmax() now returns correct max value for 32-bit sample -0x80000000. * avg(), mul(), tomono() and tostereo() now round negative result down and can return 32-bit sample -0x80000000. * add() now can return 32-bit sample -0x80000000. files: Doc/library/audioop.rst | 6 +- Lib/test/test_audioop.py | 399 ++++++++++++++++++-------- Misc/NEWS | 6 + Modules/audioop.c | 310 ++++++++++---------- 4 files changed, 435 insertions(+), 286 deletions(-) diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -36,7 +36,7 @@ Return a fragment which is the addition of the two samples passed as parameters. *width* is the sample width in bytes, either ``1``, ``2`` or ``4``. Both - fragments should have the same length. + fragments should have the same length. Samples are truncated in case of overflow. .. function:: adpcm2lin(adpcmfragment, width, state) @@ -67,7 +67,7 @@ .. function:: bias(fragment, width, bias) Return a fragment that is the original fragment with a bias added to each - sample. + sample. Samples wrap around in case of overflow. .. function:: cross(fragment, width) @@ -175,7 +175,7 @@ .. function:: mul(fragment, width, factor) Return a fragment that has all samples in the original fragment multiplied by - the floating-point value *factor*. Overflow is silently ignored. + the floating-point value *factor*. Samples are truncated in case of overflow. .. function:: ratecv(fragment, width, nchannels, inrate, outrate, state[, weightA[, weightB]]) diff --git a/Lib/test/test_audioop.py b/Lib/test/test_audioop.py --- a/Lib/test/test_audioop.py +++ b/Lib/test/test_audioop.py @@ -1,25 +1,21 @@ import audioop +import sys import unittest from test.support import run_unittest -endian = 'big' if audioop.getsample(b'\0\1', 2, 0) == 1 else 'little' +def pack(width, data): + return b''.join(v.to_bytes(width, sys.byteorder, signed=True) for v in data) -def gendata1(): - return b'\0\1\2' +packs = {w: (lambda *data, width=w: pack(width, data)) for w in (1, 2, 4)} +maxvalues = {w: (1 << (8 * w - 1)) - 1 for w in (1, 2, 4)} +minvalues = {w: -1 << (8 * w - 1) for w in (1, 2, 4)} -def gendata2(): - if endian == 'big': - return b'\0\0\0\1\0\2' - else: - return b'\0\0\1\0\2\0' - -def gendata4(): - if endian == 'big': - return b'\0\0\0\0\0\0\0\1\0\0\0\2' - else: - return b'\0\0\0\0\1\0\0\0\2\0\0\0' - -data = [gendata1(), gendata2(), gendata4()] +datas = { + 1: b'\x00\x12\x45\xbb\x7f\x80\xff', + 2: packs[2](0, 0x1234, 0x4567, -0x4567, 0x7fff, -0x8000, -1), + 4: packs[4](0, 0x12345678, 0x456789ab, -0x456789ab, + 0x7fffffff, -0x80000000, -1), +} INVALID_DATA = [ (b'abc', 0), @@ -31,171 +27,320 @@ class TestAudioop(unittest.TestCase): def test_max(self): - self.assertEqual(audioop.max(data[0], 1), 2) - self.assertEqual(audioop.max(data[1], 2), 2) - self.assertEqual(audioop.max(data[2], 4), 2) + for w in 1, 2, 4: + self.assertEqual(audioop.max(b'', w), 0) + p = packs[w] + self.assertEqual(audioop.max(p(5), w), 5) + self.assertEqual(audioop.max(p(5, -8, -1), w), 8) + self.assertEqual(audioop.max(p(maxvalues[w]), w), maxvalues[w]) + self.assertEqual(audioop.max(p(minvalues[w]), w), -minvalues[w]) + self.assertEqual(audioop.max(datas[w], w), -minvalues[w]) def test_minmax(self): - self.assertEqual(audioop.minmax(data[0], 1), (0, 2)) - self.assertEqual(audioop.minmax(data[1], 2), (0, 2)) - self.assertEqual(audioop.minmax(data[2], 4), (0, 2)) + for w in 1, 2, 4: + self.assertEqual(audioop.minmax(b'', w), + (0x7fffffff, -0x80000000)) + p = packs[w] + self.assertEqual(audioop.minmax(p(5), w), (5, 5)) + self.assertEqual(audioop.minmax(p(5, -8, -1), w), (-8, 5)) + self.assertEqual(audioop.minmax(p(maxvalues[w]), w), + (maxvalues[w], maxvalues[w])) + self.assertEqual(audioop.minmax(p(minvalues[w]), w), + (minvalues[w], minvalues[w])) + self.assertEqual(audioop.minmax(datas[w], w), + (minvalues[w], maxvalues[w])) def test_maxpp(self): - self.assertEqual(audioop.maxpp(data[0], 1), 0) - self.assertEqual(audioop.maxpp(data[1], 2), 0) - self.assertEqual(audioop.maxpp(data[2], 4), 0) + for w in 1, 2, 4: + self.assertEqual(audioop.maxpp(b'', w), 0) + self.assertEqual(audioop.maxpp(packs[w](*range(100)), w), 0) + self.assertEqual(audioop.maxpp(packs[w](9, 10, 5, 5, 0, 1), w), 10) + self.assertEqual(audioop.maxpp(datas[w], w), + maxvalues[w] - minvalues[w]) def test_avg(self): - self.assertEqual(audioop.avg(data[0], 1), 1) - self.assertEqual(audioop.avg(data[1], 2), 1) - self.assertEqual(audioop.avg(data[2], 4), 1) + for w in 1, 2, 4: + self.assertEqual(audioop.avg(b'', w), 0) + p = packs[w] + self.assertEqual(audioop.avg(p(5), w), 5) + self .assertEqual(audioop.avg(p(5, 8), w), 6) + self.assertEqual(audioop.avg(p(5, -8), w), -2) + self.assertEqual(audioop.avg(p(maxvalues[w], maxvalues[w]), w), + maxvalues[w]) + self.assertEqual(audioop.avg(p(minvalues[w], minvalues[w]), w), + minvalues[w]) + self.assertEqual(audioop.avg(packs[4](0x50000000, 0x70000000), 4), + 0x60000000) + self.assertEqual(audioop.avg(packs[4](-0x50000000, -0x70000000), 4), + -0x60000000) def test_avgpp(self): - self.assertEqual(audioop.avgpp(data[0], 1), 0) - self.assertEqual(audioop.avgpp(data[1], 2), 0) - self.assertEqual(audioop.avgpp(data[2], 4), 0) + for w in 1, 2, 4: + self.assertEqual(audioop.avgpp(b'', w), 0) + self.assertEqual(audioop.avgpp(packs[w](*range(100)), w), 0) + self.assertEqual(audioop.avgpp(packs[w](9, 10, 5, 5, 0, 1), w), 10) + self.assertEqual(audioop.avgpp(datas[1], 1), 196) + self.assertEqual(audioop.avgpp(datas[2], 2), 50534) + self.assertEqual(audioop.avgpp(datas[4], 4), 3311897002) def test_rms(self): - self.assertEqual(audioop.rms(data[0], 1), 1) - self.assertEqual(audioop.rms(data[1], 2), 1) - self.assertEqual(audioop.rms(data[2], 4), 1) + for w in 1, 2, 4: + self.assertEqual(audioop.rms(b'', w), 0) + p = packs[w] + self.assertEqual(audioop.rms(p(*range(100)), w), 57) + self.assertAlmostEqual(audioop.rms(p(maxvalues[w]) * 5, w), + maxvalues[w], delta=1) + self.assertAlmostEqual(audioop.rms(p(minvalues[w]) * 5, w), + -minvalues[w], delta=1) + self.assertEqual(audioop.rms(datas[1], 1), 77) + self.assertEqual(audioop.rms(datas[2], 2), 20001) + self.assertEqual(audioop.rms(datas[4], 4), 1310854152) def test_cross(self): - self.assertEqual(audioop.cross(data[0], 1), 0) - self.assertEqual(audioop.cross(data[1], 2), 0) - self.assertEqual(audioop.cross(data[2], 4), 0) + for w in 1, 2, 4: + self.assertEqual(audioop.cross(b'', w), -1) + p = packs[w] + self.assertEqual(audioop.cross(p(0, 1, 2), w), 0) + self.assertEqual(audioop.cross(p(1, 2, -3, -4), w), 1) + self.assertEqual(audioop.cross(p(-1, -2, 3, 4), w), 1) + self.assertEqual(audioop.cross(p(0, minvalues[w]), w), 1) + self.assertEqual(audioop.cross(p(minvalues[w], maxvalues[w]), w), 1) def test_add(self): - data2 = [] - for d in data: - str = bytearray(len(d)) - for i,b in enumerate(d): - str[i] = 2*b - data2.append(str) - self.assertEqual(audioop.add(data[0], data[0], 1), data2[0]) - self.assertEqual(audioop.add(data[1], data[1], 2), data2[1]) - self.assertEqual(audioop.add(data[2], data[2], 4), data2[2]) + for w in 1, 2, 4: + self.assertEqual(audioop.add(b'', b'', w), b'') + self.assertEqual(audioop.add(datas[w], b'\0' * len(datas[w]), w), + datas[w]) + self.assertEqual(audioop.add(datas[1], datas[1], 1), + b'\x00\x24\x7f\x80\x7f\x80\xfe') + self.assertEqual(audioop.add(datas[2], datas[2], 2), + packs[2](0, 0x2468, 0x7fff, -0x8000, 0x7fff, -0x8000, -2)) + self.assertEqual(audioop.add(datas[4], datas[4], 4), + packs[4](0, 0x2468acf0, 0x7fffffff, -0x80000000, + 0x7fffffff, -0x80000000, -2)) def test_bias(self): - # Note: this test assumes that avg() works - d1 = audioop.bias(data[0], 1, 100) - d2 = audioop.bias(data[1], 2, 100) - d4 = audioop.bias(data[2], 4, 100) - self.assertEqual(audioop.avg(d1, 1), 101) - self.assertEqual(audioop.avg(d2, 2), 101) - self.assertEqual(audioop.avg(d4, 4), 101) + for w in 1, 2, 4: + for bias in 0, 1, -1, 127, -128, 0x7fffffff, -0x80000000: + self.assertEqual(audioop.bias(b'', w, bias), b'') + self.assertEqual(audioop.bias(datas[1], 1, 1), + b'\x01\x13\x46\xbc\x80\x81\x00') + self.assertEqual(audioop.bias(datas[1], 1, -1), + b'\xff\x11\x44\xba\x7e\x7f\xfe') + self.assertEqual(audioop.bias(datas[1], 1, 0x7fffffff), + b'\xff\x11\x44\xba\x7e\x7f\xfe') + self.assertEqual(audioop.bias(datas[1], 1, -0x80000000), + datas[1]) + self.assertEqual(audioop.bias(datas[2], 2, 1), + packs[2](1, 0x1235, 0x4568, -0x4566, -0x8000, -0x7fff, 0)) + self.assertEqual(audioop.bias(datas[2], 2, -1), + packs[2](-1, 0x1233, 0x4566, -0x4568, 0x7ffe, 0x7fff, -2)) + self.assertEqual(audioop.bias(datas[2], 2, 0x7fffffff), + packs[2](-1, 0x1233, 0x4566, -0x4568, 0x7ffe, 0x7fff, -2)) + self.assertEqual(audioop.bias(datas[2], 2, -0x80000000), + datas[2]) + self.assertEqual(audioop.bias(datas[4], 4, 1), + packs[4](1, 0x12345679, 0x456789ac, -0x456789aa, + -0x80000000, -0x7fffffff, 0)) + self.assertEqual(audioop.bias(datas[4], 4, -1), + packs[4](-1, 0x12345677, 0x456789aa, -0x456789ac, + 0x7ffffffe, 0x7fffffff, -2)) + self.assertEqual(audioop.bias(datas[4], 4, 0x7fffffff), + packs[4](0x7fffffff, -0x6dcba989, -0x3a987656, 0x3a987654, + -2, -1, 0x7ffffffe)) + self.assertEqual(audioop.bias(datas[4], 4, -0x80000000), + packs[4](-0x80000000, -0x6dcba988, -0x3a987655, 0x3a987655, + -1, 0, 0x7fffffff)) def test_lin2lin(self): - # too simple: we test only the size - for d1 in data: - for d2 in data: - got = len(d1)//3 - wtd = len(d2)//3 - self.assertEqual(len(audioop.lin2lin(d1, got, wtd)), len(d2)) + for w in 1, 2, 4: + self.assertEqual(audioop.lin2lin(datas[w], w, w), datas[w]) + + self.assertEqual(audioop.lin2lin(datas[1], 1, 2), + packs[2](0, 0x1200, 0x4500, -0x4500, 0x7f00, -0x8000, -0x100)) + self.assertEqual(audioop.lin2lin(datas[1], 1, 4), + packs[4](0, 0x12000000, 0x45000000, -0x45000000, + 0x7f000000, -0x80000000, -0x1000000)) + self.assertEqual(audioop.lin2lin(datas[2], 2, 1), + b'\x00\x12\x45\xba\x7f\x80\xff') + self.assertEqual(audioop.lin2lin(datas[2], 2, 4), + packs[4](0, 0x12340000, 0x45670000, -0x45670000, + 0x7fff0000, -0x80000000, -0x10000)) + self.assertEqual(audioop.lin2lin(datas[4], 4, 1), + b'\x00\x12\x45\xba\x7f\x80\xff') + self.assertEqual(audioop.lin2lin(datas[4], 4, 2), + packs[2](0, 0x1234, 0x4567, -0x4568, 0x7fff, -0x8000, -1)) def test_adpcm2lin(self): + self.assertEqual(audioop.adpcm2lin(b'\x07\x7f\x7f', 1, None), + (b'\x00\x00\x00\xff\x00\xff', (-179, 40))) + self.assertEqual(audioop.adpcm2lin(b'\x07\x7f\x7f', 2, None), + (packs[2](0, 0xb, 0x29, -0x16, 0x72, -0xb3), (-179, 40))) + self.assertEqual(audioop.adpcm2lin(b'\x07\x7f\x7f', 4, None), + (packs[4](0, 0xb0000, 0x290000, -0x160000, 0x720000, + -0xb30000), (-179, 40))) + # Very cursory test - self.assertEqual(audioop.adpcm2lin(b'\0\0', 1, None), (b'\0' * 4, (0,0))) - self.assertEqual(audioop.adpcm2lin(b'\0\0', 2, None), (b'\0' * 8, (0,0))) - self.assertEqual(audioop.adpcm2lin(b'\0\0', 4, None), (b'\0' * 16, (0,0))) + for w in 1, 2, 4: + self.assertEqual(audioop.adpcm2lin(b'\0' * 5, w, None), + (b'\0' * w * 10, (0, 0))) def test_lin2adpcm(self): + self.assertEqual(audioop.lin2adpcm(datas[1], 1, None), + (b'\x07\x7f\x7f', (-221, 39))) + self.assertEqual(audioop.lin2adpcm(datas[2], 2, None), + (b'\x07\x7f\x7f', (31, 39))) + self.assertEqual(audioop.lin2adpcm(datas[4], 4, None), + (b'\x07\x7f\x7f', (31, 39))) + # Very cursory test - self.assertEqual(audioop.lin2adpcm(b'\0\0\0\0', 1, None), (b'\0\0', (0,0))) + for w in 1, 2, 4: + self.assertEqual(audioop.lin2adpcm(b'\0' * w * 10, w, None), + (b'\0' * 5, (0, 0))) def test_lin2alaw(self): - self.assertEqual(audioop.lin2alaw(data[0], 1), b'\xd5\xc5\xf5') - self.assertEqual(audioop.lin2alaw(data[1], 2), b'\xd5\xd5\xd5') - self.assertEqual(audioop.lin2alaw(data[2], 4), b'\xd5\xd5\xd5') + self.assertEqual(audioop.lin2alaw(datas[1], 1), + b'\xd5\x87\xa4\x24\xaa\x2a\x5a') + self.assertEqual(audioop.lin2alaw(datas[2], 2), + b'\xd5\x87\xa4\x24\xaa\x2a\x55') + self.assertEqual(audioop.lin2alaw(datas[4], 4), + b'\xd5\x87\xa4\x24\xaa\x2a\x55') def test_alaw2lin(self): - # Cursory - d = audioop.lin2alaw(data[0], 1) - self.assertEqual(audioop.alaw2lin(d, 1), data[0]) - if endian == 'big': - self.assertEqual(audioop.alaw2lin(d, 2), - b'\x00\x08\x01\x08\x02\x10') - self.assertEqual(audioop.alaw2lin(d, 4), - b'\x00\x08\x00\x00\x01\x08\x00\x00\x02\x10\x00\x00') - else: - self.assertEqual(audioop.alaw2lin(d, 2), - b'\x08\x00\x08\x01\x10\x02') - self.assertEqual(audioop.alaw2lin(d, 4), - b'\x00\x00\x08\x00\x00\x00\x08\x01\x00\x00\x10\x02') + encoded = b'\x00\x03\x24\x2a\x51\x54\x55\x58\x6b\x71\x7f'\ + b'\x80\x83\xa4\xaa\xd1\xd4\xd5\xd8\xeb\xf1\xff' + src = [-688, -720, -2240, -4032, -9, -3, -1, -27, -244, -82, -106, + 688, 720, 2240, 4032, 9, 3, 1, 27, 244, 82, 106] + for w in 1, 2, 4: + self.assertEqual(audioop.alaw2lin(encoded, w), + packs[w](*(x << (w * 8) >> 13 for x in src))) + + encoded = bytes(range(256)) + for w in 2, 4: + decoded = audioop.alaw2lin(encoded, w) + self.assertEqual(audioop.lin2alaw(decoded, w), encoded) def test_lin2ulaw(self): - self.assertEqual(audioop.lin2ulaw(data[0], 1), b'\xff\xe7\xdb') - self.assertEqual(audioop.lin2ulaw(data[1], 2), b'\xff\xff\xff') - self.assertEqual(audioop.lin2ulaw(data[2], 4), b'\xff\xff\xff') + self.assertEqual(audioop.lin2ulaw(datas[1], 1), + b'\xff\xad\x8e\x0e\x80\x00\x67') + self.assertEqual(audioop.lin2ulaw(datas[2], 2), + b'\xff\xad\x8e\x0e\x80\x00\x7e') + self.assertEqual(audioop.lin2ulaw(datas[4], 4), + b'\xff\xad\x8e\x0e\x80\x00\x7e') def test_ulaw2lin(self): - # Cursory - d = audioop.lin2ulaw(data[0], 1) - self.assertEqual(audioop.ulaw2lin(d, 1), data[0]) - if endian == 'big': - self.assertEqual(audioop.ulaw2lin(d, 2), - b'\x00\x00\x01\x04\x02\x0c') - self.assertEqual(audioop.ulaw2lin(d, 4), - b'\x00\x00\x00\x00\x01\x04\x00\x00\x02\x0c\x00\x00') - else: - self.assertEqual(audioop.ulaw2lin(d, 2), - b'\x00\x00\x04\x01\x0c\x02') - self.assertEqual(audioop.ulaw2lin(d, 4), - b'\x00\x00\x00\x00\x00\x00\x04\x01\x00\x00\x0c\x02') + encoded = b'\x00\x0e\x28\x3f\x57\x6a\x76\x7c\x7e\x7f'\ + b'\x80\x8e\xa8\xbf\xd7\xea\xf6\xfc\xfe\xff' + src = [-8031, -4447, -1471, -495, -163, -53, -18, -6, -2, 0, + 8031, 4447, 1471, 495, 163, 53, 18, 6, 2, 0] + for w in 1, 2, 4: + self.assertEqual(audioop.ulaw2lin(encoded, w), + packs[w](*(x << (w * 8) >> 14 for x in src))) + + # Current u-law implementation has two codes fo 0: 0x7f and 0xff. + encoded = bytes(range(127)) + bytes(range(128, 256)) + for w in 2, 4: + decoded = audioop.ulaw2lin(encoded, w) + self.assertEqual(audioop.lin2ulaw(decoded, w), encoded) def test_mul(self): - data2 = [] - for d in data: - str = bytearray(len(d)) - for i,b in enumerate(d): - str[i] = 2*b - data2.append(str) - self.assertEqual(audioop.mul(data[0], 1, 2), data2[0]) - self.assertEqual(audioop.mul(data[1],2, 2), data2[1]) - self.assertEqual(audioop.mul(data[2], 4, 2), data2[2]) + for w in 1, 2, 4: + self.assertEqual(audioop.mul(b'', w, 2), b'') + self.assertEqual(audioop.mul(datas[w], w, 0), + b'\0' * len(datas[w])) + self.assertEqual(audioop.mul(datas[w], w, 1), + datas[w]) + self.assertEqual(audioop.mul(datas[1], 1, 2), + b'\x00\x24\x7f\x80\x7f\x80\xfe') + self.assertEqual(audioop.mul(datas[2], 2, 2), + packs[2](0, 0x2468, 0x7fff, -0x8000, 0x7fff, -0x8000, -2)) + self.assertEqual(audioop.mul(datas[4], 4, 2), + packs[4](0, 0x2468acf0, 0x7fffffff, -0x80000000, + 0x7fffffff, -0x80000000, -2)) def test_ratecv(self): + for w in 1, 2, 4: + self.assertEqual(audioop.ratecv(b'', w, 1, 8000, 8000, None), + (b'', (-1, ((0, 0),)))) + self.assertEqual(audioop.ratecv(b'', w, 5, 8000, 8000, None), + (b'', (-1, ((0, 0),) * 5))) + self.assertEqual(audioop.ratecv(b'', w, 1, 8000, 16000, None), + (b'', (-2, ((0, 0),)))) + self.assertEqual(audioop.ratecv(datas[w], w, 1, 8000, 8000, None)[0], + datas[w]) state = None - d1, state = audioop.ratecv(data[0], 1, 1, 8000, 16000, state) - d2, state = audioop.ratecv(data[0], 1, 1, 8000, 16000, state) + d1, state = audioop.ratecv(b'\x00\x01\x02', 1, 1, 8000, 16000, state) + d2, state = audioop.ratecv(b'\x00\x01\x02', 1, 1, 8000, 16000, state) self.assertEqual(d1 + d2, b'\000\000\001\001\002\001\000\000\001\001\002') + for w in 1, 2, 4: + d0, state0 = audioop.ratecv(datas[w], w, 1, 8000, 16000, None) + d, state = b'', None + for i in range(0, len(datas[w]), w): + d1, state = audioop.ratecv(datas[w][i:i + w], w, 1, + 8000, 16000, state) + d += d1 + self.assertEqual(d, d0) + self.assertEqual(state, state0) + def test_reverse(self): - self.assertEqual(audioop.reverse(data[0], 1), b'\2\1\0') + for w in 1, 2, 4: + self.assertEqual(audioop.reverse(b'', w), b'') + self.assertEqual(audioop.reverse(packs[w](0, 1, 2), w), + packs[w](2, 1, 0)) def test_tomono(self): - data2 = bytearray() - for d in data[0]: - data2.append(d) - data2.append(d) - self.assertEqual(audioop.tomono(data2, 1, 0.5, 0.5), data[0]) + for w in 1, 2, 4: + data1 = datas[w] + data2 = bytearray(2 * len(data1)) + for k in range(w): + data2[k::2*w] = data1[k::w] + self.assertEqual(audioop.tomono(data2, w, 1, 0), data1) + self.assertEqual(audioop.tomono(data2, w, 0, 1), b'\0' * len(data1)) + for k in range(w): + data2[k+w::2*w] = data1[k::w] + self.assertEqual(audioop.tomono(data2, w, 0.5, 0.5), data1) def test_tostereo(self): - data2 = bytearray() - for d in data[0]: - data2.append(d) - data2.append(d) - self.assertEqual(audioop.tostereo(data[0], 1, 1, 1), data2) + for w in 1, 2, 4: + data1 = datas[w] + data2 = bytearray(2 * len(data1)) + for k in range(w): + data2[k::2*w] = data1[k::w] + self.assertEqual(audioop.tostereo(data1, w, 1, 0), data2) + self.assertEqual(audioop.tostereo(data1, w, 0, 0), b'\0' * len(data2)) + for k in range(w): + data2[k+w::2*w] = data1[k::w] + self.assertEqual(audioop.tostereo(data1, w, 1, 1), data2) def test_findfactor(self): - self.assertEqual(audioop.findfactor(data[1], data[1]), 1.0) + self.assertEqual(audioop.findfactor(datas[2], datas[2]), 1.0) + self.assertEqual(audioop.findfactor(b'\0' * len(datas[2]), datas[2]), + 0.0) def test_findfit(self): - self.assertEqual(audioop.findfit(data[1], data[1]), (0, 1.0)) + self.assertEqual(audioop.findfit(datas[2], datas[2]), (0, 1.0)) + self.assertEqual(audioop.findfit(datas[2], packs[2](1, 2, 0)), + (1, 8038.8)) + self.assertEqual(audioop.findfit(datas[2][:-2] * 5 + datas[2], datas[2]), + (30, 1.0)) def test_findmax(self): - self.assertEqual(audioop.findmax(data[1], 1), 2) + self.assertEqual(audioop.findmax(datas[2], 1), 5) def test_getsample(self): - for i in range(3): - self.assertEqual(audioop.getsample(data[0], 1, i), i) - self.assertEqual(audioop.getsample(data[1], 2, i), i) - self.assertEqual(audioop.getsample(data[2], 4, i), i) + for w in 1, 2, 4: + data = packs[w](0, 1, -1, maxvalues[w], minvalues[w]) + self.assertEqual(audioop.getsample(data, w, 0), 0) + self.assertEqual(audioop.getsample(data, w, 1), 1) + self.assertEqual(audioop.getsample(data, w, 2), -1) + self.assertEqual(audioop.getsample(data, w, 3), maxvalues[w]) + self.assertEqual(audioop.getsample(data, w, 4), minvalues[w]) def test_negativelen(self): # from issue 3306, previously it segfaulted self.assertRaises(audioop.error, - audioop.findmax, ''.join(chr(x) for x in range(256)), -2392392) + audioop.findmax, bytes(range(256)), -2392392) def test_issue7673(self): state = None @@ -222,9 +367,9 @@ self.assertRaises(audioop.error, audioop.lin2adpcm, data, size, state) def test_wrongsize(self): - data = b'abc' + data = b'abcdefgh' state = None - for size in (-1, 3, 5): + for size in (-1, 0, 3, 5, 1024): self.assertRaises(audioop.error, audioop.ulaw2lin, data, size) self.assertRaises(audioop.error, audioop.alaw2lin, data, size) self.assertRaises(audioop.error, audioop.adpcm2lin, data, size, state) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -215,6 +215,12 @@ Library ------- +- Issue #16686: Fixed a lot of bugs in audioop module. Fixed crashes in + avgpp(), maxpp() and ratecv(). Fixed an integer overflow in add(), bias(), + and ratecv(). reverse(), lin2lin() and ratecv() no more lose precision for + 32-bit samples. max() and rms() no more returns a negative result and + various other functions now work correctly with 32-bit sample -0x80000000. + - Issue #17073: Fix some integer overflows in sqlite3 module. - Issue #17114: IDLE now uses non-strict config parser. diff --git a/Modules/audioop.c b/Modules/audioop.c --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -26,6 +26,21 @@ #endif #endif +static const int maxvals[] = {0, 0x7F, 0x7FFF, 0x7FFFFF, 0x7FFFFFFF}; +static const int minvals[] = {0, -0x80, -0x8000, -0x800000, -0x80000000}; +static const unsigned int masks[] = {0, 0xFF, 0xFFFF, 0xFFFFFF, 0xFFFFFFFF}; + +static int +fbound(double val, double minval, double maxval) +{ + if (val > maxval) + val = maxval; + else if (val < minval + 1) + val = minval; + return val; +} + + /* Code shamelessly stolen from sox, 12.17.7, g711.c ** (c) Craig Reese, Joe Campbell and Jeff Poskanzer 1989 */ @@ -347,7 +362,7 @@ signed char *cp; Py_ssize_t len, i; int size, val = 0; - int max = 0; + unsigned int absval, max = 0; if ( !PyArg_ParseTuple(args, "s#i:max", &cp, &len, &size) ) return 0; @@ -357,10 +372,11 @@ if ( size == 1 ) val = (int)*CHARP(cp, i); else if ( size == 2 ) val = (int)*SHORTP(cp, i); else if ( size == 4 ) val = (int)*LONGP(cp, i); - if ( val < 0 ) val = (-val); - if ( val > max ) max = val; + if (val < 0) absval = (-val); + else absval = val; + if (absval > max) max = absval; } - return PyLong_FromLong(max); + return PyLong_FromUnsignedLong(max); } static PyObject * @@ -369,7 +385,7 @@ signed char *cp; Py_ssize_t len, i; int size, val = 0; - int min = 0x7fffffff, max = -0x7fffffff; + int min = 0x7fffffff, max = -0x80000000; if (!PyArg_ParseTuple(args, "s#i:minmax", &cp, &len, &size)) return NULL; @@ -406,7 +422,7 @@ if ( len == 0 ) val = 0; else - val = (int)(avg / (double)(len/size)); + val = (int)floor(avg / (double)(len/size)); return PyLong_FromLong(val); } @@ -416,6 +432,7 @@ signed char *cp; Py_ssize_t len, i; int size, val = 0; + unsigned int res; double sum_squares = 0.0; if ( !PyArg_ParseTuple(args, "s#i:rms", &cp, &len, &size) ) @@ -429,10 +446,10 @@ sum_squares += (double)val*(double)val; } if ( len == 0 ) - val = 0; + res = 0; else - val = (int)sqrt(sum_squares / (double)(len/size)); - return PyLong_FromLong(val); + res = (unsigned int)sqrt(sum_squares / (double)(len/size)); + return PyLong_FromUnsignedLong(res); } static double _sum2(short *a, short *b, Py_ssize_t len) @@ -624,52 +641,46 @@ Py_ssize_t len, i; int size, val = 0, prevval = 0, prevextremevalid = 0, prevextreme = 0; - double avg = 0.0; - int diff, prevdiff, extremediff, nextreme = 0; + double sum = 0.0; + unsigned int avg; + int diff, prevdiff, nextreme = 0; if ( !PyArg_ParseTuple(args, "s#i:avgpp", &cp, &len, &size) ) return 0; if (!audioop_check_parameters(len, size)) return NULL; - /* Compute first delta value ahead. Also automatically makes us - ** skip the first extreme value - */ + if (len <= size) + return PyLong_FromLong(0); if ( size == 1 ) prevval = (int)*CHARP(cp, 0); else if ( size == 2 ) prevval = (int)*SHORTP(cp, 0); else if ( size == 4 ) prevval = (int)*LONGP(cp, 0); - if ( size == 1 ) val = (int)*CHARP(cp, size); - else if ( size == 2 ) val = (int)*SHORTP(cp, size); - else if ( size == 4 ) val = (int)*LONGP(cp, size); - prevdiff = val - prevval; - + prevdiff = 17; /* Anything != 0, 1 */ for ( i=size; i max ) - max = extremediff; + if (val != prevval) { + diff = val < prevval; + if (prevdiff == !diff) { + /* Derivative changed sign. Compute difference to + ** last extreme value and remember. + */ + if (prevextremevalid) { + if (prevval < prevextreme) + extremediff = (unsigned int)prevextreme - + (unsigned int)prevval; + else + extremediff = (unsigned int)prevval - + (unsigned int)prevextreme; + if ( extremediff > max ) + max = extremediff; + } + prevextremevalid = 1; + prevextreme = prevval; } - prevextremevalid = 1; - prevextreme = prevval; + prevval = val; + prevdiff = diff; } - prevval = val; - if ( diff != 0 ) - prevdiff = diff; } - return PyLong_FromLong(max); + return PyLong_FromUnsignedLong(max); } static PyObject * @@ -755,7 +765,7 @@ signed char *cp, *ncp; Py_ssize_t len, i; int size, val = 0; - double factor, fval, maxval; + double factor, fval, maxval, minval; PyObject *rv; if ( !PyArg_ParseTuple(args, "s#id:mul", &cp, &len, &size, &factor ) ) @@ -763,13 +773,8 @@ if (!audioop_check_parameters(len, size)) return NULL; - if ( size == 1 ) maxval = (double) 0x7f; - else if ( size == 2 ) maxval = (double) 0x7fff; - else if ( size == 4 ) maxval = (double) 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = (double) maxvals[size]; + minval = (double) minvals[size]; rv = PyBytes_FromStringAndSize(NULL, len); if ( rv == 0 ) @@ -782,9 +787,7 @@ else if ( size == 2 ) val = (int)*SHORTP(cp, i); else if ( size == 4 ) val = (int)*LONGP(cp, i); fval = (double)val*factor; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val = (int)fval; + val = (int)floor(fbound(fval, minval, maxval)); if ( size == 1 ) *CHARP(ncp, i) = (signed char)val; else if ( size == 2 ) *SHORTP(ncp, i) = (short)val; else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)val; @@ -799,7 +802,7 @@ signed char *cp, *ncp; Py_ssize_t len, i; int size, val1 = 0, val2 = 0; - double fac1, fac2, fval, maxval; + double fac1, fac2, fval, maxval, minval; PyObject *rv; if ( !PyArg_ParseTuple(args, "s*idd:tomono", @@ -817,14 +820,8 @@ return NULL; } - if ( size == 1 ) maxval = (double) 0x7f; - else if ( size == 2 ) maxval = (double) 0x7fff; - else if ( size == 4 ) maxval = (double) 0x7fffffff; - else { - PyBuffer_Release(&pcp); - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = (double) maxvals[size]; + minval = (double) minvals[size]; rv = PyBytes_FromStringAndSize(NULL, len/2); if ( rv == 0 ) { @@ -842,9 +839,7 @@ else if ( size == 2 ) val2 = (int)*SHORTP(cp, i+2); else if ( size == 4 ) val2 = (int)*LONGP(cp, i+4); fval = (double)val1*fac1 + (double)val2*fac2; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val1 = (int)fval; + val1 = (int)floor(fbound(fval, minval, maxval)); if ( size == 1 ) *CHARP(ncp, i/2) = (signed char)val1; else if ( size == 2 ) *SHORTP(ncp, i/2) = (short)val1; else if ( size == 4 ) *LONGP(ncp, i/2)= (Py_Int32)val1; @@ -859,7 +854,7 @@ signed char *cp, *ncp; Py_ssize_t len, i; int size, val1, val2, val = 0; - double fac1, fac2, fval, maxval; + double fac1, fac2, fval, maxval, minval; PyObject *rv; if ( !PyArg_ParseTuple(args, "s#idd:tostereo", @@ -868,13 +863,8 @@ if (!audioop_check_parameters(len, size)) return NULL; - if ( size == 1 ) maxval = (double) 0x7f; - else if ( size == 2 ) maxval = (double) 0x7fff; - else if ( size == 4 ) maxval = (double) 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = (double) maxvals[size]; + minval = (double) minvals[size]; if (len > PY_SSIZE_T_MAX/2) { PyErr_SetString(PyExc_MemoryError, @@ -894,14 +884,10 @@ else if ( size == 4 ) val = (int)*LONGP(cp, i); fval = (double)val*fac1; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val1 = (int)fval; + val1 = (int)floor(fbound(fval, minval, maxval)); fval = (double)val*fac2; - if ( fval > maxval ) fval = maxval; - else if ( fval < -maxval ) fval = -maxval; - val2 = (int)fval; + val2 = (int)floor(fbound(fval, minval, maxval)); if ( size == 1 ) *CHARP(ncp, i*2) = (signed char)val1; else if ( size == 2 ) *SHORTP(ncp, i*2) = (short)val1; @@ -919,7 +905,7 @@ { signed char *cp1, *cp2, *ncp; Py_ssize_t len1, len2, i; - int size, val1 = 0, val2 = 0, maxval, newval; + int size, val1 = 0, val2 = 0, minval, maxval, newval; PyObject *rv; if ( !PyArg_ParseTuple(args, "s#s#i:add", @@ -932,13 +918,8 @@ return 0; } - if ( size == 1 ) maxval = 0x7f; - else if ( size == 2 ) maxval = 0x7fff; - else if ( size == 4 ) maxval = 0x7fffffff; - else { - PyErr_SetString(AudioopError, "Size should be 1, 2 or 4"); - return 0; - } + maxval = maxvals[size]; + minval = minvals[size]; rv = PyBytes_FromStringAndSize(NULL, len1); if ( rv == 0 ) @@ -954,12 +935,19 @@ else if ( size == 2 ) val2 = (int)*SHORTP(cp2, i); else if ( size == 4 ) val2 = (int)*LONGP(cp2, i); - newval = val1 + val2; - /* truncate in case of overflow */ - if (newval > maxval) newval = maxval; - else if (newval < -maxval) newval = -maxval; - else if (size == 4 && (newval^val1) < 0 && (newval^val2) < 0) - newval = val1 > 0 ? maxval : - maxval; + if (size < 4) { + newval = val1 + val2; + /* truncate in case of overflow */ + if (newval > maxval) + newval = maxval; + else if (newval < minval) + newval = minval; + } + else { + double fval = (double)val1 + (double)val2; + /* truncate in case of overflow */ + newval = (int)floor(fbound(fval, minval, maxval)); + } if ( size == 1 ) *CHARP(ncp, i) = (signed char)newval; else if ( size == 2 ) *SHORTP(ncp, i) = (short)newval; @@ -973,9 +961,9 @@ { signed char *cp, *ncp; Py_ssize_t len, i; - int size, val = 0; + int size, bias; + unsigned int val = 0, mask; PyObject *rv; - int bias; if ( !PyArg_ParseTuple(args, "s#ii:bias", &cp, &len, &size , &bias) ) @@ -989,15 +977,20 @@ return 0; ncp = (signed char *)PyBytes_AsString(rv); + mask = masks[size]; for ( i=0; i < len; i += size ) { - if ( size == 1 ) val = (int)*CHARP(cp, i); - else if ( size == 2 ) val = (int)*SHORTP(cp, i); - else if ( size == 4 ) val = (int)*LONGP(cp, i); + if ( size == 1 ) val = (unsigned int)(unsigned char)*CHARP(cp, i); + else if ( size == 2 ) val = (unsigned int)(unsigned short)*SHORTP(cp, i); + else if ( size == 4 ) val = (unsigned int)(Py_UInt32)*LONGP(cp, i); - if ( size == 1 ) *CHARP(ncp, i) = (signed char)(val+bias); - else if ( size == 2 ) *SHORTP(ncp, i) = (short)(val+bias); - else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)(val+bias); + val += (unsigned int)bias; + /* wrap around in case of overflow */ + val &= mask; + + if ( size == 1 ) *CHARP(ncp, i) = (signed char)(unsigned char)val; + else if ( size == 2 ) *SHORTP(ncp, i) = (short)(unsigned short)val; + else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)(Py_UInt32)val; } return rv; } @@ -1024,15 +1017,15 @@ ncp = (unsigned char *)PyBytes_AsString(rv); for ( i=0; i < len; i += size ) { - if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 8; - else if ( size == 2 ) val = (int)*SHORTP(cp, i); - else if ( size == 4 ) val = ((int)*LONGP(cp, i)) >> 16; + if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 24; + else if ( size == 2 ) val = ((int)*SHORTP(cp, i)) << 16; + else if ( size == 4 ) val = (int)*LONGP(cp, i); j = len - i - size; - if ( size == 1 ) *CHARP(ncp, j) = (signed char)(val >> 8); - else if ( size == 2 ) *SHORTP(ncp, j) = (short)(val); - else if ( size == 4 ) *LONGP(ncp, j) = (Py_Int32)(val<<16); + if ( size == 1 ) *CHARP(ncp, j) = (signed char)(val >> 24); + else if ( size == 2 ) *SHORTP(ncp, j) = (short)(val >> 16); + else if ( size == 4 ) *LONGP(ncp, j) = (Py_Int32)val; } return rv; } @@ -1066,13 +1059,13 @@ ncp = (unsigned char *)PyBytes_AsString(rv); for ( i=0, j=0; i < len; i += size, j += size2 ) { - if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 8; - else if ( size == 2 ) val = (int)*SHORTP(cp, i); - else if ( size == 4 ) val = ((int)*LONGP(cp, i)) >> 16; + if ( size == 1 ) val = ((int)*CHARP(cp, i)) << 24; + else if ( size == 2 ) val = ((int)*SHORTP(cp, i)) << 16; + else if ( size == 4 ) val = (int)*LONGP(cp, i); - if ( size2 == 1 ) *CHARP(ncp, j) = (signed char)(val >> 8); - else if ( size2 == 2 ) *SHORTP(ncp, j) = (short)(val); - else if ( size2 == 4 ) *LONGP(ncp, j) = (Py_Int32)(val<<16); + if ( size2 == 1 ) *CHARP(ncp, j) = (signed char)(val >> 24); + else if ( size2 == 2 ) *SHORTP(ncp, j) = (short)(val >> 16); + else if ( size2 == 4 ) *LONGP(ncp, j) = (Py_Int32)val; } return rv; } @@ -1136,6 +1129,10 @@ d = gcd(inrate, outrate); inrate /= d; outrate /= d; + /* divide weightA and weightB by their greatest common divisor */ + d = gcd(weightA, weightB); + weightA /= d; + weightA /= d; if ((size_t)nchannels > PY_SIZE_MAX/sizeof(int)) { PyErr_SetString(PyExc_MemoryError, @@ -1175,7 +1172,9 @@ } /* str <- Space for the output buffer. */ - { + if (len == 0) + str = PyBytes_FromStringAndSize(NULL, 0); + else { /* There are len input frames, so we need (mathematically) ceiling(len*outrate/inrate) output frames, and each frame requires bytes_per_frame bytes. Computing this @@ -1190,12 +1189,11 @@ else str = PyBytes_FromStringAndSize(NULL, q * outrate * bytes_per_frame); - - if (str == NULL) { - PyErr_SetString(PyExc_MemoryError, - "not enough memory for output buffer"); - goto exit; - } + } + if (str == NULL) { + PyErr_SetString(PyExc_MemoryError, + "not enough memory for output buffer"); + goto exit; } ncp = PyBytes_AsString(str); @@ -1229,32 +1227,32 @@ for (chan = 0; chan < nchannels; chan++) { prev_i[chan] = cur_i[chan]; if (size == 1) - cur_i[chan] = ((int)*CHARP(cp, 0)) << 8; + cur_i[chan] = ((int)*CHARP(cp, 0)) << 24; else if (size == 2) - cur_i[chan] = (int)*SHORTP(cp, 0); + cur_i[chan] = ((int)*SHORTP(cp, 0)) << 16; else if (size == 4) - cur_i[chan] = ((int)*LONGP(cp, 0)) >> 16; + cur_i[chan] = (int)*LONGP(cp, 0); cp += size; /* implements a simple digital filter */ - cur_i[ch