Mailman 3 July 2013 - Python-checkins

peps: PEP 445: cleanup
by victor.stinner July 1, 2013

July 1, 2013

http://hg.python.org/peps/rev/4de91c027ae2 changeset: 4969:4de91c027ae2 user: Victor Stinner <victor.stinner(a)gmail.com> date: Mon Jul 01 22:29:08 2013 +0200 summary: PEP 445: cleanup Avoid "should", "may" and "might". Rephrase some sentences files: pep-0445.txt | 218 ++++++++++++++++++++------------------ 1 files changed, 114 insertions(+), 104 deletions(-) diff --git a/pep-0445.txt b/pep-0445.txt --- a/pep-0445.txt +++ b/pep-0445.txt @@ -12,7 +12,8 @@ Abstract ======== -Add new APIs to customize Python memory allocators. +Add new Application Programming Interfaces (API) to customize Python +memory allocators. Rationale @@ -20,21 +21,22 @@ Use cases: -* Application embedding Python may want to isolate Python memory from - the memory of the application, or may want to use a different memory +* Applications embedding Python which want to isolate Python memory from + the memory of the application, or want to use a different memory allocator optimized for its Python usage * Python running on embedded devices with low memory and slow CPU. - A custom memory allocator may be required to use efficiently the - memory and/or to be able to use all the memory of the device. -* Debug tool to: + A custom memory allocator can be used for efficiency and/or to get + access all the memory of the device. +* Debug tools for memory allocators: - - track the memory usage (memory leaks) - - get the Python filename and line number where an object was - allocated - - detect buffer underflow, buffer overflow and detect misuse of Python - allocator APIs (builtin Python debug hooks) - - force allocation to fail to test handling of ``MemoryError`` - exception + - track the memory usage (find memory leaks) + - get the location of a memory allocation: Python filename and line + number, and the size of a memory block + - detect buffer underflow, buffer overflow and misuse of Python + allocator APIs (see `Redesign Debug Checks on Memory Block + Allocators as Hooks`_) + - force memory allocations to fail to test handling of the + ``MemoryError`` exception Proposal @@ -56,8 +58,7 @@ * Add a new ``PyMemAllocator`` structure:: typedef struct { - /* user context passed as the first argument - to the 3 functions */ + /* user context passed as the first argument to the 3 functions */ void *ctx; /* allocate a memory block */ @@ -82,7 +83,7 @@ - ``PYMEM_DOMAIN_OBJ``: ``PyObject_Malloc()``, ``PyObject_Realloc()`` and ``PyObject_Free()`` -* Add new functions to get and set memory allocators: +* Add new functions to get and set memory block allocators: - ``void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)`` - ``void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)`` @@ -94,8 +95,7 @@ * Add a new ``PyObjectArenaAllocator`` structure:: typedef struct { - /* user context passed as the first argument - to the 2 functions */ + /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate an arena */ @@ -111,18 +111,17 @@ - ``void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)`` - ``void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)`` -* Add a new function to setup the debug checks on memory allocators when - a memory allocator is replaced: +* Add a new function to reinstall the debug checks on memory allocators when + a memory allocator is replaced with ``PyMem_SetAllocator()``: - ``void PyMem_SetupDebugHooks(void)`` - - Install the debug hook on all memory block allocators. The function - can be called more than once, hooks are not reinstalled if they - were already installed. - - The function does nothing is Python is not compiled in debug mode + - Install the debug hooks on all memory block allocators. The function can be + called more than once, hooks are only installed once. + - The function does nothing is Python is not compiled in debug mode. -* Memory allocators always returns *NULL* if size is greater than - ``PY_SSIZE_T_MAX``. The check is done before calling the - inner function. +* Memory block allocators always return *NULL* if *size* is greater than + ``PY_SSIZE_T_MAX``. The check is done before calling the inner + function. The *pymalloc* allocator is optimized for objects smaller than 512 bytes with a short lifetime. It uses memory mappings with a fixed size of 256 @@ -140,8 +139,8 @@ and ``free()`` -Redesign Debug Checks on Memory Allocators as Hooks ----------------------------------------------------- +Redesign Debug Checks on Memory Block Allocators as Hooks +--------------------------------------------------------- Since Python 2.3, Python implements different checks on memory allocators in debug mode: @@ -157,7 +156,8 @@ ``PyMem_Realloc()``, ``PyMem_Free()``, ``PyObject_Malloc()``, ``PyObject_Realloc()`` and ``PyObject_Free()`` using macros. The new allocator allocates a larger buffer and write a pattern to detect buffer -underflow and overflow. It uses the original ``PyObject_Malloc()`` +underflow, buffer overflow and use after free (fill the buffer with the +pattern ``0xDB``). It uses the original ``PyObject_Malloc()`` function to allocate memory. So ``PyMem_Malloc()`` and ``PyMem_Realloc()`` call indirectly ``PyObject_Malloc()`` and ``PyObject_Realloc()``. @@ -178,13 +178,14 @@ * ``PyObject_Free()`` => ``_PyMem_DebugFree()`` => ``_PyObject_Free()`` -As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now always call -``malloc()`` and ``realloc()``, instead of calling ``PyObject_Malloc()`` -and ``PyObject_Realloc()`` in debug mode. +As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now call +``malloc()`` and ``realloc()`` in release mode and in debug mode, +instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in +debug mode. When at least one memory allocator is replaced with ``PyMem_SetAllocator()``, the ``PyMem_SetupDebugHooks()`` function must -be called to install the debug hooks on top on the new allocator. +be called to reinstall the debug hooks on top on the new allocator. Don't call malloc() directly anymore @@ -195,7 +196,7 @@ ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of ``realloc()`` -Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or +Direct calls to ``malloc()`` are replaced with ``PyMem_Malloc()``, or ``PyMem_RawMalloc()`` if the GIL is not held. Configure external libraries like zlib or OpenSSL to allocate memory @@ -205,22 +206,22 @@ For the "track memory usage" use case, it is important to track memory allocated in external libraries to have accurate reports, because these -allocations may be large. +allocations can be large (can raise a ``MemoryError`` exception). If an hook is used to the track memory usage, the memory allocated by -``malloc()`` will not be tracked. Remaining ``malloc()`` in external -libraries like OpenSSL or bz2 may allocate large memory blocks and so -would be missed in memory usage reports. +direct calls to ``malloc()`` will not be tracked. Remaining ``malloc()`` +in external libraries like OpenSSL or bz2 can allocate large memory +blocks and so would be missed in memory usage reports. Examples ======== -Use case 1: Replace Memory Allocator, keep pymalloc +Use case 1: Replace Memory Allocators, keep pymalloc ---------------------------------------------------- Dummy example wasting 2 bytes per memory block, -and 10 bytes per memory mapping:: +and 10 bytes per *pymalloc* arena:: #include <stdlib.h> @@ -267,6 +268,7 @@ PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc); PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc); + /* leave PYMEM_DOMAIN_OBJ unchanged, use pymalloc */ arena.ctx = &arena_padding; arena.alloc = my_alloc_arena; @@ -277,7 +279,7 @@ } -Use case 2: Replace Memory Allocator, override pymalloc +Use case 2: Replace Memory Allocators, override pymalloc -------------------------------------------------------- If your allocator is optimized for allocations of objects smaller than @@ -322,11 +324,14 @@ PyMem_SetupDebugHooks(); } +The *pymalloc* arena does not need to be replaced, because it is no more +used by the new allocator. -Use case 3: Setup Allocator Hooks ---------------------------------- -Example to setup hooks on all memory allocators:: +Use case 3: Setup Hooks On Memory Block Allocators +-------------------------------------------------- + +Example to setup hooks on all memory block allocators:: struct { PyMemAllocator raw; @@ -390,22 +395,23 @@ } .. note:: - ``PyMem_SetupDebugHooks()`` does not need to be called because the - allocator is not replaced: Python debug hooks are installed - automatically at startup. + ``PyMem_SetupDebugHooks()`` does not need to be called because + memory allocator are not replaced: the debug checks on memory + block allocators are installed automatically at startup. Performances ============ +The implementation of this PEP (issue #3329) has no visible overhead on +the Python benchmark suite. + Results of the `Python benchmarks suite <http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x -faster, some tests are 1.04 slower, significant is between 115 and -191. +faster, some tests are 1.04 slower. Results of pybench microbenchmark: +"+0.1%" slower globally (diff between -4.9% and +5.6%). -Results of pybench benchmark: "+0.1%" slower globally (diff between --4.9% and +5.6%). - -The full reports are attached to the issue #3329. +The full output of benchmarks is attached to the issue #3329. Rejected Alternatives @@ -428,8 +434,9 @@ * ``void PyMem_SetAllocator(PyMemAllocator *allocator)`` * ``void PyObject_SetAllocator(PyMemAllocator *allocator)`` -With more specific functions, it becomes more difficult to write generic -code, like reusing the same code for different allocator domains. +This alternative was rejected because it is not possible to write +generic code with more specific functions: code must be duplicated for +each memory allocator domain. Make PyMem_Malloc() reuse PyMem_RawMalloc() by default @@ -439,25 +446,25 @@ calling ``PyMem_SetAllocator(PYMEM_DOMAIN_RAW, alloc)`` would also also patch ``PyMem_Malloc()`` indirectly. -This option was rejected because ``PyMem_SetAllocator()`` would have a -different behaviour depending on the domain. Always having the same -behaviour is less error-prone. +This alternative was rejected because ``PyMem_SetAllocator()`` would +have a different behaviour depending on the domain. Always having the +same behaviour is less error-prone. Add a new PYDEBUGMALLOC environment variable -------------------------------------------- -To be able to use the Python builtin debug hooks even when a custom -memory allocator replaces the default Python allocator, an environment -variable ``PYDEBUGMALLOC`` can be added to setup these debug function -hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``. -If the environment variable is present, ``PyMem_SetRawAllocator()``, -``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall -automatically the hook on top of the new allocator. +Add a new ``PYDEBUGMALLOC`` environment variable to enable debug checks +on memory block allocators. The environment variable replaces the new +function ``PyMem_SetupDebugHooks()`` which is not needed anymore. +Another advantage is to allow to enable debug checks even in release +mode: debug checks are always compiled, but only enabled when the +environment variable is present and non-empty. -A new environment variable would make the Python initialization even -more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_ -tries to simply the CPython startup sequence. +This alternative was rejected because a new environment variable would +make the Python initialization even more complex. The `PEP 432 +<http://www.python.org/dev/peps/pep-0432/>`_ tries to simply the CPython +startup sequence. Use macros to get customizable allocators @@ -503,7 +510,7 @@ void* _PyMem_MallocTrace(const char *filename, int lineno, size_t size); - /* need also a function for the Python stable ABI */ + /* the function is still needed for the Python stable ABI */ void* PyMem_Malloc(size_t size); #define PyMem_Malloc(size) \ @@ -527,19 +534,19 @@ calls indirectly ``PyObject_Malloc()`` which requires the GIL to be held. That's why ``PyMem_Malloc()`` must be called with the GIL held. -This PEP proposes changes ``PyMem_Malloc()``: it now always call -``malloc()``. The "GIL must be held" restriction can be removed from +This PEP changes ``PyMem_Malloc()``: it now always call ``malloc()``. +The "GIL must be held" restriction could be removed from ``PyMem_Malloc()``. This alternative was rejected because allowing to call -``PyMem_Malloc()`` without holding the GIL might break applications +``PyMem_Malloc()`` without holding the GIL can break applications which setup their own allocators or allocator hooks. Holding the GIL is convinient to develop a custom allocator: no need to care of other threads. It is also convinient for a debug allocator hook: Python internal objects can be safetly inspected. Calling ``PyGILState_Ensure()`` in -a memory allocator may have unexpected behaviour, especially at Python +a memory allocator has unexpected behaviour, especially at Python startup and at creation of a new Python thread state. @@ -552,13 +559,14 @@ The ``PyMem_Malloc()`` is used without the GIL held in some Python functions. For example, the ``main()`` and ``Py_Main()`` functions of Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this -case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or +case, ``PyMem_Malloc()`` would be replaced with ``malloc()`` (or ``PyMem_RawMalloc()``). -If an hook is used to the track memory usage, the memory allocated by -direct calls to ``malloc()`` will not be tracked. External libraries -like OpenSSL or bz2 should not call ``malloc()`` directly, so large -allocated will be included in memory usage reports. +This alternative was rejected because ``PyMem_RawMalloc()`` is required +for accurate reports of the memory usage. When a debug hook is used to +track the memory usage, the memory allocated by direct calls to +``malloc()`` cannot be tracked. ``PyMem_RawMalloc()`` can be hooked and +so all the memory allocated by Python can be tracked. Use existing debug tools to analyze the memory @@ -571,11 +579,11 @@ <http://www.nongnu.org/failmalloc/>`_, etc. The problem is to retrieve the Python object related to a memory pointer -to read its type and/or content. Another issue is to retrieve the +to read its type and/or its content. Another issue is to retrieve the location of the memory allocation: the C backtrace is usually useless -(same reasoning than macros using ``__FILE__`` and ``__LINE__``), the -Python filename and line number (or even the Python traceback) is more -useful. +(same reasoning than macros using ``__FILE__`` and ``__LINE__``, see +`Pass the C filename and line number`_), the Python filename and line +number (or even the Python traceback) is more useful. This alternative was rejected because classic tools are unable to introspect Python internals to collect such information. Being able to @@ -586,8 +594,8 @@ Add a msize() function ---------------------- -Add another field to ``PyMemAllocator`` and ``PyObjectArenaAllocator`` -structures:: +Add another function to ``PyMemAllocator`` and +``PyObjectArenaAllocator`` structures:: size_t msize(void *ptr); @@ -607,8 +615,8 @@ platforms implement it. For example, Linux with the GNU libc does not provide a function to get the size of a memory block. ``msize()`` is not currently used in the Python source code. The function is only used to -track the memory usage, but makes the API more complex. A debug hook can -implemente the function internally, there is no need to add it to +track the memory usage, and makes the API more complex. A debug hook can +implement the function internally, there is no need to add it to ``PyMemAllocator`` and ``PyObjectArenaAllocator`` structures. @@ -653,17 +661,19 @@ * lzma: `LZMA SDK - How to Use <http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html>`_, pass an opaque pointer -* lipmpdec doesn't have this extra *ctx* parameter +* lipmpdec: no opaque pointer (classic malloc API) Other libraries: * glib: `g_mem_set_vtable() <http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-…>`_ -* libxml2: `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_, +* libxml2: + `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_, global * Oracle's OCI: `Oracle Call Interface Programmer's Guide, Release 2 (9.2) - <http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_ + <http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_, + pass an opaque pointer See also the `GNU libc: Memory Allocation Hooks <http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_. @@ -676,30 +686,30 @@ Its implementation depends on the platform and of the C library. The GNU C library uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD uses `jemalloc -<http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which +<http://www.canonware.com/jemalloc/>`_. Google provides *tcmalloc* which is part of `gperftools <http://code.google.com/p/gperftools/>`_. ``malloc()`` uses two kinds of memory: heap and memory mappings. Memory mappings are usually used for large allocations (ex: larger than 256 KB), whereas the heap is used for small allocations. -On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on -Linux, and it is contiguous. On Windows, the heap is handled by -``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by -``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be +On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls, +and it is contiguous. On Windows, the heap is handled by +``HeapAlloc()`` and can be discontiguous. Memory mappings are handled by +``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they can be discontiguous. Releasing a memory mapping gives back immediatly the memory to the -system. On UNIX, heap memory is only given back to the system if it is -at the end of the heap. Otherwise, the memory will only be given back to -the system when all the memory located after the released memory are -also released. +system. On UNIX, the heap memory is only given back to the system if the +released block is located at the end of the heap. Otherwise, the memory +will only be given back to the system when all the memory located after +the released memory is also released. -To allocate memory in the heap, the allocator tries to reuse free space. -If there is no contiguous space big enough, the heap must be increased, -even if we have more free space than required size. This issue is +To allocate memory on the heap, an allocator tries to reuse free space. +If there is no contiguous space big enough, the heap must be enlarged, +even if there is more free space than required size. This issue is called the "memory fragmentation": the memory usage seen by the system -may be much higher than real usage. On Windows, ``HeapAlloc()`` creates +is higher than real usage. On Windows, ``HeapAlloc()`` creates a new memory mapping with ``VirtualAlloc()`` if there is not enough free contiguous memory. @@ -730,8 +740,8 @@ <http://bugs.python.org/issue3329>`_ * `Issue #13483: Use VirtualAlloc to allocate memory arenas <http://bugs.python.org/issue13483>`_ -* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which - isn't thread safe <http://bugs.python.org/issue16742>`_ +* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, + which isn't thread safe <http://bugs.python.org/issue16742>`_ * `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_ * `Issue #18227: Use Python memory allocators in external libraries like -- Repository URL: http://hg.python.org/peps

1 0

cpython (merge 3.3 -> default): Issue #17097: Merge.
by richard.oudkerk July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/035577781505 changeset: 84415:035577781505 parent: 84412:879ee6762f3d parent: 84414:3ec5267f51ff user: Richard Oudkerk <shibturn(a)gmail.com> date: Mon Jul 01 19:10:39 2013 +0100 summary: Issue #17097: Merge. files: Lib/multiprocessing/connection.py | 18 ++++- Lib/test/test_multiprocessing.py | 70 ++++++++++++++++++- Misc/NEWS | 2 + 3 files changed, 86 insertions(+), 4 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -363,7 +363,10 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: - n = write(self._handle, buf) + try: + n = write(self._handle, buf) + except InterruptedError: + continue remaining -= n if remaining == 0: break @@ -374,7 +377,10 @@ handle = self._handle remaining = size while remaining > 0: - chunk = read(handle, remaining) + try: + chunk = read(handle, remaining) + except InterruptedError: + continue n = len(chunk) if n == 0: if remaining == size: @@ -578,7 +584,13 @@ self._unlink = None def accept(self): - s, self._last_accepted = self._socket.accept() + while True: + try: + s, self._last_accepted = self._socket.accept() + except InterruptedError: + pass + else: + break s.setblocking(True) return Connection(s.detach()) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -3514,13 +3514,81 @@ self.assertLessEqual(new_size, old_size) # +# Issue #17097: EINTR should be ignored by recv(), send(), accept() etc +# + +class TestIgnoreEINTR(unittest.TestCase): + + @classmethod + def _test_ignore(cls, conn): + def handler(signum, frame): + pass + signal.signal(signal.SIGUSR1, handler) + conn.send('ready') + x = conn.recv() + conn.send(x) + conn.send_bytes(b'x'*(1024*1024)) # sending 1 MB should block + + @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') + def test_ignore(self): + conn, child_conn = multiprocessing.Pipe() + try: + p = multiprocessing.Process(target=self._test_ignore, + args=(child_conn,)) + p.daemon = True + p.start() + child_conn.close() + self.assertEqual(conn.recv(), 'ready') + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + time.sleep(0.1) + conn.send(1234) + self.assertEqual(conn.recv(), 1234) + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + self.assertEqual(conn.recv_bytes(), b'x'*(1024*1024)) + time.sleep(0.1) + p.join() + finally: + conn.close() + + @classmethod + def _test_ignore_listener(cls, conn): + def handler(signum, frame): + pass + signal.signal(signal.SIGUSR1, handler) + l = multiprocessing.connection.Listener() + conn.send(l.address) + a = l.accept() + a.send('welcome') + + @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') + def test_ignore_listener(self): + conn, child_conn = multiprocessing.Pipe() + try: + p = multiprocessing.Process(target=self._test_ignore_listener, + args=(child_conn,)) + p.daemon = True + p.start() + child_conn.close() + address = conn.recv() + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + time.sleep(0.1) + client = multiprocessing.connection.Client(address) + self.assertEqual(client.recv(), 'welcome') + p.join() + finally: + conn.close() + +# # # testcases_other = [OtherTest, TestInvalidHandle, TestInitializers, TestStdinBadfiledescriptor, TestWait, TestInvalidFamily, TestFlags, TestTimeouts, TestNoForkBomb, - TestForkAwareThreadLock] + TestForkAwareThreadLock, TestIgnoreEINTR] # # diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -135,6 +135,8 @@ Library ------- +- Issue #17097: Make multiprocessing ignore EINTR. + - Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a segfault inside the _pickle C extension. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Issue #17097: Make multiprocessing ignore EINTR.
by richard.oudkerk July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/3ec5267f51ff changeset: 84414:3ec5267f51ff branch: 3.3 parent: 84409:61b6cd7b9819 user: Richard Oudkerk <shibturn(a)gmail.com> date: Mon Jul 01 18:59:26 2013 +0100 summary: Issue #17097: Make multiprocessing ignore EINTR. files: Lib/multiprocessing/connection.py | 18 ++++- Lib/test/test_multiprocessing.py | 70 ++++++++++++++++++- Misc/NEWS | 2 + 3 files changed, 86 insertions(+), 4 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -366,7 +366,10 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: - n = write(self._handle, buf) + try: + n = write(self._handle, buf) + except InterruptedError: + continue remaining -= n if remaining == 0: break @@ -377,7 +380,10 @@ handle = self._handle remaining = size while remaining > 0: - chunk = read(handle, remaining) + try: + chunk = read(handle, remaining) + except InterruptedError: + continue n = len(chunk) if n == 0: if remaining == size: @@ -581,7 +587,13 @@ self._unlink = None def accept(self): - s, self._last_accepted = self._socket.accept() + while True: + try: + s, self._last_accepted = self._socket.accept() + except InterruptedError: + pass + else: + break s.setblocking(True) return Connection(s.detach()) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -3461,13 +3461,81 @@ self.assertLessEqual(new_size, old_size) # +# Issue #17097: EINTR should be ignored by recv(), send(), accept() etc +# + +class TestIgnoreEINTR(unittest.TestCase): + + @classmethod + def _test_ignore(cls, conn): + def handler(signum, frame): + pass + signal.signal(signal.SIGUSR1, handler) + conn.send('ready') + x = conn.recv() + conn.send(x) + conn.send_bytes(b'x'*(1024*1024)) # sending 1 MB should block + + @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') + def test_ignore(self): + conn, child_conn = multiprocessing.Pipe() + try: + p = multiprocessing.Process(target=self._test_ignore, + args=(child_conn,)) + p.daemon = True + p.start() + child_conn.close() + self.assertEqual(conn.recv(), 'ready') + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + time.sleep(0.1) + conn.send(1234) + self.assertEqual(conn.recv(), 1234) + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + self.assertEqual(conn.recv_bytes(), b'x'*(1024*1024)) + time.sleep(0.1) + p.join() + finally: + conn.close() + + @classmethod + def _test_ignore_listener(cls, conn): + def handler(signum, frame): + pass + signal.signal(signal.SIGUSR1, handler) + l = multiprocessing.connection.Listener() + conn.send(l.address) + a = l.accept() + a.send('welcome') + + @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') + def test_ignore_listener(self): + conn, child_conn = multiprocessing.Pipe() + try: + p = multiprocessing.Process(target=self._test_ignore_listener, + args=(child_conn,)) + p.daemon = True + p.start() + child_conn.close() + address = conn.recv() + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + time.sleep(0.1) + client = multiprocessing.connection.Client(address) + self.assertEqual(client.recv(), 'welcome') + p.join() + finally: + conn.close() + +# # # testcases_other = [OtherTest, TestInvalidHandle, TestInitializers, TestStdinBadfiledescriptor, TestWait, TestInvalidFamily, TestFlags, TestTimeouts, TestNoForkBomb, - TestForkAwareThreadLock] + TestForkAwareThreadLock, TestIgnoreEINTR] # # diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,8 @@ Library ------- +- Issue #17097: Make multiprocessing ignore EINTR. + - Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a segfault inside the _pickle C extension. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #17097: Make multiprocessing ignore EINTR.
by richard.oudkerk July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/bc34fe4a0d58 changeset: 84413:bc34fe4a0d58 branch: 2.7 parent: 84403:26ef5d5d5c3e user: Richard Oudkerk <shibturn(a)gmail.com> date: Mon Jul 01 18:45:28 2013 +0100 summary: Issue #17097: Make multiprocessing ignore EINTR. files: Lib/multiprocessing/connection.py | 9 +- Lib/test/test_multiprocessing.py | 70 +++++++++- Misc/NEWS | 2 + Modules/_multiprocessing/socket_connection.c | 57 ++++++- 4 files changed, 127 insertions(+), 11 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -270,7 +270,14 @@ self._unlink = None def accept(self): - s, self._last_accepted = self._socket.accept() + while True: + try: + s, self._last_accepted = self._socket.accept() + except socket.error as e: + if e.args[0] != errno.EINTR: + raise + else: + break s.setblocking(True) fd = duplicate(s.fileno()) conn = _multiprocessing.Connection(fd) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2461,12 +2461,80 @@ self.assertLessEqual(new_size, old_size) # +# Issue #17097: EINTR should be ignored by recv(), send(), accept() etc +# + +class TestIgnoreEINTR(unittest.TestCase): + + @classmethod + def _test_ignore(cls, conn): + def handler(signum, frame): + pass + signal.signal(signal.SIGUSR1, handler) + conn.send('ready') + x = conn.recv() + conn.send(x) + conn.send_bytes(b'x'*(1024*1024)) # sending 1 MB should block + + @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') + def test_ignore(self): + conn, child_conn = multiprocessing.Pipe() + try: + p = multiprocessing.Process(target=self._test_ignore, + args=(child_conn,)) + p.daemon = True + p.start() + child_conn.close() + self.assertEqual(conn.recv(), 'ready') + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + time.sleep(0.1) + conn.send(1234) + self.assertEqual(conn.recv(), 1234) + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + self.assertEqual(conn.recv_bytes(), b'x'*(1024*1024)) + time.sleep(0.1) + p.join() + finally: + conn.close() + + @classmethod + def _test_ignore_listener(cls, conn): + def handler(signum, frame): + pass + signal.signal(signal.SIGUSR1, handler) + l = multiprocessing.connection.Listener() + conn.send(l.address) + a = l.accept() + a.send('welcome') + + @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') + def test_ignore_listener(self): + conn, child_conn = multiprocessing.Pipe() + try: + p = multiprocessing.Process(target=self._test_ignore_listener, + args=(child_conn,)) + p.daemon = True + p.start() + child_conn.close() + address = conn.recv() + time.sleep(0.1) + os.kill(p.pid, signal.SIGUSR1) + time.sleep(0.1) + client = multiprocessing.connection.Client(address) + self.assertEqual(client.recv(), 'welcome') + p.join() + finally: + conn.close() + +# # # testcases_other = [OtherTest, TestInvalidHandle, TestInitializers, TestStdinBadfiledescriptor, TestTimeouts, TestNoForkBomb, - TestFlags, TestForkAwareThreadLock] + TestFlags, TestForkAwareThreadLock, TestIgnoreEINTR] # # diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,8 @@ Library ------- +- Issue #17097: Make multiprocessing ignore EINTR. + - Issue #18155: The csv module now correctly handles csv files that use a delimiter character that has a special meaning in regexes, instead of throwing an exception. diff --git a/Modules/_multiprocessing/socket_connection.c b/Modules/_multiprocessing/socket_connection.c --- a/Modules/_multiprocessing/socket_connection.c +++ b/Modules/_multiprocessing/socket_connection.c @@ -23,6 +23,21 @@ #endif /* + * Wrapper for PyErr_CheckSignals() which can be called without the GIL + */ + +static int +check_signals(void) +{ + PyGILState_STATE state; + int res; + state = PyGILState_Ensure(); + res = PyErr_CheckSignals(); + PyGILState_Release(state); + return res; +} + +/* * Send string to file descriptor */ @@ -34,8 +49,14 @@ while (length > 0) { res = WRITE(h, p, length); - if (res < 0) + if (res < 0) { + if (errno == EINTR) { + if (check_signals() < 0) + return MP_EXCEPTION_HAS_BEEN_SET; + continue; + } return MP_SOCKET_ERROR; + } length -= res; p += res; } @@ -56,12 +77,16 @@ while (remaining > 0) { temp = READ(h, p, remaining); - if (temp <= 0) { - if (temp == 0) - return remaining == length ? - MP_END_OF_FILE : MP_EARLY_END_OF_FILE; - else - return temp; + if (temp < 0) { + if (errno == EINTR) { + if (check_signals() < 0) + return MP_EXCEPTION_HAS_BEEN_SET; + continue; + } + return temp; + } + else if (temp == 0) { + return remaining == length ? MP_END_OF_FILE : MP_EARLY_END_OF_FILE; } remaining -= temp; p += temp; @@ -171,9 +196,16 @@ p.revents = 0; if (timeout < 0) { - res = poll(&p, 1, -1); + do { + res = poll(&p, 1, -1); + } while (res < 0 && errno == EINTR); } else { res = poll(&p, 1, (int)(timeout * 1000 + 0.5)); + if (res < 0 && errno == EINTR) { + /* We were interrupted by a signal. Just indicate a + timeout even though we are early. */ + return FALSE; + } } if (res < 0) { @@ -209,12 +241,19 @@ FD_SET((SOCKET)conn->handle, &rfds); if (timeout < 0.0) { - res = select((int)conn->handle+1, &rfds, NULL, NULL, NULL); + do { + res = select((int)conn->handle+1, &rfds, NULL, NULL, NULL); + } while (res < 0 && errno == EINTR); } else { struct timeval tv; tv.tv_sec = (long)timeout; tv.tv_usec = (long)((timeout - tv.tv_sec) * 1e6 + 0.5); res = select((int)conn->handle+1, &rfds, NULL, NULL, &tv); + if (res < 0 && errno == EINTR) { + /* We were interrupted by a signal. Just indicate a + timeout even though we are early. */ + return FALSE; + } } if (res < 0) { -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge default -> default): Merge with current default
by lukasz.langa July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/879ee6762f3d changeset: 84412:879ee6762f3d parent: 84411:a2672dc7c805 parent: 84410:f17647ce8f8a user: Łukasz Langa <lukasz(a)langa.pl> date: Mon Jul 01 16:03:17 2013 +0200 summary: Merge with current default files: Lib/test/test_pickle.py | 7 +++++++ Misc/NEWS | 3 +++ Modules/_pickle.c | 5 +++++ 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py --- a/Lib/test/test_pickle.py +++ b/Lib/test/test_pickle.py @@ -115,6 +115,13 @@ pickler_class = _pickle.Pickler unpickler_class = _pickle.Unpickler + def test_issue18339(self): + unpickler = self.unpickler_class(io.BytesIO()) + self.assertRaises(TypeError, setattr, unpickler, "memo", object) + # used to cause a segfault + self.assertRaises(ValueError, setattr, unpickler, "memo", {-1: None}) + unpickler.memo = {1: None} + class CDispatchTableTests(AbstractDispatchTableTests): pickler_class = pickle.Pickler def get_dispatch_table(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -135,6 +135,9 @@ Library ------- +- Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a + segfault inside the _pickle C extension. + - Issue 18240: The HMAC module is no longer restricted to bytes and accepts any bytes-like object, e.g. memoryview. Original patch by Jonas Borgström. diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -5952,6 +5952,11 @@ idx = PyLong_AsSsize_t(key); if (idx == -1 && PyErr_Occurred()) goto error; + if (idx < 0) { + PyErr_SetString(PyExc_ValueError, + "memo key must be positive integers."); + goto error; + } if (_Unpickler_MemoPut(self, idx, value) < 0) goto error; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #18244: Adopt C3-based linearization in functools.singledispatch for
by lukasz.langa July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/a2672dc7c805 changeset: 84411:a2672dc7c805 parent: 84406:636947fe131e user: Łukasz Langa <lukasz(a)langa.pl> date: Mon Jul 01 16:00:38 2013 +0200 summary: Issue #18244: Adopt C3-based linearization in functools.singledispatch for improved ABC support files: Lib/functools.py | 178 ++++++++++++++++++------ Lib/test/test_functools.py | 174 +++++++++++++++++++++-- Misc/ACKS | 1 + 3 files changed, 289 insertions(+), 64 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -365,46 +365,138 @@ ### singledispatch() - single-dispatch generic function decorator ################################################################################ -def _compose_mro(cls, haystack): - """Calculates the MRO for a given class `cls`, including relevant abstract - base classes from `haystack`. +def _c3_merge(sequences): + """Merges MROs in *sequences* to a single MRO using the C3 algorithm. + + Adapted from http://www.python.org/download/releases/2.3/mro/. + + """ + result = [] + while True: + sequences = [s for s in sequences if s] # purge empty sequences + if not sequences: + return result + for s1 in sequences: # find merge candidates among seq heads + candidate = s1[0] + for s2 in sequences: + if candidate in s2[1:]: + candidate = None + break # reject the current head, it appears later + else: + break + if not candidate: + raise RuntimeError("Inconsistent hierarchy") + result.append(candidate) + # remove the chosen candidate + for seq in sequences: + if seq[0] == candidate: + del seq[0] + +def _c3_mro(cls, abcs=None): + """Computes the method resolution order using extended C3 linearization. + + If no *abcs* are given, the algorithm works exactly like the built-in C3 + linearization used for method resolution. + + If given, *abcs* is a list of abstract base classes that should be inserted + into the resulting MRO. Unrelated ABCs are ignored and don't end up in the + result. The algorithm inserts ABCs where their functionality is introduced, + i.e. issubclass(cls, abc) returns True for the class itself but returns + False for all its direct base classes. Implicit ABCs for a given class + (either registered or inferred from the presence of a special method like + __len__) are inserted directly after the last ABC explicitly listed in the + MRO of said class. If two implicit ABCs end up next to each other in the + resulting MRO, their ordering depends on the order of types in *abcs*. + + """ + for i, base in enumerate(reversed(cls.__bases__)): + if hasattr(base, '__abstractmethods__'): + boundary = len(cls.__bases__) - i + break # Bases up to the last explicit ABC are considered first. + else: + boundary = 0 + abcs = list(abcs) if abcs else [] + explicit_bases = list(cls.__bases__[:boundary]) + abstract_bases = [] + other_bases = list(cls.__bases__[boundary:]) + for base in abcs: + if issubclass(cls, base) and not any( + issubclass(b, base) for b in cls.__bases__ + ): + # If *cls* is the class that introduces behaviour described by + # an ABC *base*, insert said ABC to its MRO. + abstract_bases.append(base) + for base in abstract_bases: + abcs.remove(base) + explicit_c3_mros = [_c3_mro(base, abcs=abcs) for base in explicit_bases] + abstract_c3_mros = [_c3_mro(base, abcs=abcs) for base in abstract_bases] + other_c3_mros = [_c3_mro(base, abcs=abcs) for base in other_bases] + return _c3_merge( + [[cls]] + + explicit_c3_mros + abstract_c3_mros + other_c3_mros + + [explicit_bases] + [abstract_bases] + [other_bases] + ) + +def _compose_mro(cls, types): + """Calculates the method resolution order for a given class *cls*. + + Includes relevant abstract base classes (with their respective bases) from + the *types* iterable. Uses a modified C3 linearization algorithm. """ bases = set(cls.__mro__) - mro = list(cls.__mro__) - for needle in haystack: - if (needle in bases or not hasattr(needle, '__mro__') - or not issubclass(cls, needle)): - continue # either present in the __mro__ already or unrelated - for index, base in enumerate(mro): - if not issubclass(base, needle): - break - if base in bases and not issubclass(needle, base): - # Conflict resolution: put classes present in __mro__ and their - # subclasses first. See test_mro_conflicts() in test_functools.py - # for examples. - index += 1 - mro.insert(index, needle) - return mro + # Remove entries which are already present in the __mro__ or unrelated. + def is_related(typ): + return (typ not in bases and hasattr(typ, '__mro__') + and issubclass(cls, typ)) + types = [n for n in types if is_related(n)] + # Remove entries which are strict bases of other entries (they will end up + # in the MRO anyway. + def is_strict_base(typ): + for other in types: + if typ != other and typ in other.__mro__: + return True + return False + types = [n for n in types if not is_strict_base(n)] + # Subclasses of the ABCs in *types* which are also implemented by + # *cls* can be used to stabilize ABC ordering. + type_set = set(types) + mro = [] + for typ in types: + found = [] + for sub in typ.__subclasses__(): + if sub not in bases and issubclass(cls, sub): + found.append([s for s in sub.__mro__ if s in type_set]) + if not found: + mro.append(typ) + continue + # Favor subclasses with the biggest number of useful bases + found.sort(key=len, reverse=True) + for sub in found: + for subcls in sub: + if subcls not in mro: + mro.append(subcls) + return _c3_mro(cls, abcs=mro) def _find_impl(cls, registry): - """Returns the best matching implementation for the given class `cls` in - `registry`. Where there is no registered implementation for a specific - type, its method resolution order is used to find a more generic - implementation. + """Returns the best matching implementation from *registry* for type *cls*. - Note: if `registry` does not contain an implementation for the base - `object` type, this function may return None. + Where there is no registered implementation for a specific type, its method + resolution order is used to find a more generic implementation. + + Note: if *registry* does not contain an implementation for the base + *object* type, this function may return None. """ mro = _compose_mro(cls, registry.keys()) match = None for t in mro: if match is not None: - # If `match` is an ABC but there is another unrelated, equally - # matching ABC. Refuse the temptation to guess. - if (t in registry and not issubclass(match, t) - and match not in cls.__mro__): + # If *match* is an implicit ABC but there is another unrelated, + # equally matching implicit ABC, refuse the temptation to guess. + if (t in registry and t not in cls.__mro__ + and match not in cls.__mro__ + and not issubclass(match, t)): raise RuntimeError("Ambiguous dispatch: {} or {}".format( match, t)) break @@ -418,19 +510,19 @@ Transforms a function into a generic function, which can have different behaviours depending upon the type of its first argument. The decorated function acts as the default implementation, and additional - implementations can be registered using the 'register()' attribute of - the generic function. + implementations can be registered using the register() attribute of the + generic function. """ registry = {} dispatch_cache = WeakKeyDictionary() cache_token = None - def dispatch(typ): - """generic_func.dispatch(type) -> <function implementation> + def dispatch(cls): + """generic_func.dispatch(cls) -> <function implementation> Runs the dispatch algorithm to return the best available implementation - for the given `type` registered on `generic_func`. + for the given *cls* registered on *generic_func*. """ nonlocal cache_token @@ -440,26 +532,26 @@ dispatch_cache.clear() cache_token = current_token try: - impl = dispatch_cache[typ] + impl = dispatch_cache[cls] except KeyError: try: - impl = registry[typ] + impl = registry[cls] except KeyError: - impl = _find_impl(typ, registry) - dispatch_cache[typ] = impl + impl = _find_impl(cls, registry) + dispatch_cache[cls] = impl return impl - def register(typ, func=None): - """generic_func.register(type, func) -> func + def register(cls, func=None): + """generic_func.register(cls, func) -> func - Registers a new implementation for the given `type` on a `generic_func`. + Registers a new implementation for the given *cls* on a *generic_func*. """ nonlocal cache_token if func is None: - return lambda f: register(typ, f) - registry[typ] = func - if cache_token is None and hasattr(typ, '__abstractmethods__'): + return lambda f: register(cls, f) + registry[cls] = func + if cache_token is None and hasattr(cls, '__abstractmethods__'): cache_token = get_cache_token() dispatch_cache.clear() return func diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -929,22 +929,55 @@ self.assertEqual(g(rnd), ("Number got rounded",)) def test_compose_mro(self): + # None of the examples in this test depend on haystack ordering. c = collections mro = functools._compose_mro bases = [c.Sequence, c.MutableMapping, c.Mapping, c.Set] for haystack in permutations(bases): m = mro(dict, haystack) - self.assertEqual(m, [dict, c.MutableMapping, c.Mapping, object]) + self.assertEqual(m, [dict, c.MutableMapping, c.Mapping, c.Sized, + c.Iterable, c.Container, object]) bases = [c.Container, c.Mapping, c.MutableMapping, c.OrderedDict] for haystack in permutations(bases): m = mro(c.ChainMap, haystack) self.assertEqual(m, [c.ChainMap, c.MutableMapping, c.Mapping, c.Sized, c.Iterable, c.Container, object]) - # Note: The MRO order below depends on haystack ordering. - m = mro(c.defaultdict, [c.Sized, c.Container, str]) - self.assertEqual(m, [c.defaultdict, dict, c.Container, c.Sized, object]) - m = mro(c.defaultdict, [c.Container, c.Sized, str]) - self.assertEqual(m, [c.defaultdict, dict, c.Sized, c.Container, object]) + + # If there's a generic function with implementations registered for + # both Sized and Container, passing a defaultdict to it results in an + # ambiguous dispatch which will cause a RuntimeError (see + # test_mro_conflicts). + bases = [c.Container, c.Sized, str] + for haystack in permutations(bases): + m = mro(c.defaultdict, [c.Sized, c.Container, str]) + self.assertEqual(m, [c.defaultdict, dict, c.Sized, c.Container, + object]) + + # MutableSequence below is registered directly on D. In other words, it + # preceeds MutableMapping which means single dispatch will always + # choose MutableSequence here. + class D(c.defaultdict): + pass + c.MutableSequence.register(D) + bases = [c.MutableSequence, c.MutableMapping] + for haystack in permutations(bases): + m = mro(D, bases) + self.assertEqual(m, [D, c.MutableSequence, c.Sequence, + c.defaultdict, dict, c.MutableMapping, + c.Mapping, c.Sized, c.Iterable, c.Container, + object]) + + # Container and Callable are registered on different base classes and + # a generic function supporting both should always pick the Callable + # implementation if a C instance is passed. + class C(c.defaultdict): + def __call__(self): + pass + bases = [c.Sized, c.Callable, c.Container, c.Mapping] + for haystack in permutations(bases): + m = mro(C, haystack) + self.assertEqual(m, [C, c.Callable, c.defaultdict, dict, c.Mapping, + c.Sized, c.Iterable, c.Container, object]) def test_register_abc(self): c = collections @@ -1040,17 +1073,37 @@ self.assertEqual(g(f), "frozen-set") self.assertEqual(g(t), "tuple") + def test_c3_abc(self): + c = collections + mro = functools._c3_mro + class A(object): + pass + class B(A): + def __len__(self): + return 0 # implies Sized + @c.Container.register + class C(object): + pass + class D(object): + pass # unrelated + class X(D, C, B): + def __call__(self): + pass # implies Callable + expected = [X, c.Callable, D, C, c.Container, B, c.Sized, A, object] + for abcs in permutations([c.Sized, c.Callable, c.Container]): + self.assertEqual(mro(X, abcs=abcs), expected) + # unrelated ABCs don't appear in the resulting MRO + many_abcs = [c.Mapping, c.Sized, c.Callable, c.Container, c.Iterable] + self.assertEqual(mro(X, abcs=many_abcs), expected) + def test_mro_conflicts(self): c = collections - @functools.singledispatch def g(arg): return "base" - class O(c.Sized): def __len__(self): return 0 - o = O() self.assertEqual(g(o), "base") g.register(c.Iterable, lambda arg: "iterable") @@ -1062,35 +1115,114 @@ self.assertEqual(g(o), "sized") # because it's explicitly in __mro__ c.Container.register(O) self.assertEqual(g(o), "sized") # see above: Sized is in __mro__ - + c.Set.register(O) + self.assertEqual(g(o), "set") # because c.Set is a subclass of + # c.Sized and c.Container class P: pass - p = P() self.assertEqual(g(p), "base") c.Iterable.register(P) self.assertEqual(g(p), "iterable") c.Container.register(P) - with self.assertRaises(RuntimeError) as re: + with self.assertRaises(RuntimeError) as re_one: g(p) - self.assertEqual( - str(re), - ("Ambiguous dispatch: <class 'collections.abc.Container'> " - "or <class 'collections.abc.Iterable'>"), - ) - + self.assertIn( + str(re_one.exception), + (("Ambiguous dispatch: <class 'collections.abc.Container'> " + "or <class 'collections.abc.Iterable'>"), + ("Ambiguous dispatch: <class 'collections.abc.Iterable'> " + "or <class 'collections.abc.Container'>")), + ) class Q(c.Sized): def __len__(self): return 0 - q = Q() self.assertEqual(g(q), "sized") c.Iterable.register(Q) self.assertEqual(g(q), "sized") # because it's explicitly in __mro__ c.Set.register(Q) self.assertEqual(g(q), "set") # because c.Set is a subclass of - # c.Sized which is explicitly in - # __mro__ + # c.Sized and c.Iterable + @functools.singledispatch + def h(arg): + return "base" + @h.register(c.Sized) + def _(arg): + return "sized" + @h.register(c.Container) + def _(arg): + return "container" + # Even though Sized and Container are explicit bases of MutableMapping, + # this ABC is implicitly registered on defaultdict which makes all of + # MutableMapping's bases implicit as well from defaultdict's + # perspective. + with self.assertRaises(RuntimeError) as re_two: + h(c.defaultdict(lambda: 0)) + self.assertIn( + str(re_two.exception), + (("Ambiguous dispatch: <class 'collections.abc.Container'> " + "or <class 'collections.abc.Sized'>"), + ("Ambiguous dispatch: <class 'collections.abc.Sized'> " + "or <class 'collections.abc.Container'>")), + ) + class R(c.defaultdict): + pass + c.MutableSequence.register(R) + @functools.singledispatch + def i(arg): + return "base" + @i.register(c.MutableMapping) + def _(arg): + return "mapping" + @i.register(c.MutableSequence) + def _(arg): + return "sequence" + r = R() + self.assertEqual(i(r), "sequence") + class S: + pass + class T(S, c.Sized): + def __len__(self): + return 0 + t = T() + self.assertEqual(h(t), "sized") + c.Container.register(T) + self.assertEqual(h(t), "sized") # because it's explicitly in the MRO + class U: + def __len__(self): + return 0 + u = U() + self.assertEqual(h(u), "sized") # implicit Sized subclass inferred + # from the existence of __len__() + c.Container.register(U) + # There is no preference for registered versus inferred ABCs. + with self.assertRaises(RuntimeError) as re_three: + h(u) + self.assertIn( + str(re_three.exception), + (("Ambiguous dispatch: <class 'collections.abc.Container'> " + "or <class 'collections.abc.Sized'>"), + ("Ambiguous dispatch: <class 'collections.abc.Sized'> " + "or <class 'collections.abc.Container'>")), + ) + class V(c.Sized, S): + def __len__(self): + return 0 + @functools.singledispatch + def j(arg): + return "base" + @j.register(S) + def _(arg): + return "s" + @j.register(c.Container) + def _(arg): + return "container" + v = V() + self.assertEqual(j(v), "s") + c.Container.register(V) + self.assertEqual(j(v), "container") # because it ends up right after + # Sized in the MRO def test_cache_invalidation(self): from collections import UserDict diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -195,6 +195,7 @@ Mike Carlton Pierre Carrier Terry Carroll +Edward Catmur Lorenzo M. Catucci Donn Cave Charles Cazabon -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Singular form just like the other error message.
by christian.heimes July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/f17647ce8f8a changeset: 84410:f17647ce8f8a parent: 84408:f3372692ca20 parent: 84409:61b6cd7b9819 user: Christian Heimes <christian(a)cheimes.de> date: Mon Jul 01 15:23:48 2013 +0200 summary: Singular form just like the other error message. files: Modules/_pickle.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -5954,7 +5954,7 @@ goto error; if (idx < 0) { PyErr_SetString(PyExc_ValueError, - "memos key must be positive integers."); + "memo key must be positive integers."); goto error; } if (_Unpickler_MemoPut(self, idx, value) < 0) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Singular form just like the other error message.
by christian.heimes July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/61b6cd7b9819 changeset: 84409:61b6cd7b9819 branch: 3.3 parent: 84407:17b7af660f82 user: Christian Heimes <christian(a)cheimes.de> date: Mon Jul 01 15:23:39 2013 +0200 summary: Singular form just like the other error message. files: Modules/_pickle.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -5933,7 +5933,7 @@ goto error; if (idx < 0) { PyErr_SetString(PyExc_ValueError, - "memos key must be positive integers."); + "memo key must be positive integers."); goto error; } if (_Unpickler_MemoPut(self, idx, value) < 0) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a
by christian.heimes July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/f3372692ca20 changeset: 84408:f3372692ca20 parent: 84406:636947fe131e parent: 84407:17b7af660f82 user: Christian Heimes <christian(a)cheimes.de> date: Mon Jul 01 15:18:49 2013 +0200 summary: Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a segfault inside the _pickle C extension. files: Lib/test/test_pickle.py | 7 +++++++ Misc/NEWS | 3 +++ Modules/_pickle.c | 5 +++++ 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py --- a/Lib/test/test_pickle.py +++ b/Lib/test/test_pickle.py @@ -115,6 +115,13 @@ pickler_class = _pickle.Pickler unpickler_class = _pickle.Unpickler + def test_issue18339(self): + unpickler = self.unpickler_class(io.BytesIO()) + self.assertRaises(TypeError, setattr, unpickler, "memo", object) + # used to cause a segfault + self.assertRaises(ValueError, setattr, unpickler, "memo", {-1: None}) + unpickler.memo = {1: None} + class CDispatchTableTests(AbstractDispatchTableTests): pickler_class = pickle.Pickler def get_dispatch_table(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -135,6 +135,9 @@ Library ------- +- Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a + segfault inside the _pickle C extension. + - Issue 18240: The HMAC module is no longer restricted to bytes and accepts any bytes-like object, e.g. memoryview. Original patch by Jonas Borgström. diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -5952,6 +5952,11 @@ idx = PyLong_AsSsize_t(key); if (idx == -1 && PyErr_Occurred()) goto error; + if (idx < 0) { + PyErr_SetString(PyExc_ValueError, + "memos key must be positive integers."); + goto error; + } if (_Unpickler_MemoPut(self, idx, value) < 0) goto error; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a
by christian.heimes July 1, 2013

July 1, 2013

http://hg.python.org/cpython/rev/17b7af660f82 changeset: 84407:17b7af660f82 branch: 3.3 parent: 84404:c39ddff53694 user: Christian Heimes <christian(a)cheimes.de> date: Mon Jul 01 15:17:45 2013 +0200 summary: Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a segfault inside the _pickle C extension. files: Lib/test/test_pickle.py | 7 +++++++ Misc/NEWS | 3 +++ Modules/_pickle.c | 5 +++++ 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py --- a/Lib/test/test_pickle.py +++ b/Lib/test/test_pickle.py @@ -115,6 +115,13 @@ pickler_class = _pickle.Pickler unpickler_class = _pickle.Unpickler + def test_issue18339(self): + unpickler = self.unpickler_class(io.BytesIO()) + self.assertRaises(TypeError, setattr, unpickler, "memo", object) + # used to cause a segfault + self.assertRaises(ValueError, setattr, unpickler, "memo", {-1: None}) + unpickler.memo = {1: None} + class CDispatchTableTests(AbstractDispatchTableTests): pickler_class = pickle.Pickler def get_dispatch_table(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ Library ------- +- Issue #18339: Negative ints keys in unpickler.memo dict no longer cause a + segfault inside the _pickle C extension. + - Issue #18224: Removed pydoc script from created venv, as it causes problems on Windows and adds no value over and above python -m pydoc ... diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -5931,6 +5931,11 @@ idx = PyLong_AsSsize_t(key); if (idx == -1 && PyErr_Occurred()) goto error; + if (idx < 0) { + PyErr_SetString(PyExc_ValueError, + "memos key must be positive integers."); + goto error; + } if (_Unpickler_MemoPut(self, idx, value) < 0) goto error; } -- Repository URL: http://hg.python.org/cpython

1 0