Mailman 3 January 2015 - Python-checkins

peps: Some rephrasings. No technical change.
by antoine.pitrou Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/peps/rev/0eee0d4f2b1d changeset: 5681:0eee0d4f2b1d user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Jan 30 02:17:26 2015 +0100 summary: Some rephrasings. No technical change. files: pep-0475.txt | 186 +++++++++++++++++++++----------------- 1 files changed, 101 insertions(+), 85 deletions(-) diff --git a/pep-0475.txt b/pep-0475.txt --- a/pep-0475.txt +++ b/pep-0475.txt @@ -14,8 +14,12 @@ Abstract ======== -Retry system calls failing … [View More]with the ``EINTR`` error and recompute -timeout if needed. +System call wrappers provided in the standard library should be retried +automatically when they fail with ``EINTR``, to relieve application code +from the burden of doing so. + +By system calls, we mean the functions exposed by the standard C library +pertaining to I/O or handling of other system resources. Rationale @@ -24,10 +28,10 @@ Interrupted system calls ------------------------ -On POSIX systems, signals are common. Your program must be prepared to -handle them. Examples of signals: +On POSIX systems, signals are common. Code calling system calls must be +prepared to handle them. Examples of signals: -* The most common signal is ``SIGINT``, signal sent when CTRL+c is +* The most common signal is ``SIGINT``, the signal sent when CTRL+c is pressed. By default, Python raises a ``KeyboardInterrupt`` exception when this signal is received. * When running subprocesses, the ``SIGCHLD`` signal is sent when a @@ -37,37 +41,35 @@ * Putting the application in background (ex: press CTRL-z and then type the ``bg`` command) sends the ``SIGCONT`` signal. -Writing a signal handler is difficult, only "async-signal safe" -functions can be called. For example, ``printf()`` and ``malloc()`` -are not async-signal safe. When a signal is sent to a process calling -a system call, the system call can fail with the ``EINTR`` error to +Writing a C signal handler is difficult: only "async-signal-safe" +functions can be called (for example, ``printf()`` and ``malloc()`` +are not async-signal safe), and there are issues with reentrancy. +Therefore, when a signal is received by a process during the execution +of a system call, the system call can fail with the ``EINTR`` error to give the program an opportunity to handle the signal without the -restriction on signal safe functions. Depending on the platform, on -the system call and the ``SA_RESTART`` flag, the system call may or -may not fail with ``EINTR``. +restriction on signal-safe functions. -If the signal handler was set with the ``SA_RESTART`` flag set, the -kernel retries some the system call instead of failing with -``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is -not retried. The Python function ``signal.signal()`` clears the -``SA_RESTART`` flag when setting the signal handler: all system calls -should fail with ``EINTR`` in Python. +This behaviour is system-dependent: on certain systems, using the +``SA_RESTART`` flag, some system calls are retried automatically instead +of failing with ``EINTR``. Regardless, Python's ``signal.signal()`` +function clears the ``SA_RESTART`` flag when setting the signal handler: +all system calls will probably fail with ``EINTR`` in Python. -The problem is that handling ``EINTR`` should be done for all system -calls. The problem is similar to handling errors in the C language -which does not have exceptions: you must check all function returns to -check for error, and usually duplicate the code checking for errors. -Python does not have this issue, it uses exceptions to notify errors. +Since receiving a signal is a non-exceptional occurrence, robust POSIX code +must be prepared to handle ``EINTR`` (which, in most cases, means +retry in a loop in the hope that the call eventually succeeds). +Without special support from Python, this can make application code +much more verbose than it needs to be. Status in Python 3.4 -------------------- -In Python 3.4, the code to handle the ``InterruptedError`` -exception (``EINTR`` error) is duplicated on case by case. Only a few -Python modules handle this exception, and fixes usually took several -years to cover a whole module. Example of code retrying -``file.read()`` on ``InterruptedError``:: +In Python 3.4, handling the ``InterruptedError`` exception (``EINTR``'s +dedicated exception class) is duplicated at every call site on a case by +case basis. Only a few Python modules actually handle this exception, +and fixes usually took several years to cover a whole module. Example of +code retrying ``file.read()`` on ``InterruptedError``:: while True: try: @@ -76,7 +78,7 @@ except InterruptedError: continue -List of Python modules of the standard library which handle +List of Python modules in the standard library which handle ``InterruptedError``: * ``asyncio`` @@ -88,16 +90,17 @@ * ``socketserver`` * ``subprocess`` -Other programming languages like Perl, Java and Go already retry -system calls failing with ``EINTR``. +Other programming languages like Perl, Java and Go retry system calls +failing with ``EINTR`` at a lower level, so that libraries and applications +needn't bother. Use Case 1: Don't Bother With Signals ------------------------------------- In most cases, you don't want to be interrupted by signals and you -don't expect to get ``InterruptedError`` exceptions. For example, do -you really want to write such complex code for an "Hello World" +don't expect to get ``InterruptedError`` exceptions. For example, do +you really want to write such complex code for a "Hello World" example? :: @@ -110,48 +113,59 @@ continue ``InterruptedError`` can happen in unexpected places. For example, -``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``: +``os.close()`` and ``FileIO.close()`` may raise ``InterruptedError``: see the article `close() and EINTR <http://alobbs.com/post/54503240599/close-and-eintr>`_. The `Python issues related to EINTR`_ section below gives examples of -bugs caused by "EINTR". +bugs caused by ``EINTR``. -The expectation is that Python hides the ``InterruptedError``: retry -system calls failing with the ``EINTR`` error. +The expectation in this use case is that Python hides the +``InterruptedError`` and retries system calls automatically. Use Case 2: Be notified of signals as soon as possible ------------------------------------------------------ -Sometimes, you expect some signals and you want to handle them as soon -as possible. For example, you may want to quit immediatly a program -using the ``CTRL+c`` keyboard shortcut. +Sometimes yet, you expect some signals and you want to handle them as +soon as possible. For example, you may want to immediately quit a +program using the ``CTRL+c`` keyboard shortcut. -Some signals are not interesting and should not interrupt the the -application. There are two options to only interrupt an application -on some signals: +Besides, some signals are not interesting and should not disrupt the +application. There are two options to interrupt an application on +only *some* signals: -* Raise an exception in the signal handler, like ``KeyboardInterrupt`` for - ``SIGINT`` -* Use a I/O multiplexing function like ``select()`` with the Python - signal "wakeup" file descriptor: see the function - ``signal.set_wakeup_fd()``. +* Set up a custom signal signal handler which raises an exception, such as + ``KeyboardInterrupt`` for ``SIGINT``. +* Use a I/O multiplexing function like ``select()`` together with Python's + signal wakeup file descriptor: see the function ``signal.set_wakeup_fd()``. +The expectation in this use case is for the Python signal handler to be +executed timely, and the system call to fail if the handler raised an +exception -- otherwise restart. -Proposition -=========== -If a system call fails with ``EINTR``, Python must call signal -handlers: call ``PyErr_CheckSignals()``. If a signal handler raises -an exception, the Python function fails with the exception. -Otherwise, the system call is retried. If the system call takes a -timeout parameter, the timeout is recomputed. +Proposal +======== + +This PEP proposes to handle EINTR and retries at the lowest level, i.e. +in the wrappers provided by the stdlib (as opposed to higher-level +libraries and applications). + +Specifically, when a system call fails with ``EINTR``, its Python wrapper +must call the given signal handler (using ``PyErr_CheckSignals()``). +If the signal handler raises an exception, the Python wrapper bails out +and fails with the exception. + +If the signal handler returns successfully, the Python wrapper retries the +system call automatically. If the system call involves a timeout parameter, +the timeout is recomputed. Modified functions ------------------ -Example of functions that need to be modified: +Example of standard library functions that need to be modified to comply +with this PEP: * ``os.read()``, ``io.FileIO.read()``, ``io.FileIO.readinto()`` * ``os.write()``, ``io.FileIO.write()`` @@ -170,38 +184,40 @@ * ``select.kqueue.control()`` * ``selectors.SelectSelector.select()`` and other selector classes -Note: The ``selector`` module already retries on ``InterruptedError``, but it -doesn't recompute the timeout yet. +(note: the ``selector`` module already retries on ``InterruptedError``, but it +doesn't recompute the timeout yet) - -InterruptedError ----------------- +InterruptedError handling +------------------------- Since interrupted system calls are automatically retried, the -``InterruptedError`` exception should not occur anymore. The code handling -``InterruptedError`` can be removed from in the standard library to simply the -code. +``InterruptedError`` exception should not occur anymore when calling those +system calls. Therefore, manual handling of ``InterruptedError`` as +described in `Status in Python 3.4`_ can be removed, which will simplify +standard library code. -Backward Compatibility +Backward compatibility ====================== Applications relying on the fact that system calls are interrupted -with ``InterruptedError`` will hang. The authors of this PEP don't -think that such application exist. +with ``InterruptedError`` will hang. The authors of this PEP don't +think that such applications exist, since they would be exposed to +other issues such as race conditions (there is an opportunity for deadlock +if the signal comes before the system call). Besides, such code would +be non-portable. -If such applications exist, they are not portable and are subject to -race conditions (deadlock if the signal comes before the system call). -These applications must be fixed to handle signals differently, to -have a reliable behaviour on all platforms and all Python versions. -For example, use a signal handler which raises an exception, or use a -wakeup file descriptor. +In any case, those applications must be fixed to handle signals differently, +to have a reliable behaviour on all platforms and all Python versions. +A possible strategy is to set up a signal handler raising a well-defined +exception, or use a wakeup file descriptor. For applications using event loops, ``signal.set_wakeup_fd()`` is the -recommanded option to handle signals. The signal handler writes signal -numbers into the file descriptor and the event loop is awaken to read -them. The event loop can handle these signals without the restriction -of signal handlers. +recommanded option to handle signals. Python's low-level signal handler +will write signal numbers into the file descriptor and the event loop +will be awaken to read them. The event loop can handle those signals +without the restriction of signal handlers (for example, the loop can +be woken up in any thread, not just the main thread). Appendix @@ -212,12 +228,12 @@ Since Python 3.3, ``signal.set_wakeup_fd()`` writes the signal number into the file descriptor, whereas it only wrote a null byte before. -It becomes possible to handle different signals using the wakeup file +It becomes possible to distinguish between signals using the wakeup file descriptor. -Linux has a ``signalfd()`` which provides more information on each -signal. For example, it's possible to know the pid and uid who sent -the signal. This function is not exposed in Python yet (see the +Linux has a ``signalfd()`` system call which provides more information on +each signal. For example, it's possible to know the pid and uid who sent +the signal. This function is not exposed in Python yet (see `issue 12304 <http://bugs.python.org/issue12304>`_). On Unix, the ``asyncio`` module uses the wakeup file descriptor to @@ -227,11 +243,11 @@ Multithreading -------------- -A C signal handler can be called from any thread, but the Python -signal handler should only be called in the main thread. +A C signal handler can be called from any thread, but Python +signal handlers will always be called in the main Python thread. -Python has a ``PyErr_SetInterrupt()`` function which calls the -``SIGINT`` signal handler to interrupt the Python main thread. +Python's C API provides the ``PyErr_SetInterrupt()`` function which calls +the ``SIGINT`` signal handler in order to interrupt the main Python thread. Signals on Windows -- Repository URL: https://hg.python.org/peps [View Less]

1 0

cpython (3.4): Fix asyncio doc typo.
by r.david.murray Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/cpython/rev/fadfcb2d5a47 changeset: 94393:fadfcb2d5a47 branch: 3.4 parent: 94391:3c3f08e68c20 user: R David Murray <rdmurray(a)bitdance.com> date: Thu Jan 29 19:53:33 2015 -0500 summary: Fix asyncio doc typo. files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/… [View More]

1 0

cpython (merge 3.4 -> default): Merge: Fix asyncio doc typo.
by r.david.murray Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/cpython/rev/efb25925bd13 changeset: 94394:efb25925bd13 parent: 94392:582dd2ef9673 parent: 94393:fadfcb2d5a47 user: R David Murray <rdmurray(a)bitdance.com> date: Thu Jan 29 19:54:03 2015 -0500 summary: Merge: Fix asyncio doc typo. files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/… [View More]

1 0

cpython (3.4): asyncio doc: document the new ResourceWarning warnings
by victor.stinner Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/cpython/rev/3c3f08e68c20 changeset: 94391:3c3f08e68c20 branch: 3.4 parent: 94389:c1d92f7221a5 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 01:35:14 2015 +0100 summary: asyncio doc: document the new ResourceWarning warnings files: Doc/library/asyncio-dev.rst | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/… [View More]

1 0

cpython (merge 3.4 -> default): Merge 3.4 (asyncio doc)
by victor.stinner Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/cpython/rev/582dd2ef9673 changeset: 94392:582dd2ef9673 parent: 94390:25c1d4aea1c1 parent: 94391:3c3f08e68c20 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 01:39:01 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-dev.rst | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst … [View More]

1 0

cpython (3.4): Issue #23347, asyncio: send_signal(), terminate(), kill() don't check if the
by victor.stinner Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/cpython/rev/c1d92f7221a5 changeset: 94389:c1d92f7221a5 branch: 3.4 parent: 94387:173d4a8e1e74 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 01:20:44 2015 +0100 summary: Issue #23347, asyncio: send_signal(), terminate(), kill() don't check if the transport was closed. The check broken a Tulip example and this limitation is arbitrary. Check if _proc is None should be enough. Enhance also close(): do nothing when called … [View More]the second time. files: Lib/asyncio/base_subprocess.py | 7 ++--- Lib/test/test_asyncio/test_subprocess.py | 16 ------------ 2 files changed, 3 insertions(+), 20 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -84,6 +84,8 @@ raise NotImplementedError def close(self): + if self._closed: + return self._closed = True for proto in self._pipes.values(): @@ -100,8 +102,7 @@ except ProcessLookupError: pass - # Don't clear the _proc reference yet because _post_init() may - # still run + # Don't clear the _proc reference yet: _post_init() may still run # On Python 3.3 and older, objects with a destructor part of a reference # cycle are never destroyed. It's not more the case on Python 3.4 thanks @@ -125,8 +126,6 @@ return None def _check_proc(self): - if self._closed: - raise ValueError("operation on closed transport") if self._proc is None: raise ProcessLookupError() diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -47,22 +47,6 @@ None, None, None, 0, waiter=waiter) return (transport, protocol) - def test_close(self): - waiter = asyncio.Future(loop=self.loop) - transport, protocol = self.create_transport(waiter) - transport._process_exited(0) - transport.close() - - # The loop didn't run yet - self.assertFalse(protocol.connection_made.called) - - # methods must raise ProcessLookupError if the transport was closed - self.assertRaises(ValueError, transport.send_signal, signal.SIGTERM) - self.assertRaises(ValueError, transport.terminate) - self.assertRaises(ValueError, transport.kill) - - self.loop.run_until_complete(waiter) - def test_proc_exited(self): waiter = asyncio.Future(loop=self.loop) transport, protocol = self.create_transport(waiter) -- Repository URL: https://hg.python.org/cpython [View Less]

1 0

cpython (merge 3.4 -> default): Merge 3.4 (asyncio)
by victor.stinner Jan. 30, 2015

Jan. 30, 2015

https://hg.python.org/cpython/rev/25c1d4aea1c1 changeset: 94390:25c1d4aea1c1 parent: 94388:faf94fc9d94c parent: 94389:c1d92f7221a5 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 01:21:06 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_subprocess.py | 7 ++--- Lib/test/test_asyncio/test_subprocess.py | 16 ------------ 2 files changed, 3 insertions(+), 20 deletions(-) diff --git a/Lib/asyncio/… [View More]base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -84,6 +84,8 @@ raise NotImplementedError def close(self): + if self._closed: + return self._closed = True for proto in self._pipes.values(): @@ -100,8 +102,7 @@ except ProcessLookupError: pass - # Don't clear the _proc reference yet because _post_init() may - # still run + # Don't clear the _proc reference yet: _post_init() may still run # On Python 3.3 and older, objects with a destructor part of a reference # cycle are never destroyed. It's not more the case on Python 3.4 thanks @@ -125,8 +126,6 @@ return None def _check_proc(self): - if self._closed: - raise ValueError("operation on closed transport") if self._proc is None: raise ProcessLookupError() diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -47,22 +47,6 @@ None, None, None, 0, waiter=waiter) return (transport, protocol) - def test_close(self): - waiter = asyncio.Future(loop=self.loop) - transport, protocol = self.create_transport(waiter) - transport._process_exited(0) - transport.close() - - # The loop didn't run yet - self.assertFalse(protocol.connection_made.called) - - # methods must raise ProcessLookupError if the transport was closed - self.assertRaises(ValueError, transport.send_signal, signal.SIGTERM) - self.assertRaises(ValueError, transport.terminate) - self.assertRaises(ValueError, transport.kill) - - self.loop.run_until_complete(waiter) - def test_proc_exited(self): waiter = asyncio.Future(loop=self.loop) transport, protocol = self.create_transport(waiter) -- Repository URL: https://hg.python.org/cpython [View Less]

1 0

cpython (3.4): asyncio doc: add a section about task cancellation
by victor.stinner Jan. 29, 2015

Jan. 29, 2015

https://hg.python.org/cpython/rev/173d4a8e1e74 changeset: 94387:173d4a8e1e74 branch: 3.4 parent: 94385:21010940f8c1 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 00:55:58 2015 +0100 summary: asyncio doc: add a section about task cancellation files: Doc/library/asyncio-dev.rst | 37 +++++++++++++++++++++++++ 1 files changed, 37 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/… [View More]

1 0

cpython (merge 3.4 -> default): Merge 3.4 (asyncio doc)
by victor.stinner Jan. 29, 2015

Jan. 29, 2015

https://hg.python.org/cpython/rev/faf94fc9d94c changeset: 94388:faf94fc9d94c parent: 94386:23342733dd7e parent: 94387:173d4a8e1e74 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 00:56:10 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-dev.rst | 37 +++++++++++++++++++++++++ 1 files changed, 37 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/… [View More]

1 0

cpython (merge 3.4 -> default): Merge 3.4 (asyncio doc)
by victor.stinner Jan. 29, 2015

Jan. 29, 2015

https://hg.python.org/cpython/rev/23342733dd7e changeset: 94386:23342733dd7e parent: 94384:7e6340e67618 parent: 94385:21010940f8c1 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Jan 30 00:37:22 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-sync.rst | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/… [View More]

1 0