Mailman 3 September 2014 - Python-Dev

Python docs about comparisons vs. CPython reality
by Jan Kaliszewski Sept. 6, 2014

Sept. 6, 2014

Hello, Are they bugs in the Python docs or just some CPython implementation details that are purposely not documented? (but then, again, some of the docs seem to be at least not precise...): In https://docs.python.org/3.4/reference/datamodel.html#object.__eq__ there is the statement: > There are no implied relationships among the comparison operators. > The truth of x==y does not imply that x!=y is false. Accordingly, > when defining __eq__(), one should also define __ne__() so that the > operators will behave as expected. On the other hand, in https://docs.python.org/3.4/library/stdtypes.html#comparisons we read: > (in general, __lt__() and __eq__() are sufficient, if you want the > conventional meanings of the comparison operators) And, when I try the __eq__() stuff in CPython it seems that, indeed, the language provides a proper __ne__() implementation for me automatically (without need to implement __ne__() explicitly by myself): Python 3.4.0 (default, Mar 20 2014, 01:28:00) [...] >>> class A: ... def __eq__(self, other): ... if hasattr(self, 'x') and hasattr(other, 'x'): ... return self.x == other.x ... return NotImplemented ... >>> A() == A() False >>> A() != A() True >>> a = A() >>> a.x = 1 >>> a1 = A() >>> a1.x = 1 >>> a2 = A() >>> a2.x = 2 >>> a == a1 True >>> a != a1 False >>> a1 == a1 True >>> a1 != a1 False >>> a1 == a2 False >>> a1 != a2 True Is it a language guarantee (then, I believe, it should be documented) or just an implementation accident? (then, I believe, it still could be documented as a CPython implementation detail). See also the Python equivalent of the SimpleNamespace class (without __ne__() implemented explicitly): https://docs.python.org/3/library/types.html#types.SimpleNamespace On the other hand, the "__lt__() and __eq__() are sufficient" statement seems not to be true: >>> a < a1 False >>> a <= a1 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unorderable types: A() <= A() >>> a > a1 False >>> a >= a1 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unorderable types: A() >= A() >>> a1 < a2 True >>> a1 <= a2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unorderable types: A() <= A() >>> a1 > a2 False >>> a1 >= a2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unorderable types: A() >= A() On yet another hand, adding __le__() to that class seems to be perfectly sufficient (without adding __gt__() and __ge__()): >>> def le(self, other): ... if hasattr(self, 'x') and hasattr(other, 'x'): ... return self.x <= other.x ... return NotImplemented ... >>> A.__le__ = le >>> a < a1 False >>> a <= a1 True >>> a > a1 False >>> a >= a1 True >>> a1 < a2 True >>> a1 <= a2 True >>> a1 > a2 False >>> a1 >= a2 False What of all this stuff is a language guarantee and what is just an implementation accident? Shouldn't it be documented more accurately? Cheers. *j

2 1

Summary of Python tracker Issues
by Python tracker Sept. 5, 2014

Sept. 5, 2014

ACTIVITY SUMMARY (2014-08-29 - 2014-09-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4640 ( +2) closed 29471 (+40) total 34111 (+42) Open issues with patches: 2191 Issues opened (29) ================== #16037: httplib: header parsing is unlimited http://bugs.python.org/issue16037 reopened by Arfrever #22301: smtplib.SMTP.starttls' documentation is just confusing http://bugs.python.org/issue22301 opened by maker #22302: Windows os.path.isabs UNC path bug http://bugs.python.org/issue22302 opened by akima #22303: Write better test for SSLContext.load_verify_locations http://bugs.python.org/issue22303 opened by pitrou #22305: Documentation on deepcopy's problems is misleading http://bugs.python.org/issue22305 opened by shlsh #22308: add {implementation} to sysconfig.py http://bugs.python.org/issue22308 opened by mattip #22309: distutils/spawn.py handle fork() not implemented. http://bugs.python.org/issue22309 opened by John.Malmberg #22310: Report actual EOF character instead of assuming Ctrl-D http://bugs.python.org/issue22310 opened by John.Malmberg #22313: Make PYLONG_BITS_IN_DIGIT always available to non-core extensi http://bugs.python.org/issue22313 opened by scoder #22314: pydoc.py: TypeError with a $LINES defined to anything http://bugs.python.org/issue22314 opened by arigo #22317: action argument is not documented in argparse add_subparser() http://bugs.python.org/issue22317 opened by Ubik #22319: mailbox.MH chokes on directories without .mh_sequences http://bugs.python.org/issue22319 opened by gumnos #22323: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString( http://bugs.python.org/issue22323 opened by haypo #22324: Use PyUnicode_AsWideCharString() instead of PyUnicode_AsUnicod http://bugs.python.org/issue22324 opened by haypo #22326: tempfile.TemporaryFile fails on NFS v4 filesystems http://bugs.python.org/issue22326 opened by drosera #22327: test_gdb failures on Ubuntu 14.10 http://bugs.python.org/issue22327 opened by barry #22329: Windows installer can't recover partially installed state http://bugs.python.org/issue22329 opened by LlelanD #22330: PyOS_mystricmp is broken http://bugs.python.org/issue22330 opened by kakkoko #22331: test_io.test_interrupted_write_text() hangs on the buildbot Fr http://bugs.python.org/issue22331 opened by haypo #22332: test_multiprocessing_main_handling fail on buildbot "x86 FreeB http://bugs.python.org/issue22332 opened by haypo #22333: test_threaded_import.test_parallel_meta_path() failed on x86 W http://bugs.python.org/issue22333 opened by haypo #22334: test_tcl.test_split() fails on "x86 FreeBSD 7.2 3.x" buildbot http://bugs.python.org/issue22334 opened by haypo #22335: Python 3: Segfault instead of MemoryError when bytearray too b http://bugs.python.org/issue22335 opened by swanson #22336: _tkinter should use Python PyMem_Malloc() instead of Tcl ckall http://bugs.python.org/issue22336 opened by haypo #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 opened by haypo #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 opened by Walkman #22340: Fix Python 3 warnings in Python 2 tests http://bugs.python.org/issue22340 opened by haypo #22341: Python 3 crc32 documentation clarifications http://bugs.python.org/issue22341 opened by vadmium #22342: Fix typo in PEP 380 -- Syntax for Delegating to a Subgenerator http://bugs.python.org/issue22342 opened by Gael.Robin Most recent 15 issues with no replies (15) ========================================== #22342: Fix typo in PEP 380 -- Syntax for Delegating to a Subgenerator http://bugs.python.org/issue22342 #22341: Python 3 crc32 documentation clarifications http://bugs.python.org/issue22341 #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 #22327: test_gdb failures on Ubuntu 14.10 http://bugs.python.org/issue22327 #22317: action argument is not documented in argparse add_subparser() http://bugs.python.org/issue22317 #22314: pydoc.py: TypeError with a $LINES defined to anything http://bugs.python.org/issue22314 #22313: Make PYLONG_BITS_IN_DIGIT always available to non-core extensi http://bugs.python.org/issue22313 #22303: Write better test for SSLContext.load_verify_locations http://bugs.python.org/issue22303 #22300: PEP 446 What's New Updates for 2.7.9 http://bugs.python.org/issue22300 #22294: 2to3 consuming_calls: len, min, max, zip, map, reduce, filter http://bugs.python.org/issue22294 #22289: support.transient_internet() doesn't catch timeout on FTP test http://bugs.python.org/issue22289 #22286: Allow backslashreplace error handler to be used on input http://bugs.python.org/issue22286 #22268: mrohasattr and mrogetattr http://bugs.python.org/issue22268 #22255: Multiprocessing freeze_support raises RuntimeError http://bugs.python.org/issue22255 #22251: Various markup errors in documentation http://bugs.python.org/issue22251 Most recent 15 issues waiting for review (15) ============================================= #22340: Fix Python 3 warnings in Python 2 tests http://bugs.python.org/issue22340 #22339: Incorrect behavior when subclassing enum.Enum http://bugs.python.org/issue22339 #22338: test_json crash on memory allocation failure http://bugs.python.org/issue22338 #22336: _tkinter should use Python PyMem_Malloc() instead of Tcl ckall http://bugs.python.org/issue22336 #22330: PyOS_mystricmp is broken http://bugs.python.org/issue22330 #22324: Use PyUnicode_AsWideCharString() instead of PyUnicode_AsUnicod http://bugs.python.org/issue22324 #22323: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString( http://bugs.python.org/issue22323 #22319: mailbox.MH chokes on directories without .mh_sequences http://bugs.python.org/issue22319 #22308: add {implementation} to sysconfig.py http://bugs.python.org/issue22308 #22301: smtplib.SMTP.starttls' documentation is just confusing http://bugs.python.org/issue22301 #22300: PEP 446 What's New Updates for 2.7.9 http://bugs.python.org/issue22300 #22294: 2to3 consuming_calls: len, min, max, zip, map, reduce, filter http://bugs.python.org/issue22294 #22292: pickle whichmodule RuntimeError http://bugs.python.org/issue22292 #22290: "AMD64 OpenIndiana 3.x" buildbot: assertion failed in PyObject http://bugs.python.org/issue22290 #22289: support.transient_internet() doesn't catch timeout on FTP test http://bugs.python.org/issue22289 Top 10 most discussed issues (10) ================================= #22194: access to cdecimal / libmpdec API http://bugs.python.org/issue22194 11 msgs #21951: tcl test change crashes AIX http://bugs.python.org/issue21951 8 msgs #22298: Lib/warnings.py _show_warning does not protect against being c http://bugs.python.org/issue22298 6 msgs #22299: resolve() on Windows makes some pathological paths unusable http://bugs.python.org/issue22299 6 msgs #22334: test_tcl.test_split() fails on "x86 FreeBSD 7.2 3.x" buildbot http://bugs.python.org/issue22334 6 msgs #17620: Python interactive console doesn't use sys.stdin for input http://bugs.python.org/issue17620 5 msgs #22043: Use a monotonic clock to compute timeouts http://bugs.python.org/issue22043 5 msgs #22166: test_codecs leaks references http://bugs.python.org/issue22166 5 msgs #22257: PEP 432: Redesign the interpreter startup sequence http://bugs.python.org/issue22257 5 msgs #22297: 2.7 json encoding broken for enums http://bugs.python.org/issue22297 5 msgs Issues closed (39) ================== #1598: unexpected response in imaplib http://bugs.python.org/issue1598 closed by terry.reedy #5833: readline update http://bugs.python.org/issue5833 closed by terry.reedy #6588: insert cookies into cookie jar - cookielib.py http://bugs.python.org/issue6588 closed by terry.reedy #10240: dict.update.__doc__ is misleading http://bugs.python.org/issue10240 closed by rhettinger #15418: 2to3 docs should mention setup.py fixes required to install co http://bugs.python.org/issue15418 closed by terry.reedy #16040: nntplib: unlimited readline() from connection http://bugs.python.org/issue16040 closed by terry.reedy #19461: RawConfigParser modifies empty strings unconditionally http://bugs.python.org/issue19461 closed by lukasz.langa #19546: configparser leaks implementation detail http://bugs.python.org/issue19546 closed by lukasz.langa #19826: Document that bug reporting by email is possible http://bugs.python.org/issue19826 closed by brett.cannon #20421: expose SSL socket protocol version http://bugs.python.org/issue20421 closed by pitrou #20957: test_smptnet Fail instead of Skip if SSL-port is unavailable http://bugs.python.org/issue20957 closed by haypo #21186: RawConfigParser __name__ option handling inconsistent http://bugs.python.org/issue21186 closed by lukasz.langa #21204: multiprocessing example does not work on Windows http://bugs.python.org/issue21204 closed by terry.reedy #21527: concurrent.futures.ThreadPoolExecutor does not use a default v http://bugs.python.org/issue21527 closed by python-dev #21933: Allow the user to change font sizes with the text pane of turt http://bugs.python.org/issue21933 closed by terry.reedy #22098: Behavior of Structure inconsistent with BigEndianStructure whe http://bugs.python.org/issue22098 closed by pitrou #22185: Occasional RuntimeError from Condition.notify http://bugs.python.org/issue22185 closed by pitrou #22216: smtplip STARTTLS fails at second attampt due to unsufficiant q http://bugs.python.org/issue22216 closed by r.david.murray #22221: ast.literal_eval confused by coding declarations http://bugs.python.org/issue22221 closed by serhiy.storchaka #22258: set_inheritable(): ioctl(FIOCLEX) is available but fails with http://bugs.python.org/issue22258 closed by python-dev #22270: cache version selection for documentation http://bugs.python.org/issue22270 closed by terry.reedy #22275: asyncio: enhance documentation of OS support http://bugs.python.org/issue22275 closed by python-dev #22283: "AMD64 FreeBSD 9.0 3.x" fails to build the _decimal module: #e http://bugs.python.org/issue22283 closed by skrah #22296: cookielib uses time.time(), making incorrect checks of expirat http://bugs.python.org/issue22296 closed by regu0004 #22304: Update multiprocessing examples to Py3 and test http://bugs.python.org/issue22304 closed by terry.reedy #22306: Inconsistent division by 0 behaviour in decimal module http://bugs.python.org/issue22306 closed by rhettinger #22307: os.getlogin() documentation has misleading side note http://bugs.python.org/issue22307 closed by python-dev #22311: Pip 404's http://bugs.python.org/issue22311 closed by pitrou #22312: ntpath.splitdrive('//') -> IndexError http://bugs.python.org/issue22312 closed by python-dev #22315: TypeError in error handling in distutils.dir_util.copy_tree http://bugs.python.org/issue22315 closed by python-dev #22316: Add rule about "extraneous whitespace around colon" to "Whites http://bugs.python.org/issue22316 closed by gvanrossum #22318: unitest documentation on assertItemsEqual misleading http://bugs.python.org/issue22318 closed by r.david.murray #22320: Invalid link in General Python FAQ http://bugs.python.org/issue22320 closed by ned.deily #22321: odd result for datetime.time.strftime("%s") http://bugs.python.org/issue22321 closed by r.david.murray #22322: Zip files created by `git archive` result in a SyntaxError (du http://bugs.python.org/issue22322 closed by r.david.murray #22325: wrong subtraction result http://bugs.python.org/issue22325 closed by r.david.murray #22328: ur'foo\d' raises SyntaxError http://bugs.python.org/issue22328 closed by r.david.murray #22337: Modulus not returning proper remainder http://bugs.python.org/issue22337 closed by r.david.murray #1660009: continuing problem with httplib multiple set-cookie headers http://bugs.python.org/issue1660009 closed by terry.reedy

1 0

PEP 477: selected ensurepip backports for Python 2.7
by Nick Coghlan Sept. 4, 2014

Sept. 4, 2014

Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 maintenance release in addition to including it with Python 3.4. That part of the proposal proved to be controversial, so we dropped it from the original PEP in order to focus on meeting the Python 3.4 specific release deadlines. This also had the benefit of working out the kinks in the bootstrapping processing as part of the Python 3.4 release cycle. However, we still think we should start providing pip by default to Python 2.7 users as well, at least as part of the Windows and Mac OS X installers. One notable difference from PEP 453 is that because there is no venv module in 2.7, and hence no integration between venv and ensurepip, we can give redistributors the option of just disabling ensurepip entirely and redirecting users to platform specific installation tools. Regards, Nick. ====================== PEP: 477 Title: Backport ensurepip (PEP 453) to Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald(a)stufft.io> Nick Coghlan <ncoghlan(a)gmail.com> Status: Active Type: Process Content-Type: text/x-rst Created: 26-Aug-2014 Post-History: 1-Sep-2014 Abstract ======== This PEP proposes that the ``ensurepip`` module, added to Python 3.4 by PEP 453, be backported to Python 2.7. It also proposes that automatic invocation of ``ensurepip`` be added to the Python 2.7 Windows and OSX installers. However it does **not** propose that automatic invocation be added to the ``Makefile``. It also proposes that the documentation changes for the package distribution and installation guides be updated to match that in 3.4, which references using the ``ensurepip`` module to bootstrap the installer. Rationale ========= Python 2.7 is effectively a LTS release of Python which represents the end of the 2.x series and there is still a very large contingent of users whom are still using Python 2.7 as their primary version. These users, in order to participate in the wider Python ecosystem, must manually attempt to go out and find the correct way to bootstrap the packaging tools. It is the opinion of this PEP that making it as easy as possible for end users to participate in the wider Python ecosystem is important for 3 primary reasons: 1. The Python 2.x to 3.x migration has a number of painpoints that are eased by a number of third party modules such as six [#six]_, modernize [#modernize]_, or future [#future]_. However relying on these tools requires that everyone who uses the project have a tool to install these packages. 2. In addition to tooling to aid in migration from Python 2.x to 3.x, there are also a number of modules that are *new* in Python 3 for which there are backports available on PyPI. This can also aid in the ability for people to write 2.x and 3.x compatible software as well as enable them to use some of the newer features of Python 3 on Python 2. 3. Users also will need a number of tools in order to create python packages that conform to the newer standards that are being proposed. Things like setuptools [#setuptools]_, Wheel [#wheel]_, and twine [#twine]_ are enabling a safer, faster, and more reliable packaging tool chain. These tools can be difficult for people to use if first they must be told how to go out and install the package manager. 4. One of Pythons biggest strengths is in the huge ecosystem of libraries and projects that have been built on top of it, most of which are distributed through PyPI. However in order to benefit from this wide ecosystem meaningfully requires end users, some of which are going to be new, to make a decision on which package manager they should get, how to get it, and then finally actually installing it first. Furthermore, alternative implementations of Python are recognizing the benefits of PEP 453 and both PyPy and Jython have plans to backport ensurepip to their 2.7 runtimes. Automatic Invocation ==================== PEP 453 has ``ensurepip`` automatically invoked by default in the ``Makefile`` and the Windows and OSX Installers. This allowed it to ensure that, by default, all users would get Python with pip already installed. This PEP however believes that while this is fine for the Python 2.7 Windows and Mac OS X installers it is *not* ok for the Python 2.7 ``Makefile`` in general. The primary consumers of the ``Makefile`` are downstream package managers which distribute Python themselves. These downstream distributors typically do not want pip to be installed via ``ensurepip`` and would prefer that end users install it with their own package manager. Not invoking ``ensurepip`` automatically from the ``Makefile`` would allow these distributors to simply ignore the fact that ``ensurepip`` has been backported and still not end up with pip installed via it. The primary consumers of the OSX and Windows installers are end users who are attempting to install Python on their own machine. There is not a package manager available where these users can install pip into their Python through a more supported mechanism. For this reason it is the belief of this PEP that installing by default on OSX and Windows is the best course of action. Documentation ============= As part of this PEP, the updated packaging distribution and installation guides for Python 3.4 would be backported to Python 2.7. Disabling ensurepip by Downstream Distributors ============================================== Due to its use in the ``venv`` module, downstream distributors cannot disable the ``ensurepip`` module in Python 3.4. However since Python 2.7 has no such module it is explicitly allowed for downstream distributors to patch the ``ensurepip`` module to prevent it from installing anything. If a downstream distributor wishes to disable ``ensurepip`` completely in Python 2.7, they should still at least provide the module and allow `python -m ensurepip` style invocation. However it should raise errors or otherwise exit with a non-zero exit code and print out an error on stderr directing users to what they can/should use instead of ``ensurepip``. References ========== .. [#six] `six.py <https://pypi.python.org/pypi/six>`__ .. [#modernize] `modernize <https://pypi.python.org/pypi/modernize>`__ .. [#future] `python-future <https://pypi.python.org/pypi/future>`__ .. [#setuptools] `setuptools <https://pypi.python.org/pypi/setuptools>`__ .. [#wheel] `Wheel <https://pypi.python.org/pypi/wheel>`__ .. [#twine] `twine <https://pypi.python.org/pypi/twine>`__ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 11

RFC: PEP 475, Retry system calls failing with EINTR
by Victor Stinner Sept. 3, 2014

Sept. 3, 2014

HTML version: http://legacy.python.org/dev/peps/pep-0475/ PEP: 475 Title: Retry system calls failing with EINTR Version: $Revision$ Last-Modified: $Date$ Author: Charles-François Natali <cf.natali(a)gmail.com>, Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-July-2014 Python-Version: 3.5 Abstract ======== Retry system calls failing with the ``EINTR`` error and recompute timeout if needed. Rationale ========= Interrupted system calls ------------------------ On POSIX systems, signals are common. Your program must be prepared to handle them. Examples of signals: * The most common signal is ``SIGINT``, signal sent when CTRL+c is pressed. By default, Python raises a ``KeyboardInterrupt`` exception when this signal is received. * When running subprocesses, the ``SIGCHLD`` signal is sent when a child process exits. * Resizing the terminal sends the ``SIGWINCH`` signal to the applications running in the terminal. * Putting the application in background (ex: press CTRL-z and then type the ``bg`` command) sends the ``SIGCONT`` signal. Writing a signal handler is difficult, only "async-signal safe" functions can be called. For example, ``printf()`` and ``malloc()`` are not async-signal safe. When a signal is sent to a process calling a system call, the system call can fail with the ``EINTR`` error to give the program an opportunity to handle the signal without the restriction on signal safe functions. Depending on the platform, on the system call and the ``SA_RESTART`` flag, the system call may or may not fail with ``EINTR``. If the signal handler was set with the ``SA_RESTART`` flag set, the kernel retries some the system call instead of failing with ``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is not retried. The Python function ``signal.signal()`` clears the ``SA_RESTART`` flag when setting the signal handler: all system calls should fail with ``EINTR`` in Python. The problem is that handling ``EINTR`` should be done for all system calls. The problem is similar to handling errors in the C language which does not have exceptions: you must check all function returns to check for error, and usually duplicate the code checking for errors. Python does not have this issue, it uses exceptions to notify errors. Current status -------------- Currently in Python, the code to handle the ``InterruptedError`` exception (``EINTR`` error) is duplicated on case by case. Only a few Python modules handle this exception, and fixes usually took several years to cover a whole module. Example of code retrying ``file.read()`` on ``InterruptedError``:: while True: try: data = file.read(size) break except InterruptedError: continue List of Python modules of the standard library which handle ``InterruptedError``: * ``asyncio`` * ``asyncore`` * ``io``, ``_pyio`` * ``multiprocessing`` * ``selectors`` * ``socket`` * ``socketserver`` * ``subprocess`` Other programming languages like Perl, Java and Go already retry system calls failing with ``EINTR``. Use Case 1: Don't Bother With Signals ------------------------------------- In most cases, you don't want to be interrupted by signals and you don't expect to get ``InterruptedError`` exceptions. For example, do you really want to write such complex code for an "Hello World" example? :: while True: try: print("Hello World") break except InterruptedError: continue ``InterruptedError`` can happen in unexpected places. For example, ``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``: see the article `close() and EINTR <http://alobbs.com/post/54503240599/close-and-eintr>`_. The `Python issues related to EINTR`_ section below gives examples of bugs caused by "EINTR". The expectation is that Python hides the ``InterruptedError``: retry system calls failing with the ``EINTR`` error. Use Case 2: Be notified of signals as soon as possible ------------------------------------------------------ Sometimes, you expect some signals and you want to handle them as soon as possible. For example, you may want to quit immediatly a program using the ``CTRL+c`` keyboard shortcut. Some signals are not interesting and should not interrupt the the application. There are two options to only interrupt an application on some signals: * Raise an exception in the signal handler, like ``KeyboardInterrupt`` for ``SIGINT`` * Use a I/O multiplexing function like ``select()`` with the Python signal "wakeup" file descriptor: see the function ``signal.set_wakeup_fd()``. Proposition =========== If a system call fails with ``EINTR``, Python must call signal handlers: call ``PyErr_CheckSignals()``. If a signal handler raises an exception, the Python function fails with the exception. Otherwise, the system call is retried. If the system call takes a timeout parameter, the timeout is recomputed. Modified functions ------------------ Example of functions that need to be modified: * ``os.read()``, ``io.FileIO.read()``, ``io.FileIO.readinto()`` * ``os.write()``, ``io.FileIO.write()`` * ``os.waitpid()`` * ``socket.accept()`` * ``socket.connect()`` * ``socket.recv()``, ``socket.recv_into()`` * ``socket.recv_from()`` * ``socket.send()`` * ``socket.sendto()`` * ``time.sleep()`` * ``select.select()`` * ``select.poll()`` * ``select.epoll.poll()`` * ``select.devpoll.poll()`` * ``select.kqueue.control()`` * ``selectors.SelectSelector.select()`` and other selector classes Note: The ``selector`` module already retries on ``InterruptedError``, but it doesn't recompute the timeout yet. Backward Compatibility ====================== Applications relying on the fact that system calls are interrupted with ``InterruptedError`` will hang. The authors of this PEP don't think that such application exist. If such applications exist, they are not portable and are subject to race conditions (deadlock if the signal comes before the system call). These applications must be fixed to handle signals differently, to have a reliable behaviour on all platforms and all Python versions. For example, use a signal handler which raises an exception, or use a wakeup file descriptor. For applications using event loops, ``signal.set_wakeup_fd()`` is the recommanded option to handle signals. The signal handler writes signal numbers into the file descriptor and the event loop is awaken to read them. The event loop can handle these signals without the restriction of signal handlers. Appendix ======== Wakeup file descriptor ---------------------- Since Python 3.3, ``signal.set_wakeup_fd()`` writes the signal number into the file descriptor, whereas it only wrote a null byte before. It becomes possible to handle different signals using the wakeup file descriptor. Linux has a ``signalfd()`` which provides more information on each signal. For example, it's possible to know the pid and uid who sent the signal. This function is not exposed in Python yet (see the `issue 12304 <http://bugs.python.org/issue12304>`_). On Unix, the ``asyncio`` module uses the wakeup file descriptor to wake up its event loop. Multithreading -------------- A C signal handler can be called from any thread, but the Python signal handler should only be called in the main thread. Python has a ``PyErr_SetInterrupt()`` function which calls the ``SIGINT`` signal handler to interrupt the Python main thread. Signals on Windows ------------------ Control events ^^^^^^^^^^^^^^ Windows uses "control events": * ``CTRL_BREAK_EVENT``: Break (``SIGBREAK``) * ``CTRL_CLOSE_EVENT``: Close event * ``CTRL_C_EVENT``: CTRL+C (``SIGINT``) * ``CTRL_LOGOFF_EVENT``: Logoff * ``CTRL_SHUTDOWN_EVENT``: Shutdown The `SetConsoleCtrlHandler() function <http://msdn.microsoft.com/en-us/library/windows/desktop/ms686016%28v=vs.85%…>`_ can be used to install a control handler. The ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` events can be sent to a process using the `GenerateConsoleCtrlEvent() function <http://msdn.microsoft.com/en-us/library/windows/desktop/ms683155%28v=vs.85%…>`_. This function is exposed in Python as ``os.kill()``. Signals ^^^^^^^ The following signals are supported on Windows: * ``SIGABRT`` * ``SIGBREAK`` (``CTRL_BREAK_EVENT``): signal only available on Windows * ``SIGFPE`` * ``SIGILL`` * ``SIGINT`` (``CTRL_C_EVENT``) * ``SIGSEGV`` * ``SIGTERM`` SIGINT ^^^^^^ The default Python signal handler for ``SIGINT`` sets a Windows event object: ``sigint_event``. ``time.sleep()`` is implemented with ``WaitForSingleObjectEx()``, it waits for the ``sigint_event`` object using ``time.sleep()`` parameter as the timeout. So the sleep can be interrupted by ``SIGINT``. ``_winapi.WaitForMultipleObjects()`` automatically adds ``sigint_event`` to the list of watched handles, so it can also be interrupted. ``PyOS_StdioReadline()`` also used ``sigint_event`` when ``fgets()`` failed to check if Ctrl-C or Ctrl-Z was pressed. Links ----- Misc ^^^^ * `glibc manual: Primitives Interrupted by Signals <http://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.ht…>`_ * `Bug #119097 for perl5: print returning EINTR in 5.14 <https://rt.perl.org/Public/Bug/Display.html?id=119097>`_. Python issues related to EINTR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The main issue is: `handle EINTR in the stdlib <http://bugs.python.org/issue18885>`_. Open issues: * `Add a new signal.set_wakeup_socket() function <http://bugs.python.org/issue22018>`_ * `signal.set_wakeup_fd(fd): set the fd to non-blocking mode <http://bugs.python.org/issue22042>`_ * `Use a monotonic clock to compute timeouts <http://bugs.python.org/issue22043>`_ * `sys.stdout.write on OS X is not EINTR safe <http://bugs.python.org/issue22007>`_ * `platform.uname() not EINTR safe <http://bugs.python.org/issue21772>`_ * `asyncore does not handle EINTR in recv, send, connect, accept, <http://bugs.python.org/issue11266>`_ * `socket.create_connection() doesn't handle EINTR properly <http://bugs.python.org/issue20611>`_ Closed issues: * `Interrupted system calls are not retried <http://bugs.python.org/issue9867>`_ * `Solaris: EINTR exception in select/socket calls in telnetlib <http://bugs.python.org/issue1049450>`_ * `subprocess: Popen.communicate() doesn't handle EINTR in some cases <http://bugs.python.org/issue12493>`_ * `multiprocessing.util._eintr_retry doen't recalculate timeouts <http://bugs.python.org/issue12338>`_ * `file readline, readlines & readall methods can lose data on EINTR <http://bugs.python.org/issue12268>`_ * `multiprocessing BaseManager serve_client() does not check EINTR on recv <http://bugs.python.org/issue17097>`_ * `selectors behaviour on EINTR undocumented <http://bugs.python.org/issue19849>`_ * `asyncio: limit EINTR occurrences with SA_RESTART <http://bugs.python.org/issue19850>`_ * `smtplib.py socket.create_connection() also doesn't handle EINTR properly <http://bugs.python.org/issue21602>`_ * `Faulty RESTART/EINTR handling in Parser/myreadline.c <http://bugs.python.org/issue11650>`_ * `test_httpservers intermittent failure, test_post and EINTR <http://bugs.python.org/issue3771>`_ * `os.spawnv(P_WAIT, ...) on Linux doesn't handle EINTR <http://bugs.python.org/issue686667>`_ * `asyncore fails when EINTR happens in pol <http://bugs.python.org/issue517554>`_ * `file.write and file.read don't handle EINTR <http://bugs.python.org/issue10956>`_ * `socket.readline() interface doesn't handle EINTR properly <http://bugs.python.org/issue1628205>`_ * `subprocess is not EINTR-safe <http://bugs.python.org/issue1068268>`_ * `SocketServer doesn't handle syscall interruption <http://bugs.python.org/issue7978>`_ * `subprocess deadlock when read() is interrupted <http://bugs.python.org/issue17367>`_ * `time.sleep(1): call PyErr_CheckSignals() if the sleep was interrupted <http://bugs.python.org/issue12462>`_ * `siginterrupt with flag=False is reset when signal received <http://bugs.python.org/issue8354>`_ * `need siginterrupt() on Linux - impossible to do timeouts <http://bugs.python.org/issue1089358>`_ * `[Windows] Can not interrupt time.sleep() <http://bugs.python.org/issue581232>`_ Python issues related to signals ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Open issues: * `signal.default_int_handler should set signal number on the raised exception <http://bugs.python.org/issue17182>`_ * `expose signalfd(2) in the signal module <http://bugs.python.org/issue12304>`_ * `missing return in win32_kill? <http://bugs.python.org/issue14484>`_ * `Interrupts are lost during readline PyOS_InputHook processing <http://bugs.python.org/issue3180>`_ * `cannot catch KeyboardInterrupt when using curses getkey() <http://bugs.python.org/issue1687125>`_ * `Deferred KeyboardInterrupt in interactive mode <http://bugs.python.org/issue16151>`_ Closed issues: * `sys.interrupt_main() <http://bugs.python.org/issue753733>`_ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 37

https:bugs.python.org -- Untrusted Connection (Firefox)
by Terry Reedy Sept. 2, 2014

Sept. 2, 2014

Firefox does not want to connect to https:bugs.python.org. Plain bugs.python.org works fine. Has the certificate expired? -- Terry Jan Reedy

10 18

Re: [Python-Dev] Daily reference leaks (640c575ab3e1): sum=151940
by Antoine Pitrou Sept. 1, 2014

Sept. 1, 2014

Is anyone working on those? On Mon, 01 Sep 2014 10:41:45 +0200 solipsis(a)pitrou.net wrote: > results for 640c575ab3e1 on branch "default" > -------------------------------------------- > > test_codecs leaked [5825, 5825, 5825] references, sum=17475 > test_codecs leaked [1172, 1174, 1174] memory blocks, sum=3520 > test_collections leaked [0, 2, 0] references, sum=2 > test_distutils leaked [37735, 37735, 37735] references, sum=113205 > test_distutils leaked [5909, 5911, 5911] memory blocks, sum=17731 > test_functools leaked [0, 0, 3] memory blocks, sum=3 > test_site leaked [0, 0, 2] references, sum=2 > test_site leaked [0, 0, 2] memory blocks, sum=2 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogHPiXJp', '-x']

2 1

Bug 19494 ... urllib2.HTTPBasicAuthHandler for GitHub et al.
by Matěj Cepl Sept. 1, 2014

Sept. 1, 2014

Hi, now when the vacations even in Europe are over could I ask for some movement on http://bugs.python.org/issue19494? Demanding a half-megabyte amount of packages from PIP ('just use requests' mentioned by some comments in the thread) or for that matter any package from PIP (including mine https://pypi.python.org/pypi/urllib2_prior_auth/) for something which is essentially a ten-lines patch seems to me rather crazy. What happened to all those batteries which were supposed to be included? Best, Matěj

1 0