Mailman 3 December 2014 - Python-Dev

PEP 479 (Change StopIteration handling inside generators) -- hopefully final text
by Guido van Rossum Dec. 6, 2014

Dec. 6, 2014

For those who haven't followed along, here's the final text of PEP 479, with a brief Acceptance section added. The basic plan hasn't changed, but there's a lot more clarifying text and discussion of a few counter-proposals. Please send suggestions for editorial improvements to peps(a)python.org. The official reference version of the PEP is at https://www.python.org/dev/peps/pep-0479/; the repo is https://hg.python.org/peps/ (please check out the repo and send diffs relative to the repo if you have edits). PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico <rosuav(a)gmail.com>, Guido van Rossum < guido(a)python.org> Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014, 19-Nov-2014, 5-Dec-2014 Abstract ======== This PEP proposes a change to generators: when ``StopIteration`` is raised inside a generator, it is replaced it with ``RuntimeError``. (More precisely, this happens when the exception is about to bubble out of the generator's stack frame.) Because the change is backwards incompatible, the feature is initially introduced using a ``__future__`` statement. Acceptance ========== This PEP was accepted by the BDFL on November 22. Because of the exceptionally short period from first draft to acceptance, the main objections brought up after acceptance were carefully considered and have been reflected in the "Alternate proposals" section below. However, none of the discussion changed the BDFL's mind and the PEP's acceptance is now final. (Suggestions for clarifying edits are still welcome -- unlike IETF RFCs, the text of a PEP is not cast in stone after its acceptance, although the core design/plan/specification should not change after acceptance.) Rationale ========= The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct. The main goal of the proposal is to ease debugging in the situation where an unguarded ``next()`` call (perhaps several stack frames deep) raises ``StopIteration`` and causes the iteration controlled by the generator to terminate silently. (When another exception is raised, a traceback is printed pinpointing the cause of the problem.) This is particularly pernicious in combination with the ``yield from`` construct of PEP 380 [1]_, as it breaks the abstraction that a subgenerator may be factored out of a generator. That PEP notes this limitation, but notes that "use cases for these [are] rare to non- existent". Unfortunately while intentional use is rare, it is easy to stumble on these cases by accident:: import contextlib @contextlib.contextmanager def transaction(): print('begin') try: yield from do_it() except: print('rollback') raise else: print('commit') def do_it(): print('Refactored preparations') yield # Body of with-statement is executed here print('Refactored finalization') def gene(): for i in range(2): with transaction(): yield i # return raise StopIteration # This is wrong print('Should not be reached') for i in gene(): print('main: i =', i) Here factoring out ``do_it`` into a subgenerator has introduced a subtle bug: if the wrapped block raises ``StopIteration``, under the current behavior this exception will be swallowed by the context manager; and, worse, the finalization is silently skipped! Similarly problematic behavior occurs when an ``asyncio`` coroutine raises ``StopIteration``, causing it to terminate silently. Additionally, the proposal reduces the difference between list comprehensions and generator expressions, preventing surprises such as the one that started this discussion [2]_. Henceforth, the following statements will produce the same result if either produces a result at all:: a = list(F(x) for x in xs if P(x)) a = [F(x) for x in xs if P(x)] With the current state of affairs, it is possible to write a function ``F(x)`` or a predicate ``P(x)`` that causes the first form to produce a (truncated) result, while the second form raises an exception (namely, ``StopIteration``). With the proposed change, both forms will raise an exception at this point (albeit ``RuntimeError`` in the first case and ``StopIteration`` in the second). Finally, the proposal also clears up the confusion about how to terminate a generator: the proper way is ``return``, not ``raise StopIteration``. As an added bonus, the above changes bring generator functions much more in line with regular functions. If you wish to take a piece of code presented as a generator and turn it into something else, you can usually do this fairly simply, by replacing every ``yield`` with a call to ``print()`` or ``list.append()``; however, if there are any bare ``next()`` calls in the code, you have to be aware of them. If the code was originally written without relying on ``StopIteration`` terminating the function, the transformation would be that much easier. Background information ====================== When a generator frame is (re)started as a result of a ``__next__()`` (or ``send()`` or ``throw()``) call, one of three outcomes can occur: * A yield point is reached, and the yielded value is returned. * The frame is returned from; ``StopIteration`` is raised. * An exception is raised, which bubbles out. In the latter two cases the frame is abandoned (and the generator object's ``gi_frame`` attribute is set to None). Proposal ======== If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with ``RuntimeError``, which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. >From then on it's just like any old exception. [4]_ This affects the third outcome listed above, without altering any other effects. Furthermore, it only affects this outcome when the exception raised is ``StopIteration`` (or a subclass thereof). Note that the proposed replacement happens at the point where the exception is about to bubble out of the frame, i.e. after any ``except`` or ``finally`` blocks that could affect it have been exited. The ``StopIteration`` raised by returning from the frame is not affected (the point being that ``StopIteration`` means that the generator terminated "normally", i.e. it did not raise an exception). A subtle issue is what will happen if the caller, having caught the ``RuntimeError``, calls the generator object's ``__next__()`` method again. The answer is that it from this point on it will raise ``StopIteration`` -- the behavior is the same as when any other exception was raised by the generator. Another logical consequence of the proposal: if somone uses ``g.throw(StopIteration)`` to throw a ``StopIteration`` exception into a generator, if the generator doesn't catch it (which it could do using a ``try/except`` around the ``yield``), it will be transformed into ``RuntimeError``. During the transition phase, the new feature must be enabled per-module using:: from __future__ import generator_stop Any generator function constructed under the influence of this directive will have the ``REPLACE_STOPITERATION`` flag set on its code object, and generators with the flag set will behave according to this proposal. Once the feature becomes standard, the flag may be dropped; code should not inspect generators for it. Consequences for existing code ============================== This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [3]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail. Other examples abound, e.g. [6]_, [7]_. (Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""") There are also examples of generator expressions floating around that rely on a ``StopIteration`` raised by the expression, the target or the predicate (rather than by the ``__next__()`` call implied in the ``for`` loop proper). Writing backwards and forwards compatible code ---------------------------------------------- With the exception of hacks that raise ``StopIteration`` to exit a generator expression, it is easy to write code that works equally well under older Python versions as under the new semantics. This is done by enclosing those places in the generator body where a ``StopIteration`` is expected (e.g. bare ``next()`` calls or in some cases helper functions that are expected to raise ``StopIteration``) in a ``try/except`` construct that returns when ``StopIteration`` is raised. The ``try/except`` construct should appear directly in the generator function; doing this in a helper function that is not itself a generator does not work. If ``raise StopIteration`` occurs directly in a generator, simply replace it with ``return``. Examples of breakage -------------------- Generators which explicitly raise ``StopIteration`` can generally be changed to simply return instead. This will be compatible with all existing Python versions, and will not be affected by ``__future__``. Here are some illustrations from the standard library. Lib/ipaddress.py:: if other == self: raise StopIteration Becomes:: if other == self: return In some cases, this can be combined with ``yield from`` to simplify the code, such as Lib/difflib.py:: if context is None: while True: yield next(line_pair_iterator) Becomes:: if context is None: yield from line_pair_iterator return (The ``return`` is necessary for a strictly-equivalent translation, though in this particular file, there is no further code, and the ``return`` can be omitted.) For compatibility with pre-3.3 versions of Python, this could be written with an explicit ``for`` loop:: if context is None: for line in line_pair_iterator: yield line return More complicated iteration patterns will need explicit ``try/except`` constructs. For example, a hypothetical parser like this:: def parser(f): while True: data = next(f) while True: line = next(f) if line == "- end -": break data += line yield data would need to be rewritten as:: def parser(f): while True: try: data = next(f) while True: line = next(f) if line == "- end -": break data += line yield data except StopIteration: return or possibly:: def parser(f): for data in f: while True: line = next(f) if line == "- end -": break data += line yield data The latter form obscures the iteration by purporting to iterate over the file with a ``for`` loop, but then also fetches more data from the same iterator during the loop body. It does, however, clearly differentiate between a "normal" termination (``StopIteration`` instead of the initial line) and an "abnormal" termination (failing to find the end marker in the inner loop, which will now raise ``RuntimeError``). This effect of ``StopIteration`` has been used to cut a generator expression short, creating a form of ``takewhile``:: def stop(): raise StopIteration print(list(x for x in range(10) if x < 5 or stop())) # prints [0, 1, 2, 3, 4] Under the current proposal, this form of non-local flow control is not supported, and would have to be rewritten in statement form:: def gen(): for x in range(10): if x >= 5: return yield x print(list(gen())) # prints [0, 1, 2, 3, 4] While this is a small loss of functionality, it is functionality that often comes at the cost of readability, and just as ``lambda`` has restrictions compared to ``def``, so does a generator expression have restrictions compared to a generator function. In many cases, the transformation to full generator function will be trivially easy, and may improve structural clarity. Explanation of generators, iterators, and StopIteration ======================================================= Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. An iterator is an object with a ``__next__`` method. Like many other special methods, it may either return a value, or raise a specific exception - in this case, ``StopIteration`` - to signal that it has no value to return. In this, it is similar to ``__getattr__`` (can raise ``AttributeError``), ``__getitem__`` (can raise ``KeyError``), and so on. A helper function for an iterator can be written to follow the same protocol; for example:: def helper(x, y): if x > y: return 1 / (x - y) raise StopIteration def __next__(self): if self.a: return helper(self.b, self.c) return helper(self.d, self.e) Both forms of signalling are carried through: a returned value is returned, an exception bubbles up. The helper is written to match the protocol of the calling function. A generator function is one which contains a ``yield`` expression. Each time it is (re)started, it may either yield a value, or return (including "falling off the end"). A helper function for a generator can also be written, but it must also follow generator protocol:: def helper(x, y): if x > y: yield 1 / (x - y) def gen(self): if self.a: return (yield from helper(self.b, self.c)) return (yield from helper(self.d, self.e)) In both cases, any unexpected exception will bubble up. Due to the nature of generators and iterators, an unexpected ``StopIteration`` inside a generator will be converted into ``RuntimeError``, but beyond that, all exceptions will propagate normally. Transition plan =============== - Python 3.5: Enable new semantics under ``__future__`` import; silent deprecation warning if ``StopIteration`` bubbles out of a generator not under ``__future__`` import. - Python 3.6: Non-silent deprecation warning. - Python 3.7: Enable new semantics everywhere. Alternate proposals =================== Raising something other than RuntimeError ----------------------------------------- Rather than the generic ``RuntimeError``, it might make sense to raise a new exception type ``UnexpectedStopIteration``. This has the downside of implicitly encouraging that it be caught; the correct action is to catch the original ``StopIteration``, not the chained exception. Supplying a specific exception to raise on return ------------------------------------------------- Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed. This subproposal has been withdrawn in favour of better options, but is retained for reference. Making return-triggered StopIterations obvious ---------------------------------------------- For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` (``GeneratorReturn``) which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement. The inspiration for this alternative proposal was Nick's observation [8]_ that if an ``asyncio`` coroutine [9]_ accidentally raises ``StopIteration``, it currently terminates silently, which may present a hard-to-debug mystery to the developer. The main proposal turns such accidents into clearly distinguishable ``RuntimeError`` exceptions, but if that is rejected, this alternate proposal would enable ``asyncio`` to distinguish between a ``return`` statement and an accidentally-raised ``StopIteration`` exception. Of the three outcomes listed above, two change: * If a yield point is reached, the value, obviously, would still be returned. * If the frame is returned from, ``GeneratorReturn`` (rather than ``StopIteration``) is raised. * If an instance of ``GeneratorReturn`` would be raised, instead an instance of ``StopIteration`` would be raised. Any other exception bubbles up normally. In the third case, the ``StopIteration`` would have the ``value`` of the original ``GeneratorReturn``, and would reference the original exception in its ``__cause__``. If uncaught, this would clearly show the chaining of exceptions. This alternative does *not* affect the discrepancy between generator expressions and list comprehensions, but allows generator-aware code (such as the ``contextlib`` and ``asyncio`` modules) to reliably differentiate between the second and third outcomes listed above. However, once code exists that depends on this distinction between ``GeneratorReturn`` and ``StopIteration``, a generator that invokes another generator and relies on the latter's ``StopIteration`` to bubble out would still be potentially wrong, depending on the use made of the distinction between the two exception types. Converting the exception inside next() -------------------------------------- Mark Shannon suggested [12]_ that the problem could be solved in ``next()`` rather than at the boundary of generator functions. By having ``next()`` catch ``StopIteration`` and raise instead ``ValueError``, all unexpected ``StopIteration`` bubbling would be prevented; however, the backward-incompatibility concerns are far more serious than for the current proposal, as every ``next()`` call now needs to be rewritten to guard against ``ValueError`` instead of ``StopIteration`` - not to mention that there is no way to write one block of code which reliably works on multiple versions of Python. (Using a dedicated exception type, perhaps subclassing ``ValueError``, would help this; however, all code would still need to be rewritten.) Sub-proposal: decorator to explicitly request current behaviour --------------------------------------------------------------- Nick Coghlan suggested [13]_ that the situations where the current behaviour is desired could be supported by means of a decorator:: from itertools import allow_implicit_stop @allow_implicit_stop def my_generator(): ... yield next(it) ... Which would be semantically equivalent to:: def my_generator(): try: ... yield next(it) ... except StopIteration return but be faster, as it could be implemented by simply permitting the ``StopIteration`` to bubble up directly. Single-source Python 2/3 code would also benefit in a 3.7+ world, since libraries like six and python-future could just define their own version of "allow_implicit_stop" that referred to the new builtin in 3.5+, and was implemented as an identity function in other versions. However, due to the implementation complexities required, the ongoing compatibility issues created, the subtlety of the decorator's effect, and the fact that it would encourage the "quick-fix" solution of just slapping the decorator onto all generators instead of properly fixing the code in question, this sub-proposal has been rejected. [14]_ Criticism ========= Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [5]_ Code does exist which relies on the current behaviour (e.g. [3]_, [6]_, [7]_), and there is the concern that this would be unnecessary code churn to achieve little or no gain. Steven D'Aprano started an informal survey on comp.lang.python [10]_; at the time of writing only two responses have been received: one was in favor of changing list comprehensions to match generator expressions (!), the other was in favor of this PEP's main proposal. The existing model has been compared to the perfectly-acceptable issues inherent to every other case where an exception has special meaning. For instance, an unexpected ``KeyError`` inside a ``__getitem__`` method will be interpreted as failure, rather than permitted to bubble up. However, there is a difference. Special methods use ``return`` to indicate normality, and ``raise`` to signal abnormality; generators ``yield`` to indicate data, and ``return`` to signal the abnormal state. This makes explicitly raising ``StopIteration`` entirely redundant, and potentially surprising. If other special methods had dedicated keywords to distinguish between their return paths, they too could turn unexpected exceptions into ``RuntimeError``; the fact that they cannot should not preclude generators from doing so. References ========== .. [1] PEP 380 - Syntax for Delegating to a Subgenerator (https://www.python.org/dev/peps/pep-0380) .. [2] Initial mailing list comment (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html ) .. [3] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby) .. [4] Proposal by GvR (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html ) .. [5] Response by Steven D'Aprano (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html ) .. [6] Split a sequence or generator using a predicate ( http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-us… ) .. [7] wrap unbounded generator to restrict its output ( http://code.activestate.com/recipes/66427-wrap-unbounded-generator-to-restr… ) .. [8] Post from Nick Coghlan mentioning asyncio (https://mail.python.org/pipermail/python-ideas/2014-November/029961.html ) .. [9] Coroutines in asyncio (https://docs.python.org/3/library/asyncio-task.html#coroutines) .. [10] Thread on comp.lang.python started by Steven D'Aprano (https://mail.python.org/pipermail/python-list/2014-November/680757.html) .. [11] Tracker issue with Proof-of-Concept patch (http://bugs.python.org/issue22906) .. [12] Post from Mark Shannon with alternate proposal (https://mail.python.org/pipermail/python-dev/2014-November/137129.html) .. [13] Idea from Nick Coghlan (https://mail.python.org/pipermail/python-dev/2014-November/137201.html) .. [14] Rejection by GvR (https://mail.python.org/pipermail/python-dev/2014-November/137243.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- --Guido van Rossum (python.org/~guido)

2 1

Summary of Python tracker Issues
by Python tracker Dec. 5, 2014

Dec. 5, 2014

ACTIVITY SUMMARY (2014-11-28 - 2014-12-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4666 ( -2) closed 30095 (+39) total 34761 (+37) Open issues with patches: 2173 Issues opened (28) ================== #9179: Lookback with group references incorrect (two issues?) http://bugs.python.org/issue9179 reopened by serhiy.storchaka #22619: Possible implementation of negative limit for traceback functi http://bugs.python.org/issue22619 reopened by vlth #22922: asyncio: call_soon() should raise an exception if the event lo http://bugs.python.org/issue22922 reopened by haypo #22964: dbm.open(..., "x") http://bugs.python.org/issue22964 opened by Antony.Lee #22968: Lib/types.py nit: isinstance != PyType_IsSubtype http://bugs.python.org/issue22968 opened by gmt #22969: Compile fails with --without-signal-module http://bugs.python.org/issue22969 opened by KHH #22970: Cancelling wait() after notification leaves Condition in an in http://bugs.python.org/issue22970 opened by dcoles #22971: test_pickle: "Fatal Python error: Cannot recover from stack ov http://bugs.python.org/issue22971 opened by haypo #22972: Timeout making ajax calls to SimpleHTTPServer from internet ex http://bugs.python.org/issue22972 opened by Andrew.Burrows #22976: multiprocessing Queue empty() is broken on Windows http://bugs.python.org/issue22976 opened by Radosław.Szkodziński #22977: Unformatted “Windows Error 0x%X” exception message on Wine http://bugs.python.org/issue22977 opened by vadmium #22980: C extension naming doesn't take bitness into account http://bugs.python.org/issue22980 opened by pitrou #22981: Use CFLAGS when extracting multiarch http://bugs.python.org/issue22981 opened by pitrou #22982: BOM incorrectly inserted before writing, after seeking in text http://bugs.python.org/issue22982 opened by MarkIngramUK #22983: Cookie parsing should be more permissive http://bugs.python.org/issue22983 opened by demian.brecht #22984: test_json.test_endless_recursion(): "Fatal Python error: Canno http://bugs.python.org/issue22984 opened by haypo #22985: Segfault on time.sleep http://bugs.python.org/issue22985 opened by Omer.Katz #22986: Improved handling of __class__ assignment http://bugs.python.org/issue22986 opened by njs #22988: No error when yielding from `finally` http://bugs.python.org/issue22988 opened by fov #22989: HTTPResponse.msg not as documented http://bugs.python.org/issue22989 opened by bastik #22990: bdist installation dialog http://bugs.python.org/issue22990 opened by Alan #22991: test_gdb leaves the terminal in raw mode with gdb 7.8.1 http://bugs.python.org/issue22991 opened by xdegaye #22992: Adding a git developer's guide to Mercurial to devguide http://bugs.python.org/issue22992 opened by demian.brecht #22993: Plistlib fails on certain valid plist values http://bugs.python.org/issue22993 opened by Connor.Wolf #22995: Restrict default pickleability http://bugs.python.org/issue22995 opened by serhiy.storchaka #22996: Order of _io objects finalization can lose data in reference c http://bugs.python.org/issue22996 opened by pitrou #22997: Minor improvements to "Functional API" section of Enum documen http://bugs.python.org/issue22997 opened by simeon.visser #22998: inspect.Signature and default arguments http://bugs.python.org/issue22998 opened by doerwalter Most recent 15 issues with no replies (15) ========================================== #22990: bdist installation dialog http://bugs.python.org/issue22990 #22989: HTTPResponse.msg not as documented http://bugs.python.org/issue22989 #22985: Segfault on time.sleep http://bugs.python.org/issue22985 #22981: Use CFLAGS when extracting multiarch http://bugs.python.org/issue22981 #22970: Cancelling wait() after notification leaves Condition in an in http://bugs.python.org/issue22970 #22969: Compile fails with --without-signal-module http://bugs.python.org/issue22969 #22964: dbm.open(..., "x") http://bugs.python.org/issue22964 #22962: ipaddress: Add optional prefixlen argument to ip_interface and http://bugs.python.org/issue22962 #22958: Constructors of weakref mapping classes don't accept "self" an http://bugs.python.org/issue22958 #22956: Improved support for prepared SQL statements http://bugs.python.org/issue22956 #22947: Enable 'imageop' - "Multimedia Srvices Feature module" for 64- http://bugs.python.org/issue22947 #22942: Language Reference - optional comma http://bugs.python.org/issue22942 #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 #22907: Misc/python-config.sh.in: ensure sed invocations only match be http://bugs.python.org/issue22907 #22893: Idle: __future__ does not work in startup code. http://bugs.python.org/issue22893 Most recent 15 issues waiting for review (15) ============================================= #22998: inspect.Signature and default arguments http://bugs.python.org/issue22998 #22997: Minor improvements to "Functional API" section of Enum documen http://bugs.python.org/issue22997 #22992: Adding a git developer's guide to Mercurial to devguide http://bugs.python.org/issue22992 #22991: test_gdb leaves the terminal in raw mode with gdb 7.8.1 http://bugs.python.org/issue22991 #22986: Improved handling of __class__ assignment http://bugs.python.org/issue22986 #22984: test_json.test_endless_recursion(): "Fatal Python error: Canno http://bugs.python.org/issue22984 #22970: Cancelling wait() after notification leaves Condition in an in http://bugs.python.org/issue22970 #22968: Lib/types.py nit: isinstance != PyType_IsSubtype http://bugs.python.org/issue22968 #22955: Pickling of methodcaller, attrgetter, and itemgetter http://bugs.python.org/issue22955 #22952: multiprocessing doc introduction not in affirmative tone http://bugs.python.org/issue22952 #22947: Enable 'imageop' - "Multimedia Srvices Feature module" for 64- http://bugs.python.org/issue22947 #22946: urllib gives incorrect url after open when using HTTPS http://bugs.python.org/issue22946 #22941: IPv4Interface arithmetic changes subnet mask http://bugs.python.org/issue22941 #22935: Disabling SSLv3 support http://bugs.python.org/issue22935 #22932: email.utils.formatdate uses unreliable time.timezone constant http://bugs.python.org/issue22932 Top 10 most discussed issues (10) ================================= #22980: C extension naming doesn't take bitness into account http://bugs.python.org/issue22980 25 msgs #17852: Built-in module _io can loose data from buffered files at exit http://bugs.python.org/issue17852 23 msgs #22356: mention explicitly that stdlib assumes gmtime(0) epoch is 1970 http://bugs.python.org/issue22356 8 msgs #22906: PEP 479: Change StopIteration handling inside generators http://bugs.python.org/issue22906 8 msgs #22955: Pickling of methodcaller, attrgetter, and itemgetter http://bugs.python.org/issue22955 8 msgs #9179: Lookback with group references incorrect (two issues?) http://bugs.python.org/issue9179 6 msgs #16329: mimetypes does not support webm type http://bugs.python.org/issue16329 6 msgs #22922: asyncio: call_soon() should raise an exception if the event lo http://bugs.python.org/issue22922 6 msgs #22931: cookies with square brackets in value http://bugs.python.org/issue22931 6 msgs #22959: http.client.HTTPSConnection checks hostname when SSL context h http://bugs.python.org/issue22959 6 msgs Issues closed (37) ================== #3068: IDLE - Add an extension configuration dialog http://bugs.python.org/issue3068 closed by terry.reedy #12987: Demo/scripts/newslist.py has non-free licensing terms http://bugs.python.org/issue12987 closed by terry.reedy #13027: python 2.6.6 interpreter core dumps on modules command from he http://bugs.python.org/issue13027 closed by serhiy.storchaka #14099: ZipFile.open() should not reopen the underlying file http://bugs.python.org/issue14099 closed by serhiy.storchaka #16569: Preventing errors of simultaneous access in zipfile http://bugs.python.org/issue16569 closed by serhiy.storchaka #18053: Add checks for Misc/NEWS in make patchcheck http://bugs.python.org/issue18053 closed by terry.reedy #19834: Unpickling exceptions pickled by Python 2 http://bugs.python.org/issue19834 closed by doerwalter #20335: bytes constructor accepts more than one argument even if the f http://bugs.python.org/issue20335 closed by serhiy.storchaka #21032: Socket leak if HTTPConnection.getresponse() fails http://bugs.python.org/issue21032 closed by serhiy.storchaka #22389: Add contextlib.redirect_stderr() http://bugs.python.org/issue22389 closed by berker.peksag #22407: re.LOCALE is nonsensical for Unicode http://bugs.python.org/issue22407 closed by serhiy.storchaka #22429: asyncio: pending call to loop.stop() if a coroutine raises a B http://bugs.python.org/issue22429 closed by python-dev #22473: The gloss on asyncio "future with run_forever" example is conf http://bugs.python.org/issue22473 closed by python-dev #22475: asyncio task get_stack documentation seems to contradict itsel http://bugs.python.org/issue22475 closed by python-dev #22599: traceback: errors in the linecache module at exit http://bugs.python.org/issue22599 closed by haypo #22768: Add a way to get the peer certificate of a SSL Transport http://bugs.python.org/issue22768 closed by haypo #22838: Convert re tests to unittest http://bugs.python.org/issue22838 closed by serhiy.storchaka #22895: test failure introduced by the fix for issue #22462 http://bugs.python.org/issue22895 closed by pitrou #22902: Use 'ip' for uuid.getnode() http://bugs.python.org/issue22902 closed by serhiy.storchaka #22909: [argparse] Using parse_known_args, unknown arg with space in v http://bugs.python.org/issue22909 closed by berker.peksag #22914: Rewrite of Python 2/3 porting HOWTO http://bugs.python.org/issue22914 closed by brett.cannon #22924: Use of deprecated cgi.escape http://bugs.python.org/issue22924 closed by serhiy.storchaka #22943: bsddb: test_queue fails on Windows http://bugs.python.org/issue22943 closed by serhiy.storchaka #22951: unexpected return from float.__repr__() for inf, -inf, nan http://bugs.python.org/issue22951 closed by terry.reedy #22960: xmlrpc.client.ServerProxy() should accept a custom SSL context http://bugs.python.org/issue22960 closed by benjamin.peterson #22963: broken link in PEP 102 http://bugs.python.org/issue22963 closed by berker.peksag #22965: smtplib.py: senderrs[each] -> TypeError: unhashable instance http://bugs.python.org/issue22965 closed by r.david.murray #22966: py_compile: foo.bar.py → __pycache__/foo.cpython-34.pyc http://bugs.python.org/issue22966 closed by barry #22967: tempfile.py does not work in windows8 http://bugs.python.org/issue22967 closed by zach.ware #22973: hash() function gives the same result for -1 and for -2 argume http://bugs.python.org/issue22973 closed by eric.smith #22974: Make traceback functions support negative limits http://bugs.python.org/issue22974 closed by vlth #22975: Crosses initialization? http://bugs.python.org/issue22975 closed by serhiy.storchaka #22978: Logical Negation of NotImplemented http://bugs.python.org/issue22978 closed by r.david.murray #22979: Use of None in min and max http://bugs.python.org/issue22979 closed by r.david.murray #22987: ssl module documentation: incorrect compatibility matrix http://bugs.python.org/issue22987 closed by pitrou #22994: datetime buggy http://bugs.python.org/issue22994 closed by r.david.murray #22999: Copying emoji to Windows clipboard corrupts string in Python 3 http://bugs.python.org/issue22999 closed by amaury.forgeotdarc

1 0

WebM MIME type in mimetypes module
by Chris Rebert Dec. 3, 2014

Dec. 3, 2014

Hi all, I'm seeking to move http://bugs.python.org/issue16329 towards conclusion. Since the discussion on the issue itself seems to have petered out, I thought I'd bring it up here. To summarize the issue, it proposes adding an entry for WebM ( http://www.webmproject.org/docs/container/#naming ) to the mimetypes standard library module's file-extension to MIME-type database. (Specifically: .webm => video/webm ) Mozilla, Microsoft, Opera, and freedesktop.org (the de facto standard *nix MIME type database package) all acknowledge the existence of a video/webm MIME type (see the issue for relevant links), and this MIME type is in WebM's documentation. However, there is no official IANA registration for WebM's MIME type, and none seems to be forthcoming/planned. As R.D.M. said in the issue: > So we have two choices: > leave it to the platform mime types file to define because it is not even on track to be an official IANA standard, > or include it with a comment that it is a de-facto standard. [...] > I guess I'd be OK with adding it as a de-facto standard, though I'm not entirely comfortable with it. But that would represent a change in policy, so others may want to weigh in. Nobody has weighed in during the subsequent ~2 years, so I'm hoping a few of y'all could weigh in one way or the other, and thus bring the issue to a definitive conclusion. Cheers, Chris -- https://github.com/cvrebert

6 6

advice needed: best approach to enabling "metamodules"?
by Nathaniel Smith Dec. 2, 2014

Dec. 2, 2014

Hi all, There was some discussion on python-ideas last month about how to make it easier/more reliable for a module to override attribute access. This is useful for things like autoloading submodules (accessing 'foo.bar' triggers the import of 'bar'), or for deprecating module attributes that aren't functions. (Accessing 'foo.bar' emits a DeprecationWarning, "the bar attribute will be removed soon".) Python has had some basic support for this for a long time -- if a module overwrites its entry in sys.modules[__name__], then the object that's placed there will be returned by 'import'. This allows one to define custom subclasses of module and use them instead of the default, similar to how metaclasses allow one to use custom subclasses of 'type'. In practice though it's very difficult to make this work safely and correctly for a top-level package. The main problem is that when you create a new object to stick into sys.modules, this necessarily means creating a new namespace dict. And now you have a mess, because now you have two dicts: new_module.__dict__ which is the namespace you export, and old_module.__dict__, which is the globals() for the code that's trying to define the module namespace. Keeping these in sync is extremely error-prone -- consider what happens, e.g., when your package __init__.py wants to import submodules which then recursively import the top-level package -- so it's difficult to justify for the kind of large packages that might be worried about deprecating entries in their top-level namespace. So what we'd really like is a way to somehow end up with an object that (a) has the same __dict__ as the original module, but (b) is of our own custom module subclass. If we can do this then metamodules will become safe and easy to write correctly. (There's a little demo of working metamodules here: https://github.com/njsmith/metamodule/ but it uses ctypes hacks that depend on non-stable parts of the CPython ABI, so it's not a long-term solution.) I've now spent some time trying to hack this capability into CPython and I've made a list of the possible options I can think of to fix this. I'm writing to python-dev because none of them are obviously The Right Way so I'd like to get some opinions/ruling/whatever on which approach to follow up on. Option 1: Make it possible to change the type of a module object in-place, so that we can write something like sys.modules[__name__].__class__ = MyModuleSubclass Option 1 downside: The invariants required to make __class__ assignment safe are complicated, and only implemented for heap-allocated type objects. PyModule_Type is not heap-allocated, so making this work would require lots of delicate surgery to typeobject.c. I'd rather not go down that rabbit-hole. ---- Option 2: Make PyModule_Type into a heap type allocated at interpreter startup, so that the above just works. Option 2 downside: PyModule_Type is exposed as a statically-allocated global symbol, so doing this would involve breaking the stable ABI. ---- Option 3: Make it legal to assign to the __dict__ attribute of a module object, so that we can write something like new_module = MyModuleSubclass(...) new_module.__dict__ = sys.modules[__name__].__dict__ sys.modules[__name__].__dict__ = {} # *** sys.modules[__name__] = new_module The line marked *** is necessary because the way modules are designed, they expect to control the lifecycle of their __dict__. When the module object is initialized, it fills in a bunch of stuff in the __dict__. When the module object (not the dict object!) is deallocated, it deletes everything from the __dict__. This latter feature in particular means that having two module objects sharing the same __dict__ is bad news. Option 3 downside: The paragraph above. Also, there's stuff inside the module struct besides just the __dict__, and more stuff has appeared there over time. ---- Option 4: Add a new function sys.swap_module_internals, which takes two module objects and swaps their __dict__ and other attributes. By making the operation a swap instead of an assignment, we avoid the lifecycle pitfalls from Option 3. By making it a builtin, we can make sure it always handles all the module fields that matter, not just __dict__. Usage: new_module = MyModuleSubclass(...) sys.swap_module_internals(new_module, sys.modules[__name__]) sys.modules[__name__] = new_module Option 4 downside: Obviously a hack. ---- Option 3 or 4 both seem workable, it just depends on which way we prefer to hold our nose. Option 4 is slightly more correct in that it works for *all* modules, but OTOH at the moment the only time Option 3 *really* fails is for compiled modules with PEP 3121 metadata, and compiled modules can already use a module subclass via other means (since they instantiate their own module objects). Thoughts? Suggestions on other options I've missed? Should I go ahead and write a patch for one of these? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

13 42

Python 2.x vs 3.x survey - new owner?
by Dan Stromberg Dec. 2, 2014

Dec. 2, 2014

Last year in late December, I did a brief, 9 question survey of 2.x vs 3.x usage. I like the think the results were interesting, but I don't have the spare cash to do it again this year. I probably shouldn't have done it last year. ^_^ Is anyone interested in taking over the survey? It's on SurveyMonkey. It was mentioned last year, that it might be interesting to see how things change, year to year. It was also reported that some people felt that late December wasn't necessarily the best time of year to do the survey, as a lot of people were on vacation. The Python wiki has last year's results: https://wiki.python.org/moin/2.x-vs-3.x-survey

1 0

Re: [Python-Dev] [Python-checkins] cpython (3.4): - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is
by Jeremy Kloth Dec. 2, 2014

Dec. 2, 2014

On Mon, Dec 1, 2014 at 4:17 PM, barry.warsaw <python-checkins(a)python.org> wrote: > summary: > - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is > asked to compile a source file containing multiple dots in the source file > name. > > diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py > --- a/Lib/test/test_py_compile.py > +++ b/Lib/test/test_py_compile.py > @@ -99,5 +99,21 @@ > self.assertFalse(os.path.exists( > importlib.util.cache_from_source(bad_coding))) > > + def test_double_dot_no_clobber(self): > + # http://bugs.python.org/issue22966 > + # py_compile foo.bar.py -> __pycache__/foo.cpython-34.pyc > + weird_path = os.path.join(self.directory, 'foo.bar.py') > + cache_path = importlib.util.cache_from_source(weird_path) > + pyc_path = weird_path + 'c' > + self.assertEqual( > + '/'.join(cache_path.split('/')[-2:]), > + '__pycache__/foo.bar.cpython-34.pyc') > + with open(weird_path, 'w') as file: > + file.write('x = 123\n') > + py_compile.compile(weird_path) > + self.assertTrue(os.path.exists(cache_path)) > + self.assertFalse(os.path.exists(pyc_path)) > + > + This test is failing on the Windows buildbots due to the hard-coded path separator. Using `os.pathsep` should work assuming that importlib returns normalized paths. -- Jeremy Kloth

2 1

LTTng-UST support for CPython
by Francis Giraldeau Dec. 2, 2014

Dec. 2, 2014

Here is a working prototype for CPython to record all function call/return using LTTng-UST, a fast tracer. https://github.com/giraldeau/python-profile-ust However, there are few issues and questions: - I was not able to get PyTrace_EXCEPTION using "raise" or other error conditions. How can we trigger this event in Python code (PyTrace_C_EXCEPTION works)? - How could be the best way to get the full name of an object (such as package, module, class and function). Maybe it's too Java-ish, and it is better to record file/lineno instead? - On the C-API side: I did a horrible and silly function show_type() to run every Py*_Check() to determine the type of a PyObject *. What would be the sane way to do that? Your comments are very valuable. Thanks! Francis

2 1

Joining the PEP Editors team
by Chris Angelico Dec. 1, 2014

Dec. 1, 2014

In response to Guido's call for volunteers, I'm offering myself as a PEP editor. Who is in charge of this kind of thing? Who manages public key lists etc? ChrisA

2 1