Mailman 3 June 2016 - Python-Dev

Proper way to specify that a method is not defined for a type
by Ethan Furman 08 Jun '16

08 Jun '16

For binary methods, such as __add__, either do not implement or return NotImplemented if the other operand/class is not supported. For non-binary methods, simply do not define. Except for subclasses when the super-class defines __hash__ and the subclass is not hashable -- then set __hash__ to None. Question: Are there any other methods that should be set to None to tell the run-time that the method is not supported? Or is this a general mechanism for subclasses to declare any method is unsupported? -- ~Ethan~

4 3

PEP 492: __aiter__ should return async iterator directly instead of awaitable
by Yury Selivanov 07 Jun '16

07 Jun '16

There is a small flaw in PEP 492 design -- __aiter__ should not return an awaitable object that resolves to an asynchronous iterator. It should return an asynchronous iterator directly. Let me explain this by showing some examples. I've discovered this while working on a new asynchronous generators PEP. Let's pretend that we have them already: if we have a 'yield' expression in an 'async def' function, the function becomes an "asynchronous generator function": async def foo(): await bar() yield 1 await baz() yield 2 # foo -- is an `asynchronous generator function` # foo() -- is an `asynchronous generator` If we iterate through "foo()", it will await on "bar()", yield "1", await on "baz()", and yield "2": >>> async for el in foo(): ... print(el) 1 2 If we decide to have a class with an __aiter__ that is an async generator, we'd write something like this: class Foo: async def __aiter__(self): await bar() yield 1 await baz() yield 2 However, with the current PEP 492 design, the above code would be invalid! The interpreter expects __aiter__ to return a coroutine, not an async generator. I'm still working on the PEP for async generators, targeting CPython 3.6. And once it is ready, it might still be rejected or deferred. But in any case, this PEP 492 flaw has to be fixed now, in 3.5.2 (since PEP 492 is provisional). I've created an issue on the bug tracker: http://bugs.python.org/issue27243 The proposed patch fixes the __aiter__ in a backwards compatible way: 1. ceval/GET_AITER opcode calls the __aiter__ method. 2. If the returned object has an '__anext__' method, GET_AITER silently wraps it in an awaitable, which is equivalent to the following coroutine: async def wrapper(aiter_result): return aiter_result 3. If the returned object does not have an '__anext__' method, a DeprecationWarning is raised.

4 4

Re: [Python-Dev] cpython: replace custom validation logic in the parse module with a simple DFA validator
by Christian Heimes 06 Jun '16

06 Jun '16

On 2016-06-02 11:32, benjamin.peterson wrote: > https://hg.python.org/cpython/rev/4a9159ea2536 > changeset: 101601:4a9159ea2536 > user: Benjamin Peterson <benjamin(a)python.org> > date: Thu Jun 02 11:30:18 2016 -0700 > summary: > replace custom validation logic in the parse module with a simple DFA validator (closes #26526) > > Patch from A. Skrobov. > > files: > Misc/NEWS | 3 + > Modules/parsermodule.c | 2545 +-------------------------- > 2 files changed, 96 insertions(+), 2452 deletions(-) > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -22,6 +22,9 @@ > Library > ------- > > +- Issue #26526: Replace custom parse tree validation in the parser > + module with a simple DFA validator. > + > - Issue #27114: Fix SSLContext._load_windows_store_certs fails with > PermissionError > > diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c > --- a/Modules/parsermodule.c > +++ b/Modules/parsermodule.c > @@ -670,9 +670,75 @@ > > > static node* build_node_tree(PyObject *tuple); > -static int validate_expr_tree(node *tree); > -static int validate_file_input(node *tree); > -static int validate_encoding_decl(node *tree); > + > +static int > +validate_node(node *tree) > +{ > + int type = TYPE(tree); > + int nch = NCH(tree); > + dfa *nt_dfa; > + state *dfa_state; > + int pos, arc; > + > + assert(ISNONTERMINAL(type)); > + type -= NT_OFFSET; > + if (type >= _PyParser_Grammar.g_ndfas) { > + PyErr_Format(parser_error, "Unrecognized node type %d.", TYPE(tree)); > + return 0; > + } > + nt_dfa = &_PyParser_Grammar.g_dfa[type]; > + REQ(tree, nt_dfa->d_type); > + > + /* Run the DFA for this nonterminal. */ > + dfa_state = &nt_dfa->d_state[nt_dfa->d_initial]; > + for (pos = 0; pos < nch; ++pos) { > + node *ch = CHILD(tree, pos); > + int ch_type = TYPE(ch); > + for (arc = 0; arc < dfa_state->s_narcs; ++arc) { > + short a_label = dfa_state->s_arc[arc].a_lbl; > + assert(a_label < _PyParser_Grammar.g_ll.ll_nlabels); > + if (_PyParser_Grammar.g_ll.ll_label[a_label].lb_type == ch_type) { > + /* The child is acceptable; if non-terminal, validate it recursively. */ > + if (ISNONTERMINAL(ch_type) && !validate_node(ch)) > + return 0; > + > + /* Update the state, and move on to the next child. */ > + dfa_state = &nt_dfa->d_state[dfa_state->s_arc[arc].a_arrow]; > + goto arc_found; > + } > + } > + /* What would this state have accepted? */ > + { > + short a_label = dfa_state->s_arc->a_lbl; > + int next_type; > + if (!a_label) /* Wouldn't accept any more children */ > + goto illegal_num_children; > + > + next_type = _PyParser_Grammar.g_ll.ll_label[a_label].lb_type; > + if (ISNONTERMINAL(next_type)) > + PyErr_Format(parser_error, "Expected node type %d, got %d.", > + next_type, ch_type); > + else > + PyErr_Format(parser_error, "Illegal terminal: expected %s.", > + _PyParser_TokenNames[next_type]); Coverity doesn't that line: CID 1362505 (#1 of 1): Out-of-bounds read (OVERRUN) 20. overrun-local: Overrunning array _PyParser_TokenNames of 58 8-byte elements at element index 255 (byte offset 2040) using index next_type (which evaluates to 255). Can you add a check to verify, that next_type is not out-of-bounds, e.g. + else if (next_type > N_TOKENS) + PyErr_Format(parser_error, "Illegal node type %d", next_type); > + return 0; > + } > + > +arc_found: > + continue; > + } > + /* Are we in a final state? If so, return 1 for successful validation. */ > + for (arc = 0; arc < dfa_state->s_narcs; ++arc) { > + if (!dfa_state->s_arc[arc].a_lbl) { > + return 1; > + } > + } > + > +illegal_num_children: > + PyErr_Format(parser_error, > + "Illegal number of children for %s node.", nt_dfa->d_name); > + return 0; > +}

2 1

Improving the bytecode
by Serhiy Storchaka 06 Jun '16

06 Jun '16

Following the converting 8-bit bytecode to 16-bit bytecode (wordcode), there are other issues for improving the bytecode. 1. http://bugs.python.org/issue27129 Make the bytecode more 16-bit oriented. 2. http://bugs.python.org/issue27140 Add new opcode BUILD_CONST_KEY_MAP for building a dict with constant keys. This optimize the common case and especially helpful for two following issues (creating and calling functions). 3. http://bugs.python.org/issue27095 Simplify MAKE_FUNCTION/MAKE_CLOSURE. Instead packing three numbers in oparg the new MAKE_FUNCTION takes built tuples and dicts from the stack. MAKE_FUNCTION and MAKE_CLOSURE are merged in the single opcode. 4. http://bugs.python.org/issue27213 Rework CALL_FUNCTION* opcodes. Replace four existing opcodes with three simpler and more efficient opcodes. 5. http://bugs.python.org/issue27127 Rework the for loop implementation. 6. http://bugs.python.org/issue17611 Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

5 5

I broke the 3.5 branch, apparently
by R. David Murray 04 Jun '16

04 Jun '16

I don't understand how it happened, but apparently I got a merge commit backward and merged 3.6 into 3.5 and pushed it without realizing what had happened. If anyone has any clue how to reverse this cleanly, please let me know. (There are a couple people at the sprints looking in to it, but the mercurial guys aren't here so we are short on experts). My apologies for the mess :( --David

2 2

Summary of Python tracker Issues
by Python tracker 04 Jun '16

04 Jun '16

ACTIVITY SUMMARY (2016-05-27 - 2016-06-03) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5537 ( +8) closed 33416 (+52) total 38953 (+60) Open issues with patches: 2416 Issues opened (41) ================== #22331: test_io.test_interrupted_write_text() hangs on the buildbot Fr http://bugs.python.org/issue22331 reopened by martin.panter #27137: Python implementation of `functools.partial` is not a class http://bugs.python.org/issue27137 opened by ebarry #27139: Increased test coverage for statistics.median_grouped http://bugs.python.org/issue27139 opened by juliojr77 #27140: Opcode for creating dict with constant keys http://bugs.python.org/issue27140 opened by serhiy.storchaka #27141: Fix collections.UserList shallow copy http://bugs.python.org/issue27141 opened by bar.harel #27142: Default int value with xmlrpclib / xmlrpc.client http://bugs.python.org/issue27142 opened by julienc #27144: concurrent.futures.as_completed() memory inefficiency http://bugs.python.org/issue27144 opened by grzgrzgrz3 #27145: long_add and long_sub might return a new int where &small_ints http://bugs.python.org/issue27145 opened by Oren Milman #27149: Implement socket.sendmsg() for Windows http://bugs.python.org/issue27149 opened by mmarkk #27150: PEP446 (CLOEXEC by default) violation with fcntl.fcntl(..., fc http://bugs.python.org/issue27150 opened by mmarkk #27151: multiprocessing.Process leaves read pipes open (Process.sentin http://bugs.python.org/issue27151 opened by Roman Bolshakov #27152: Additional assert methods for unittest http://bugs.python.org/issue27152 opened by serhiy.storchaka #27154: Regression in file.writelines behavior http://bugs.python.org/issue27154 opened by snaury #27156: IDLE: remove unused code http://bugs.python.org/issue27156 opened by terry.reedy #27157: Unhelpful error message when one calls a subclass of type with http://bugs.python.org/issue27157 opened by ppperry #27161: Confusing exception in Path().with_name http://bugs.python.org/issue27161 opened by Antony.Lee #27162: Add idlelib.interface module http://bugs.python.org/issue27162 opened by terry.reedy #27163: IDLE entry for What's New in Python 3.6 http://bugs.python.org/issue27163 opened by terry.reedy #27164: zlib can't decompress DEFLATE using shared dictionary http://bugs.python.org/issue27164 opened by Vladimir Mihailenco #27165: Skip callables when displaying exception fields in cgitb http://bugs.python.org/issue27165 opened by Adam.Bielański #27167: subprocess reports signal as negative exit status, not documen http://bugs.python.org/issue27167 opened by dmacnet #27168: Yury isn't sure comprehensions and await interact correctly http://bugs.python.org/issue27168 opened by njs #27169: __debug__ is not optimized out at compile time for anything bu http://bugs.python.org/issue27169 opened by josh.r #27170: IDLE: remove Toggle Auto Coloring or add to edit menu & doc http://bugs.python.org/issue27170 opened by terry.reedy #27172: Add skip_bound_arg argument to inspect.Signature.from_callable http://bugs.python.org/issue27172 opened by ryan.petrello #27173: Modern Unix key bindings for IDLE http://bugs.python.org/issue27173 opened by serhiy.storchaka #27175: Unpickling Path objects http://bugs.python.org/issue27175 opened by Antony.Lee #27177: re match.group should support __index__ http://bugs.python.org/issue27177 opened by jdemeyer #27179: subprocess uses wrong encoding on Windows http://bugs.python.org/issue27179 opened by davispuh #27180: Doc/pathlib: Please describe the behaviour of Path().rename() http://bugs.python.org/issue27180 opened by hashimo #27181: Add geometric mean to `statistics` module http://bugs.python.org/issue27181 opened by cool-RR #27182: PEP 519 support in the stdlib http://bugs.python.org/issue27182 opened by ethan.furman #27184: Support path objects in the ntpath module http://bugs.python.org/issue27184 opened by ethan.furman #27186: add os.fspath() http://bugs.python.org/issue27186 opened by ethan.furman #27187: Relax __all__ location requirement in PEP 8 http://bugs.python.org/issue27187 opened by barry #27188: sqlite3 execute* methods return value not documented http://bugs.python.org/issue27188 opened by Dave Sawyer #27189: configure --with-lto with clang should find the appropriate ll http://bugs.python.org/issue27189 opened by gregory.p.smith #27190: Check sqlite3_version before allowing check_same_thread = Fals http://bugs.python.org/issue27190 opened by Dave Sawyer #27194: Tarfile superfluous truncate calls slows extraction. http://bugs.python.org/issue27194 opened by fried #27195: Crash when RawIOBase.write(b) evaluates b.format http://bugs.python.org/issue27195 opened by martin.panter #27196: Eliminate 'ThemeChanged' warning when running IDLE tests http://bugs.python.org/issue27196 opened by terry.reedy Most recent 15 issues with no replies (15) ========================================== #27195: Crash when RawIOBase.write(b) evaluates b.format http://bugs.python.org/issue27195 #27189: configure --with-lto with clang should find the appropriate ll http://bugs.python.org/issue27189 #27188: sqlite3 execute* methods return value not documented http://bugs.python.org/issue27188 #27180: Doc/pathlib: Please describe the behaviour of Path().rename() http://bugs.python.org/issue27180 #27175: Unpickling Path objects http://bugs.python.org/issue27175 #27168: Yury isn't sure comprehensions and await interact correctly http://bugs.python.org/issue27168 #27165: Skip callables when displaying exception fields in cgitb http://bugs.python.org/issue27165 #27163: IDLE entry for What's New in Python 3.6 http://bugs.python.org/issue27163 #27162: Add idlelib.interface module http://bugs.python.org/issue27162 #27151: multiprocessing.Process leaves read pipes open (Process.sentin http://bugs.python.org/issue27151 #27144: concurrent.futures.as_completed() memory inefficiency http://bugs.python.org/issue27144 #27139: Increased test coverage for statistics.median_grouped http://bugs.python.org/issue27139 #27123: Allow `install_headers` command to follow specific directory s http://bugs.python.org/issue27123 #27121: imghdr does not support jpg files with Lavc bytes http://bugs.python.org/issue27121 #27115: IDLE/tkinter: in simpledialog, <Return> != [OK] click http://bugs.python.org/issue27115 Most recent 15 issues waiting for review (15) ============================================= #27194: Tarfile superfluous truncate calls slows extraction. http://bugs.python.org/issue27194 #27190: Check sqlite3_version before allowing check_same_thread = Fals http://bugs.python.org/issue27190 #27186: add os.fspath() http://bugs.python.org/issue27186 #27179: subprocess uses wrong encoding on Windows http://bugs.python.org/issue27179 #27177: re match.group should support __index__ http://bugs.python.org/issue27177 #27173: Modern Unix key bindings for IDLE http://bugs.python.org/issue27173 #27172: Add skip_bound_arg argument to inspect.Signature.from_callable http://bugs.python.org/issue27172 #27165: Skip callables when displaying exception fields in cgitb http://bugs.python.org/issue27165 #27164: zlib can't decompress DEFLATE using shared dictionary http://bugs.python.org/issue27164 #27161: Confusing exception in Path().with_name http://bugs.python.org/issue27161 #27157: Unhelpful error message when one calls a subclass of type with http://bugs.python.org/issue27157 #27152: Additional assert methods for unittest http://bugs.python.org/issue27152 #27145: long_add and long_sub might return a new int where &small_ints http://bugs.python.org/issue27145 #27144: concurrent.futures.as_completed() memory inefficiency http://bugs.python.org/issue27144 #27141: Fix collections.UserList shallow copy http://bugs.python.org/issue27141 Top 10 most discussed issues (10) ================================= #27157: Unhelpful error message when one calls a subclass of type with http://bugs.python.org/issue27157 26 msgs #19611: inspect.getcallargs doesn't properly interpret set comprehensi http://bugs.python.org/issue19611 15 msgs #27179: subprocess uses wrong encoding on Windows http://bugs.python.org/issue27179 12 msgs #20699: Document that binary IO classes work with bytes-likes objects http://bugs.python.org/issue20699 11 msgs #27137: Python implementation of `functools.partial` is not a class http://bugs.python.org/issue27137 11 msgs #27161: Confusing exception in Path().with_name http://bugs.python.org/issue27161 10 msgs #27136: sock_connect fails for bluetooth (and probably others) http://bugs.python.org/issue27136 9 msgs #22558: Missing doc links to source code for Python-coded modules. http://bugs.python.org/issue22558 8 msgs #26546: Provide translated french translation on docs.python.org http://bugs.python.org/issue26546 8 msgs #27033: Change the decode_data default in smtpd to False http://bugs.python.org/issue27033 8 msgs Issues closed (50) ================== #5252: 2to3 should detect and delete import of removed statvfs module http://bugs.python.org/issue5252 closed by r.david.murray #8519: doc: termios and ioctl reference links http://bugs.python.org/issue8519 closed by orsenthil #9327: doctest DocFileCase setUp/tearDown asymmetry http://bugs.python.org/issue9327 closed by berker.peksag #9363: data_files are not installed relative to sys.prefix http://bugs.python.org/issue9363 closed by berker.peksag #12243: getpass.getuser works on OSX http://bugs.python.org/issue12243 closed by berker.peksag #12691: tokenize.untokenize is broken http://bugs.python.org/issue12691 closed by terry.reedy #13784: Documentation of xml.sax.xmlreader: Locator.getLineNumber() a http://bugs.python.org/issue13784 closed by r.david.murray #17352: Be clear that __prepare__ must be declared as a class method http://bugs.python.org/issue17352 closed by berker.peksag #18384: Add devhelp build instructions to the documentation makefile http://bugs.python.org/issue18384 closed by berker.peksag #18478: Class bodies: when does a name become local? http://bugs.python.org/issue18478 closed by terry.reedy #20496: function definition tutorial encourages bad practice http://bugs.python.org/issue20496 closed by berker.peksag #20973: Implement proper comparison operations for in _TotalOrderingMi http://bugs.python.org/issue20973 closed by r.david.murray #21271: reset_mock needs parameters to also reset return_value and sid http://bugs.python.org/issue21271 closed by kushal.das #21776: distutils.upload uses the wrong order of exceptions http://bugs.python.org/issue21776 closed by berker.peksag #23116: Python Tutorial 4.7.1: Improve ask_ok() to cover more input va http://bugs.python.org/issue23116 closed by berker.peksag #24647: Document argparse.REMAINDER as being equal to "..." http://bugs.python.org/issue24647 closed by r.david.murray #24671: idlelib 2.7: finish converting print statements http://bugs.python.org/issue24671 closed by terry.reedy #25570: urllib.request > Request.add_header("abcd","efgh") fails with http://bugs.python.org/issue25570 closed by martin.panter #25926: Clarify that the itertools pure python equivalents are only ap http://bugs.python.org/issue25926 closed by rhettinger #25931: os.fork() command distributed in windows Python27 (in SocketSe http://bugs.python.org/issue25931 closed by gregory.p.smith #26526: In parsermodule.c, replace over 2KLOC of hand-crafted validati http://bugs.python.org/issue26526 closed by python-dev #26553: Write HTTP in uppercase http://bugs.python.org/issue26553 closed by martin.panter #26632: @public - an __all__ decorator http://bugs.python.org/issue26632 closed by barry #26739: idle: Errno 10035 a non-blocking socket operation could not be http://bugs.python.org/issue26739 closed by zach.ware #26829: update docs: when creating classes a new dict is created for t http://bugs.python.org/issue26829 closed by r.david.murray #27043: Describe what ‘inspect.cleandoc’ does to synopsis line. http://bugs.python.org/issue27043 closed by orsenthil #27113: sqlite3 connect parameter "check_same_thread" not documented http://bugs.python.org/issue27113 closed by orsenthil #27117: turtledemo does not work with IDLE's new dark theme. http://bugs.python.org/issue27117 closed by terry.reedy #27124: binascii.a2b_hex raises binascii.Error and ValueError, not Typ http://bugs.python.org/issue27124 closed by martin.panter #27125: Typo in Python 2 multiprocessing documentation http://bugs.python.org/issue27125 closed by martin.panter #27138: FileFinder.find_spec() docstring needs to be corrected. http://bugs.python.org/issue27138 closed by eric.snow #27143: python 3.5 conflict with Mailman, ebtables and firewalld http://bugs.python.org/issue27143 closed by barry #27146: posixmodule.c needs stdio.h http://bugs.python.org/issue27146 closed by gregory.p.smith #27147: importlib docs do not mention PEP 420 http://bugs.python.org/issue27147 closed by eric.snow #27148: Make VENV_DIR relative to Script directory http://bugs.python.org/issue27148 closed by vinay.sajip #27153: Default value shown by argparse.ArgumentDefaultsHelpFormatter http://bugs.python.org/issue27153 closed by r.david.murray #27155: '-' sign typo in example http://bugs.python.org/issue27155 closed by r.david.murray #27158: `isinstance` function does not handle types that are their own http://bugs.python.org/issue27158 closed by ebarry #27159: Python 3.5.1's websocket's lib crashes in event that internet http://bugs.python.org/issue27159 closed by r.david.murray #27160: str.format: Silent truncation of kwargs when passing keywords http://bugs.python.org/issue27160 closed by ebarry #27166: Spam http://bugs.python.org/issue27166 closed by ebarry #27171: Fix various typos http://bugs.python.org/issue27171 closed by martin.panter #27174: Update URL to IPython in interactive.rst http://bugs.python.org/issue27174 closed by berker.peksag #27176: Addition of assertNotRaises http://bugs.python.org/issue27176 closed by rhettinger #27178: Unconverted RST marking in interpreter tutorial http://bugs.python.org/issue27178 closed by berker.peksag #27183: Clarify that Py_VISIT(NULL) does nothing http://bugs.python.org/issue27183 closed by python-dev #27185: Clarify Test Coverage for the String Module (test_pep292 is no http://bugs.python.org/issue27185 closed by erinspace #27191: Add formatting around methods in PEP 8 http://bugs.python.org/issue27191 closed by berker.peksag #27192: Keyboard Shortcuts Consistently Cause Crashes http://bugs.python.org/issue27192 closed by ebarry #27193: Tkinter Unresponsive With Special Keys http://bugs.python.org/issue27193 closed by ned.deily

1 0

Start of the Python Language Summit coverage at LWN
by Jake Edge 03 Jun '16

03 Jun '16

Howdy python-dev, I was able to sit in on the Python Language Summit again this year (thanks Larry and Barry!) and have some of the coverage available for your viewing pleasure now. The starting point is here: https://lwn.net/Articles/688969/ (or here for non-subscribers: https://lwn.net/SubscriberLink/688969/91cbeeaf32807914/ ) So far, I have written up the first three sessions. The rest will be coming in over the next week or so and be added to the page above (and will also appear in next week's weekly edition). The future of the ssl module: https://lwn.net/Articles/688974/ https://lwn.net/SubscriberLink/688974/31cfa9f818c834e1/ Twisted and Python 3: https://lwn.net/Articles/689068/ https://lwn.net/SubscriberLink/689068/34b68a2aea6ddd2d/ Gilectomy: https://lwn.net/Articles/689548/ https://lwn.net/SubscriberLink/689548/4328423f85a47679/ The articles will be freely available (without using the SubscriberLink) to the world at large in a week (and the next batch the week after that) ... until then, feel free to share the SubscriberLinks Hopefully I have captured things reasonably well. If there are corrections or clarifications needed, though, I recommend posting them as comments on the article. enjoy! jake -- Jake Edge - LWN - jake(a)lwn.net - http://lwn.net

2 1

Adding NewType() to PEP 484
by Ivan Levkivskyi 02 Jun '16

02 Jun '16

Hi, It has been proposed to enhance the typing module with a NewType function that allows to define simple unique types with almost zero runtime overhead. The PR containing actual implementation and PEP 484 update is here: https://github.com/python/typing/pull/226 Review comments are very welcome. Best regards, Ivan

22 40