Python-Dev April 2018

python-dev@python.org

103 participants
90 discussions

by Anthony Flury

All, Can someone please review Pull Request 5974 <https://github.com/python/cpython/pull/5974> on Python3.8 - the Pull request was submitted on 4th March - this pull request is associated with bpo-32933 <https://bugs.python.org/issue32933> To summarize the point of this pull request: It fixes a bug of omission within mock_open <https://docs.python.org/3/library/unittest.mock.html?highlight=mock_open#un…> (part of unittest.mock) The functionality of mock_open enables the test code to mock a file being opened with some data which can be read. Importantly, mock_open has a read_data attrribute which can be used to specify the data to read from the file. The mocked file which is opened correctly supports file.read(), file.readlines(), file.readline(). These all make use of the read_data as expected, and the mocked file also supports being opened as a context manager. But the mock_open file does not support iteration - so pythonic code which uses a for loop to iterate around the file content will only ever appear to iterate around an empty file, regardless of the read_data attribute when the mock_open is created So non-pythonic methods to iterate around the file contents - such as this : data = opened_file.readlines() for line in data: process_line(line) and this : line = opened_file.readline() while line: process_line(line) line = opened_file.readline() Can both be tested with the mocked file containing simulated data (using the read_data attribute) as expected. But this code (which by any standard is the 'correct' way to iterate around the file content of a text file): for line in opened_file: process_line(line) Will only ever appear to iterate around an empty file when tested using mock_open. I would like this to be reviewed so it can be back-ported into Python3.7 and 3.6 if at all possible. I know that the bug has existed since the original version of mock_open, but it does seem strange that code under test which uses a pythonic code structure can't be fully tested fully using the standard library. -- Anthony Flury email : *Anthony.flury(a)btinternet.com* Twitter : *@TonyFlury <https://twitter.com/TonyFlury/>*

2 years, 10 months

Reserve ':=' for type-inferred variable initialization (was PEP 572)

by Fatty Morgan

I would like to urge you to reconsider the use of the token ':=' for assignment expressions. The natural interpretation of 'name := expr' is a PEP 526 type-annotated variable initialization 'name : T = expr' with the type annotation T omitted, the tokens ':' and '=' coalesced, and the implied type T inferred as 'type(expr)'. There is an ongoing tendency to introduce possibilities for static analysis into python, perhaps eventually resulting in a statically typed variant of python were (optional) type annotations are enforced (and used for optimizations), and it would be a pity if this specific piece of obvious syntax had been wasted on a comparatively unimportant feature. The following should probably be discussed in python-ideas and would only be relevant for a more distant future, but while I have your attention, here are a couple of thoughts: Distinguishing in a rather unobtrusive way (no 'var' or 'let') the initialization of a previously unbound variable 'name := expr' and the re-assignment of a previously bound variable 'name = expr' would be beneficial in itself (this has been discussed before but AFAIK not with this syntax). For example, 'global' and 'nonlocal' would be redundant and could be deprecated. The use of variable initializations with explicit type annotations as in PEP 526 would only be required in situations where the declared type is deliberately different from the inferred type, e.g. 'any_var: object = 42; any_var = "foo"; #OK, no TypeError'. In the majority of cases a simple 'name := expr' would be sufficient and type-safe. Destructuring initialization, e.g., 'a,b,c: int = range(3)' or 'a,b,c := range(3)', is possible (the annotated or inferred type refers not to the type of the iterable but to the generated type). If it were forbidden (i.e., only a single name-variable allowed) PEP 572 with ':=' could be simply implemented by parsing 'name := expr' as an expression rather than a statement (assuming it is OK to introduce 'name' into the current scope) but destructuring initialization appears to be the more important feature and would make assignment expressions with ':=' ambiguous. Implementation would be straightforward if 'name := expr' were lowered to 'name: auto = expr' where the newly introduced abstract type 'auto' simply acts as a token informing compile-time and/or run-time what to do. An explicit type annotation with type 'auto' could be allowed and would help to teach the feature.

2 years, 10 months

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2018-04-20 - 2018-04-27) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6615 (+28) closed 38510 (+26) total 45125 (+54) Open issues with patches: 2586 Issues opened (45) ================== #3692: improper scope in list comprehension, when used in class decla https://bugs.python.org/issue3692 reopened by Mariatta #32799: <class 'contextlib._GeneratorContextManager'> returned a resul https://bugs.python.org/issue32799 reopened by xiang.zhang #33144: random._randbelow optimization https://bugs.python.org/issue33144 reopened by serhiy.storchaka #33321: Add a Linux clang ubsan undefined behavior sanitizer buildbot https://bugs.python.org/issue33321 opened by gregory.p.smith #33325: Optimize sequences of constants in the compiler https://bugs.python.org/issue33325 opened by serhiy.storchaka #33326: Convert collections (cmp_op, hasconst, hasname and others) in https://bugs.python.org/issue33326 opened by godaygo #33327: Add a method to move messages to IMAPlib https://bugs.python.org/issue33327 opened by mcepl #33328: pdb error when stepping through generator https://bugs.python.org/issue33328 opened by Ricyteach #33330: Better error handling in PyImport_Cleanup() https://bugs.python.org/issue33330 opened by serhiy.storchaka #33331: Clean modules in the reversed order https://bugs.python.org/issue33331 opened by serhiy.storchaka #33332: Expose valid signal set (sigfillset()) https://bugs.python.org/issue33332 opened by pitrou #33333: ConfigParser.items returns items present in `DEFAULTSECT` when https://bugs.python.org/issue33333 opened by chrBrd #33335: turtle.onkey doesn't pass key information when key is None https://bugs.python.org/issue33335 opened by je1234 #33336: [imaplib] MOVE is a legal command https://bugs.python.org/issue33336 opened by mcepl #33337: Provide a supported Concrete Syntax Tree implementation in the https://bugs.python.org/issue33337 opened by lukasz.langa #33338: [lib2to3] Synchronize token.py and tokenize.py with the standa https://bugs.python.org/issue33338 opened by lukasz.langa #33339: Using default encoding with `subprocess.run()` is not obvious https://bugs.python.org/issue33339 opened by pekka.klarck #33340: Inaccurate docs on `import` behaviour https://bugs.python.org/issue33340 opened by sam_b #33341: python3 fails to build if directory or sysroot contains "*icc* https://bugs.python.org/issue33341 opened by locutusofborg #33342: urllib IPv6 parsing fails with special characters in passwords https://bugs.python.org/issue33342 opened by benaryorg #33343: [argparse] Add subcommand abbreviations https://bugs.python.org/issue33343 opened by porton #33345: Documentation for PowerShell instructions https://bugs.python.org/issue33345 opened by stevepiercy #33346: Syntax error with async generator inside dictionary comprehens https://bugs.python.org/issue33346 opened by pablogsal #33347: zlibmodule undefined reference https://bugs.python.org/issue33347 opened by Lucian Cristian #33348: lib2to3 doesn't parse f(*[] or []) https://bugs.python.org/issue33348 opened by zsol #33349: 2to3 fails to parse async generators in non-async functions https://bugs.python.org/issue33349 opened by zsol #33350: WinError 10038 is raised when loop.sock_connect is wrapped wit https://bugs.python.org/issue33350 opened by Alisue Lambda #33351: Support compiling with clang-cl on Windows https://bugs.python.org/issue33351 opened by Ethan Smith #33352: Windows: test_regrtest fails on installed Python https://bugs.python.org/issue33352 opened by vstinner #33353: test_asyncio: test_sock_sendfile_mix_with_regular_send() hangs https://bugs.python.org/issue33353 opened by vstinner #33354: Python2: test_ssl fails on non-ASCII path https://bugs.python.org/issue33354 opened by vstinner #33355: Windows 10 buildbot: 15 min timeout on test_mmap.test_large_fi https://bugs.python.org/issue33355 opened by vstinner #33356: Windows 10 buildbot: test__xxsubinterpreters.test_already_runn https://bugs.python.org/issue33356 opened by vstinner #33357: [EASY C] test_posix.test_posix_spawn_file_actions() leaks memo https://bugs.python.org/issue33357 opened by vstinner #33358: [EASY] x86 Ubuntu Shared 3.x: test_embed.test_pre_initializati https://bugs.python.org/issue33358 opened by vstinner #33360: ALternative recipe for password using secrets https://bugs.python.org/issue33360 opened by jpc4242 #33361: readline() + seek() on io.EncodedFile breaks next readline() https://bugs.python.org/issue33361 opened by da #33363: async for statement is not a syntax error in sync context https://bugs.python.org/issue33363 opened by zsol #33365: http/client.py does not print correct headers in debug https://bugs.python.org/issue33365 opened by mstrigl #33366: `contextvars` documentation incorrectly refers to "non-local s https://bugs.python.org/issue33366 opened by tomchristie #33367: Multiprocessing Pool workers initiated with maxtasksperchild d https://bugs.python.org/issue33367 opened by Soumyadipta Das #33369: Removing Popen log files in threads is racy on Windows https://bugs.python.org/issue33369 opened by pbos #33370: Addition of mypy cache to gitignore https://bugs.python.org/issue33370 opened by onlined #33373: ttk modules Label class does not respect background config opt https://bugs.python.org/issue33373 opened by cmaceachern #33374: generate-posix-vars failed when building Python 2.7.14 on Linu https://bugs.python.org/issue33374 opened by piotr.dobrogost Most recent 15 issues with no replies (15) ========================================== #33374: generate-posix-vars failed when building Python 2.7.14 on Linu https://bugs.python.org/issue33374 #33373: ttk modules Label class does not respect background config opt https://bugs.python.org/issue33373 #33370: Addition of mypy cache to gitignore https://bugs.python.org/issue33370 #33367: Multiprocessing Pool workers initiated with maxtasksperchild d https://bugs.python.org/issue33367 #33365: http/client.py does not print correct headers in debug https://bugs.python.org/issue33365 #33356: Windows 10 buildbot: test__xxsubinterpreters.test_already_runn https://bugs.python.org/issue33356 #33353: test_asyncio: test_sock_sendfile_mix_with_regular_send() hangs https://bugs.python.org/issue33353 #33352: Windows: test_regrtest fails on installed Python https://bugs.python.org/issue33352 #33349: 2to3 fails to parse async generators in non-async functions https://bugs.python.org/issue33349 #33348: lib2to3 doesn't parse f(*[] or []) https://bugs.python.org/issue33348 #33346: Syntax error with async generator inside dictionary comprehens https://bugs.python.org/issue33346 #33342: urllib IPv6 parsing fails with special characters in passwords https://bugs.python.org/issue33342 #33339: Using default encoding with `subprocess.run()` is not obvious https://bugs.python.org/issue33339 #33336: [imaplib] MOVE is a legal command https://bugs.python.org/issue33336 #33335: turtle.onkey doesn't pass key information when key is None https://bugs.python.org/issue33335 Most recent 15 issues waiting for review (15) ============================================= #33370: Addition of mypy cache to gitignore https://bugs.python.org/issue33370 #33366: `contextvars` documentation incorrectly refers to "non-local s https://bugs.python.org/issue33366 #33365: http/client.py does not print correct headers in debug https://bugs.python.org/issue33365 #33363: async for statement is not a syntax error in sync context https://bugs.python.org/issue33363 #33358: [EASY] x86 Ubuntu Shared 3.x: test_embed.test_pre_initializati https://bugs.python.org/issue33358 #33354: Python2: test_ssl fails on non-ASCII path https://bugs.python.org/issue33354 #33349: 2to3 fails to parse async generators in non-async functions https://bugs.python.org/issue33349 #33348: lib2to3 doesn't parse f(*[] or []) https://bugs.python.org/issue33348 #33338: [lib2to3] Synchronize token.py and tokenize.py with the standa https://bugs.python.org/issue33338 #33337: Provide a supported Concrete Syntax Tree implementation in the https://bugs.python.org/issue33337 #33336: [imaplib] MOVE is a legal command https://bugs.python.org/issue33336 #33333: ConfigParser.items returns items present in `DEFAULTSECT` when https://bugs.python.org/issue33333 #33332: Expose valid signal set (sigfillset()) https://bugs.python.org/issue33332 #33331: Clean modules in the reversed order https://bugs.python.org/issue33331 #33330: Better error handling in PyImport_Cleanup() https://bugs.python.org/issue33330 Top 10 most discussed issues (10) ================================= #33337: Provide a supported Concrete Syntax Tree implementation in the https://bugs.python.org/issue33337 14 msgs #33312: ubsan undefined behavior sanitizer flags struct _dictkeysobjec https://bugs.python.org/issue33312 9 msgs #33315: Allow queue.Queue to be used in type annotations https://bugs.python.org/issue33315 7 msgs #33277: Deprecate __loader__, __package__, __file__, and __cached__ on https://bugs.python.org/issue33277 6 msgs #33361: readline() + seek() on io.EncodedFile breaks next readline() https://bugs.python.org/issue33361 6 msgs #33275: glob.glob should explicitly note that results aren't sorted https://bugs.python.org/issue33275 5 msgs #33330: Better error handling in PyImport_Cleanup() https://bugs.python.org/issue33330 5 msgs #21822: KeyboardInterrupt during Thread.join hangs that Thread https://bugs.python.org/issue21822 4 msgs #31141: Start should be a keyword argument of the built-in sum https://bugs.python.org/issue31141 4 msgs #31463: test_multiprocessing_fork hangs test_subprocess https://bugs.python.org/issue31463 4 msgs Issues closed (26) ================== #25427: Remove the pyvenv script in Python 3.8 https://bugs.python.org/issue25427 closed by brett.cannon #27485: urllib.splitport -- is it official or not? https://bugs.python.org/issue27485 closed by lukasz.langa #31606: [Windows] Tests failing on installed Python 3.7a1 on Windows https://bugs.python.org/issue31606 closed by vstinner #32209: Crash in set_traverse Within the Garbage Collector's collect_g https://bugs.python.org/issue32209 closed by connorwfitzgerald #32232: building extensions as builtins is broken in 3.7 https://bugs.python.org/issue32232 closed by ncoghlan #32652: test_distutils: BuildRpmTestCase tests fail on RHEL buildbots https://bugs.python.org/issue32652 closed by vstinner #33128: PathFinder is twice on sys.meta_path https://bugs.python.org/issue33128 closed by ncoghlan #33131: Upgrade to pip 10 for Python 3.7 https://bugs.python.org/issue33131 closed by ncoghlan #33258: Unable to install 3.6.5 on Windows Server 2008 https://bugs.python.org/issue33258 closed by hpo0016 #33266: 2to3 doesn't parse all valid string literals https://bugs.python.org/issue33266 closed by zsol #33276: Clarify that __path__ can't be set to just anything https://bugs.python.org/issue33276 closed by brett.cannon #33280: Update link to Tcl/Tk 8.6 man pages in tkinter.rst https://bugs.python.org/issue33280 closed by serhiy.storchaka #33297: Mention Pillow package on tkinter.rst to work with more image https://bugs.python.org/issue33297 closed by serhiy.storchaka #33317: `repr()` of string in NFC and NFD forms does not differ https://bugs.python.org/issue33317 closed by benjamin.peterson #33322: Overridden __getitem__ not called on use of slice syntax when https://bugs.python.org/issue33322 closed by steven.daprano #33323: inconsistent stack trace for exceptions thrown in generators p https://bugs.python.org/issue33323 closed by serhiy.storchaka #33324: Bug in documentation 3.6: string-methods - str.center https://bugs.python.org/issue33324 closed by Alfonso Chavez #33329: sigaddset() can fail on some signal numbers https://bugs.python.org/issue33329 closed by pitrou #33334: Add support of NOP and EXTENDED_ARG in stack_effect() https://bugs.python.org/issue33334 closed by serhiy.storchaka #33344: Use logical negation of integers directly in arithmatic propos https://bugs.python.org/issue33344 closed by Kasra Vand #33359: curses.has_key failure https://bugs.python.org/issue33359 closed by benjamin.peterson #33362: string strip() strips extra characters that it shouldn't https://bugs.python.org/issue33362 closed by steven.daprano #33364: Conditionals not evaluating propertly https://bugs.python.org/issue33364 closed by serhiy.storchaka #33368: Inaccuracy in https://docs.python.org/3/library/re.html#re.mat https://bugs.python.org/issue33368 closed by serhiy.storchaka #33371: Clarify the predicate parameter of inspect.getmembers https://bugs.python.org/issue33371 closed by brian.curtin #33372: Wrong calculation https://bugs.python.org/issue33372 closed by tim.peters

2 years, 10 months

Is PEP 572 really the most effective way to solve the problems it's targeting?

by Ryan Gonzalez

<opinion> I have to say I'm not overly thrilled with PEP 572...it's almost odd, because if you asked me back when I first joined this list when I was 13, I would've no doubt said *YES*. But, since then, I've gone across many projects and languages, and fundamentally *I have never felt hurt by the lack of assignment in an expression*, and I always regretted every time I tried it in C or Crystal. I understand this experience is pretty insignificant in comparison to many of the wizards here, but I thought I'd still share it as an opener for what I'm about to say. </opinion> With this being said, I'd encourage everyone to take a bit of a step back: what exactly are people looking for in PEP 572? I see two main goals: - Assignment in a conditional structure. - Assignment in a list comprehension. Most other use cases would significantly hurt readability and seem pretty rare. Now let's break down the top one: - Assignment in an if condition. - Assignment in a while condition. So there are roughly three main goals here overall. Now, are there better ways to solve these? (FWIW C solved the while condition one with the C-style for loop, but I'm pretty sure few people here would really go for that.) C++ has recently solved the if condition by allowing declarations inside the conditions: if (auto a = 123; a != 456) { Many languages have a 'let' expression (using Felix as my example): if let a = 1, b = 2 in a == b then Swift has taken a bit of a hybrid between the above two: if let a = 1, b = 2, a == b { Now, what's the common theme here? **Declarations should be separate from expressions.** We've got languages that range from baggage-filled to functional to a bit of all of the above, and none of them have added assignment *inside* an expression. The argument is roughly the same across all boards: you're putting major but easy-to-miss side effects in the midst of expressions that *seem* pure. All this is to say: I'd really encourage everyone here to think a bit more about *why* exactly you want this feature, and then think if there's really no better way. Any solution that separates declarations would be far more readable, (arguably) more Pythonic, and play more nicely with the new-ish typing features to boot I understand reluctance to add a syntax exception like this, but I really feel it'd be worth it. As a side note, I was a strong supporter of comprehension generalizations, f-strings, *and* dataclasses. However, this proposal seems a bit ugly and excessive for what it's trying to accomplish. P.S. Yes, the unmatched curly braces were intentional to drive you crazy for a few hours. I blame Randall Monroe. You're welcome. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/

2 years, 10 months

PEP 575 (Unifying function/method classes) update

by Jeroen Demeyer

Hello all, I have updated PEP 575 and its reference implementation. See https://www.python.org/dev/peps/pep-0575/ The main differences with respect to the previous version are: * METH_PASS_FUNCTION now passes the function *in addition* to self (previously, it was passed *instead* of self). * __objclass__ was generalized to __parent__ and stores either the defining class or the defining module of a built-in function/method. * Proposed two-phase implementation for better backwards compatibility (at the cost of added complexity). The first two items on the above list are meant to prepare for PEP 573 but are sufficiently useful by itself to add them to PEP 575. On this mailing list, there have been concerns about backwards compatibility. This PEP does indeed affect code not using duck typing, but using type checks or things like inspect.isbuiltin(). Note that "affect" != "break". I don't know how bad this presumed breakage is. Personally, I think it will be acceptable, but others may disagree. What I *do* know for sure is that very little breaks in the Python standard library. If anybody has a clever idea to estimate the breakage, I would love to know. Jeroen.

2 years, 10 months

Order of positional and keyword arguments

by Serhiy Storchaka

There is an inconsistence in passing arguments to functions. Explicit positional arguments should precede keyword arguments (both explicit and variable), but variable positional arguments can follow explicit keyword arguments and precede variable keyword arguments. For example, for function def f(a=None, b=None): return a, b the following is valid syntax: f(1, b=2) f(1, **{'b': 2}) f(*[1], b=2) f(*[1], **{'b': 2}) f(b=2, *[1]) but the following is an error: f(b=2, 1) f(**{'b': 2}, 1) f(**{'b': 2}, *[1]) f(b=2, *[1]) is surprised in two ways: 1. Argument values are passed not in order. The first value is assigned to the second parameter, and the second value is assigned to the first parameter. 2. Argument values are evaluated not from left to right. This contradicts the general rule that expressions are evaluated from left to right (with few known exceptions). I never seen the form `f(b=2, *[1])` in practice (though the language reference contains an explicit example for it), and it looks weird to me. I don't see reasons of writing `f(b=2, *[1])` instead of more natural `f(*[1], b=2)`. I propose to disallow it. This will also make the grammar simpler. Current grammar: argument_list: `positional_arguments` ["," `starred_and_keywords`] : ["," `keywords_arguments`] : | `starred_and_keywords` ["," `keywords_arguments`] : | `keywords_arguments` positional_arguments: ["*"] `expression` ("," ["*"] `expression`)* starred_and_keywords: ("*" `expression` | `keyword_item`) : ("," "*" `expression` | "," `keyword_item`)* keywords_arguments: (`keyword_item` | "**" `expression`) : ("," `keyword_item` | "," "**" `expression`)* keyword_item: `identifier` "=" `expression` Proposed grammar: argument_list: `positional_arguments` ["," `keywords_arguments`] : | `keywords_arguments` positional_arguments: ["*"] `expression` ("," ["*"] `expression`)* keywords_arguments: `keyword_argument` ("," `keyword_argument`)* keyword_argument: `identifier` "=" `expression` | "**" `expression`

2 years, 10 months

The new and improved PEP 572, same great taste with 75% less complexity!

by Chris Angelico

The most notable change since last posting is that the assignment target is no longer as flexible as with the statement form of assignment, but is restricted to a simple name. Note that the reference implementation has not been updated. ChrisA PEP: 572 Title: Assignment Expressions Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018, 17-Apr-2018, 25-Apr-2018 Abstract ======== This is a proposal for creating a way to assign to variables within an expression. Additionally, the precise scope of comprehensions is adjusted, to maintain consistency and follow expectations. Rationale ========= Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts. Merely introducing a way to assign as an expression would create bizarre edge cases around comprehensions, though, and to avoid the worst of the confusions, we change the definition of comprehensions, causing some edge cases to be interpreted differently, but maintaining the existing behaviour in the majority of situations. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a **named expression** can appear. This is of the form ``name := expr`` where ``expr`` is any valid Python expression, and ``name`` is an identifier. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:: # Handle a matched regex if (match := pattern.search(data)) is not None: ... # A more explicit alternative to the 2-arg form of iter() invocation while (value := read_next_item()) is not None: ... # Share a subexpression between a comprehension filter clause and its output filtered_data = [y for x in data if (y := f(x)) is not None] Differences from regular assignment statements ---------------------------------------------- Most importantly, since ``:=`` is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions. An assignment statement can assign to multiple targets, left-to-right:: x = y = z = 0 The equivalent assignment expression is parsed as separate binary operators, and is therefore processed right-to-left, as if it were spelled thus:: assert 0 == (x := (y := (z := 0))) Statement assignment can include annotations. This would be syntactically noisy in expressions, and is of minor importance. An annotation can be given separately from the assignment if needed:: x:str = "" # works (x:str := "") # SyntaxError x:str # possibly before a loop (x := "") # fine Augmented assignment is not supported in expression form:: >>> x +:= 1 File "<stdin>", line 1 x +:= 1 ^ SyntaxError: invalid syntax Statement assignment is able to set attributes and subscripts, but expression assignment is restricted to names. (This restriction may be relaxed in a future version of Python.) Otherwise, the semantics of assignment are identical in statement and expression forms. Alterations to comprehensions ----------------------------- The current behaviour of list/set/dict comprehensions and generator expressions has some edge cases that would behave strangely if an assignment expression were to be used. Therefore the proposed semantics are changed, removing the current edge cases, and instead altering their behaviour *only* in a class scope. As of Python 3.7, the outermost iterable of any comprehension is evaluated in the surrounding context, and then passed as an argument to the implicit function that evaluates the comprehension. Under this proposal, the entire body of the comprehension is evaluated in its implicit function. Names not assigned to within the comprehension are located in the surrounding scopes, as with normal lookups. As one special case, a comprehension at class scope will **eagerly bind** any name which is already defined in the class scope. A list comprehension can be unrolled into an equivalent function. With Python 3.7 semantics:: numbers = [x + y for x in range(3) for y in range(4)] # Is approximately equivalent to def <listcomp>(iterator): result = [] for x in iterator: for y in range(4): result.append(x + y) return result numbers = <listcomp>(iter(range(3))) Under the new semantics, this would instead be equivalent to:: def <listcomp>(): result = [] for x in range(3): for y in range(4): result.append(x + y) return result numbers = <listcomp>() When a class scope is involved, a naive transformation into a function would prevent name lookups (as the function would behave like a method):: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " prefixed_names = [prefix + name for name in names] With Python 3.7 semantics, this will evaluate the outermost iterable at class scope, which will succeed; but it will evaluate everything else in a function:: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " def <listcomp>(iterator): result = [] for name in iterator: result.append(prefix + name) return result prefixed_names = <listcomp>(iter(names)) The name ``prefix`` is thus searched for at global scope, ignoring the class name. Under the proposed semantics, this name will be eagerly bound; and the same early binding then handles the outermost iterable as well. The list comprehension is thus approximately equivalent to:: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " def <listcomp>(names=names, prefix=prefix): result = [] for name in names: result.append(prefix + name) return result prefixed_names = <listcomp>() With list comprehensions, this is unlikely to cause any confusion. With generator expressions, this has the potential to affect behaviour, as the eager binding means that the name could be rebound between the creation of the genexp and the first call to ``next()``. It is, however, more closely aligned to normal expectations. The effect is ONLY seen with names that are looked up from class scope; global names (eg ``range()``) will still be late-bound as usual. One consequence of this change is that certain bugs in genexps will not be detected until the first call to ``next()``, where today they would be caught upon creation of the generator. Recommended use-cases ===================== Simplifying list comprehensions ------------------------------- A list comprehension can map and filter efficiently by capturing the condition:: results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0] Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:: stuff = [[y := f(x), x/y] for x in range(5)] # There are a number of less obvious ways to spell this in current # versions of Python, such as: # Inline helper function stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] # Extra 'for' loop - potentially could be optimized internally stuff = [[y, x/y] for x in range(5) for y in [f(x)]] # Using a mutable cache object (various forms possible) c = {} stuff = [[c.update(y=f(x)) or c['y'], x/c['y']] for x in range(5)] In all cases, the name is local to the comprehension; like iteration variables, it cannot leak out into the surrounding context. Capturing condition values -------------------------- Assignment expressions can be used to good effect in the header of an ``if`` or ``while`` statement:: # Proposed syntax while (command := input("> ")) != "quit": print("You entered:", command) # Capturing regular expression match objects # See, for instance, Lib/pydoc.py, which uses a multiline spelling # of this effect if match := re.search(pat, text): print("Found:", match.group(0)) # Reading socket data until an empty string is returned while data := sock.read(): print("Received data:", data) # Equivalent in current Python, not caring about function return value while input("> ") != "quit": print("You entered a command.") # To capture the return value in current Python demands a four-line # loop header. while True: command = input("> "); if command == "quit": break print("You entered:", command) Particularly with the ``while`` loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Rejected alternative proposals ============================== Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. Alternative spellings --------------------- Broadly the same semantics as the current proposal, but spelled differently. 1. ``EXPR as NAME``:: stuff = [[f(x) as y, x/y] for x in range(5)] Since ``EXPR as NAME`` already has meaning in ``except`` and ``with`` statements (with different semantics), this would create unnecessary confusion or require special-casing (eg to forbid assignment within the headers of these statements). 2. ``EXPR -> NAME``:: stuff = [[f(x) -> y, x/y] for x in range(5)] This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow ``y <- f(x)`` is not possible in Python, as it would be interpreted as less-than and unary minus.) This syntax has a slight advantage over 'as' in that it does not conflict with ``with`` and ``except`` statements, but otherwise is equivalent. 3. Adorning statement-local names with a leading dot:: stuff = [[(f(x) as .y), x/.y] for x in range(5)] # with "as" stuff = [[(.y := f(x)), x/.y] for x in range(5)] # with ":=" This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable's scope is encoded into its name, making refactoring harder. 4. Adding a ``where:`` to any statement to create local name bindings:: value = x**2 + 2*x where: x = spam(1, 4, 7, q) Execution order is inverted (the indented body is performed first, followed by the "header"). This requires a new keyword, unless an existing keyword is repurposed (most likely ``with:``). See PEP 3150 for prior discussion on this subject (with the proposed keyword being ``given:``). 5. ``TARGET from EXPR``:: stuff = [[y from f(x), x/y] for x in range(5)] This syntax has fewer conflicts than ``as`` does (conflicting only with the ``raise Exc from Exc`` notation), but is otherwise comparable to it. Instead of paralleling ``with expr as target:`` (which can be useful but can also be confusing), this has no parallels, but is evocative. Special-casing conditional statements ------------------------------------- One of the most popular use-cases is ``if`` and ``while`` statements. Instead of a more general solution, this proposal enhances the syntax of these two statements to add a means of capturing the compared value:: if re.search(pat, text) as match: print("Found:", match.group(0)) This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return `''` when done), and completely useless in more complicated cases (eg where the condition is ``f(x) < 0`` and you want to capture the value of ``f(x)``). It also has no benefit to list comprehensions. Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction of possible use-cases, even in ``if``/``while`` statements. Special-casing comprehensions ----------------------------- Another common use-case is comprehensions (list/set/dict, and genexps). As above, proposals have been made for comprehension-specific solutions. 1. ``where``, ``let``, or ``given``:: stuff = [(y, x/y) where y = f(x) for x in range(5)] stuff = [(y, x/y) let y = f(x) for x in range(5)] stuff = [(y, x/y) given y = f(x) for x in range(5)] This brings the subexpression to a location in between the 'for' loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, ``where`` reads the most cleanly, but also has the greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` methods, as does ``tkinter.dnd.Icon`` in the standard library). 2. ``with NAME = EXPR``:: stuff = [(y, x/y) with y = f(x) for x in range(5)] As above, but reusing the `with` keyword. Doesn't read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into "longhand" for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why "with NAME = EXPR:" cannot be used as a statement on its own. 3. ``with EXPR as NAME``:: stuff = [(y, x/y) with f(x) as y for x in range(5)] As per option 2, but using ``as`` rather than an equals sign. Aligns syntactically with other uses of ``as`` for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of ``with`` inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax. Regardless of the spelling chosen, this introduces a stark difference between comprehensions and the equivalent unrolled long-hand form of the loop. It is no longer possible to unwrap the loop into statement form without reworking any name bindings. The only keyword that can be repurposed to this task is ``with``, thus giving it sneakily different semantics in a comprehension than in a statement; alternatively, a new keyword is needed, with all the costs therein. Lowering operator precedence ---------------------------- There are two logical precedences for the ``:=`` operator. Either it should bind as loosely as possible, as does statement-assignment; or it should bind more tightly than comparison operators. Placing its precedence between the comparison and arithmetic operators (to be precise: just lower than bitwise OR) allows most uses inside ``while`` and ``if`` conditions to be spelled without parentheses, as it is most likely that you wish to capture the value of something, then perform a comparison on it:: pos = -1 while pos := buffer.find(search_term, pos + 1) >= 0: ... Once find() returns -1, the loop terminates. If ``:=`` binds as loosely as ``=`` does, this would capture the result of the comparison (generally either ``True`` or ``False``), which is less useful. While this behaviour would be convenient in many situations, it is also harder to explain than "the := operator behaves just like the assignment statement", and as such, the precedence for ``:=`` has been made as close as possible to that of ``=``. Migration path ============== The semantic changes to list/set/dict comprehensions, and more so to generator expressions, may potentially require migration of code. In many cases, the changes simply make legal what used to raise an exception, but there are some edge cases that were previously legal and now are not, and a few corner cases with altered semantics. The Outermost Iterable ---------------------- As of Python 3.7, the outermost iterable in a comprehension is special: it is evaluated in the surrounding context, instead of inside the comprehension. Thus it is permitted to contain a ``yield`` expression, to use a name also used elsewhere, and to reference names from class scope. Also, in a genexp, the outermost iterable is pre-evaluated, but the rest of the code is not touched until the genexp is first iterated over. Class scope is now handled more generally (see above), but if other changes require the old behaviour, the iterable must be explicitly elevated from the comprehension:: # Python 3.7 def f(x): return [x for x in x if x] def g(): return [x for x in [(yield 1)]] # With PEP 572 def f(x): return [y for y in x if y] def g(): sent_item = (yield 1) return [x for x in [sent_item]] This more clearly shows that it is g(), not the comprehension, which is able to yield values (and is thus a generator function). The entire comprehension is consistently in a single scope. The following expressions would, in Python 3.7, raise exceptions immediately. With the removal of the outermost iterable's special casing, they are now equivalent to the most obvious longhand form:: gen = (x for x in rage(10)) # NameError gen = (x for x in 10) # TypeError (not iterable) gen = (x for x in range(1/0)) # ZeroDivisionError def <genexp>(): for x in rage(10): yield x gen = <genexp>() # No exception yet tng = next(gen) # NameError Open questions ============== Importing names into comprehensions ----------------------------------- A list comprehension can use and update local names, and they will retain their values from one iteration to another. It would be convenient to use this feature to create rolling or self-effecting data streams:: progressive_sums = [total := total + value for value in data] This will fail with UnboundLocalError due to ``total`` not being initalized. Simply initializing it outside of the comprehension is insufficient - unless the comprehension is in class scope:: class X: total = 0 progressive_sums = [total := total + value for value in data] At other scopes, it may be beneficial to have a way to fetch a value from the surrounding scope. Should this be automatic? Should it be controlled with a keyword? Hypothetically (and using no new keywords), this could be written:: total = 0 progressive_sums = [total := total + value import nonlocal total for value in data] Translated into longhand, this would become:: total = 0 def <listcomp>(total=total): result = [] for value in data: result.append(total := total + value) return result progressive_sums = <listcomp>() ie utilizing the same early-binding technique that is used at class scope. Frequently Raised Objections ============================ Why not just turn existing assignment into an expression? --------------------------------------------------------- C and its derivatives define the ``=`` operator as an expression, rather than a statement as is Python's way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between ``if (x == y)`` and ``if (x = y)`` belies their drastically different semantics. Thus this proposal uses ``:=`` to clarify the distinction. This could be used to create ugly code! --------------------------------------- So can anything else. This is a tool, and it is up to the programmer to use it where it makes sense, and not use it where superior constructs can be used. With assignment expressions, why bother with assignment statements? ------------------------------------------------------------------- The two forms have different flexibilities. The ``:=`` operator can be used inside a larger expression; the ``=`` statement can be augmented to ``+=`` and its friends, can be chained, and can assign to attributes and subscripts. Why not use a sublocal scope and prevent namespace pollution? ------------------------------------------------------------- Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from ``for`` loops or other constructs, and can be solved the same way: ``del`` the name once it is no longer needed, or prefix it with an underscore. Names bound within a comprehension are local to that comprehension, even in the outermost iterable, and can thus be used freely without polluting the surrounding namespace. (The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2]_) Style guide recommendations =========================== As expression assignments can sometimes be used equivalently to statement assignments, the question of which should be preferred will arise. For the benefit of style guides such as PEP 8, two recommendations are suggested. 1. If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent. 2. If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead. Acknowledgements ================ The author wishes to thank Guido van Rossum and Nick Coghlan for their considerable contributions to this proposal, and to members of the core-mentorship mailing list for assistance with implementation. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/assignment-expressions) .. [2] Pivotal post regarding inline assignment semantics (https://mail.python.org/pipermail/python-ideas/2018-March/049409.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

2 years, 10 months

PEP 572: Why not := as standard assignment operator?

by Martin Teichmann

Hi list, when reading PEP 572 I actually liked it a lot - I think it's actually a cool idea. I think it's actually that cool an idea that it should be made the default way of doing an assignment, over time phasing out the good ole =. This would have several benefits: - people wouldn't have to worry about two different options - different things would have a different look: assignment is :=, keyword args is =, while comparison is ==. Especially beginners would benefit from this clarity. in this case, for sure, we should make it possible to chain :=s, for example by making it bind right-to-left, so that a := b := 3 would be a := (b := 3) I'm sorry if somebody brought that up already, but the discussion has grown so huge that I couldn't read through it entirely. Greetings Martin

2 years, 10 months

Nickname Binding (PEP 572)

by Jim J. Jewett

I think part of the disconnect is that this enhancement could very easily be abused, and it seems likely that it will be, because the problems aren't visible while writing the code -- only when reading it later. I therefore suggest making it very clear in the PEP -- and probably in PEP 8 -- how these expressions should be limited. Simply renaming them to "nickname binding" would be start, but here is a rough draft for wording. When scanning code by eye, it is helpful that assignments are (almost) always at the start of a line. Even def and class statements can cause confusion if the reader didn't realize that the name referred to a class, rather than an instance. Moving assignments to the middle of a line will make it harder for someone else to read your code -- so don't do that. A nickname is just a regular name, except that it also suggests an intimate environment. If the name is purely for documentation, or will be used only later in the same expression (or, possibly, the same block or just after), then a nickname may be appropriate. But * If you are wondering what to do about type hints, then the expression is probably too complex to leave inline. Separate it out into a regular assignment statement; nicknames do not support type hints. * If you will be saving the value -- even as an attribute on self -- there is a chance it will be retrieved in a very different context. Use a regular assignment statement; nicknames are just simple names, not attributes or keys. * If you will be using the value somewhere else in your code, use a regular assignment statement. This makes it easier to find, and warns people that the value may be used again later. -jJ

2 years, 10 months

Boundaries between numbers and identifiers

by Serhiy Storchaka

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize this example: $ echo '0or[]' | ./python -m tokenize 1,0-1,1: NUMBER '0' 1,1-1,3: NAME 'or' 1,3-1,4: OP '[' 1,4-1,5: OP ']' 1,5-1,6: NEWLINE '\n' 2,0-2,0: ENDMARKER '' On other hand, all these examples look weird. There is an assymmetry: `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize visually the boundary between a number and the following identifier or keyword, especially if numbers can contain letters ("b", "e", "j", "o", "x") and underscores, and identifiers can contain digits. On both sides of the boundary can be letters, digits, and underscores. I propose to change the Python syntax by adding a requirement that there should be a whitespace or delimiter between a numeric literal and the following keyword.

2 years, 10 months

← Newer
1
2
3
4
5
6
7
8
9
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev April 2018