Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

On 22 June 2018 at 02:26, Antoine Pitrou <solipsis@pitrou.net> wrote:
I asked a handful of folks at the Education Summit the next day about it: * for the basic notion of allowing expression level name binding using the "NAME := EXPR" notation, the reactions ranged from mildly negative (I read it as only a "-0" rather than a "-1") to outright positive. * for the reactions to my description of the currently proposed parent local scoping behaviour in comprehensions, I'd use the word "horrified", and feel I wasn't overstating the response :) While I try to account for the fact that I implemented the current comprehension semantics for the 3.x series, and am hence biased towards considering them the now obvious interpretation, it's also the case that generator expressions have worked like nested functions since they were introduced in Python 2.4 (more than 13 years ago now), and comprehensions have worked the same way as generator expressions since Python 3.0 (which has its 10th birthday coming up in December this year). This means that I take any claims that the legacy Python 2.x interpretation of comprehension behaviour is intuitively obvious with an enormous grain of salt - for the better part of a decade now, every tool at a Python 3 user's disposal (the fact that the iteration variable is hidden from the current scope, reading the language reference [1], printing out locals(), using the dis module, stepping through code in a debugger, writing their own tracing function, and even observing the quirky interaction with class scopes) will have nudged them towards the "it's a hidden nested function" interpretation of expected comprehension behaviour. Acquiring the old mental model for the way comprehensions work pretty much requires a developer to have started with Python 2.x themselves (perhaps even before comprehensions and lexical closures were part of the language), or else have been taught the Python 2 comprehension model by someone else - there's nothing in Python 3's behaviour to encourage that point of view, and plenty of functional-language-inspired documentation to instead encourage folks to view comprehensions as tightly encapsulated declarative container construction syntax. I'm currently working on a concept proposal at https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572 than any of my previous `given` based suggestions: for already declared locals, it devolves to being the same as PEP 572 (except that expressions are allowed as top level statements), but for any names that haven't been previously introduced, it prohibits assigning to a name that doesn't already have a defined scope, and instead relies on a new `given` clause on various constructs that allows new target declarations to be introduced into the current scope (such that "if x:= f():" implies "x" is already defined as a target somewhere else in the current scope, while "if x := f() given x:" potentially introduces "x" as a new local target the same way a regular assignment statement does). One of the nicer features of the draft proposal is that if all you want to do is export the iteration variable from a comprehension, you don't need to use an assignment expression at all: you can just append "... given global x" or "... given nonlocal x" and export the iteration variable directly to the desired outer scope, the same way you can in the fully spelled out nested function equivalent. Cheers, Nick. [1] From https://docs.python.org/3.0/reference/expressions.html#displays-for-lists-se...: 'Note that the comprehension is executed in a separate scope, so names assigned to in the target list don’t “leak” in the enclosing scope.' -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, 23 Jun 2018 00:22:33 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Thank you. Personally, I'd like to see feedback from educators/teachers after they take the time to read the PEP and take some time to think about its consequences. My main concern is we're introducing a second different way of doing something which is really fundamental.
Hmm... I don't think conflating the assignment expression proposal with comprehension semantics issues is helping the discussion. Regards Antoine.

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
I've started testing the proposed syntax when I teach. I don't have a large sample yet, but most students either dislike it or don't appreciate the benefits. They state a clear preference for shorter, simpler lines at the consequence of more lines of code. This may partially be caused by the smaller screen real estate on a projector or large TV than a desktop monitor. My intuition is that one strength of Python for beginners is the relative lack of punctuation and operators compared with most other languages. This proposal encourages denser lines with more punctuation. Because of the order of operations, many uses of ``:=`` will also require parentheses. Even relatively simple uses, like ``if (match := pattern.search(data)) is not None:`` require doubled parentheses on one side or the other. Beginners are especially prone to typographical errors with mismatched parentheses and missing colons and get easily frustrated by the associated syntax errors. Given the following options: A. if (row := cursor.fetchone()) is None: raise NotFound return row B. row = cursor.fetchone() if row is None: raise NotFound return row C. if (row := cursor.fetchone()) is not None: return row raise NotFound D. row = cursor.fetchone() if row is not None: return row raise NotFound The majority of students preferred option B. I also tested some regex match examples. Results were similar.
My main concern is we're introducing a second different way of doing something which is really fundamental.
The few students who like the proposal ask why it requires creating a new operator instead of repurposing the ``=`` operator. I'll reserve my personal opinions for a different thread.

On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mike@selik.org> wrote:
This is partly because students, lacking the experience to instantly recognize larger constructs, prefer a more concrete approach to coding. "Good code" is code where the concrete behaviour is more easily understood. As a programmer gains experience, s/he learns to grok more complex expressions, and is then better able to make use of the more expressive constructs such as list comprehensions. ChrisA

On Fri, Jun 22, 2018 at 10:19 AM Chris Angelico <rosuav@gmail.com> wrote:
I don't think that's the only dynamic going on here. List comprehensions are more expressive, but also more declarative and in Python they have nice parallels with SQL and speech patterns in natural language. The concept of a comprehension is separate from its particular expression in Python. For example, Mozilla's array comprehensions in Javascript are/were ugly [0]. Students who are completely new to programming can see the similarity of list comprehensions to spoken language. They also appreciate the revision of certain 3-line and 4-line for-loops to comprehensions. I didn't get the same sense of "Oh! That looks better!" from my students when revising code with an assignment expression. Despite my best efforts to cheerlead, some students initially dislike list comprehensions. However, they come around to the idea that there's a tradeoff between line density and code block density. Comprehensions have a 3-to-1 or 4-to-1 ratio of code line shrinkage. They're also often used in sequence, like piping data through a series of transforms. Even if students dislike a single comprehension, they agree that turning 15 lines into 5 lines improves the readability. In contrast, an assignment expression only has a 2-to-1 code line compression ratio. It might save a level of indentation, but I think there are usually alternatives. Also, the assignment expression is less likely to be used several times in the same block. A good pitch for an assignment expression is refactoring a cascade of regular expressions: for line in f: mo = foo_re.search(line) if mo is not None: foo(mo.groups()) continue mo = bar_re.search(line) if mo is not None: bar(mo.groups()) continue mo = baz_re.search(line) if mo is not None: baz(mo.groups()) continue Here the assignment operator makes a clear improvement: for line in f: if (mo := foo_re.search(line)) is not None: foo(mo.groups()) elif (mo := bar_re.search(line)) is not None: bar(mo.groups()) elif (mo := baz_re.search(line)) is not None: baz(mo.groups()) However, I think this example is cheating a bit. While I've written similar code many times, it's almost never just a function call in each if-block. It's nearly always a handful of lines of logic which I wouldn't want to cut out into a separate function. The refactor is misleading, because I'd nearly always make a visual separation with a newline and the code would still look similar to the initial example. [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/...

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
Of course they do -- they're less fluent at reading code. They don't have the experience to judge good code from bad. The question we should be asking is, do we only add features to Python if they are easy for beginners? It's not that I especially want to add features which *aren't* easy for beginners, but Python isn't Scratch and "easy for beginners" should only be a peripheral concern.
Mozilla's array comprehensions are almost identical to Python's, aside from a couple of trivial differences: evens = [for (i of numbers) if (i % 2 === 0) i]; compared to: evens = [i for i in numbers if (i % 2 == 0)] - the inexplicable (to me) decision to say "for x of array" instead of "for x in array"; - moving the expression to the end, instead of the beginning. The second one is (arguably, though not by me) an improvement, since it preserves a perfect left-to-right execution order within the comprehension.
Students who are completely new to programming can see the similarity of list comprehensions to spoken language.
o_O I've been using comprehensions for something like a decade, and I can't :-) The closest analogy to comprehensions I know of is set builder notation in mathematics, which is hardly a surprise. That's where Haskell got the inspiration from, and their syntax is essentially an ASCIIfied version of set builder notation: Haskell: [(i,j) | i <- [1,2], j <- [1..4]] Maths: {(i,j) : i ∈ {1, 2}, j ∈ {1...4}} I teach secondary school children maths, and if there's a plain English natural language equivalent to list builder notation, neither I nor any of my students, nor any of the text books I've read, have noticed it. -- Steve

On 23.06.2018 5:46, Steven D'Aprano wrote:
Python's design principles are expressed in the Zen. They rather focus on being no more complex than absolutely necessary, without prioritizing either beginners or old-timers ("simple is better than complex", "complex is better than complicated").
-- Regards, Ivan

On 2018-06-22 19:46, Steven D'Aprano wrote:
- the inexplicable (to me) decision to say "for x of array" instead of "for x in array";
Believe JavaScript has for…in, but as usual in the language it is broken and they needed a few more tries to get it right. for…of is the latest version and works as expected. -Mike

On Fri, Jun 22, 2018 at 10:02 AM Michael Selik <mike@selik.org> wrote:
I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mike@selik.org> wrote:
nor would I. For a while, anyway.... But once it becomes a more common idiom, students will see it in the wild pretty early in their path to learning python. So we'll need to start introducing it earlier than later. I think this reflects that the "smaller" a language is, the easier it is to learn. Python has already grown a fair bit since 1.5 (when I started using it :-) ). Some things, like generators, are special purpose enough that I can wait pretty far into the program before teaching them. But others, like comprehensions (and lambda) are common enough that I have to introduce them pretty early on. Adding := is not a HUGE change, but it IS an expansion of the language, and one that we WILL have to introduce in an introductory course once it starts seeing common use. I really have no idea how much harder thats going to make the langauge to teach, but it will make it a bit harder -- I see enough confusion with "is" vs == already... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Jun 22, 2018 at 11:28:45AM -0700, Chris Barker via Python-Dev wrote:
Students see many features early in their path. I've had people still struggling with writing functions ask about metaclasses. People will see async code everywhere. We don't have to teach *everything* at once. The *subtleties* of assignment expressions might have some funny corner cases, but the high-level overview is simple. It is like ordinary assignment, but it is an expression that returns the value being assigned. So if you absolutely need to teach it to a beginner, it shouldn't be difficult once they understand the difference between an expression and a statement. [...]
I think that the biggest source of confusion with "is" is that it *sometimes* seems to do what is wanted, i.e. test equality, but other times doesn't. It is that inconsistency that bites. Whereas with assignment expressions, there's no such inconsistency: - regular assignment using = only works as a statement, always; - assignment expression can go anywhere an expression can go, always; - regular assignment never returns a value; - assignment expression always returns a value; - regular assignments have lots of complex forms, such as sequence unpacking, and complex targets like spam[eggs](arg).attr; - assignment expressions only takes a plain name, always. Although there is some overlap in behaviour between the two, unlike "is", there's no inconsist behaviour to lead people astray. A better syntax error for things like this: py> if mo = regex.match(string): File "<stdin>", line 1 if mo = regex.match(string): ^ SyntaxError: invalid syntax will also help, although of course some users won't read error messages for love or money. -- Steve

On Fri, Jun 22, 2018 at 7:23 PM, Steven D'Aprano <steve@pearwood.info> wrote:
These are not similar at all -- if you want similar examples, I"d say comprehensions, and lambda, both of which I DO introduce fairly early While newbies will *ask* about metaclasses, it's probably because they read about them somewhere, not because someone actually used a metaclass in a simple script or answer to a common question on SO. As for async, you are either doing async or not -- you can't even run an async def function without an event loop -- so again, it won't show up in real code newbies need to understand (at least until async becomes common practice with python...) -CHB So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an expression and a statement.
probably not, though that's a distinction that's mostly academic in the early stages of learning, it may become more critical now... again, not a huge deal, just a little bit more complexity -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Jun 22, 2018 at 09:08:37PM -0700, Chris Barker wrote:
I don't think so. People do try to use assignment in expressions, even if only by mistake writing = when they meant == and need to distinguish between them. In Python 2, the most common clash between statements and expressions was print, but at least that's gone. https://www.quora.com/Whats-the-difference-between-a-statement-and-an-expres... https://stackoverflow.com/questions/4728073/what-is-the-difference-between-a... https://stackoverflow.com/questions/43435850/what-is-the-difference-between-... Even without assignment expressions, people still need to know why they can't write "if mo = re.match(pattern, text)".
again, not a huge deal, just a little bit more complexity
Every new feature is added complexity. -- Steve

On Fri, Jun 22, 2018 at 7:28 PM, Chris Barker via Python-Dev < python-dev@python.org> wrote:
For what it's worth, Chris's thoughts are close to my own here. I and several of my colleagues teach week-long Python courses for Enthought. The target audience is mostly scientists and data scientists (many of whom are coming from MATLAB or R or IDL or Excel/VBA or some other development environment, but some of whom are new to programming altogether), and our curriculum is Python, NumPy, SciPy, Pandas, plus additional course-specific bits and pieces (scikit-learn, NLTK, seaborn, statsmodels, GUI-building, Cython, HPC, etc., etc.). There's a constant struggle to keep the Python portion of the course large enough to be coherent and useful, but small enough to allow time for the other topics. To that end, we separate the Python piece of the course into "core topics" that are essential for the later parts, and "advanced topics" that can be covered if time allows, or if we get relevant questions. I can't see a way that the assignment expression wouldn't have to be part of the core topics. async stuff only appears in async code, and it's easy to compartmentalize; in contrast, I'd expect that once the assignment expression took hold we'd be seeing it in a lot of code, independent of the domain. And yes, I too see enough confusion with "is" vs == already, and don't relish the prospect of teaching := in addition to those. That's with my Python-teaching hat on. With my Python-developer hat on, my thoughts are slightly different, but that's off-topic for this thread, and I don't think I have anything to say that hasn't already been said many times by others, so I'll keep quiet about that bit. :-) -- Mark

Nick Coghlan wrote:
Noooo..... this is just taking a bad idea and making it worse, IMO. I'm -1 on any contortions designed to allow comprehensions to assign to things in outer scopes. All the proposed use cases I've seen for this have not improved readability over writing a function that does things the usual way. Can we please leave comprehensions as declarative constructs? The best tools do just one thing and do it well. These proposals seem to be trying to turn comprehensions into swiss army knives. -- Greg

On 23 June 2018 at 09:06, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If PEP 572 was proposing the use of regular local scoping for assignment expressions in comprehensions, such that they could still be used to avoiding repeating subexpressions within an iteration, but couldn't be used to export progress data, or to maintain a partial sum without having to rely on `locals().get("total", 0)` to provide an initial value, then I wouldn't be feeling obliged to present an alternative that offers the same state export capabilities in a more explicit form. Given that PEP 572 *is* proposing implicit comprehension state export, though, then I think it's important to make the case that seeing the proposed semantics as intuitive is only going to be the case for folks that have used Python 2 style comprehensions extensively - anyone that's never encountered the old state-leaking behaviour for iteration variables is going to be surprised when assignment expressions ignore the existence of the comprehension scope (even though the iteration variable pays attention to it). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
Given that PEP 572 *is* proposing implicit comprehension state export,
"Implicit" and "explicit" are two terms which often get misused to mean "I don't like it" and "I do like it". Making the intentional choice to use an assignment expression is not really "implicit" in any meaningful sense. One might as well complain that "import this" implicitly creates a local variable "this". Well, yes, it does, in a very loose sense, but that's what imports are defined as do and it is the whole purpose for making them. If PEP 572's proposal goes ahead, the behaviour of assignment expressions will be *defined* as creating assignments in the local scope rather than the sublocal comprehension scope. To call that "implicit" is rather like saying that regular assignment is implicit.
You are making the assumption that most people are even aware of "comprehension scope". I don't think that is the case. In my experience, scoping in Python is still typically seen as the LGB rule (locals/globals/builtins). See for example this StackOverflow post from 2016: https://stackoverflow.com/questions/37211910/override-lgb-scope-rule Sometimes people remember the E/N (enclosing function/nonlocal) part. Hardly anyone remembers the C (class) part unless they are actively thinking in terms of code running inside a class definition, and even if they do, they typically aren't sure of exactly how it interacts with the rest. And I predict that even fewer think of comprehensions as a separate scope, except by ommission: they *don't think about* the scope of the loop variable until it bites them. But as Tim Peters has previously discussed, the loop variable is special, and is especially prone to accidental shadowing. That won't be the case for assignment expressions. If there's shadowing going on, it will be deliberate. Aside: I've said before that I'm not a fan of sublocal comprehension scoping, since I personally found it helpful on occasion for the loop variable to be visible outside of the comprehension. But given that the only experience most people apparently had with comprehension scoping was to be bitten by it, I grudgingly accept that encapsulating the loop variable was the right decision to make, even if it personally inconvenienced me more than it saved me. Nor was I the only one: others have been bitten by the change to comprehension scope, see for example: https://www.reddit.com/r/Python/comments/425qmb/strange_python_27_34_differe... There is no consensus that the change to comprehensions was a good thing or justified. The bottom line is that I don't think people will be surprised by assignment expression scope being local instead of sublocal. Rather I expect that they won't even think about it, until they do, and then *whatever* behaviour we pick, we'll annoy somebody. -- Steve

On 24 June 2018 at 15:56, Steven D'Aprano <steve@pearwood.info> wrote:
No, it's actually implicit: there's an extra "global NAME" or "nonlocal NAME" in the equivalent code for a comprehension that isn't there in the as-written source code, and doesn't get emitted for a regular assignment expression or for the iteration variable in a comprehension - it only shows up due to the defined interaction between comprehensions and assignment expressions.
And they behave the same way in every context where they're permitted to appear.
I do say that regular assignments implicitly declare a name as local. "Python has implicit local variable declarations" is also regularly cited as one of the differences between it and languages that require explicit declarations, like C. Even augmented assignments implicitly declare a name as being a local (hence the infamous UnboundLocalError that arises when you attempt to use an augmented assignment to rebind a name from an outer scope). The problem I have with PEP 572 is that it proposes *breaking that otherwise universal pattern* - instead of having assignment expressions in comprehensions implicitly declare the name as local in the nested comprehension scope, it instead has them: 1. implicitly declare the name as global or as nonlocal in the comprehension (or else raise an exception), depending on the nature of the parent scope where the comprehension is used 2. in the nonlocal reference case, amend the symbol table analysis to act like there was a preceding "if 0:\n for NAME in ():\n pass" in the parent scope (so the compiler knows which outer function scope to target) The rationale being given for why that is OK is: 1. "Everyone" thinks comprehensions are just a for loop (even though that hasn't been true for the better part of a decade, and was never true for generator expressions) 2. If comprehensions are just a for loop, then assignment expressions inside them should be local to the containing scope 3. Therefore the implicit declarations required to tie everything together and allow folks to continue with an incorrect understanding of how comprehensions work aren't really implicit - they're explicit in the inaccurate expansion of the construct! Can you imagine the reaction if anyone other than Guido or Tim was attempting to argue for a change to the language that only makes sense if we grant a completely inaccurate understanding of how a particular language construct works as being a credible starting point? Because that's how this comprehension scoping argument feels to me: Proposal author: "If the language worked in a way other than it does, then this proposal would make perfect sense." Proposal reviewer: "Yes, but it doesn't work that way, it works this way. We deliberately changed it because the old way caused problems." Proposal author: "Ah, but it *used* to work that way, and a lot of people still think it works that way, and we can get the compiler to jump through hoops to pretend it still works that way, except for the parts of the new way that we want to keep." Proposal reviewer: "..." Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The implicit "nonlocal NAME" is only because there is an equally implicit function boundary. Why is there a function boundary marked by square brackets? It's not saying "def" or "lambda", which obviously create functions. It's a 'for' loop wrapped inside a list display. What part of that says "hey, I'm a nested function"? So if there's an implicit function, with implicit declaration of a magical parameter called ".0", why can't it have an equally implicit declaration that "spam" is a nonlocal name? ChrisA

On 24 June 2018 at 16:53, Chris Angelico <rosuav@gmail.com> wrote:
Nothing - that's why I refer to them as implicitly nested scopes (vs the explicitly nested scopes in functions and lambda expressions, where the scope is introduced via keyword). However, there's still a major behavioural tell at runtime that they're running in a nested scope: the iteration variables don't leak. (There are other tells as well, but not ones that most folks are likely to encounter)
Because comprehensions don't do that for their iteration variables, because assignment expressions don't do that when used in explicitly nested scopes, because the required implicit scope declarations are context dependent, and because even such gyrations still can't hide the existence of the comprehension's implicitly nested scope when dealing with classes and the two-argument form of exec(). Since the implicitly nested scopes can't be hidden, it makes far more sense to me to just admit that they're there, and provide explicit syntax for cases where folks decide they really do want name bindings to leak out of that scope (whether those name bindings are assignment expression targets or the iteration variables themselves). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 24.06.2018 9:53, Chris Angelico wrote:
My 2c. An expression is intuitively thought to be self-contained i.e. without side effects. if I write `a=b+1`, I'm not expecting it to do anything except assigning `a'. Expressions with side effects has long since proven to be problematic because of the implicit (thus hard to see and track) links they create (and because the result depends on the order of evaluation). Moreover, Python's other design elements have been consistently discouraging expressions with side effects, too (e.g. mutator methods intentionally return None instead of the new value, making them useless in expressions), so the proposition is in direct conflict with the language's design. Assignment expressions are a grey area: they carry the full implications of expressions with side effects described above, but their side effect is their only effect, i.e. they are explicit and prominent about the "evil" they do.
-- Regards, Ivan

On Sun, Jun 24, 2018 at 05:24:12PM +0300, Ivan Pozdeev via Python-Dev wrote:
a = d.pop(1) a = d.setdefault(key, 0) chars_written = file.write(text)
If you're going to take a hard-core functional approach to side-effects, I think you are using the wrong language. Nearly everything in Python *could* have side-effects (even if usually it won't). Even your own example of "b+1" (depending on what b.__add__ does).
I don't think that's the reason why mutator methods return None. They return None rather than self to avoid confusion over whether they return a copy or not. https://docs.python.org/3/faq/design.html#why-doesn-t-list-sort-return-the-s...
so the proposition is in direct conflict with the language's design.
Python is full of operations with side-effects. Besides, they're not quite useless: (alist.append() or alist) is functionally equivalent to alist.append returning self. Just a bit more verbose. Methods (and functions) all return a value, even if that value is None, so they can be used in expressions. If Guido wanted Pascal style procedures, which cannot be used in expressions, we would have them by now :-) -- Steve

A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline assignment, (b) when := is used in a comprehension, set the scope for the target as if the assignment occurred outside any comprehensions. It seems we have more support for (a) than for (b) -- at least Nick and Greg seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his essay on the topic, included as Appendix A ( https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings) does not even mention comprehensions. However, he did post his motivation for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the PEP gives a strong motivation ( https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless, maybe we should compromise and drop (b)? -- --Guido van Rossum (python.org/~guido)

On Sun, Jun 24, 2018 at 09:24:39AM -0700, Guido van Rossum wrote:
I'm not sure who came up with the idea first, but as I remember it, the first mention of this came in a separate thread on Python-Ideas: https://mail.python.org/pipermail/python-ideas/2018-April/049631.html so possibly I'm to blame :-) That thread starts here: https://mail.python.org/pipermail/python-ideas/2018-April/049622.html If I did get the idea from Tim, I don't remember doing so.
I will have more to say about the whole "comprehensions are their own scope" issue later. But I'd like to see Nick's proposed PEP, or at least a draft of it, before making any final decisions. If it came down to it, I'd be happy with the ability to declare an assignment target nonlocal in the comprehension if that's what it takes. What do you think of this syntax? [global|nonlocal] simple_target := expression Inside a comprehension, without a declaration, the target would be sublocal (comprehension scope); that should make Nick happier :-) -- Steve

On Sun, Jun 24, 2018 at 11:50 AM Steven D'Aprano <steve@pearwood.info> wrote:
Actually that post sounds like the OP of that thread (Peter O'Connor) is to blame -- he proposed a similar thing using `=` for the assignment and custom syntax (`from <value>`) to specify the initial value, and it looks like that inspired you.
Agreed, though I assume it's just `given` again.
It's more special syntax. Just taking part (a) of PEP 572 would make most people happy enough. -- --Guido van Rossum (python.org/~guido)

On 25 June 2018 at 09:02, Guido van Rossum <guido@python.org> wrote:
While I still have some TODO notes of my own to resolve before posting it to python-ideas, the examples section at https://github.com/ncoghlan/peps/pull/2/files#diff-7a25ca1769914c1141cb5c63d... already gives a pretty good idea of the differences relative to PEP 572: rebinding existing names is unchanged from PEP 572, but introducing new names requires a bit of "Yes, I really do want to introduce this new name here" repetition. The big difference from previous iterations of the "given" idea is that it doesn't try to completely replace the proposed inline assignments, it just supplements them by providing a way to do inline name *declarations* (which may include declaring targets as global or nonlocal, just as regular function level declarations can). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

[Guido]
I was writing up my observations about simple changes to existing code. Since there's nothing sanely akin to binding non-for-targets possible in comprehensions now, comprehensions were out of scope for that effort (which was limited to staring at existing code already doing bindings). :> However, he did post his motivation for (b) on python-ideas, IIRC a bit
Two things to say about that. First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say) p = None # to establish the intended scope of `p` while any(<nonlocal p> # split across lines just for readability n % p == 0 for p in small_primes): n //= p It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour. Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can become visible when bindings can be embedded: def f(): y = -1 ys = [y for _ in range(y := 5)] print(y, ys) Here `range(y := 5)` is executed in f's scope. Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced. It's incoherent without detailed knowledge of the implementation. def g(): y = -1 ys = [y for y in range(y := 5)] print(y, ys) And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call. That's incoherent in its own way. Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

On Sun, Jun 24, 2018 at 12:03 PM Tim Peters <tim.peters@gmail.com> wrote:
Hm, that's more special syntax. The nice bit about (b) as currently specified is that it adds no syntax -- it adds a scope rule, but (as IIRC Steven has convincingly argued) few people care about those. Python's scope rules, when fully specified, are intricate to the point of being arcane (e.g. for class scopes) but all that has a purpose -- to make them so DWIM ("Do what I Mean") that in practice you almost never have to worry about them, *especially* when reading non-obfuscated code (and also when writing, except for a few well-known patterns).
That code should have the same meaning regardless of whether we accept (b) or not -- there is only one `y`, in f's scope. I don't mind if we have to add more words to the PEP's scope rules to make this explicit, though I doubt it -- the existing weirdness (in the comprehension spec) about the "outermost iterable" being evaluated in the surrounding scope specifies this. I wouldn't call it incoherent -- I think what I said about scope rules above applies here, it just does what you expect.
And under the (b)-less proposal, `g` would interpret `y for y` as both referring to a new variable created just for the comprehension, and `y := 5` as referring to g's scope. Again I don't think it needs extra words in the spec. And the end user docs might just say "don't do that" (with a link to the reference manual's rule about the "outermost iterable"). Even if in the end we did find a case where we'd have to write an explicit rule to make what happens here a consequence of the spec rather than the implementation, that doesn't count as an argument for keeping (b) to me. In favor of (b) we have a few examples (see https://www.python.org/dev/peps/pep-0572/#scope-of-the-target) that require it, and more that you described on python-ideas (and also the motivating use case from the thread that Steven dug up, starting here: https://mail.python.org/pipermail/python-ideas/2018-April/049622.html). A "neutral" argument about (b) is that despite the "horrified" reactions that Nick saw, in practice it's going to confuse very few people (again, due to my point about Python's scope rules). I'd wager that the people who might be most horrified about it would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement, and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3. -- --Guido van Rossum (python.org/~guido)

[Guido]
Hm, that's more special syntax.
Of course - I'm anticipating that the PEP will be changed to throw out useful assignment expressions in comprehensions, but I still want a way to "export" comprehension for-targets at times ;-)
You and Steven and i appear to be on the same page here - but it's in a book nobody else seems to own :-( To me it's just screamingly obvious that total = 0 cumsums = [total := total + value for value in data] "should do" what it obviously intends to do - and that the only thing stopping that is a bass-ackwards focus on what most trivially falls out of the current implementation. ... def f():
Remove "y = -1" and - voila! - we have the dreaded "parent local scoping" Nick finds so baffling to explain (or so he claims). That is, "has exactly the same scope in the comprehension as in the parent block, and will create a local in the latter if the name is otherwise unknown in the parent" comes with assignment expressions, regardless of whether _all_ such targets "leak" (the current PEP) or only targets in the expression defining the iterable of the outermost `for` (the PEP without leaking assignment expressions in comprehensions). As to whether it "does what you expect", no, not really! In a world where _all_ binding targets in a comprehension are claimed to be local to the comprehension, I _expect_ that `y := 5` appearing inside the listcomp means `y` is local to the listcomp. "Oh - unless the binding appears in the expression defining the iterable of the outermost `for`" comes from Mars. Not that it really matters much, but (b) provides consistent semantics in these cases. No need to search Mars for weird exceptions ;-) ...
I also doubt it will generally confuse people in practice (to the contrary, I expect they _will_ be confused if things like the cumulative sums example blow up with UnboundLocalError). But I still don't get the source of the "horror". Assignment expression semantics are wholly consistent with ordinary nested lexical scoping, with or without (b). The only difference is in the scopes picked for assignment expression target names (except for those appearing in the expression defining the iterable yadda yadda yadda).

On 25 June 2018 at 09:25, Guido van Rossum <guido@python.org> wrote:
FWIW, the most cryptic parent local scoping related exception I've been able to devise so far still exhibits PEP 572's desired "Omitting the comprehension scope entirely would give you the same name lookup behaviour" semantics: >>> def outer(x=1): ... def middle(): ... return [x := x + i for i in range(10)] ... return middle() ... >>> outer() Traceback (most recent call last): ... NameError: free variable 'x' referenced before assignment in enclosing scope It isn't the parent local scoping, or even the assignment expression, that's at fault there, since you'd get exactly the same exception for: def outer(x=1): def middle(): x = x +1 return x return middle() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2018 at 22:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
Oops, I didn't mean to say "exactly the same exception" here, as the whole reason I'd settled on this example as the most cryptic one I'd found so far was the fact that the doubly nested version *doesn't* give you the same exception as the singly nested version: the version without the comprehension throws UnboundLocalError instead. However, the resolution is the same either way: either 'x' has to be declared as 'nonlocal x' in 'middle', or else it has to be passed in to 'middle' as a parameter. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2018 at 13:24, Nick Coghlan <ncoghlan@gmail.com> wrote:
At the level of "what my intuition says" the result is the same in both cases - "it throws an exception". I have no intuition on *which* exception would be raised and would experiment (or look up the details) if I cared.
Once someone told me that's what I needed, it's sufficiently obvious that I'm fine with that. If no-one was able to tell me what to do, I'd simply rewrite the code to be less obfuscated :-) I've probably explained my intuition enough here. If we debate any further I'll just end up knowing what's going on, destroying my value as an "uninformed user" :-) Paul

On 25 June 2018 at 13:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again offering an "intuition" based response: 1. That definition of outer() is very complicated, I don't *expect* to understand it without checking the details. So the NameError is simply "hmm, wonder what triggered that?" not "OMG that's not what I'd expect!" :-) 2. Given that your version with no assignment expression or comprehension exhibits the same behaviour, I'm not sure what your argument is here anyway... Paul

On 6/24/2018 7:25 PM, Guido van Rossum wrote:
I'd wager that the people who might be most horrified about it
the (b) scoping rule change
would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement,
I might not be one of those 'most horrified' by (b), but I increasingly don't like it, and I was at best -0 on the comprehension scope change. To me, iteration variable assignment in the current scope is a non-problem. So to me the change was mostly useless churn. Little benefit, little harm. And not worth fighting when others saw a benefit. However, having made the change to nested scopes, I think we should stick with them. Or repeal them. (I believe there is another way to isolate iteration names -- see below). To me, (b) amounts to half repealing the nested scope change, making comprehensions half-fowl, half-fish chimeras.
and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3.
That I pretty much am, I think. In Python 2, comprehensions (the fish) were, at least in effect, expanded in-line to a normal for loop. Generator expressions (the fowls) were different. They were, and still are, expanded into a temporary generator function whose return value is dropped back into the original namespace. Python 3 turned comprehensions (with 2 news varieties thereof) into fowls also, temporary functions whose return value is dropped back in the original namespace. The result is that a list comprehension is equivalent to list(generator_ expression), even though, for efficiency, it is not implemented that way. (To me, this unification is more a benefit than name hiding.) (b) proposes to add extra hidden code in and around the temporary function to partly undo the isolation. list comprehensions would no longer be equivalent to list(generator_expression), unless generator_expressions got the same treatment, in which case they would no longer be equivalent to calling the obvious generator function. Breaking either equivalence might break someone's code. --- How loop variables might be isolated without a nested scope: After a comprehension is parsed, so that names become strings, rename the loop variables to something otherwise illegal. For instance, i could become '<i>', just as lambda becomes '<lambda>' as the name of the resulting function. Expand the comprehension as in Python 2, except for deleting the loop names along with the temporary result name. Assignment expressions within a comprehension would become assignment expressions within the for loop expansion and would automatically add or replace values in the namespace containing the comprehension. In other words, I am suggesting that if we want name expressions in comprehensions to act as they would in Python 2, then we should consider reverting to an altered version of the Python 2 expansion. --- In any case, I think (b) should be a separate PEP linked to a PEP for (a). The decision for (a) could be reject (making (b) moot), accept with (b), or accept unconditionally (but still consider (b)). -- Terry Jan Reedy

On 26.06.2018 1:58, Greg Ewing wrote:
Chris Angelico wrote:
The wheel turns round and round, and the same spokes come up.
Unless there's a repository of prior discussion no-one can be bothered to gather scraps from around the Net. Wikis solve this by all the discussion being in one place, and even they struggle is there were multiple.
A discussion long past, and a discussion yet to come.
There are no beginnings or endings in the Wheel of Python...
-- Regards, Ivan

On Tue, Jun 26, 2018 at 05:42:43AM +1000, Chris Angelico wrote:
So..... sublocal scopes, like in the earliest versions of PEP 572?
The wheel turns round and round, and the same spokes come up.
It isn't as if comprehensions (and generator expressions) run in a proper separate scope. It is more half-and-half, sometimes it is seperate, sometimes it isn't: py> def show_subscope(): ... a, b = 1, 2 ... print("Comprehension scope, Part A") ... print(next(locals() for x in (1,))) ... print("Comprehension scope, Part B") ... print(next(obj for obj in (locals(),))) ... py> show_subscope() Comprehension scope, Part A {'x': 1, '.0': <tuple_iterator object at 0xb799cf8c>} Comprehension scope, Part B {'b': 2, 'a': 1} Comprehensions already run partly in the surrounding scope. I tried to take a survey of people on the Python-List mailing list, so see what their expectations of comprehension scope was. Disappointingly, not many people responded, but those who did, invariably think in terms of comprehensions running inside their enclosing scope, like any other expression: https://mail.python.org/pipermail/python-list/2018-June/734838.html (Please excuse the doubled-up posts, some misconfigured news server is periodically sending duplicate posts.) (Oh and ignore my comment about Python 2 -- I was thinking of something else.) Given the code shown: def test(): a = 1 b = 2 result = [value for key, value in locals().items()] return result nobody suggested that the result ought to be the empty list, which is what you should get if the comprehension ran in its own scope. Instead, they all expected some variation of [1, 2], which is what you would get if the comprehension ran in the enclosing scope. A decade or more since generator expressions started running in their own half-local-half-sublocal scope, people still think of scoping in terms of LEGB and don't think of comprehensions as running in their own scope *except* to the very limited degree that sometimes they are either surprised or pleased that "the loop variable doesn't leak". For example: http://nbviewer.jupyter.org/github/rasbt/python_reference/blob/master/tutori... doesn't mention comprehensions until the very end, almost in passing, and doesn't describe them as a separate scope at all. Rather, they are described as using closures "to prevent the for-loop variable to cut [sic] into the global namespace." This doesn't mention comprehension subscope at all: https://www.python-course.eu/python3_global_vs_local_variables.php Even the official documentation doesn't explicitly state that comprehensions are a separate scope: https://docs.python.org/3/reference/executionmodel.html#resolution-of-names rather leaving it to an after thought, to mention in passing almost as if it were an implementation-dependent accident, that comprehensions cannot see variables defined in any surrounding class scope. Aside from the loop variable (which PEP 572 will not change!) I see no evidence that the average non-core developer Python programmer considers comprehensions as a separate scope, or wants them to be a separate scope. Regardless of comprehensions being implicitly wrapped in a function or not, the average developer doesn't want the loop variable to "leak", and that's as far as their consideration has needed to go until now. But when pressed to explicitly consider the scope inside a comprehension, the evidence I have seen is that they consider it the same as the local scope surrounding it. Which is not wrong, as can be seen from the example above. Unlike the loop variable, I don't believe that assignment-expression bindings quote-unquote "leaking" from comprehensions will come as a surprise. On the contrary -- given that Nick et al have gone to great lengths to ensure that as a first approximation, comprehensions are equivalent to a simple for-loop running in the current scope: result = [expr for a in seq] # is almost the same as result = [] for a in seq: result.append(expr) I expect that people will be surprised if explicit, non-loop variable assignments *don't* occur in the current scope. If all that takes to implement is something like an implicit "nonlocal", that's hardly worse than the implicit functions already used. -- Steve

On 27 June 2018 at 07:54, Steven D'Aprano <steve@pearwood.info> wrote:
But test() returns [1, 2]. So does that say (as you claim above) that "the comprehension ran in the enclosing scope"? Doesn't it just say that the outermost iterable runs in the enclosing scope? So everybody expected the actual behaviour? (Disclaimer: in my response, I said that I had no clear expectation, which I stand by - locals() exposes implementation details that I don't normally feel that I need to know - but certainly the majority of respondents expected 1 and 2 to appear). On the other hand,
and I bet no-one would have expected that if you'd posed that question (I certainly wouldn't). Although some might have said [('v', 'a')]. I suspect some would have expected a and b to appear there too, but that's just a guess... So yes, it's likely that people would have found the current behaviour unexpected in respect of locals(). But I imagine most people only care about the effective results when referencing variables, and
i.e., thanks to scope nesting, you can still reference locals from the enclosing scope. The problem is that := allows you to *change* values in a scope, and at that point you need to know *which* scope. So to that extent, the locals() question is important. However, I still suspect that most people would answer that they would like := to assign values *as if* they were in the enclosing scope, which is not really something that I think people would express in answer to a question about locals(). This can be achieved with an implicit "nonlocal" (and some extra shenanigans if the enclosing scope has a nonlocal or global declaration itself). Which, AIUI, is what the current proposal tries to do. IMO, the big question over the current PEP 572 proposal is whether it goes too far in the direction of "do what I mean". Superficially, the semantics are pretty clearly "what people would expect", and indeed that's been the whole focus recently to capture and satisfy *expected* behaviour. But there are edge cases (there always are when you work from messy real-world requirements rather than nice clean mathematical definitions ;-)) and the question is essentially whether any of those are bad enough to be an issue. I'm starting to feel that they aren't, and I'm moving towards a cautious +0 (or even +1) on the proposal. Paul

On Wed, Jun 27, 2018 at 5:30 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Yes - because the *outermost iterable* runs in the enclosing scope. But suppose you worded it like this: def test(): a = 1 b = 2 vars = {key: locals()[key] for key in locals()} return vars What would your intuition say? Should this be equivalent to dict(locals()) ? ChrisA

On 27 June 2018 at 08:52, Chris Angelico <rosuav@gmail.com> wrote:
As I said on python-list, my intuition doesn't apply to locals() - I simply have no idea what I'd "expect" from that code, other than a request to go back and write it more clearly :-) *After* staring at it for a while and trying to interpret it base on the detailed knowledge I've gained from this thread, I'd say it does nothing remotely useful, and if you want dict(locals()) you should write it. (No, test() is not equivalent, because the two instances of locals() refer to different scopes, but I can't imagine why I'd ever need to know that outside of solving artificial puzzles like this). Paul

On Wed, Jun 27, 2018 at 7:19 PM, Steven D'Aprano <steve@pearwood.info> wrote:
It gets funnier with nested loops. Or scarier. I've lost the ability to distinguish those two. def test(): spam = 1 ham = 2 vars = [key1+key2 for key1 in locals() for key2 in locals()] return vars Wanna guess what that's gonna return? ChrisA

On Wed, Jun 27, 2018 at 08:00:20AM -0400, Eric V. Smith wrote:
You can just copy and paste the function into the interactive interpreter and run it :-) But where's the fun in that? The point of the exercise is to learn first hand just how complicated it is to try to predict the *current* scope behaviour of comprehensions. Without the ability to perform assignment inside them, aside from the loop variable, we've managed to avoid thinking too much about this until now. It also demonstrates the unrealisticness of treating comprehensions as a separate scope -- they're hybrid scope, with parts of the comprehension running in the surrounding local scope, and parts running in an sublocal scope. Earlier in this thread, Nick tried to justify the idea that comprehensions run in their own scope, no matter how people think of them -- but that's an over-simplification, as Chris' example above shows. Parts of the comprehension do in fact behave exactly as the naive model would suggest (even if Nick is right that other parts don't). As complicated and hairy as the above example is, (1) it is a pretty weird thing to do, so most of us will almost never need to consider it; and (2) backwards compatibility requires that we live with it now (at least unless we introduce a __future__ import). If we can't simplify the scope of comprehensions, we can at least simplify the parts that actually matters. What matters are the loop variables (already guaranteed to be sublocal and not "leak" out of the comprehension) and the behaviour of assignment expressions (open to discussion). Broadly speaking, there are two positions we can take: 1. Let the current implementation of comprehensions as an implicit hidden function drive the functionality; that means we duplicate the hairiness of the locals() behaviour seen above, although it won't be obvious at first glance. What this means in practice is that assignments will go to different scopes depending on *where* they are in the comprehension: [ expr for x in iter1 for y in iter2 if cond ...] [ BBBBBB for x in AAAAAA for y in BBBBBB if BBBBBB ...] Assignments in the section marked "AAAAAA" will be in the local scope; assignments in the BBBBBB sections will be in the sublocal scope. That's not too bad, up to the point you try to assign to the same name in AAAAAA and BBBBBB. And then you are likely to get confusing hard to debug UnboundLocalErrors. 2. Or we can keep the current behaviour for locals and the loop variables, but we can keep assignment expressions simple by ensuring they always bind to the enclosing scope. Compared to the complexity of the above, we have the relatively straight forward: [ AAAAAA for x in AAAAAA for y in AAAAAA if AAAAAA ...] The loop variables continue to be hidden away in the invisible, implicit comprehension function, where they can't leak out, while explicit assignments to variables (using := or given or however it is spelled) will always go into the surrounding local scope, like they do in every other expression. Does it matter that the implementation of this requires an implicit nonlocal declaration for each assignment? No more than it matters that comprehensions themselves require an implicit function. And what we get out of this is simpler semantics at the Python level: - Unless previous declared global, assignment expressions always bind to the current scope, even if they're inside a comprehension; - and we don't have to deal with the oddity that different bits of a comprehension run in different scopes (unless we go out of our way to use locals()); merely using assignment expressions will just work consistently and simply, and loop variables will still be confined to the comprehension as they are now. -- Steve

On 27.06.2018 16:49, Steven D'Aprano wrote:
This isn't as messy as you make it sound if you remember that the outermost iterable is evaluated only once at the start and all the others -- each iteration. Anyone using comprehensions has to know this fact. The very readable syntax also makes it rather straightforward (though admittedly requiring some hand-tracing) to figure out what is evaluated after what.
-- Regards, Ivan

On Wed, Jun 27, 2018 at 05:52:16PM +0300, Ivan Pozdeev via Python-Dev wrote:
The question isn't *how often* they are evaluated, or how many loops you have, but *what scope* they are evaluated in. Even in a single loop comprehension, parts of it are evaluated in the local scope and parts are evaluated in an implicit sublocal scope. The overlap between the two is the trap, if you try to assign to the same variable in the loop header and then update it in the loop body. Not to mention the inconsistency that some assignments are accessible from the surrounding code: [expr for a in (x := func(), ...) ] print(x) # works while the most useful ones, those in the body, will be locked up in an implicit sublocal scope where they are unreachable from outside of the comprehension: [x := something ... for a in sequence ] print(x) # fails -- Steve

On 28.06.2018 1:42, Steven D'Aprano wrote:
All expressions inside the comprehension other than the initial iterable have access to the loop variables generated by the previous parts. So they are necessarily evaluated in the internal scope for that to be possible. Since this is too an essential semantics that one has to know to use the construct sensibly, I kinda assumed you could make that connection... E.g.: [(x*y) for x in range(5) if x%2 for y in range(x,5) if not (x+y)%2] A B C D E C and D have access to the current x; E and A to both x and y.
-- Regards, Ivan

On 28.06.2018 2:31, Ivan Pozdeev via Python-Dev wrote:
This means btw that users cannot rely on there being a single internal scope, or a scope at all. The public guarantee is only the access to the loop variables (and, with the PEP, additional variables from assignments), of the current iteration, generated by the previous parts.
-- Regards, Ivan

On 28.06.2018 2:45, Ivan Pozdeev via Python-Dev wrote:
The expressions in the comprehension just somehow automagically determine which of the variables are internal and which are local. How they do that is an implementation detail. And the PEP doesn't need to (and probably shouldn't) make guarantees here other than where the variables from expressions are promised to be accessible.
-- Regards, Ivan

Ivan Pozdeev via Python-Dev wrote:
That fact alone doesn't imply anthing about the *scopes* in which those iterators are evaluated, however. Currently the only situation where the scoping makes a difference is a generator expression that isn't immediately used, and you can get a long way into your Python career without ever encountering that case. -- Greg

On Wed, Jun 27, 2018 at 08:30:00AM +0100, Paul Moore wrote:
I think I was careful enough to only say that this was the same result you would get *if* the comprehension ran in the outer scope. Not to specifically say it *did* run in the outer scope. (If I slipped up anywhere, sorry.) I did say that the comprehension runs *partly* in the surrounding scope, and the example shows that the local namespace in the "... in iterable" part is not the same as the (sub)local namespace in the "expr for x in ..." part. *Parts* of the comprehension run in the surrounding scope, and parts of it run in an implicit sublocal scope inside a hidden function, giving us a quite complicated semantics for "comprehension scope": [expression for a in first_sequence for b in second ... ] |------sublocal-----|----local-----|------sublocal------| Try fitting *that* in the LEGB (+class) acronym :-) This becomes quite relevant once we include assignment expressions. To make the point that this is not specific to := but applies equally to Nick's "given" syntax as well, I'm going to use his syntax: result = [a for a in (x given x = expensive_function(), x+1, 2*x, x**3)] Here, the assignment to x runs in the local part. I can simulate that right now, using locals, but only outside of a function due to CPython's namespace optimization inside functions. (For simplicity, I'm just going to replace the call to "expensive_function" with just a constant.) py> del x py> [a for a in (locals().__setitem__('x', 2) or x, x+1, 2*x, x**3)] [2, 3, 4, 8] py> x 2 This confirms that the first sequence part of the comprehension runs in the surrounding local scope. So far so good. What if we move that assignment one level deep? Unfortunately, I can no longer use locals for this simulation, due to a peculiarity of the CPython function implementation. But replacing the call to locals() with globals() does the trick: del x # simulate [b*a for b in (1,) for a in (x given x = 2, x+1, 2*x, x**3)] [b*a for b in (1,) for a in (globals().__setitem__('x', 2) or x, x+1, 2*x, x**3)] That also works. But the problem comes if the user tries to assign to x in both the local and a sublocal section: # no simulation here, sorry [b*a for b in (x given x = 2, x**2) for a in (x given x = x + 1, x**3)] That looks like it should work. You're assigning to the same x in two parts of the same expression. Where's the problem? But given the "implicit function" implementation of comprehensions, I expect that this ought to raise an UnboundLocalError. The local scope part is okay: # needs a fixed-width font for best results [b*a for b in (x given x = 2, x**2) for a in (x given x = x + 1, x**3)] ..............|-----local part----|.....|--------sublocal part--------| but the sublocal part defines x as a sublocal variable, shadowing the surrounding local x, then tries to get a value for that sublocal x before it is defined. If we had assignment expressions before generator expressions and comprehensions, I don't think this would have been the behaviour we desired. (We might, I guess, accept it as an acceptable cost of the implicit function implementation. But we surely wouldn't argue for this complicated scoping behaviour as a good thing in and of itself.) In any case, we can work around this (at some cost of clarity and unobviousness) by changing the name of the variable. Not a big burden when the variable is a single character x: [b*a for b in (x given x = 2, x**2) for a in (y given y = x + 1, y**3)] but if x is a more descriptive name, that becomes more annoying. Nevermind, it is a way around this. Or we could Just Make It Work by treating the entire comprehension as the same scope for assignment expressions. (I stress, not for the loop variable.) Instead of having to remember which bits of the comprehension run in which scope, we have a conceptually much simpler rule: - comprehensions are expressions, and assignments inside them bind to the enclosing local scope, just like other expressions: - except for the loop variables, which are intentionally encapsulated inside the comprehension and don't "leak". The *implementation details* of how that works are not conceptually relevant. We may or may not want to advertise the fact that comprehensions use an implicit hidden function to do the encapsulation, and implicit hidden nonlocal to undo the effects of that hidden function. Or whatever implementation we happen to use.
So everybody expected the actual behaviour?
More or less, if we ignore a few misapprehensions about how locals works.
I suspect not. To be honest, I didn't even think of asking that question until after I had asked the first.
That is my belief as well. But that was intentionally not the question I was asking. I was interested in seeing whether people thought of comprehensions as a separate scope, or part of the enclosing scope. -- Steve

On Mon, Jun 25, 2018 at 8:37 PM, Terry Reedy <tjreedy@udel.edu> wrote:
[...]
this email have ever deliberately taken advantage of the limited Python 3 scope in comprehensions and generator expressions to use what would otherwise be a conflicting local variable name? I appreciate that the scope limitation can sidestep accidental naming errors, which is a good thing. Unfortunately, unless we anticipate Python 4 (or whatever) also making for loops have an implicit scope, I am left wondering whether it's not too large a price to pay. After all, special cases aren't special enough to break the rules, and unless the language is headed towards implicit scope for all uses of "for" one could argue that the scope limitation is a special case too far. It certainly threatens to be yet another confusion for learners, and while that isn't the only consideration, it should be given due weight.

On Mon, Jun 25, 2018 at 2:16 PM Steve Holden <steve@holdenweb.com> wrote:
No, never, but the opposite has bitten me in production code (as I related several months back, a class-level variable was being used on the lhs of a comprehension and that failed when it was run in Py3). The caveat is that our code base is Py2+Py3, so we have the mindset that comprehension variables always leak.

On Mon, Jun 25, 2018 at 5:14 PM Steve Holden <steve@holdenweb.com> wrote:
I have never once *deliberately* utilized the Python 3 local scoping in comprehensions. There were a few times in Python 2 where I made an error of overwriting a surrounding name by using it in a comprehension, and probably Python 3 has saved me from that a handful of times. Where I ever made such an error, it was with names like 'x' and 'i' and 'n'. They are useful for quick use, but "more important" variables always get more distinctive names anyway. Had the Python 2 behavior remained, I would have been very little inconvenienced; and I suppose comprehensions would have been slightly less "magic" (but less functional-programming).
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

Terry Reedy wrote:
This doesn't change the situation conceptually, though, since the question arises of why not do the same mangling for names assigned within the comprehension. A decision still needs to be made about whether we *want* semantics that leak some things but not others. -- Greg

On Tue, Jun 26, 2018 at 10:54:12AM +1200, Greg Ewing wrote:
A decision still needs to be made about whether we *want* semantics that leak some things but not others.
My sense (or bias, if you prefer) is that the answer to that depends on how you word the question. If you talk about "leaking", or give examples with trivial 1-character names that look all too easy to accidentally clobber, people will say "No": # Given this: x = 999 [(x := i)*x for i in (1, 2)] # should print(x) afterwards result in 4? but if you show a useful example that doesn't look like an accident waiting to happen, but a deliberate feature: # Given this: previous = 0 [previous + (previous := i) for i in (1, 2, 3)] # what value would you expect previous to have # at the completion of the loop? they'll be more receptive to the idea. (If they're not opposed to assignment expressions at all.) Avoiding leading questions is *hard*, and I believe that in general people don't know what they want until they've got it. I say that from considering all the times I've made a radical about face, features which I was *sure* would be awful actually turned out to be not awful at all -- augmented assignment, for instance. -- Steve

[This is my one response today] On Mon, Jun 25, 2018 at 12:40 PM Terry Reedy <tjreedy@udel.edu> wrote:
Fair enough, and by itself this might not have been enough reason to make the change. But see below.
That depends on how you see it -- to me (b) just means that there's an implicit nonlocal[1] to make the assignment have the (desirable) side-effect. The key thing to consider here is whether that side-effect is in fact desirable. For me, the side-effect of the comprehension's loop control variable was never desirable -- it was just an implementation detail leaking out. (And that's different from leaking a regular for-loop's control variable -- since we have 'break' (and 'else') there are some legitimate use cases. But comprehensions try to be expressions, and here the side effect is at best useless and at worst a nasty surprise.)
Right, and this consistency convinced me that the change was worth it. I just really like to be able to say "[... for ...]" is equivalent to "list(... for ...)", and similar for set and dict.
(b) proposes to add extra hidden code in and around the temporary function to partly undo the isolation.
But it just adds a nonlocal declaration. There's always some hidden code ('def' and 'return' at the very least).
Ah, there's the rub! I should probably apologize for not clarifying my terminology more. In the context of PEP 572, when I say "comprehensions" I include generators! PEP 572 states this explicitly ( https://github.com/python/peps/blame/master/pep-0572.rst#L201-L202). Certainly PEP 572 intends to add that implicit nonlocal to both comprehensions and generator expressions. (I just got really tired of writing that phrase over and over, and at some point I forgot that this is only a parenthetical remark added in the PEP's latest revision, and not conventional terminology -- alas. :-) Part (b) of PEP 572 does several things of things to *retain* consistency: - The target of := lives in the same scope regardless of whether it occurs in a comprehension, a generator expression, or just in some other expression. - When it occurs in a comprehension or generator expression, the scope is the same regardless of whether it occurs in the "outermost iterable" or not. If we didn't have (b) the target would live in the comprehension/genexpr scope if it occurred in a comprehension/genexp but outside its "outermost iterable", and in the surrounding scope otherwise.
Possibly this is based on a misunderstanding of my use of "comprehensions". Also, since your trick can only be used for list/set/dict comprehensions, but not for generator expressions (at least I assume you don't want it there) it would actually *reduce* consistency between list/set/dict comprehensions and generator expressions.
For me personally, (b) makes the PEP more consistent, so I'm not in favor of breaking up the PEP. But we can certainly break up the discussion -- that's why I started using the labels (a) and (b). ---------- [1] Sometimes it's an implicit global instead of an implicit nonlocal -- when there's already a global for the same variable in the target scope. -- --Guido van Rossum (python.org/~guido)

On 6/26/2018 10:36 PM, Guido van Rossum wrote:
[This is my one response today]
Thank you for clearly presenting how you see 'comprehension', 'generator expression' and by implication 'equivalent code'. The latter can either be a definition or an explanation. The difference is subtle but real, and, I believe, the essence of the disagreement over iteration variables. If the code equivalent to a comprehension is its definition, like a macro expansion, then survival of the iteration variable is to be expected. If the equivalent code is an explanation of the *result* of evaluating a *self-contained expression*, then leakage is easily seen a wart, just as leakage of temporaries from any other expression would be. My interpretation of what you say below is that you always wanted, for instance, [i*i for i in iterable] == [j*j for j in iterable] to be true, and saw the leakage making this not quite true as a wart. In other words, within comprehensions (including generator expressions) iterations names should be regarded as dummy placeholders and not part of the value. If this is correct, the list comprehension syntax could have been [\0 * \0 for \0 in iterable] with \1, \2, ... used as needed. (I am using the regex back-reference notation in a way similar to the use of str.format forward reference notation.) I will stop here for now, as it is 1:30 am for me. Terry
-- Terry Jan Reedy

Guido van Rossum wrote:
Greg seem to be +0 or better for (a)
Actually, I'm closer to -1 on (a) as well. I don't like := as a way of getting assignment in an expression. The only thing I would give a non-negative rating is some form of "where" or "given". Brief summary of reasons for disliking ":=": * Cryptic use of punctuation * Too much overlap in functionality with "=" * Asymmetry between first and subsequent uses of the bound value * Makes expressions cluttered and hard to read to my eyes -- Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
+1 to this; ‘:=’ doesn't convey the meaning well. Python's syntax typically eschews cryptic punctuation in faviour of existing words that convey an appropriate meaning, and I agree with Greg that would be a better way to get this effect. -- \ “Self-respect: The secure feeling that no one, as yet, is | `\ suspicious.” —Henry L. Mencken | _o__) | Ben Finney

Not giving a vote, as I'm just a lurker, but: Le 25/06/2018 à 01:30, Greg Ewing a écrit :
This resonates with me for a yet different reason: expressing the feature with a new operator makes it feel very important and fundamental, so that beginners would feel compelled to learn it early, and old-timers tend to have a strong gut reaction to it. Using merely a keyword makes it less prominent. Baptiste

On 25.06.2018 2:30, Greg Ewing wrote:
"as" was suggested even before is became a keyword in `with'. ( if (re.match(regex,line) as m) is not None: <do smth> ) The only objective objection I've heard is it's already used in `import' and `with' -- but that's perfectly refutable.
-- Regards, Ivan

On 26.06.2018 1:34, Greg Ewing wrote:
What do you mean by "asymmetry"? The fact that the first time around, it's the expression and after that, the variable? If that, it's not a "problem". The whole idea is to assign the result of a subexpression to something. If you force any assignments to be outside, it won't be a subexpression anymore, but effectively a separate statement -- if not syntactically, then visually at least -- both of which are the things the feature's purpose is to avoid. If you seek to force assignments outside, you should've rather suggested inline code blocks e.g. like anonymous methods in C# ( { a=foo(); b=bar(); return a+b;} ). Using this assigned result elsewhere in the same expression (akin to regex backreferences) is not a part of the basic idea actually. It depends on the evaluation order (and whether something is evaluated at all), so I doubt it should even be allowed -- but even if it is, it's a side benefit at best. -- Regards, Ivan

Ivan Pozdeev via Python-Dev wrote:
Using this assigned result elsewhere in the same expression (akin to regex backreferences) is not a part of the basic idea actually.
If that's true, then the proposal has mutated into something that has *no* overlap whatsoever with the use case that started this whole discussion, which was about binding a temporary variable in a comprehension, for use *within* the comprehension.
It depends on the evaluation order (and whether something is evaluated at all),
Which to my mind is yet another reason not to like ":=". -- Greg

Why is this discussion talking about comprehensions at all? Is there a decent use case for using assignments in comprehensions (as opposed to language lawyering or deliberate obfuscation)? Regards Antoine. On Thu, 28 Jun 2018 01:25:14 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On Wed, Jun 27, 2018 at 03:41:23PM +0200, Antoine Pitrou wrote:
Yes. The *very first* motivating example for this proposal came from a comprehension. I think it is both unfortunate and inevitable that the discussion bogged down in comprehension-hell. Unfortunate because I don't think that the most compelling use-cases involve comprehensions at all. But inevitable because *comprehensions are the hard case*, thanks to the (justifiable!) decision to implement them as implicit hidden functions. In my opinion, the really two BIG wins for assignment expressions are while loops and cascades of if... blocks. Tim Peters has also given a couple of good examples of mathematical code that would benefit strongly from this feature. Going back a few months now, they were the examples that tipped me over from the opinion "Oh, just re-write the comprehension as a loop" to the opinion "You know, I think this feature actually is useful... and as a bonus, you can keep using the comprehension" But that requires that we get the comprehension scoping right. Not just leave it as an unspecified implementation detail. -- Steve

Steven D'Aprano wrote:
I think the unfortunateness started when we crossed over from talking about binding a temporary name for use *within* a comprehension or expression, to binding a name for use *outside* the comprehension or expression where it's bound. As long as it's for internal use, whether it's in a comprehension or not isn't an issue.
Well, I remain profoundly unconvinced that writing comprehensions with side effects is ever a good idea, and Tim's examples did nothing to change that. -- Greg

On 28.06.2018 2:31, Greg Ewing wrote:
I've shown in <05f368c2-3cd2-d7e0-9f91-27afb40d5b35@mail.mipt.ru> (27 Jun 2018 17:07:24 +0300) that assignment expressions are fine in most use cases without any changes to scoping whatsoever. So, as Guido suggested in <CAP7+vJ+xBAT4ZvAo4B7qSqxnnpj8jJ1VZ-Le7EwT8=n-UTjE3Q@mail.gmail.com> (26 Jun 2018 19:36:14 -0700), the scoping matter can be split into a separate PEP and discussion.
-- Regards, Ivan

Le 28/06/2018 à 01:31, Greg Ewing a écrit :
Comprehensions with side effects feel scary indeed. But I could see myself using some variant of the "cumsum" example (for scientific work at the command prompt):
x=0; [x:=x+i for i in range(5)]
Here the side effects are irrelevant, the "x" variable won't be reused. But it needs to be initialized at the start of the comprehension. I would happily get rid of the side-effects, but then what would be a non-cryptic alternative to the above example? Baptiste

On 6/28/2018 8:05 AM, Baptiste Carvello wrote:
Creating an unneeded list with a comprehension purely for side effects is considered a bad idea by many. x = 0 for i in range(5): x += i
Here the side effects are irrelevant, the "x" variable won't be reused.
If we ignore the side effect on x, the above is equivalent to 'pass' ;-) Perhaps you meant x = 0 cum = [x:=x+i for i in range(5)] which is equivalent to x, cum = 0, [] for i in range(5): x += i; cum.append(x)
The above as likely intended can also be written import itertools as it cum = list(it.accumulate(range(5))) We have two good existing alternatives to the proposed innovation. -- Terry Jan Reedy

On 27.06.2018 16:25, Greg Ewing wrote:
I don't know what and where "started" it (AFAIK the idea has been around for years) but for me, the primary use case for an assignment expression is to be able to "catch" a value into a variable in places where I can't put an assignment statement in, like the infamous `if re.match() is not None'.
which was about binding a temporary variable in a comprehension, for use *within* the comprehension.
Then I can't understand all the current fuss about scoping. AFAICS, it's already like I described in https://mail.python.org/pipermail/python-dev/2018-June/154067.html : the outermost iterable is evaluated in the local scope while others in the internal one: In [13]: [(l,i) for l in list(locals())[:5] for i in locals()] Out[13]: [('__name__', 'l'), ('__name__', '.0'), ('__builtin__', 'l'), ('__builtin__', '.0'), ('__builtin__', 'i'), ('__builtins__', 'l'), ('__builtins__', '.0'), ('__builtins__', 'i'), ('_ih', 'l'), ('_ih', '.0'), ('_ih', 'i'), ('_oh', 'l'), ('_oh', '.0'), ('_oh', 'i')] (note that `i' is bound after the first evaluation of internal `locals()' btw, as to be expected) If the "temporary variables" are for use inside the comprehension only, the assignment expression needs to bind in the current scope like the regular assignment statement, no changes are needed!
-- Regards, Ivan

Ivan Pozdeev via Python-Dev wrote:
This seems to be one of only about two uses for assignment expressions that gets regularly brought up. The other is the loop-and-a-half, which is already adequately addressed by iterators. So maybe instead of introducing an out-of-control sledgehammer in the form of ":=", we could think about addressing this particular case. Like maybe adding an "as" clause to if-statements: if pattern.match(s) as m: do_something_with(m) -- Greg

On 28.06.2018 2:44, Greg Ewing wrote:
I've skimmed for the origins of "as" (which I remember seeing maybe even before Py3 was a thing) and found this excellent analysis of modern languages which is too a part of the PEP 572 discussion: https://mail.python.org/pipermail/python-ideas/2018-May/050920.html It basically concludes that most recently-created languages do not have assignment expressions; they rather allow assignment statement(s?) before the tested expression in block statements (only if/while is mentioned. `for' is not applicable because its exit condition in Python is always the iterable's exhaustion, there's nothing in it that could be used as a variable). It, however, doesn't say anything about constructs that are not block statements but are equivalent to them, like the ternary operator. (In comprehensions, filter conditions are the bits equivalent to if/while statements.) -- Regards, Ivan

On Thu, Jun 28, 2018 at 11:29 AM, Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
Now read this response. https://mail.python.org/pipermail/python-ideas/2018-May/050938.html ChrisA

On Mon, Jun 25, 2018 at 11:30:24AM +1200, Greg Ewing wrote:
":=" is the second most common syntax used for assignment in common programming languages, not just Pascal. Even modern languages like Go use it. If that's "cryptic", what word would you use to describe @decorator syntax? *wink* Honestly Greg, can you put your hand on your heart and swear that if you came across "name := expression" in source code you wouldn't be able to hazard a guess as the meaning of the := operator?
* Too much overlap in functionality with "="
If you are willing to consider a non-negative rating under the "given/willing"spelling, presumably the "overlap in functionality" isn't that important. (Otherwise it would be an argument against the feature *regardless of spelling*.) So why should it be an argument against the := spelling?
* Asymmetry between first and subsequent uses of the bound value
I don't know what this means.
* Makes expressions cluttered and hard to read to my eyes
And Nick's more verbose "given" proposal makes expressions less cluttered? result = process(first=(spam := ham or eggs), second=spam*5) result = process(first=(spam given spam = ham or eggs), second=spam*5) The := spelling has three syntactic elements: the target name, the := operator itself, and the expression being assigned. The syntax you are willing to consider has five elements: an arbitrarily complex return expression, the keyword "given", the target name, the = operator, and the expression being assigned. It isn't rational to say that adding extra complexity and more syntactic elements *reduces* clutter. At the minimum, Nick's syntax requires: - an extra keyword ("given" or "where") - a repetitive, redundant, repeated use of the target name just to save one : character. That adds, not subtracts, clutter. Aside from the asymmetry issue (which I don't understand) it seems that most of your arguments against := apply equally, or even more strongly, to the "expr given name = expr" version. I know matters of taste are deeply subjective, but we ought to distinguish between *subjective* and *irrational* reasons for disliking proposed features, and try to resist the irrational ones: "We should change the spelling of set.add to set.append, as that will remove the troublesome double-letter, and reduce typing." *wink* -- Steve

On 25 June 2018 at 02:24, Guido van Rossum <guido@python.org> wrote:
Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or 'NAME := EXPR'?" bothers me a bit, but it's the implementation implications of parent local scoping that I fear will create a semantic tar pit we can't get out of later.
Unfortunately, I think the key rationale for (b) is that if you *don't* do something along those lines, then there's a different strange scoping discrepancy that arises between the non-comprehension forms of container displays and the comprehension forms: (NAME := EXPR,) # Binds a local tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local [NAME := EXPR] # Binds a local [NAME := EXPR for __ in range(1)] # Doesn't bind a local list(NAME := EXPR for __ in range(1)) # Doesn't bind a local {NAME := EXPR} # Binds a local {NAME := EXPR for __ in range(1)} # Doesn't bind a local set(NAME := EXPR for __ in range(1)) # Doesn't bind a local {NAME := EXPR : EXPR2} # Binds a local {NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local Those scoping inconsistencies aren't *new*, but provoking them currently involves either class scopes, or messing about with locals(). The one virtue that choosing this particular set of discrepancies has is that the explanation for why they happen is the same as the explanation for how the iteration variable gets hidden from the containing scope: because "(EXPR for ....)" et al create an implicitly nested scope, and that nested scope behaves the same way as an explicitly nested scope as far as name binding and name resolution is concerned. Parent local scoping tries to mitigate the surface inconsistency by changing how write semantics are defined for implicitly nested scopes, but that comes at the cost of making those semantics inconsistent with explicitly nested scopes and with the read semantics of implicitly nested scopes. The early iterations of PEP 572 tried to duck this whole realm of potential semantic inconsistencies by introducing sublocal scoping instead, such that the scoping for assignment expression targets would be unusual, but they'd be consistently unusual regardless of where they appeared, and their quirks would clearly be the result of how assignment expressions were defined, rather than only showing up in how they interacted with other scoping design decisions made years ago. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2018 at 12:44, Nick Coghlan <ncoghlan@gmail.com> wrote:
I've been mostly ignoring this proposal for a while now, so I'm going to respond here in the context of someone with a bit of an idea of the underlying complexities, but otherwise coming at it as a new proposal.
None of those "discrepancies" bother me in the slightest, when taken in isolation as you present them here. I suspect you could lead me through a chain of logic that left me understanding why you describe them as discrepancies, but without that explanation, I'm fine with all of them. I'd also say that they seem contrived (not just in the use of artificial names, but also in the sense that I'm not sure why I'd want to use this *pattern*) so I'd happily say "well, don't do that then" if things started behaving non-intuitively.
And to reinforce my point above, I already consider putting significant code in class scopes, or using locals() to be techniques that should only be used sparingly and with a clear understanding of the subtleties. I'm sure you could say "but the examples above would be much more common" in response to which I'd like to see real use cases that behave non-intuitively in the way you're concerned about.
But that's precisely why I find the behaviour intuitive - the nested scope is the *reason* things behave this way, not some sort of easily-overlooked way the "problem" can be explained away.
Those last two paragraphs made my head explode, as far as I can see by virtue of the fact that they try to over-analyze the fairly simple intuition I have that "there's a nested scope involved". Disclaimer: I may well have got a *lot* of subtleties wrong here, and it's quite likely that my impressions don't stand up to the harsh reality of how the implementation works. But my comments are on the basis of my *intuition*, whether that's right or wrong. And if the reality violates my intuition, it's *other* constructs that I find non-intuitive, not this one. (I'm perfectly happy to concede that it's not possible to avoid *any* non-intuitive behaviour - all I'm trying to say is that my intuition doesn't balk at this one, unlike yours). Paul

On 6/25/2018 8:25 AM, Paul Moore wrote:
Of course not, in local scopes where is it not executed. But it would, in the nested function where the assignment *is* executed. Ditto for all of the following.
Me neither. I pretty much agree with the rest of what Paul said. If we don't want comprehensions to execute in a nested scope, then we should not create one. See my response to Guido for a possible alternative. -- Terry Jan Reedy

On 25.06.2018 14:44, Nick Coghlan wrote:
I've got an idea about this. The fact is, assignments don't make much sense in an arbitrary part of a comprehension: `for' variables are assigned every iteration, so when the result is returned, only the final value will be seen. (And if you need a value every iteration, just go the explicit way and add it to the returned tuple.) Contrary to that, the "feeder" expression is only evaluated once at the start -- there, assignments do make sense. Effectively, it's equivalent to an additional line: seq = range(calculate_b() as bottom, calculate_t() as top) results = [calculate_r(bottom,r,top) for r in seq] So, I suggest to evaluate the "feeder" expression in a local scope but expressions that are evaluated every iteration in a private scope. -- Regards, Ivan

[This is my one reply in this thread today. I am trying to limit the amount of time I spend to avoid another overheated escalation.] On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Others have remarked this too, but it really bother me that you are focusing so much on the implementation of parent local scoping rather than on the "intuitive" behavior which is super easy to explain -- especially to someone who isn't all that familiar (or interested) with the implicit scope created for the loop control variable(s). According to Steven (who noticed that this is barely mentioned in most tutorials about comprehensions) that is most people, however very few of them read python-dev. It's not that much work for the compiler, since it just needs to do a little bit of (new) static analysis and then it can generate the bytecode to manipulate closure(s). The runtime proper doesn't need any new implementation effort. The fact that sometimes a closure must be introduced where no explicit initialization exists is irrelevant to the runtime -- this only affects the static analysis, at runtime it's no different than if the explicit initialization was inside `if 0`. Unfortunately, I think the key rationale for (b) is that if you
In what sense are they not new? This syntax doesn't exist yet.
Yeah, but most people don't think much about that explanation. You left out another discrepancy, which is more likely to hit people in the face: according to your doctrine, := used in the "outermost iterable" would create a local in the containing scope, since that's where the outermost iterable is evaluated. So in this example a = [x := i+1 for i in range(y := 2)] the scope of x would be the implicit function (i.e. it wouldn't leak) while the scope of y would be the same as that of a. (And there's an even more cryptic example, where the same name is assigned in both places.) This is another detail of comprehensions that I assume tutorials (rightly, IMO) gloss over because it's so rarely relevant. But it would make the explanation of how := works in comprehensions more cumbersome: you'd have to draw attention to the outermost iterable, otherwise "inline assignment in comprehensions has the same scope as the comprehension's loop control variable(s)" would lead one to believe that y's scope above would also be that of the implicit function.
Nobody thinks about write semantics though -- it's simply not the right abstraction to use here, you've introduced it because that's how *you* think about this.
There was also another variant in some iteration or PEP 572, after sublocal scopes were already eliminated -- a change to comprehensions that would evaluate the innermost iterable in the implicit function. This would make the explanation of inline assignment in comprehensions consistent again (they were always local to the comprehension in that iteration of the PEP), at the cost of a backward incompatibility that was ultimately withdrawn. -- --Guido van Rossum (python.org/~guido)

On 26 June 2018 at 02:27, Guido van Rossum <guido@python.org> wrote:
[This is my one reply in this thread today. I am trying to limit the amount of time I spend to avoid another overheated escalation.]
Aye, I'm trying to do the same, and deliberately spending some evenings entirely offline is helping with that :)
One of the things I prize about Python's current code generator is how many of the constructs can be formulated as simple content-and-context independent boilerplate removal, which is why parent local scoping (as currently defined in PEP 572) bothers me: rather than being a new primitive in its own right, the PEP instead makes the notion of "an assignment expression in a comprehension or generator expression" a construct that can't readily decomposed into lower level building blocks the way that both assignment expressions on their own and comprehensions and generator expressions on their own can be. Instead, completely new language semantics arise from the interaction between two otherwise independent features. Even changes as complicated as PEP 343's with statement, PEP 380's yield from, and PEP 492's native coroutines all include examples of how they could be written *without* the benefit of the new syntax. By contrast, PEP 572's parent local scoping can't currently be defined that way. Instead, to explain how the code generator is going to be expected to handle comprehensions, you have to take the current comprehension semantics and add two new loops to link up the bound names correctly:: [item := x for x in items] becomes: # Each bound name gets declared as local in the parent scope if 0: for item in (): pass def _list_comp(_outermost_iter): # Each bound name gets declared as: # - nonlocal if outer scope is a function scope # - global item if outer scope is a module scope # - an error, otherwise _result = [] for x in _outermost_iter: _result.append(x) return _result _expr_result = _list_comp(items) This is why my objections would be reduced significantly if the PEP explicitly admitted that it was defining a new kind of scoping semantics, and actually made those semantics available as an explicit "parentlocal NAME" declaration (behind a "from __future__ import parent_locals" guard), such that the translation of the above example to an explicitly nested scope could just be the visually straightforward:: def _list_comp(_outermost_iter): parentlocal item _result = [] for x in _outermost_iter: item = x _result.append(x) return _result _expr_result = _list_comp(items) That splits up the learning process for anyone trying to really understand how this particular aspect of Python's code generation works into two distinct pieces: - "assignment expressions inside comprehensions and generator expressions use parent local scoping" - "parent local scoping works <the way that PEP 572 defines it>" If the PEP did that, we could likely even make parent locals work sensibly for classes by saying that "parent local" for a method definition in a class body refers to the closure namespace where we already stash __class__ references for the benefit of zero-arg super (this would also be a far more robust way of defining private class variables than name mangling is able to offer). Having parent locals available as a language level concept (rather than solely as an interaction between assignment expressions and implicitly nested scopes) also gets us to a point where context-independent code thunks that work both at module level and inside another function can be built as nested functions which declare all their working variables as parentlocal (you still need to define the thunks inline in the scope you want them to affect, since this isn't dynamic scoping, but when describing the code, you don't need to say "as a module level function define it this way, as a nested function define it that way"). An explicit "parentlocal NAME" concept at the PEP 572 layer would also change the nature of the draft "given" proposal from competing with PEP 572, to instead being a follow-up proposal that focused on providing control of target name declarations in lambda expressions, comprehensions, and generator expressions such that: - (lambda arg: value := arg given parentlocal value) # Exports "value" to parent scope - any(x for x in items given parentlocal x) # Exports "x" to parent scope - [y for x in data if (y := f(x)) given y] # *Avoids* exporting "y" to parent scope With parent local scoping in the mix the proposed "given" syntax could also dispense with initialiser and type hinting support entirely and instead only allow: - "... given NAME" (always local, no matter the default scoping) - "... given parentlocal NAME" (always parent local, declaring if necessary) - "... given nonlocal NAME" (always nonlocal, error if not declared in outer scope) - "... given global NAME" (always global, no matter how nested the current scope is) - "... given (TARGET1, TARGET2, ...)" (declaring multiple assignment targets) If you want an initialiser or a type hint, then you'd use parentlocal semantics. If you want to keep names local (e.g. to avoid exporting them as part of a module's public API) then you can do that, too.
The simplest way to illustrate the scope distinction today is with "len(locals())": >>> [len(locals()) for i in range(1)] [2] >>> [len(locals())] [7] But essentially nobody ever does that, so the distinction doesn't currently matter. By contrast, where assignment expressions bind their targets matters a *lot*, so PEP 572 makes the existing scoping oddities a lot more significant.
Yeah, the fact it deals with this problem nicely is one aspect of the parent local scoping that I find genuinely attractive.
The truth of the last part of that paragraph means that the only way for the first part of it to be true is to decide that my way of thinking is *so* unusual that nobody else in the 10 years that Python 3 has worked the way it does now has used the language reference, the source code, the disassembler, or the debugger to formulate a similar mental model of how they expect comprehensions and generator expressions to behave. I'll grant that I may be unusual in thinking about comprehensions and generator expressions the way I do, and I definitely accept that most folks simply don't think about the subtleties of how they handle scopes in the first place, but I *don't* accept the assertion that I'm unique in thinking about them that way. There are simply too many edge cases in their current runtime behaviour where the "Aha!" moment at the end of a debugging effort is going to be the realisation that they're implemented as an implicitly nested scope, and we've had a decade of Python 3 use where folks prone towards writing overly clever comprehensions have been in a position to independently make that discovery.
The early iterations of PEP 572 tried to duck this whole realm of potential semantic inconsistencies by introducing sublocal scoping
Yeah, the current "given" draft has an open question around the idea of having the presence of a "given" clause pull the outermost iterable evaluation inside the nested scope. It still doesn't really solve the problem, though, so I think I'd actually consider PEP-572-with-explicit-parent-local-scoping-support the version of assignment expressions that most cleanly handles the interaction with comprehension scopes without making that interaction rely on opaque magic (instead, it would be relying on an implicit target scope declaration, the same as any other name binding - the only unusual aspect is that the implicit declaration would be "parentlocal NAME" rather than the more typical local variable declaration). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
I'm not sure that's possible. If I understand correctly, part of the definition of "parent local" is that "parent" refers to the nearest enclosing *non-comprehension* scope, to give the expected result for nested comprehensions. If that's so, then it's impossible to fully decouple its definition from comprehensions. -- Greg

On 27 June 2018 at 23:38, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I'm OK with a target scope declaration construct having lexical-scope-dependent behaviour - exactly what "nonlocal NAME" will do depends on both the nature of the current scope, and on which names are declared as local in which outer scopes, and that's also implicitly the case for all name lookups. However, PEP 572 in its current form takes the position "parent local scoping is sufficiently useful to make it a required pre-requisite for adding assignment expressions, but not useful enough to expose as a new scope declaration primitive", and I've come to the view that it really is the "A+B=MAGIC!" aspect of the current proposal that bothers me, whereas "A+B implies C for <pragmatic reasons>" doesn't bother me any more than the implicit non-local references introduced as part of the original lexical scoping changes bother me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jun 27, 2018 at 9:27 AM Paul Moore <p.f.moore@gmail.com> wrote:
So, my interpretation is that it will behave like this? x = 2 y = [x := 3 for i in range(1)] print(x) 3 def f(): x = 4 y = [x := 5 for i in range(1)] print(x) f() 5 class C: x = 6 y = [x := 7 for i in range(1)] print(x) C() 6 print(x) 7

[Nick Coghlan]
Of course the PEP doesn't take that position at all: it doesn't even contain the term "parent local scoping". That's your term, which nobody else uses unless they're replying to you ;-) What the PEP does say: """ an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as "comprehensions") binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope. """ It's a small collection of plainly stated rules for specifying the intended semantics. If you want to claim that this _is_ "useful enough to expose as a new scope declaration primitive", it's really on you to present use cases to justify that claim. I'd present some for you, but I don't have any (I don't care that "by hand" conversion of nested comprehensions to workalike Python nested functions may require a bit of thought to establish the intended scope of assignment expression target names - all of which is easily doable without adding any new statements). I don't _expect_ that other good use cases exist. The gimmick's purpose is to make code that visually _appears_ to belong to a block act as if embedded assignments do occur in that block. If there's an explicitly nested function, that fundamental motivation no longer applies.

Nick Coghlan wrote:
Yes, but my point is that having an explicit "parentlocal" scope declaration doesn't help to make anything more orthogonal, because there's no way it can have *exactly* the same effect as a comprehension's implicit parent-local scoping. In other words, taking a comprehension and manually expanding it into a function with parentlocal declarations wouldn't give you something exactly equivalent to the original. If that's the purpose of having an explicit parentlocal, then it fails at that purpose. If that's *not* the purpose, then I'm not really sure what the purpose is, because I can't think of a situation where I'd choose to use parentlocal instead of nonlocal with an explicit assignment in the outer scope. Except maybe for the class-scope situation, which seems like an extremely obscure reason to introduce a whole new scoping concept with its own keyword. -- Greg

[Nick Coghlan]
[Greg Ewing]
Yes, but my point is that having an explicit "parentlocal" scope
Sure it can - but I already explained that. This is the analogy to "nonlocal" Nick is making: neither "nonlocal" nor "parentlocal" tell you which scope a declared name _does_ belong to. Instead they both say "it's not this scope" and specify algorithms you can follow to determine the scope to which the name does belong. "parentlocal" isn't an accurate name because the owning scope may not be the parent block at all, and it may even be a synonym for "global". I think "by hand" translations of nested comprehensions into nested functions are clearer _without_ the "parentlocal" invention.- then you have to be explicit about what the context requires. Nick hates that because it isn't uniform. I like that because I don't want to pretend a non-uniform thing is uniform ;-) The only real use case here is synthesizing nested functions to implement comprehensions/genexps. In other words, taking a comprehension and manually expanding
You can add (a sufficient number of) parentlocal declarations to get the precise intended semantics. Then again, that can also be done today (without the "parentlocal" invention).
For example, if the name is declared "global" in the outer scope, you'll get a compile-time error if you try to declare it "nonlocal" in the contained scope. "parentlocal" adjusts its meaning accordingly, becoming a synonym for "global" in that specific case.

On Wed, Jun 27, 2018 at 07:29:52PM -0500, Tim Peters wrote: [...]
"Parentlocal" is only a thing if we buy into the paradigm that inside comprehensions is a separate "local". And *that* is only true under two circumstances: - if you are utterly immersed in the implementation of comprehensions as invisible, implicit functions; - or if you start from the premise that comprehensions ought to encapsulate not just the loop variable, but anything else as well. But experimenting with locals() inside comprehensions shows that comprehension-scope *isn't* a well-defined thing. It already bleeds out of the comprehension, and so would some (but only some!) assignment expressions. Instead, if we start from the premise that comprehensions (like any other expression) run in the current scope, then there is no need to invent a term "parentlocal". There's just the usual LEGB scopes, plus class (which people usually forget). With no sublocal scopes (a term we never even had prior to this PEP) assignments inside the comprehension are no more special than assignments inside any other expression. They bind in the current scope, same as always, and keep the sensible identity that these two expressions are exactly equivalent in their visible semantics: [x:=0, x:=1, x:=2] [x:=i for i in (0, 1, 2)] including assignments. What about the loop variable? They ARE special, which is completely justified by the Zen: Although practicality beats purity. We can take a series of ever-more-detailed explanations, starting from the highest "bird's eye" view and gradually dropping further into the murky details of the implementation when, and if, required: - assignment within comprehensions is no different from assignment in any other expression, it occurs in the local scope; - loop variables? they're a special case, for good reason, and are encapsulated inside the comprehension; - how? they're hidden in an implicit, invisible scope, same as .0 the implicit, invisible iterator object; - oh, you didn't know about the .0 variable? well forget about it, it's an undocumented implementation detail, just like the invisible, implicit function used by comprehensions; - oh, you didn't know about that either? read the source code. Only the first two levels of explanation are part of Python the language. The rest is CPython implementation. -- Steve

Thank you all. I will accept the PEP as is. I am happy to accept *clarification* updates to the PEP if people care to submit them as PRs to the peps repo (https://github.com/python/peps), and that could even (to some extent) include summaries of discussion we've had, or outright rejected ideas. But even without any of those I think the PEP is very clear so I will not wait very long (maybe a week). On Mon, Jul 2, 2018 at 8:38 AM Steven D'Aprano <steve@pearwood.info> wrote:
-- --Guido van Rossum (python.org/~guido)

On 02/07/2018 19:19, Guido van Rossum wrote:
It's late to raise this, but what exactly are the objections to the syntax expr -> name # or variations such as expr => name instead of name := expr The PEP mentions that this syntax does not have a problem that "as" does, but does not list any downsides of it. It conforms to left-to-right evaluation, where name:=expr does not. It (I would argue) reduces the asymmetry of the first use of a sub-expression in cases such as [ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ] vs [ ( (y := f(x))**2, y**3, y**4) for x in iterable ] because the first "y" is closer to the way it is used, viz "**2". Regards Rob Cliffe

[Rob Cliffe]
It's late to raise this,
By months, yes ;-)
but what exactly are the objections to the syntax
expr -> name # or variations such as expr => name
instead of
name := expr
The PEP mentions that this syntax does not have a problem that "as"
does, but does not list any downsides of it.
My guess: it probably strikes too many as "excessive novelty", These are assignment expressions. Python's assignment statements put the target at the left. Why change that? ":=" is used for assignment in many more other languages than "->" is. Why fight that?
It conforms to left-to-right evaluation, where name:=expr does not.
? Only "expr" is evaluated, so left-to-right seems irrelevant here. The "evaluation" of a simple name as a binding target is a no-op (no code is generated). If you really do see this as a wart anyway, then it's positively a Good Thing that it's exactly the same "wart" as in Python's assignment statements.
It (I would argue) reduces the asymmetry of the first use of a sub-expression in cases such as
[ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ]
vs
[ ( (y := f(x))**2, y**3, y**4) for x in iterable ]
because the first "y" is closer to the way it is used, viz "**2".
The first form reads a little better to me too, but not a lot better. The problem I have with variations of this example on its own (which comes up surprisingly often with minor changes) is that it's clearer spelled today via [(y**2, y**3, y**4) for y in map(f, iterable)] Spelling that with either form of assignment expression reads significantly worse than that to my eyes But more importantly, it's expected that assignment expressions will be used _most_ often to make some common `if` and `while` patterns briefer. Hardly all. Our eyes are already trained to "look at the far right end" for the value being tested, and, e.g., while data := sock.recv(): preserves that. Especially in code that doesn't _always_ use assignment expressions in such contexts (which is likely all significant blobs of code), it would be visually jarring to have to "sometimes look in the middle instead" to extract the important part of: while sock.recv() -> data: "Look to the left for the name, look to the right for the value" is the rule for assignment statements, assignment expressions, and `for` loop targets. But there's no "QED" here because this isn't a deductive science. The final answer is "because that's what Guido liked best" ;-)

2018-07-02 20:19 GMT+02:00 Guido van Rossum <guido@python.org>:
Thank you all. I will accept the PEP as is. (...)
I see more and more articles ("on the Internet") saying that Guido van Rossum already accepted the PEP. Is the PEP already accepted or will be accepted? Right now, https://www.python.org/dev/peps/pep-0572/ status is "Draft". Victor

On Tue, Jul 3, 2018 at 4:25 PM Victor Stinner <vstinner@redhat.com> wrote:
That's a rather philosophical question. I clearly said "I will" not "I might". And if you're asking whether it's likely that I'll change my mind, no. I would like help with updates to the PEP to summarize some of the discussions and rejected proposals. And I am giving myself a week to "cool off". But I am muting threads that bring up objections that I've heard before (e.g. "design principles"). So those articles aren't wrong. Your patience is appreciated. -- --Guido van Rossum (python.org/~guido)

On Jul 2, 2018, at 8:34 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Guido has decided — and despite my concerns, I’m going to enjoy my new loop-and-a half construct:-) But a comment on this:
Sure — and I don’t think that’s confusing. However, generator expressions ( why don’t we call them generator comprehensions?) are a different story, as they may be run at some arbitrary time in the future. This hasn’t been an issue (except for the loop variable, which has been addressed) because: 1) Much of the time, the gen_ex is run right away, in-line. 2) There aren’t many ways to manipulate the local namespace in a gen_ex. With assignment expressions, it will be much easier to manipulate the local namespace, so there is room for some real confusion here. So a real local namespace gen_exp (and comprehensions, for consistency) would be nice. However, that ship has pretty much sailed. Will it end up being a common problem? Probably not, because (a) is still the case, and := will be used infrequently, and hopefully with unlikely to clash names. And as for all the other languages that have assignment expressions? Do they have constructs like generator expressions? -CHB

[Chris Barker]
However, generator expressions ( why don’t we call them generator
comprehensions?)
Because nobody really liked the "iterator comprehensions" or "accumulator displays" they were variously called at the start. https://mail.python.org/pipermail/python-dev/2003-October/039186.html As that explains, "generator expressions" was an attempt to break away from that "comprehensions" was always a dubious term, carried over from set theory where the term focuses on the optional "if" part rather than the more fundamental iterator or computation parts. At the start, for some (forgotten by me) reason it seemed important to make a distinction between "things like this" that were evaluated at once (list, dict, and set comprehensions) and the new-fangled accumulator displays that got evaluated lazily. But the "generator" in "generator comprehensions" would really be enough all by itself to make that clear enough. So if we had it to do over again I'd sigh and accept "generator comprehensions" anyway. It's been an eternal PITA - and especially in the PEP 572 threads! - to keep typing "comprehensions or generator expressions". Then again, if I had the power of Guido's time machine, I'd go back more, and not use "comprehensions" for anything to begin with. Instead we'd have list, dict, set, and generator twizzlers, affectionately called listwiz, dictwiz, setwiz, and gentwiz by the cool kids :-)

On Mon, Jul 2, 2018 at 11:42 PM, Tim Peters <tim.peters@gmail.com> wrote:
I always wondered about that :-) -- I'd say for most of us that aren't familiar with set theory, it's kind of a "sounds something like putting thing together" word and I just left it at that, and learned what they are.
Well, too late to change the official name, but not too late to start using the term in threads like these -- and other documentation, etc.... I find there is a lot of confusion about the word "generator", as it implies a "thing that generates values on the fly" (like, say the range() object. But then, in Python, a generator is something that gets crated by a generator function, and CAN be an "thing (iterator) that generates things on the fly", but can also be a more generic coroutine, and can be used in nifty ways that really have nothing to do with generating a bunch of value. (like pytest fixtures, for example) So we have generators, iterators, and iterables, and generators can be iterators, but aren't always, and any number of iterators can generate values on the fly, and .... so it's all a bit of a mess to explain to a newbie. Then again, if I had the power of Guido's time machine, I'd go back more,
I'd like that! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 7/3/2018 2:42 AM, Tim Peters wrote:
Amen. I cannot make 'comprehension' in this context comprehensible without some linguistic twisting.
I learned the set notion, such as {n^2: n in N; 1 <= n < 100, n even} # math {n*n for n in range(1,100) if not n%2} # python as 'set builder' notation. If we had followed the math precedent, instead of <other computer language>, we would have set builders, list builders, dict builders, and generator builders. I half seriously think we should consider this for 3.8 for the benefit of future Python programmers as well as ourselves. Comprehensions that can contain assignment expressions are a slightly new thing. -- Terry Jan Reedy

Terry Reedy wrote:
I was intending to suggest something like that back when comprehensions were first being discussed, but people raced ahead and adopted the term "comprehension" before I got the chance. "List builder" and "dict builder" make a lot of sense, but "generator builder" not so much -- it *is* a generator, not something that builds a generator. In fact it doesn't build anything in the sense that the others do. So maybe "generator expression" is the best we could have done. -- Greg

On Wed, Jul 04, 2018 at 11:13:20AM +1200, Greg Ewing wrote:
But [expr for x in seq] is a list, just as (expr for ...) is a generator. If you don't believe me, try it: py> type([x for x in (1,)]) <class 'list'> py> type(x for x in (1,)) <class 'generator'> So I think the similarity is complete. Further, if we think of "list builder" as an abbreviation of "list builder syntax", we have: - list builder syntax is syntax which returns a list; - dict builder syntax is syntax which returns a dict; - set builder syntax is syntax which returns a set; - generator builder syntax is syntax which returns a generator. Of course, there are other ways to build lists, such as calling the constructor, or using a list display ("list literal", except it isn't always a literal). But they're not *builder syntax* :-) In hindsight, I think "spam builder (syntax)" would have been better than the rather mysterious technical word "comprehension" and the not very felicitous term "generator expression". -- Steve

Steven D'Aprano wrote:
You only get a list/dict/set from the first three after you've run the iterators within it, but with a generator expression, you already have a generator before you've run it. That makes it feel different to me. -- Greg

On 7/6/2018 11:20 AM, Brett Cannon wrote:
"Comprehension" was an incomprehensible term to me, when I first heard it. After reading the documentation, I comprehended it, but the term is meaningless. "generator expression" is actually a little more comprehensible, by analogy to a power (electricity) generator... values are generated. But comprehensions would have been more intuitively understood by me, had they been called "list generators" or "dict generators" or "set generators". The difference between comprehensions and generators seems to be one of timing and sequencing: the comprehensions are comprehended as fast as possible, while generators are lazy (unlike those electricity generators, unless you turn them on and off repeatedly). So neither term is very intuitive to "plain old programmers" unless perhaps they happen to know some of the advanced math concepts where the term "comprehension" is claimed to come from (obviously, in spite of a math degree from some years back, I never encountered that term). Focusing just on lists, at this time, "list builder" would be better than "list generator" because using "generator" to replace "comprehension" would be confusing because of the history of the terminology as used/documented currently. A generator seems to be a "lazy list builder". If the names are changed, for a time both terminologies would have to coexist, so it would be critical to use terms not already in use in Python terminology. I would find it an improvement to use terms like "list builder" rather than "comprehension".

On 7/6/2018 11:51 AM, Chris Barker - NOAA Federal via Python-Dev wrote: via phone...
Are we just having fun here?
I floated the idea as a trial balloon to see what response it got.
Or might we actually start using a new naming convention for the-syntax-formerly-known-as-generator-expressions?
Since Guido, the first respondent, did not immediately shoot the idea down, I intend to flesh it out and make it more concrete.
To expand on this: an iterable represents a collection of information objects. Some iterables are concrete collections. Others are more abstract, generating objects on demand in an order that may or may not be significant. Set builders in math define a set in terms of 1 set, 0 to 1 filters, and 1 transform: defined-set = map(tranform, filter(predicate, known-set)). (One could say that there is always a filter, which defaults to passing everything.) Python builders generalize the type 'set' to 'iterable' and the first and second numbers 1 to n and specify a particular nested order of iteration and filtration. For n left as 1, the type generalization is new_iterable = output_type(map(transform, filter(predicate, iter(iterable)). I omitted above the potential dependence of iterable, predicate, and transform pre-existing arguments. For generator builders, define output type 'generator' as a identity function when the input is a generator. The generalization to n > 1 is tricky to specify with functions call, as I did above, because the filtered iterations are nested rather than crossed. Consequently, each iterable and filter (as well as the tranform) could depend not only on values existing before the top call but also the current values of surrounding iterations.
What all 4 results have in common is that they are (mutable) iterables produced from iterators and other inputs with builder syntax. Aside from that, it is true that there are differences between concrete iterables like set, list, and dict versus generator iterators. But to me, this is secondary in this context. One could note that lists and dicts can be subscripted, sets and generators cannot. Or note that dict builders are 'different' because they use the funny dict item ':' notation instead of the (key,value) notation that would make {dict-builder} = dict(dict-builder) true without needing an asterisk. (But then something else would be needed to mark {(k,v) for k,v in it} as a dict rather than set. The use of ':' is quite clever.) -- Terry Jan Reedy

On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy <tjreedy@udel.edu> wrote:
Since Guido, the first respondent, did not immediately shoot the idea down, I intend to flesh it out and make it more concrete.
Maybe I should have shot it down. The term is entrenched in multiple languages by now (e.g. https://en.wikipedia.org/wiki/List_comprehension). Regarding "list builder" one could argue that it would just add more confusion, since there's already an unrelated Builder Pattern ( https://en.wikipedia.org/wiki/Builder_pattern) commonly used in Java. (Though I worry about the presence of a Python example in that Wikipedia page. :-) Also, "generator builder" is not much more expressive than "generator expression", and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions". Maybe it's not too late to start calling the latter "generator comprehensions" so that maybe by the year 2025 we can say "comprehensions" and everyone will understand we mean all four types? FWIW more people should start using "list display" etc. for things like [a, b, c]. -- --Guido van Rossum (python.org/~guido)

On 07.07.2018 2:31, Guido van Rossum wrote:
According to https://en.wikipedia.org/wiki/List_comprehension#History, the term's known from at least 1977 and comes from such influential languages as NPL, Miranda and Haskell. So it's not you to blame for it :-)
Also, "generator builder" is not much more expressive than "generator expression",
"generator builder" is simply incorrect. The GE doesn't "build" generators, it's a generator itself. It's a generator _and_ an expression. What could be a more obvious name? This suggestion looks like coming from someone who hasn't quite grasped generators yet.
and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions".
Since "X comprehensions" are advertised as and intended to be functionally equivalent to `X(generator expression)', I use just "generator expressions" to refer to all. That's accurate because the common part with the distinctive syntax -- which is the thing referred to when addressing them all -- effectively _is_ a generator expression (the syntax differences in the leading term are insignificant), what wraps it is of no concern. So, no new terms are necessary, but someone who cares may add a note to the docs to this effect.

On 07.07.2018 2:58, Ivan Pozdeev via Python-Dev wrote:
Oh, I see. So, "comprehension" is actually the official term for this "distinctive syntax", and the fact that "generator expressions" came to use it is but a coincidence. In that case, we can do a Solomon's decision: mention _both_ that "comprehension" is the official term for the syntax in GE's reference entry, _and_ the fact that "X comprehensions" are effectively wrapped GEs in their reference entries. Then everyone will learn both terminologies and could choose which is more convenient to use.

https://github.com/python/cpython/pull/8145 On 07.07.2018 3:33, Ivan Pozdeev via Python-Dev wrote:

On Sat, Jul 07, 2018 at 02:58:23AM +0300, Ivan Pozdeev via Python-Dev wrote:
Also, "generator builder" is not much more expressive than "generator expression",
I agree with Guido on that comment. The only advantage (such little as it is) is that we can refer to them all using the same terminology: [list | set | dict | generator] builder syntax but given how prevalent the comprehension terminology has become, maybe the best we can hope for is to start using "generator comprehension".
"generator builder" is simply incorrect. The GE doesn't "build" generators, it's a generator itself.
Nobody suggested that it was an thing that you call to build a generator. The name refers to the syntax, not the object. I did refer to it as *generator builder syntax* in my earlier post, and explicitly noted that "list/set/dict/generator builder" was the abbreviated form. But as Guido says, the possible confusion with the Builder design pattern makes this terminology confusing. If we were back in Python 2.2 days when neither the feature nor the terminology "comprehension" were so well established, perhaps we could have gone with "builder" instead, but I think that ship has sailed.
It's a generator _and_ an expression. What could be a more obvious name?
It's not about the obviousness, it is about it being a mouthful to say "comprehension or generator expression" to represent something which is conceptually a single kind of thing. We can use "comprehension" to group "list comprehension or dict comprehension or dict comprehension", but generator expressions are the odd one out.
This suggestion looks like coming from someone who hasn't quite grasped generators yet.
I assure you that both Greg and I understand generators quite well. -- Steve

In response to Guido's reply to my post fleshing out my idea to use 'generator|list|set|dict builder', On 7/6/2018 7:58 PM, Ivan Pozdeev via Python-Dev wrote:
"A list comprehension is a syntactic construct available in some programming languages for creating a list based on existing lists. It follows the form of the mathematical set-builder notation (set comprehension) as distinct from the use of map and filter functions." Mathematicians do not always agree on terminology and notation. I believe that 'set builder notatation' is both older and was and perhaps is more widespread than 'set comprehension'. I have read that it is at least a century old. But https://en.wikipedia.org/wiki/Set-builder_notation does not seem to be the place In any case, Python's comprehensions use an English-syntax version of extended set builder notation. "In Python, the set-builder's braces are replaced with square brackets, parentheses, or curly braces, giving list, generator, and set objects, respectively. Python uses an English-based syntax."
Also, "generator builder" is not much more expressive than "generator expression",
I looked for an alternative 'x' to 'comprehension' such that 'generator|list|set|dict x' works and is specific to the notation. 'Builder' is a reasonable choice. 'expression' is way too general. A 'list expression', for instance, is any expression that evaluated to a list. In this context, I consider that the specific term 'says more' than the general term. On the face of it, a generator expression is an expression that evaluates to a generator. In this sense, 'f(args)', where f is a generator function, is a generator expression. In any case, 'generator comprehension' is an awkward 8 syllable mouthful. -- Terry Jan Reedy

On 7/6/2018 9:01 PM, Terry Reedy wrote:
I'm not sure if your quote above was quoting documentation, or was a suggested quote to add to the documentation, I think the latter, as Google didn't find it. The conflict between the "Builder pattern" and "set-builder notation" can be disambiguated by consistently using the hyphenated "set-builder" (as wikipedia does). And happily, by using wikipedia terms, they would be easily found with explanations outside of python docs as well as (if this is done) inside. We do not need [ typ + ' builder' for typ in ('set', 'list', 'dict', 'generator')] only set-builder. The fencing and : determine the type of the result. We could use [ typ + ' form of set-builder' for typ in ('set', 'list', 'dict', 'generator')] in the few places where the type of the set-builder must be disambiguated, avoiding the need for the compound terms. The result of ( set-builder ) is a generator. We do not need the term "generator expression" or "generator comprehension". Use "generator form of set-builder"... yes, it is one or two syllables longer, but is clearer. A generator can be produced in one of two ways: either a function containing a yield, or a set-builder delimited by parentheses or used as an actual parameter to a function, both of which can be referred to as the "generator form of set-builder". Glenn

On 7/6/2018 7:31 PM, Guido van Rossum wrote:
I glad you did not do so immediately since some of what I worked out since applies to the alternative of consistently using 'comprehension'.
I was not aware of that. I read enough to see that as a relevant conflict.
If one views 'generator expression' as a 2-word phrase, as opposed to a defined compound word, it could mean either 'expression that contains a generator' or 'expression that evaluates to a generator. With the latter meaning, 'generator_func(*args)' is a generator expression. I intended 'generator builder' to be more clearly delimited. So is 'generator comprehension'.
and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions".
That was part of my motivation also.
Maybe it's not too late to start calling the latter "generator comprehensions"
Having proposed a more drastic change, I obviously think it is not too late to change the doc at least for 3.8. (If we do it now, I would consider 3.7 also.) Rename the Expressions section to just 'Comprehensions'. Define 'comprehension' perhaps as "an expression that defines an iterable using Python's adaptation and generalization of extended set builder notation". Comprehensions have to be fenced for use (except for gencomps in function calls) to determine the concrete type of iterable. The key:value syntax that separates dict from set displays separates dict from set comprehensions. Otherwise: Change to 'generator comprehension'. Do whatever to the doc grammar. Adjust glossary entries. If allowed in our reference format, perhaps link to Wikipedia articles on 'set builder notation' and 'list comprehension'. The 8 syllables of 'generator comprehension' is bit long for a compound word. Python uses '<genexpr>' as the pseudo-name for generators. Some people use 'genexp' as an abbreviation (do they pronounce the 'p'?), along with listcomp. 'Gencomp' should do as well.
Definitely. -- Terry Jan Reedy

TL;DR- +1 on advocating the term “generator comprehension” as a synonym for “generator expression” in future documentation and instructional materials. The long version: If we were starting from scratch, maybe it would makes sense to use “ builder” or something else, rather than comprehension. But we’re not starting from scratch. And “list comprehension” has become a pretty universal (or at least common) term in the python world. So we really can’t expect to re-name that. And despite the term having its roots in an esoteric corner of mathematics, I’ve never seen anyone confused by it — it’s just a new word for a new thing to most folks. Which is actually why “generator expression” is a bit confusing — it seems to have more in common with comprehensions than with generators (As in: the things functions with a yield in them produce) In fact, the term “generator” is a bit confusing itself — due to it being a common English word, it seems to refer to the generic idea of “a thing that generates values on the fly”, which, of course it can be in Python. But it actually is a specific type. And you can make a “thing that generates values in the fly” with a regular class that follows the iterator protocol. So re-using the word in “generator expression” just adds a bit to the confusion. (Even though it’s technically correct, it does “make” an actual generator object, doesn’t it? (On a phone, so I can’t check) but that’s kind of an implementation detail. Also: if you google: “python list comprehension” you get a LOT of hits to tutorials, etc. So it is WAY too late to change that name. When you google “python generator expression” the top hit is the python docs, and the next few are about comparing generator expressions and list comprehensions. So I think we could start a new naming scheme for those. ‘cause then we’d get lost of questions about the difference between generator expressions and generator comprehensions ;-) -CHB

On Mon, 9 Jul 2018 at 12:05 Guido van Rossum <guido@python.org> wrote:
I think this is worth a try.
How far do we want to go with this? Update the docs? Update the grammar and/or code? I think the former is probably good enough for now to see if it takes, and if it does then we can talk about updating code to not confuse people.

Definitely docs first. And we should keep references to generator expressions too, if only to explain that they've been renamed. Perhaps someone should try it out in a 3rd party tutorial to see how it goes? I'm CC'ing Raymond Hettinger. On Mon, Jul 9, 2018 at 12:57 PM Brett Cannon <brett@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)

On Mon, Jul 9, 2018 at 3:18 PM, Guido van Rossum <guido@python.org> wrote:
I'm not sure what "trying it out in a tutorial" would look like. I try to be pretty clear about terminology when I teach newbies -- so I don't want to tell anyone this new thing is called a "generator comprehension" if they aren't going to see that term anywhere else. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 13 July 2018 at 15:30, Chris Barker via Python-Dev <python-dev@python.org> wrote:
Nina Zakharenko made the "they're officially called generator expressions, but I find it more helpful to think of them as generator comprehensions" observation in her PyCon 2018 presentation on "Elegant Solutions for Everyday Python Problems": https://www.slideshare.net/nnja/elegant-solutions-for-everyday-python-proble... The article from Ned Batchelder mentioned in that slide is this one, which goes through the history of Raymond originally proposing the notion as generator comprehensions, them getting changed to generator expressions during the PEP 289 discussions, and then asking if it might be worth going back to the originally suggested name: https://nedbatchelder.com/blog/201605/generator_comprehensions.html And then in PEP 572, it was found that being able to group all 4 constructs (list/set/dict comps + generator expressions) under a single term was a genuinely useful shorthand: https://www.python.org/dev/peps/pep-0572/#scope-of-the-target So trying out the terminology in a tutorial context would be to do something similar to what Nina did in her talk: introduce the notion of list/set/dict/generator comprehensions, and then make a side note that the latter are officially referred to as "generator expressions". This wouldn't be the first time that terminology has differed between Python-as-commonly-taught and Python-as-formally-defined, as I've yet to hear anyone refer to container displays outside a language design discussion - everyone else calls them container literals (or, more typically, a literal for the specific builtin container type being discussed). In this case, though, we'd be considering eventually changing the language reference as well, and perhaps even some day the AST node name (from GeneratorExp to GeneratorComp). We wouldn't need to change anything in the grammar definition (since that already shares the comp_for and comp_if syntax definitions between container comprehensions and generator expressions), or the AST node structure (since GeneratorExp already uses a "comprehensions" attribute, the same as the ListComp/SetComp/DictComp nodes). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Thanks Nick, I'll adopt this approach when I update my teaching materials. If I think of it, I"ll post here when I do that -CHB On Sun, Jul 15, 2018 at 12:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

I've pretty much always taught using the "comprehension" term. It makes sense to introduce comprehensions on concrete collections first. Then I typically say something like "This special kind of comprehension is called a 'generator expression'". Usually that's accompanied by a motivation like creating a listcomp of a hundred million items, then showing that creating the generator expression is instantaneous. However, after the initial introduction, I consistently call it a generator expression. Albeit, for the students I teach—data scientists and quants, mostly—there's not a lot of time spent on that being the introduction (by then I'm talking about NumPy and Pandas, and scikit-learn, and Seaborn, abd Statsmodels, and so on. If things are "lazy" it's because they are Dask deferred or Dask DataFrame. On Sun, Jul 15, 2018, 1:02 PM Chris Barker via Python-Dev < python-dev@python.org> wrote:

[Nick Coghlan]>
actually made those semantics available as an explicit
[Greg Ewing] I'm not sure that's possible. If I understand correctly,
So a nested comprehension would declare its assignment expression targets as parentlocal in its synthesized function, and in all the containing synthesized functions generated for containing comprehensions. This appears in some strained ;-) way "natural" only because there is no explicit way to declare something "local" in Python. In just about any other language with closures and nested lexical scopes, comprehensions and generator expressions would have been implemented via nested functions that explicitly declared their "for" target names "local". and nothing else. The only change needed then for PEP 572 (b) semantics would be to declare assignment expression target names local (if their scope wasn't already known) in the closest containing non-synthesized block. None of which really matters. The real question is which semantics are desired.

[Tim]
If the parent has a matching parentlocal declaration for the same name then the original
really refers to the grandparent - and so on.
[Greg]
Ah, I missed that part, sorry -- I withdraw that particular objecttion.
Good! I have another reply that crossed in the mail.
I don't even think it makes "explaining" easier. It doesn't eliminate any corner cases, it just pushes them into the definition of what "parentllocal" means. What it would do is make writing synthetic functions "by hand" to implement comprehensions more uniform, because "parentlocal" would handle the corner cases by itself instead of making the programmer figure out when and where they need to type "nonlocal", "global", and/or cruft to establish a name as local to a block in which the name otherwise does't appear as a binding target. But to the extent that doing such translations by hand is meant to be "educational", it's more educational to learn how to do that stuff yourself.

So IIUC you are okay with the behavior described by the PEP but you want an explicit language feature to specify it? I don't particularly like adding a `parentlocal` statement to the language, because I don't think it'll be generally useful. (We don't have `goto` in the language even though it could be used in the formal specification of `if`, for example. :-) But as a descriptive mechanism to make the PEP's spec clearer I'm fine with it. Let's call it `__parentlocal` for now. It would work a bit like `nonlocal` but also different, since in the normal case (when there's no matching `nonlocal` in the parent scope) it would make the target a local in that scope rather than trying to look for a definition of the target name in surrounding (non-class, non-global) scopes. Also if there's a matching `global` in the parent scope, `__parentlocal` itself changes its meaning to `global`. If you want to push a target through several level of target scopes you can do that by having a `__parentlocal` in each scope that it should push through (this is needed for nested comprehensions, see below). Given that definition of `__parentlocal`, in first approximation the scoping rule proposed by PEP 572 would then be: In comprehensions (which in my use in the PEP 572 discussion includes generator expressions) the targets of inline assignments are automatically endowed with a `__parentlocal` declaration, except inside the "outermost iterable" (since that already runs in the parent scope). There would have to be additional words when comprehensions themselves are nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's intention is that inline assignments anywhere there end up targeting the scope containing the outermost comprehension. But this can all be expressed by adding `__parentlocal` for various variables in various places (including in the "outermost iterable" of inner comprehensions). I'd also like to keep the rule prohibiting use of the same name as a comprehension loop control variable and as an inline assignment target; this rule would also prohibit shenanigans with nested comprehensions (for any set of nested comprehensions, any name that's a loop control variable in any of them cannot be an inline assignment target in any of them). This would also apply to the "outermost iterable". Does this help at all, or did I miss something? --Guido On Wed, Jun 27, 2018 at 5:27 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

[Guido]
If this has to be done ;-) , I suggest removing that last exception. That is, "[all] targets of inline assignments in comprehensions are declared __parentlocal", period, should work fine for (b). In case one appears in the outermost iterable of the outermost comprehension, I believe such declaration is merely semantically redundant, not harmful. Where "redundant" means someone is so familiar with the implementation that the scope implications of "already runs in the parent scope" are immediately clear. For someone muddy about that, it would be a positive help to have the intent clarified by removing the exception. Plus 99% of the point of "parentlocal" seemed to be to allow mindless ("uniform") by-hand translation of nested comprehensions to nested Python functions, and an exception for the outermost iterable would work against that intent.

On 28 June 2018 at 08:31, Guido van Rossum <guido@python.org> wrote:
Yep, it does, and I don't think you missed anything. Using "__parentlocal" to indicate "parent local scoping semantics apply here" still gives the concept a name and descriptive shorthand for use in pseudo-code expansions of assignment expressions in comprehensions, without needing to give it an actually usable statement level syntax, similar to the way we use "_expr_result" and "_outermost_iter" to indicate references that in reality are entries in an interpreter's stack or register set, or else a pseudo-variable that doesn't have a normal attribute identifier. And if anyone does want to make the case for the syntax being generally available, they don't need to specify how it should work - they just need to provide evidence of cases where it would clarify code unrelated to the PEP 572 use case. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 24, 2018 at 04:33:38PM +1000, Nick Coghlan wrote: [...]
You seem to be talking about an implementation which could change in the future. I'm talking semantics of the proposed language feature. As a programmer writing Python code, I have no visibility into the implementation. The implementation could change ten times a day for all I care, so long as the semantics remain the same. I want the desired semantics to drive the implementation, not the other way around. You seem to want the implementation to drive the semantics, by eliminating the proposed feature because it doesn't match your deep understanding of the implementation as a nested function. I want this feature because its useful, and without it the use-cases for assignment expressions are significantly reduced. As far as "implicit", for the sake of the discussion, I'll grant you that one. Okay, the proposed behaviour will implicitly enable comprehensions to export their state. Now what? Is that a good thing or a bad thing? If "implicit" (with or without the scare quotes) is such a bad thing to be avoided, why are comprehensions implemented using an implicit function?
You talk about "nested comprehension scope", and that's a critical point, but I'm going to skip answering that for now. I have a draft email responding to another of your posts on that topic, which I hope to polish in the next day.
If it is okay for you to amend the list comprehension to behave as if it were wrapped in an implicit nested function, why shouldn't it be okay to behave as if assignments inside the comprehension included an implicit nonlocal declaration?
Obviously "everyone" is an exaggeration, but, yes, I stand by that -- most people don't even give comprehension scope a thought until they get bitten by it. Either because (Python 2) they don't realise the loop variable is local to their current scope: http://www.librador.com/2014/07/10/Variable-scope-in-list-comprehension-vs-g... or (Python 3) they get bitten by the change: https://old.reddit.com/r/Python/comments/425qmb/strange_python_27_34_differe... (As is so often the case, whatever behaviour we choose, we're going to surprise somebody.) It is hardly surprising that people don't think too hard about scoping of comprehensions. Without a way to perform assignments inside comprehensions, aside from the loop variables themselves, there's nothing going on inside a comprehension where it makes a visible difference whether it is a local scope or a sublocal scope. *IF* assignment expressions are introduced, that is going to change. We have some choices: 1. Keep assignment expressions encapsulated in their implicit function, and be prepared for people to be annoyed because (with no way to declare them global or non-local inside an expression), they can't use them to get data in and out of the comprehension. 2. Allow assignment expressions to be exported out of the comprehension, and be prepared for people to be annoyed because they clobbered a local. (But for the reasons Tim Peters has already set out, I doubt this will happen often.) 3. Allow some sort of extra comprehension syntax to allow global/nonlocal declarations inside comprehensions. x = 1 [nonlocal x := x+i for i in sequence] (Hmmm... I thought I would hate that more than I actually do.) 4. Have some sort of cunning plan whereby if the variable in question exists in the local scope, it is implicitly local inside the comprehension: x = 1 [x := i+1 for i in (1, 2)] assert x == 3 but if it doesn't, then the variable is implicitly sublocal inside the comprehension: del x [x := i+1 for i in (1, 2)] x # raises NameError Remember, the driving use-case which started this (ever-so-long) discussion was the ability to push data into a comprehension and then update it on each iteration, something like this: x = initial_value() results = [x := transform(x, i) for i in sequence] Please, Nick, take your implementor's hat off, forget everything you know about the implementation of comprehensions and their implicit nested function, and tell me that doesn't look like it should work. -- Steve

On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Which means there is another option. 5. Have the assignment be local to the comprehension, but the initial value of ANY variable is looked up from the surrounding scopes. That is: you will NEVER get UnboundLocalError from a comprehension/genexp; instead, you will look up the name as if it were in the surrounding scope, either getting a value or bombing with regular old NameError. Or possibly variations on this such as "the immediately surrounding scope only", rather than full name lookups. It'd have an awkward boundary somewhere, whichever way you do it. This isn't able to send information *out* of a comprehension, but it is able to send information *in*. ChrisA

On Sun, Jun 24, 2018 at 2:10 PM Chris Angelico <rosuav@gmail.com> wrote:
But this "horrifies" me for a slightly different reason: it effectively introduces a new case of dynamic scoping, which Python used to do everywhere but has long switched away from, with the exception of class scopes (whose difference with function scopes sometimes confuses people -- usually people who put too much code in their class scope). -- --Guido van Rossum (python.org/~guido)

This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected. On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <steve@pearwood.info> wrote:
If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner. results = accumulate(sequence, transform) The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section. While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first. Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases. On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev < python-dev@python.org> wrote:
again, not a huge deal, just a little bit more complexity
I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language. On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
On the other hand, an "expert" may be so steeped in a particular subculture that he no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.
On the contrary, I believe that "easy for beginners" should be a major concern. Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts. Mozilla's array comprehensions are almost identical to Python's, aside
from a couple of trivial differences:
I can't prove it, but I think the phrase ordering difference is not trivial.
Python: any(line.startswith('#') for line in file) English: Any line starts with "#" in the file?

On Sun, Jun 24, 2018 at 2:41 PM Michael Selik <mike@selik.org> wrote:
Sure. Since the target for the PEP is Python 3.8 I am in no particular hurry. It would be important to know how you present it to your students.
I think that's a misunderstanding. At the very least the typical use case is *not* using an existing transform function which is readily passed to accumulate -- instead, it's typically written as a simple expression (e.g. `total := total + v` in the PEP) which would require a lambda. Plus, I don't know what kind of students you are teaching, but for me, whenever the solution requires a higher-order function (like accumulate), this implies a significant speed bump -- both when writing and when reading code. (Honestly, whenever I read code that uses itertools, I end up making a trip to StackOverflow :-).
The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section.
(I think you mean first_true().)
I trust Tim's intuition here, he's written about this. Also, Python's predecessor, ABC, had quantifiers (essentially any() and all()) built into the language, and the semantics included making the first (counter-)example available ( https://homepages.cwi.nl/~steven/abc/qr.html#TESTS). Essentially IF SOME x IN values HAS x < 0: WRITE "Found a negative x:", x equivalently IF EACH x IN values HAS x >= 0: # ... ELSE: WRITE "Found a negative x:", x and even IF NO x IN values HAS x < 0: # ... ELSE: WRITE "Found a negative x:", x
Perhaps, but IMO the new itertool would be much less useful than the new syntax.
I don't think that appeal to beginners is Python's greatest strength. I'd rather say that it is its appeal to both beginners and experts (and everyone in between). If true appeal to beginners is needed, Scratch or Processing would probably be better.
Nor should we cater to them excessively though. While the user is indeed king, it's also well known that most users when they are asking for a feature don't know what they want (same for kings, actually, that's why they have advisors :-).
I'm sorry, but this offends me, and I don't believe it's true at all. Python is *not* a beginners language, and you are mixing ease of use and ease of learning. Python turns beginners into experts at an unprecedented rate, and that's the big difference with Scratch. -- --Guido van Rossum (python.org/~guido)

On Sun, Jun 24, 2018 at 4:57 PM Guido van Rossum <guido@python.org> wrote:
Absolutely. Since this has come up, I'll make an effort to be more systematic in data collection.
Plus, I don't know what kind of students you are teaching, but for me,
Mostly mid-career professionals, of highly varying backgrounds. The higher-order functions do require some cushioning getting into, but I have some tricks I've learned over the years to make it go over pretty well. On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
By saying "Scratch for adults" I meant that Python is a language that can be adopted by beginners and rapidly make them professionals, not that it's exclusively a beginner's language. Also, Scratch and similar languages, like NetLogo, have some interesting features that allow beginners to write some sophisticated parallelism. I don't mean "beginner's language" in that it's overly simplistic, but that it enables what would be complex in other languages. I realize that my phrasing was likely to be misunderstood without knowing the context that I teach working professionals who are asked to be immediately productive at high-value tasks.

Steven D'Aprano wrote:
You seem to be talking about an implementation which could change in the future. I'm talking semantics of the proposed language feature.
The way I see it, it's not about implementation details, it's about having a mental model that's easy to reason about. "Comprehensions run in their own scope, like a def or lambda" is a clear and simple mental model. It's easy to explain and keep in your head. The proposed semantics are much more complicated, and as far as I can see, are only motivated by use cases that you shouldn't really be doing in the first place. -- Greg

On Sun, Jun 24, 2018 at 03:56:47PM +1000, Steven D'Aprano wrote:
There is no consensus that the change to comprehensions was a good thing or justified.
On re-reading that, I think its wrong -- it wasn't really what I intended to say. What I intended to say was, in hindsight, more like: *Despite the consensus to change comprehension scope*, there's a contingent of people who are not convinced that the change was a good thing or justified. Sorry for the inaccurate comment. Mea culpa. -- Steve

On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote: [...]
Without knowing how you worded the question, and the reasons for this horrified reaction, I'm afraid that isn't really helpful. It is nothing more than an appeal to emotion: https://en.wikipedia.org/wiki/Wisdom_of_repugnance Such strong emotions as "horrified" are typically a sign of an immediate, emotional gut reaction, not careful thought. We often see those sorts of reactions attached to the most objectively trivial matters. Immediate gut reactions are rarely a good guide because they tend to over-value the status quo, exaggerate the difficulty and costs of change, and under-estimate the benefits. Speaking personally, I've learned to question my immediately gut reaction. (And I remember to do so at least half the time.) PEP 572 is an example: when the issue was first raised back in February, my gut reaction was "Not in MY Python!!!" but by taking it seriously and running through some examples over the course of the discussion, I realised that, actually, I cautiously favour the idea. Of course, matters of *personal taste* cannot be anything but gut reaction, but in those matters, what one person holds strongly another can legitimately reject strongly. We ought to try to look beyond personal taste, and try (even if only imperfectly) to consider rational reasons for and against a proposal. If we do, reactions like "horrified" are rarely justified. It's just a minor feature in a programming language, the world will go on one way or the other, and Python already has trickier gotchas.
While we certainly don't want to make "non-obvious" a virtue for its own sake, obviousness (obvious to who?) ought to take a distant second place to *useful*. Otherwise we'd have to give up an awful lot of existing Python, starting with the fundamental execution model. (Oh, the number and length of arguments about whether Python uses call by value or call by reference, why mutable defaults and [[]]*3 are "broken"... if you think Python's execution model is "obvious" you've been using Python too long ;-) But as Tim Peters has said on a number of occasions, nobody is suggesting changing the interpretation of current comprehension semantics. Comprehension loop variables will continue to remain isolated to the comprehension. (And for the record, that makes *comprehensions* a weird special case, not assignment expressions. All other expressions run in the current lexical scope. Comprehensions introduce an implicit, invisible, sub-local scope that doesn't match up with a change in indentation as class and def statements do.) The behaviour in question is a matter of *assignment expression* semantics, not comprehensions. And honestly, I don't see why the proposed behaviour is "horrifying". Here's the high-level overview: - at the top level of a module, assignment expressions assign in the global scope; - inside a class, assignment expressions assign in the class scope; - inside a function, assignment expressions assign in the function local scope (unless declared global or nonlocal); - inside a comprehension, assignment expressions assign in the surrounding lexical scope (the surrounding function, class or module). The first three are the same as ordinary statement assignment. The last one is what you would expect if you treat comprehensions as any other expression which run in the current lexical scope. (The current function or class or module.) Even if we treat it as a "weird special case" (I don't think it is, but for the sake of the argument let's say it is) its not hard to explain. As I discuss below, you can get a very long way indeed working with comprehensions without once thinking about the scope they run in. By the time you need to think about comprehension scope, it shouldn't be hard to deal with the rule: - loop variables are hidden in a comprehension private scope; - explicit assignment expression variables are not. This is not async, or metaclasses, or even Unicode. [...]
I can't say I've done a broad survey, but the third-party documentation I've read on comprehensions typically glosses over the scoping issues without mentioning them. To the extent that scoping is even hinted at, comprehensions are treated as expressions which are exactly equivalent to re-writing them as a for-loop in the current scope. This is a typical example, found as the top result on googling for "python comprehensions": https://www.google.com/search?q=python+comprehensions http://www.pythonforbeginners.com/basics/list-comprehensions-in-python Nothing is mentioned about scope, and it repeats the inaccurate but simple equivalency: for item in list: if conditional: expression But perhaps that tutorial is too old. Okay this recent one is only a little more than a year old: https://hackernoon.com/list-comprehension-in-python-8895a785550b Again, no mention of scoping issues, comprehensions are simply expressions which presumably run in the same scope as any other expression. I think you over-estimate how many newcomers to Python are even aware that the scope of comprehensions is something to consider.
I look forward to reading it, and I promise I won't go by my gut reaction :-) -- Steve

On Sat, Jun 23, 2018 at 1:48 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Even first-party documentation elides that distinction. The same inaccurate-but-simple equivalency - even using the word "equivalent" - comes up here: https://docs.python.org/3/howto/functional.html?highlight=equivalent#generat... So I'm very sympathetic to the desire to have assignment expressions inside comprehensions behave like assignment expressions outside comprehensions. The trouble is that they are then _not_ the same as other names inside comprehensions. One way or another, there's a confusing distinction, especially at class scope. Unless this comes with an actual semantic change that affects existing code, there is going to be a bizarre disconnect *somewhere*, and it's just a matter of where. ChrisA

On 23 June 2018 at 13:48, Steven D'Aprano <steve@pearwood.info> wrote:
I put quite a bit of work into making it possible for folks to gloss over the distinction and still come to mostly-correct conclusions about how particular code snippets would behave. I was only able to achieve it because the folks that designed lexical scoping before me had made *read* access to lexical scopes almost entirely transparent, and because generator expressions were designed to fail fast if there was a bug in the expression defining the outermost iterable (which meant that even at class scope, the outermost iterable expression still had access to class level variables, because it was evaluated *outside* the nested scope). *Write* access to lexically nested scopes, by contrast, was omitted entirely from the original lexical scoping design, and when it was later added by https://www.python.org/dev/peps/pep-3104/, it was done using an explicit "nonlocal" declaration statement (expressly akin to "global"), and PEP 3099 explicitly ruled out the use of ":=" to implicitly declare the target name as being non-local. PEP 572 is thus taking the position that: - we now want to make write access to outer scopes implicit (despite PEP 3099 explicitly ruling that out as desired design feature) - but only in comprehensions and generator expressions (not lambda expressions, and not full nested functions) - and only for assignment expressions, not for loop iteration variables - and we want it to implicitly choose between a "global NAME" declaration and a "nonlocal NAME" declaration based on where the comprehension is defined - and this is OK because "nobody" actually understands how comprehensions hide the iteration variable in practice, and "everybody" thinks they're still a simple for loop like they were in Python 2 - the fact that the language reference, the behaviour at class scopes, the disassembly output, and the behaviour in a debugger all indicate that comprehensions are full nested scopes isn't important This level of additional complication and complexity in the scoping semantics simply isn't warranted for such a minor readability enhancement as assignment expressions. Cheers, Nick. P.S. "You did such a good job of minimising the backwards compatibility breakage when we changed the semantics of scoping in comprehensions that we now consider your opinion on reasonable scoping semantics for comprehensions to be irrelevant, because everybody else still thinks they work the same way as they did in Python 2" is such a surreal position for folks to be taking that I'm having trouble working out how to effectively respond to it. Guido has complained that "I keep changing my mind about what I want", but that's not what's actually going on: what I want is to keep folks from taking our already complicated scoping semantics and making it close to impossible for anyone to ever infer how they work from experimentation at the interactive prompt. That goal has pretty much stayed consistent since the parent local scoping proposal was first put forward. What keeps changing is my tactics in pursuing that goal, based on my current perception of what the folks pushing that proposal are actually trying to achieve (which seems to be some combination of "We want to pretend that the Python 3 scoping changes in comprehensions never happened, but we still want to avoid leaking the iteration variables somehow" and "We want to enable new clever tricks with state export from comprehensions and generator expressions"), as well as repeatedly asking myself what *actually* bothers me about the proposal (which I've now distilled down to just the comprehension scoping issue, and the reliance on an arbitrary syntactic restriction against top level usage to avoid competing with traditional assignment statements). -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, 23 Jun 2018 00:22:33 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Thank you. Personally, I'd like to see feedback from educators/teachers after they take the time to read the PEP and take some time to think about its consequences. My main concern is we're introducing a second different way of doing something which is really fundamental.
Hmm... I don't think conflating the assignment expression proposal with comprehension semantics issues is helping the discussion. Regards Antoine.

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
I've started testing the proposed syntax when I teach. I don't have a large sample yet, but most students either dislike it or don't appreciate the benefits. They state a clear preference for shorter, simpler lines at the consequence of more lines of code. This may partially be caused by the smaller screen real estate on a projector or large TV than a desktop monitor. My intuition is that one strength of Python for beginners is the relative lack of punctuation and operators compared with most other languages. This proposal encourages denser lines with more punctuation. Because of the order of operations, many uses of ``:=`` will also require parentheses. Even relatively simple uses, like ``if (match := pattern.search(data)) is not None:`` require doubled parentheses on one side or the other. Beginners are especially prone to typographical errors with mismatched parentheses and missing colons and get easily frustrated by the associated syntax errors. Given the following options: A. if (row := cursor.fetchone()) is None: raise NotFound return row B. row = cursor.fetchone() if row is None: raise NotFound return row C. if (row := cursor.fetchone()) is not None: return row raise NotFound D. row = cursor.fetchone() if row is not None: return row raise NotFound The majority of students preferred option B. I also tested some regex match examples. Results were similar.
My main concern is we're introducing a second different way of doing something which is really fundamental.
The few students who like the proposal ask why it requires creating a new operator instead of repurposing the ``=`` operator. I'll reserve my personal opinions for a different thread.

On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mike@selik.org> wrote:
This is partly because students, lacking the experience to instantly recognize larger constructs, prefer a more concrete approach to coding. "Good code" is code where the concrete behaviour is more easily understood. As a programmer gains experience, s/he learns to grok more complex expressions, and is then better able to make use of the more expressive constructs such as list comprehensions. ChrisA

On Fri, Jun 22, 2018 at 10:19 AM Chris Angelico <rosuav@gmail.com> wrote:
I don't think that's the only dynamic going on here. List comprehensions are more expressive, but also more declarative and in Python they have nice parallels with SQL and speech patterns in natural language. The concept of a comprehension is separate from its particular expression in Python. For example, Mozilla's array comprehensions in Javascript are/were ugly [0]. Students who are completely new to programming can see the similarity of list comprehensions to spoken language. They also appreciate the revision of certain 3-line and 4-line for-loops to comprehensions. I didn't get the same sense of "Oh! That looks better!" from my students when revising code with an assignment expression. Despite my best efforts to cheerlead, some students initially dislike list comprehensions. However, they come around to the idea that there's a tradeoff between line density and code block density. Comprehensions have a 3-to-1 or 4-to-1 ratio of code line shrinkage. They're also often used in sequence, like piping data through a series of transforms. Even if students dislike a single comprehension, they agree that turning 15 lines into 5 lines improves the readability. In contrast, an assignment expression only has a 2-to-1 code line compression ratio. It might save a level of indentation, but I think there are usually alternatives. Also, the assignment expression is less likely to be used several times in the same block. A good pitch for an assignment expression is refactoring a cascade of regular expressions: for line in f: mo = foo_re.search(line) if mo is not None: foo(mo.groups()) continue mo = bar_re.search(line) if mo is not None: bar(mo.groups()) continue mo = baz_re.search(line) if mo is not None: baz(mo.groups()) continue Here the assignment operator makes a clear improvement: for line in f: if (mo := foo_re.search(line)) is not None: foo(mo.groups()) elif (mo := bar_re.search(line)) is not None: bar(mo.groups()) elif (mo := baz_re.search(line)) is not None: baz(mo.groups()) However, I think this example is cheating a bit. While I've written similar code many times, it's almost never just a function call in each if-block. It's nearly always a handful of lines of logic which I wouldn't want to cut out into a separate function. The refactor is misleading, because I'd nearly always make a visual separation with a newline and the code would still look similar to the initial example. [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/...

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
Of course they do -- they're less fluent at reading code. They don't have the experience to judge good code from bad. The question we should be asking is, do we only add features to Python if they are easy for beginners? It's not that I especially want to add features which *aren't* easy for beginners, but Python isn't Scratch and "easy for beginners" should only be a peripheral concern.
Mozilla's array comprehensions are almost identical to Python's, aside from a couple of trivial differences: evens = [for (i of numbers) if (i % 2 === 0) i]; compared to: evens = [i for i in numbers if (i % 2 == 0)] - the inexplicable (to me) decision to say "for x of array" instead of "for x in array"; - moving the expression to the end, instead of the beginning. The second one is (arguably, though not by me) an improvement, since it preserves a perfect left-to-right execution order within the comprehension.
Students who are completely new to programming can see the similarity of list comprehensions to spoken language.
o_O I've been using comprehensions for something like a decade, and I can't :-) The closest analogy to comprehensions I know of is set builder notation in mathematics, which is hardly a surprise. That's where Haskell got the inspiration from, and their syntax is essentially an ASCIIfied version of set builder notation: Haskell: [(i,j) | i <- [1,2], j <- [1..4]] Maths: {(i,j) : i ∈ {1, 2}, j ∈ {1...4}} I teach secondary school children maths, and if there's a plain English natural language equivalent to list builder notation, neither I nor any of my students, nor any of the text books I've read, have noticed it. -- Steve

On 23.06.2018 5:46, Steven D'Aprano wrote:
Python's design principles are expressed in the Zen. They rather focus on being no more complex than absolutely necessary, without prioritizing either beginners or old-timers ("simple is better than complex", "complex is better than complicated").
-- Regards, Ivan

On 2018-06-22 19:46, Steven D'Aprano wrote:
- the inexplicable (to me) decision to say "for x of array" instead of "for x in array";
Believe JavaScript has for…in, but as usual in the language it is broken and they needed a few more tries to get it right. for…of is the latest version and works as expected. -Mike

On Fri, Jun 22, 2018 at 10:02 AM Michael Selik <mike@selik.org> wrote:
I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mike@selik.org> wrote:
nor would I. For a while, anyway.... But once it becomes a more common idiom, students will see it in the wild pretty early in their path to learning python. So we'll need to start introducing it earlier than later. I think this reflects that the "smaller" a language is, the easier it is to learn. Python has already grown a fair bit since 1.5 (when I started using it :-) ). Some things, like generators, are special purpose enough that I can wait pretty far into the program before teaching them. But others, like comprehensions (and lambda) are common enough that I have to introduce them pretty early on. Adding := is not a HUGE change, but it IS an expansion of the language, and one that we WILL have to introduce in an introductory course once it starts seeing common use. I really have no idea how much harder thats going to make the langauge to teach, but it will make it a bit harder -- I see enough confusion with "is" vs == already... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Jun 22, 2018 at 11:28:45AM -0700, Chris Barker via Python-Dev wrote:
Students see many features early in their path. I've had people still struggling with writing functions ask about metaclasses. People will see async code everywhere. We don't have to teach *everything* at once. The *subtleties* of assignment expressions might have some funny corner cases, but the high-level overview is simple. It is like ordinary assignment, but it is an expression that returns the value being assigned. So if you absolutely need to teach it to a beginner, it shouldn't be difficult once they understand the difference between an expression and a statement. [...]
I think that the biggest source of confusion with "is" is that it *sometimes* seems to do what is wanted, i.e. test equality, but other times doesn't. It is that inconsistency that bites. Whereas with assignment expressions, there's no such inconsistency: - regular assignment using = only works as a statement, always; - assignment expression can go anywhere an expression can go, always; - regular assignment never returns a value; - assignment expression always returns a value; - regular assignments have lots of complex forms, such as sequence unpacking, and complex targets like spam[eggs](arg).attr; - assignment expressions only takes a plain name, always. Although there is some overlap in behaviour between the two, unlike "is", there's no inconsist behaviour to lead people astray. A better syntax error for things like this: py> if mo = regex.match(string): File "<stdin>", line 1 if mo = regex.match(string): ^ SyntaxError: invalid syntax will also help, although of course some users won't read error messages for love or money. -- Steve

On Fri, Jun 22, 2018 at 7:23 PM, Steven D'Aprano <steve@pearwood.info> wrote:
These are not similar at all -- if you want similar examples, I"d say comprehensions, and lambda, both of which I DO introduce fairly early While newbies will *ask* about metaclasses, it's probably because they read about them somewhere, not because someone actually used a metaclass in a simple script or answer to a common question on SO. As for async, you are either doing async or not -- you can't even run an async def function without an event loop -- so again, it won't show up in real code newbies need to understand (at least until async becomes common practice with python...) -CHB So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an expression and a statement.
probably not, though that's a distinction that's mostly academic in the early stages of learning, it may become more critical now... again, not a huge deal, just a little bit more complexity -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Jun 22, 2018 at 09:08:37PM -0700, Chris Barker wrote:
I don't think so. People do try to use assignment in expressions, even if only by mistake writing = when they meant == and need to distinguish between them. In Python 2, the most common clash between statements and expressions was print, but at least that's gone. https://www.quora.com/Whats-the-difference-between-a-statement-and-an-expres... https://stackoverflow.com/questions/4728073/what-is-the-difference-between-a... https://stackoverflow.com/questions/43435850/what-is-the-difference-between-... Even without assignment expressions, people still need to know why they can't write "if mo = re.match(pattern, text)".
again, not a huge deal, just a little bit more complexity
Every new feature is added complexity. -- Steve

On Fri, Jun 22, 2018 at 7:28 PM, Chris Barker via Python-Dev < python-dev@python.org> wrote:
For what it's worth, Chris's thoughts are close to my own here. I and several of my colleagues teach week-long Python courses for Enthought. The target audience is mostly scientists and data scientists (many of whom are coming from MATLAB or R or IDL or Excel/VBA or some other development environment, but some of whom are new to programming altogether), and our curriculum is Python, NumPy, SciPy, Pandas, plus additional course-specific bits and pieces (scikit-learn, NLTK, seaborn, statsmodels, GUI-building, Cython, HPC, etc., etc.). There's a constant struggle to keep the Python portion of the course large enough to be coherent and useful, but small enough to allow time for the other topics. To that end, we separate the Python piece of the course into "core topics" that are essential for the later parts, and "advanced topics" that can be covered if time allows, or if we get relevant questions. I can't see a way that the assignment expression wouldn't have to be part of the core topics. async stuff only appears in async code, and it's easy to compartmentalize; in contrast, I'd expect that once the assignment expression took hold we'd be seeing it in a lot of code, independent of the domain. And yes, I too see enough confusion with "is" vs == already, and don't relish the prospect of teaching := in addition to those. That's with my Python-teaching hat on. With my Python-developer hat on, my thoughts are slightly different, but that's off-topic for this thread, and I don't think I have anything to say that hasn't already been said many times by others, so I'll keep quiet about that bit. :-) -- Mark

Nick Coghlan wrote:
Noooo..... this is just taking a bad idea and making it worse, IMO. I'm -1 on any contortions designed to allow comprehensions to assign to things in outer scopes. All the proposed use cases I've seen for this have not improved readability over writing a function that does things the usual way. Can we please leave comprehensions as declarative constructs? The best tools do just one thing and do it well. These proposals seem to be trying to turn comprehensions into swiss army knives. -- Greg

On 23 June 2018 at 09:06, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If PEP 572 was proposing the use of regular local scoping for assignment expressions in comprehensions, such that they could still be used to avoiding repeating subexpressions within an iteration, but couldn't be used to export progress data, or to maintain a partial sum without having to rely on `locals().get("total", 0)` to provide an initial value, then I wouldn't be feeling obliged to present an alternative that offers the same state export capabilities in a more explicit form. Given that PEP 572 *is* proposing implicit comprehension state export, though, then I think it's important to make the case that seeing the proposed semantics as intuitive is only going to be the case for folks that have used Python 2 style comprehensions extensively - anyone that's never encountered the old state-leaking behaviour for iteration variables is going to be surprised when assignment expressions ignore the existence of the comprehension scope (even though the iteration variable pays attention to it). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
Given that PEP 572 *is* proposing implicit comprehension state export,
"Implicit" and "explicit" are two terms which often get misused to mean "I don't like it" and "I do like it". Making the intentional choice to use an assignment expression is not really "implicit" in any meaningful sense. One might as well complain that "import this" implicitly creates a local variable "this". Well, yes, it does, in a very loose sense, but that's what imports are defined as do and it is the whole purpose for making them. If PEP 572's proposal goes ahead, the behaviour of assignment expressions will be *defined* as creating assignments in the local scope rather than the sublocal comprehension scope. To call that "implicit" is rather like saying that regular assignment is implicit.
You are making the assumption that most people are even aware of "comprehension scope". I don't think that is the case. In my experience, scoping in Python is still typically seen as the LGB rule (locals/globals/builtins). See for example this StackOverflow post from 2016: https://stackoverflow.com/questions/37211910/override-lgb-scope-rule Sometimes people remember the E/N (enclosing function/nonlocal) part. Hardly anyone remembers the C (class) part unless they are actively thinking in terms of code running inside a class definition, and even if they do, they typically aren't sure of exactly how it interacts with the rest. And I predict that even fewer think of comprehensions as a separate scope, except by ommission: they *don't think about* the scope of the loop variable until it bites them. But as Tim Peters has previously discussed, the loop variable is special, and is especially prone to accidental shadowing. That won't be the case for assignment expressions. If there's shadowing going on, it will be deliberate. Aside: I've said before that I'm not a fan of sublocal comprehension scoping, since I personally found it helpful on occasion for the loop variable to be visible outside of the comprehension. But given that the only experience most people apparently had with comprehension scoping was to be bitten by it, I grudgingly accept that encapsulating the loop variable was the right decision to make, even if it personally inconvenienced me more than it saved me. Nor was I the only one: others have been bitten by the change to comprehension scope, see for example: https://www.reddit.com/r/Python/comments/425qmb/strange_python_27_34_differe... There is no consensus that the change to comprehensions was a good thing or justified. The bottom line is that I don't think people will be surprised by assignment expression scope being local instead of sublocal. Rather I expect that they won't even think about it, until they do, and then *whatever* behaviour we pick, we'll annoy somebody. -- Steve

On 24 June 2018 at 15:56, Steven D'Aprano <steve@pearwood.info> wrote:
No, it's actually implicit: there's an extra "global NAME" or "nonlocal NAME" in the equivalent code for a comprehension that isn't there in the as-written source code, and doesn't get emitted for a regular assignment expression or for the iteration variable in a comprehension - it only shows up due to the defined interaction between comprehensions and assignment expressions.
And they behave the same way in every context where they're permitted to appear.
I do say that regular assignments implicitly declare a name as local. "Python has implicit local variable declarations" is also regularly cited as one of the differences between it and languages that require explicit declarations, like C. Even augmented assignments implicitly declare a name as being a local (hence the infamous UnboundLocalError that arises when you attempt to use an augmented assignment to rebind a name from an outer scope). The problem I have with PEP 572 is that it proposes *breaking that otherwise universal pattern* - instead of having assignment expressions in comprehensions implicitly declare the name as local in the nested comprehension scope, it instead has them: 1. implicitly declare the name as global or as nonlocal in the comprehension (or else raise an exception), depending on the nature of the parent scope where the comprehension is used 2. in the nonlocal reference case, amend the symbol table analysis to act like there was a preceding "if 0:\n for NAME in ():\n pass" in the parent scope (so the compiler knows which outer function scope to target) The rationale being given for why that is OK is: 1. "Everyone" thinks comprehensions are just a for loop (even though that hasn't been true for the better part of a decade, and was never true for generator expressions) 2. If comprehensions are just a for loop, then assignment expressions inside them should be local to the containing scope 3. Therefore the implicit declarations required to tie everything together and allow folks to continue with an incorrect understanding of how comprehensions work aren't really implicit - they're explicit in the inaccurate expansion of the construct! Can you imagine the reaction if anyone other than Guido or Tim was attempting to argue for a change to the language that only makes sense if we grant a completely inaccurate understanding of how a particular language construct works as being a credible starting point? Because that's how this comprehension scoping argument feels to me: Proposal author: "If the language worked in a way other than it does, then this proposal would make perfect sense." Proposal reviewer: "Yes, but it doesn't work that way, it works this way. We deliberately changed it because the old way caused problems." Proposal author: "Ah, but it *used* to work that way, and a lot of people still think it works that way, and we can get the compiler to jump through hoops to pretend it still works that way, except for the parts of the new way that we want to keep." Proposal reviewer: "..." Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The implicit "nonlocal NAME" is only because there is an equally implicit function boundary. Why is there a function boundary marked by square brackets? It's not saying "def" or "lambda", which obviously create functions. It's a 'for' loop wrapped inside a list display. What part of that says "hey, I'm a nested function"? So if there's an implicit function, with implicit declaration of a magical parameter called ".0", why can't it have an equally implicit declaration that "spam" is a nonlocal name? ChrisA

On 24 June 2018 at 16:53, Chris Angelico <rosuav@gmail.com> wrote:
Nothing - that's why I refer to them as implicitly nested scopes (vs the explicitly nested scopes in functions and lambda expressions, where the scope is introduced via keyword). However, there's still a major behavioural tell at runtime that they're running in a nested scope: the iteration variables don't leak. (There are other tells as well, but not ones that most folks are likely to encounter)
Because comprehensions don't do that for their iteration variables, because assignment expressions don't do that when used in explicitly nested scopes, because the required implicit scope declarations are context dependent, and because even such gyrations still can't hide the existence of the comprehension's implicitly nested scope when dealing with classes and the two-argument form of exec(). Since the implicitly nested scopes can't be hidden, it makes far more sense to me to just admit that they're there, and provide explicit syntax for cases where folks decide they really do want name bindings to leak out of that scope (whether those name bindings are assignment expression targets or the iteration variables themselves). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 24.06.2018 9:53, Chris Angelico wrote:
My 2c. An expression is intuitively thought to be self-contained i.e. without side effects. if I write `a=b+1`, I'm not expecting it to do anything except assigning `a'. Expressions with side effects has long since proven to be problematic because of the implicit (thus hard to see and track) links they create (and because the result depends on the order of evaluation). Moreover, Python's other design elements have been consistently discouraging expressions with side effects, too (e.g. mutator methods intentionally return None instead of the new value, making them useless in expressions), so the proposition is in direct conflict with the language's design. Assignment expressions are a grey area: they carry the full implications of expressions with side effects described above, but their side effect is their only effect, i.e. they are explicit and prominent about the "evil" they do.
-- Regards, Ivan

On Sun, Jun 24, 2018 at 05:24:12PM +0300, Ivan Pozdeev via Python-Dev wrote:
a = d.pop(1) a = d.setdefault(key, 0) chars_written = file.write(text)
If you're going to take a hard-core functional approach to side-effects, I think you are using the wrong language. Nearly everything in Python *could* have side-effects (even if usually it won't). Even your own example of "b+1" (depending on what b.__add__ does).
I don't think that's the reason why mutator methods return None. They return None rather than self to avoid confusion over whether they return a copy or not. https://docs.python.org/3/faq/design.html#why-doesn-t-list-sort-return-the-s...
so the proposition is in direct conflict with the language's design.
Python is full of operations with side-effects. Besides, they're not quite useless: (alist.append() or alist) is functionally equivalent to alist.append returning self. Just a bit more verbose. Methods (and functions) all return a value, even if that value is None, so they can be used in expressions. If Guido wanted Pascal style procedures, which cannot be used in expressions, we would have them by now :-) -- Steve

A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline assignment, (b) when := is used in a comprehension, set the scope for the target as if the assignment occurred outside any comprehensions. It seems we have more support for (a) than for (b) -- at least Nick and Greg seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his essay on the topic, included as Appendix A ( https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings) does not even mention comprehensions. However, he did post his motivation for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the PEP gives a strong motivation ( https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless, maybe we should compromise and drop (b)? -- --Guido van Rossum (python.org/~guido)

On Sun, Jun 24, 2018 at 09:24:39AM -0700, Guido van Rossum wrote:
I'm not sure who came up with the idea first, but as I remember it, the first mention of this came in a separate thread on Python-Ideas: https://mail.python.org/pipermail/python-ideas/2018-April/049631.html so possibly I'm to blame :-) That thread starts here: https://mail.python.org/pipermail/python-ideas/2018-April/049622.html If I did get the idea from Tim, I don't remember doing so.
I will have more to say about the whole "comprehensions are their own scope" issue later. But I'd like to see Nick's proposed PEP, or at least a draft of it, before making any final decisions. If it came down to it, I'd be happy with the ability to declare an assignment target nonlocal in the comprehension if that's what it takes. What do you think of this syntax? [global|nonlocal] simple_target := expression Inside a comprehension, without a declaration, the target would be sublocal (comprehension scope); that should make Nick happier :-) -- Steve

On Sun, Jun 24, 2018 at 11:50 AM Steven D'Aprano <steve@pearwood.info> wrote:
Actually that post sounds like the OP of that thread (Peter O'Connor) is to blame -- he proposed a similar thing using `=` for the assignment and custom syntax (`from <value>`) to specify the initial value, and it looks like that inspired you.
Agreed, though I assume it's just `given` again.
It's more special syntax. Just taking part (a) of PEP 572 would make most people happy enough. -- --Guido van Rossum (python.org/~guido)

On 25 June 2018 at 09:02, Guido van Rossum <guido@python.org> wrote:
While I still have some TODO notes of my own to resolve before posting it to python-ideas, the examples section at https://github.com/ncoghlan/peps/pull/2/files#diff-7a25ca1769914c1141cb5c63d... already gives a pretty good idea of the differences relative to PEP 572: rebinding existing names is unchanged from PEP 572, but introducing new names requires a bit of "Yes, I really do want to introduce this new name here" repetition. The big difference from previous iterations of the "given" idea is that it doesn't try to completely replace the proposed inline assignments, it just supplements them by providing a way to do inline name *declarations* (which may include declaring targets as global or nonlocal, just as regular function level declarations can). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

[Guido]
I was writing up my observations about simple changes to existing code. Since there's nothing sanely akin to binding non-for-targets possible in comprehensions now, comprehensions were out of scope for that effort (which was limited to staring at existing code already doing bindings). :> However, he did post his motivation for (b) on python-ideas, IIRC a bit
Two things to say about that. First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say) p = None # to establish the intended scope of `p` while any(<nonlocal p> # split across lines just for readability n % p == 0 for p in small_primes): n //= p It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour. Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can become visible when bindings can be embedded: def f(): y = -1 ys = [y for _ in range(y := 5)] print(y, ys) Here `range(y := 5)` is executed in f's scope. Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced. It's incoherent without detailed knowledge of the implementation. def g(): y = -1 ys = [y for y in range(y := 5)] print(y, ys) And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call. That's incoherent in its own way. Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

On Sun, Jun 24, 2018 at 12:03 PM Tim Peters <tim.peters@gmail.com> wrote:
Hm, that's more special syntax. The nice bit about (b) as currently specified is that it adds no syntax -- it adds a scope rule, but (as IIRC Steven has convincingly argued) few people care about those. Python's scope rules, when fully specified, are intricate to the point of being arcane (e.g. for class scopes) but all that has a purpose -- to make them so DWIM ("Do what I Mean") that in practice you almost never have to worry about them, *especially* when reading non-obfuscated code (and also when writing, except for a few well-known patterns).
That code should have the same meaning regardless of whether we accept (b) or not -- there is only one `y`, in f's scope. I don't mind if we have to add more words to the PEP's scope rules to make this explicit, though I doubt it -- the existing weirdness (in the comprehension spec) about the "outermost iterable" being evaluated in the surrounding scope specifies this. I wouldn't call it incoherent -- I think what I said about scope rules above applies here, it just does what you expect.
And under the (b)-less proposal, `g` would interpret `y for y` as both referring to a new variable created just for the comprehension, and `y := 5` as referring to g's scope. Again I don't think it needs extra words in the spec. And the end user docs might just say "don't do that" (with a link to the reference manual's rule about the "outermost iterable"). Even if in the end we did find a case where we'd have to write an explicit rule to make what happens here a consequence of the spec rather than the implementation, that doesn't count as an argument for keeping (b) to me. In favor of (b) we have a few examples (see https://www.python.org/dev/peps/pep-0572/#scope-of-the-target) that require it, and more that you described on python-ideas (and also the motivating use case from the thread that Steven dug up, starting here: https://mail.python.org/pipermail/python-ideas/2018-April/049622.html). A "neutral" argument about (b) is that despite the "horrified" reactions that Nick saw, in practice it's going to confuse very few people (again, due to my point about Python's scope rules). I'd wager that the people who might be most horrified about it would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement, and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3. -- --Guido van Rossum (python.org/~guido)

[Guido]
Hm, that's more special syntax.
Of course - I'm anticipating that the PEP will be changed to throw out useful assignment expressions in comprehensions, but I still want a way to "export" comprehension for-targets at times ;-)
You and Steven and i appear to be on the same page here - but it's in a book nobody else seems to own :-( To me it's just screamingly obvious that total = 0 cumsums = [total := total + value for value in data] "should do" what it obviously intends to do - and that the only thing stopping that is a bass-ackwards focus on what most trivially falls out of the current implementation. ... def f():
Remove "y = -1" and - voila! - we have the dreaded "parent local scoping" Nick finds so baffling to explain (or so he claims). That is, "has exactly the same scope in the comprehension as in the parent block, and will create a local in the latter if the name is otherwise unknown in the parent" comes with assignment expressions, regardless of whether _all_ such targets "leak" (the current PEP) or only targets in the expression defining the iterable of the outermost `for` (the PEP without leaking assignment expressions in comprehensions). As to whether it "does what you expect", no, not really! In a world where _all_ binding targets in a comprehension are claimed to be local to the comprehension, I _expect_ that `y := 5` appearing inside the listcomp means `y` is local to the listcomp. "Oh - unless the binding appears in the expression defining the iterable of the outermost `for`" comes from Mars. Not that it really matters much, but (b) provides consistent semantics in these cases. No need to search Mars for weird exceptions ;-) ...
I also doubt it will generally confuse people in practice (to the contrary, I expect they _will_ be confused if things like the cumulative sums example blow up with UnboundLocalError). But I still don't get the source of the "horror". Assignment expression semantics are wholly consistent with ordinary nested lexical scoping, with or without (b). The only difference is in the scopes picked for assignment expression target names (except for those appearing in the expression defining the iterable yadda yadda yadda).

On 25 June 2018 at 09:25, Guido van Rossum <guido@python.org> wrote:
FWIW, the most cryptic parent local scoping related exception I've been able to devise so far still exhibits PEP 572's desired "Omitting the comprehension scope entirely would give you the same name lookup behaviour" semantics: >>> def outer(x=1): ... def middle(): ... return [x := x + i for i in range(10)] ... return middle() ... >>> outer() Traceback (most recent call last): ... NameError: free variable 'x' referenced before assignment in enclosing scope It isn't the parent local scoping, or even the assignment expression, that's at fault there, since you'd get exactly the same exception for: def outer(x=1): def middle(): x = x +1 return x return middle() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2018 at 22:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
Oops, I didn't mean to say "exactly the same exception" here, as the whole reason I'd settled on this example as the most cryptic one I'd found so far was the fact that the doubly nested version *doesn't* give you the same exception as the singly nested version: the version without the comprehension throws UnboundLocalError instead. However, the resolution is the same either way: either 'x' has to be declared as 'nonlocal x' in 'middle', or else it has to be passed in to 'middle' as a parameter. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2018 at 13:24, Nick Coghlan <ncoghlan@gmail.com> wrote:
At the level of "what my intuition says" the result is the same in both cases - "it throws an exception". I have no intuition on *which* exception would be raised and would experiment (or look up the details) if I cared.
Once someone told me that's what I needed, it's sufficiently obvious that I'm fine with that. If no-one was able to tell me what to do, I'd simply rewrite the code to be less obfuscated :-) I've probably explained my intuition enough here. If we debate any further I'll just end up knowing what's going on, destroying my value as an "uninformed user" :-) Paul

On 25 June 2018 at 13:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again offering an "intuition" based response: 1. That definition of outer() is very complicated, I don't *expect* to understand it without checking the details. So the NameError is simply "hmm, wonder what triggered that?" not "OMG that's not what I'd expect!" :-) 2. Given that your version with no assignment expression or comprehension exhibits the same behaviour, I'm not sure what your argument is here anyway... Paul

On 6/24/2018 7:25 PM, Guido van Rossum wrote:
I'd wager that the people who might be most horrified about it
the (b) scoping rule change
would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement,
I might not be one of those 'most horrified' by (b), but I increasingly don't like it, and I was at best -0 on the comprehension scope change. To me, iteration variable assignment in the current scope is a non-problem. So to me the change was mostly useless churn. Little benefit, little harm. And not worth fighting when others saw a benefit. However, having made the change to nested scopes, I think we should stick with them. Or repeal them. (I believe there is another way to isolate iteration names -- see below). To me, (b) amounts to half repealing the nested scope change, making comprehensions half-fowl, half-fish chimeras.
and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3.
That I pretty much am, I think. In Python 2, comprehensions (the fish) were, at least in effect, expanded in-line to a normal for loop. Generator expressions (the fowls) were different. They were, and still are, expanded into a temporary generator function whose return value is dropped back into the original namespace. Python 3 turned comprehensions (with 2 news varieties thereof) into fowls also, temporary functions whose return value is dropped back in the original namespace. The result is that a list comprehension is equivalent to list(generator_ expression), even though, for efficiency, it is not implemented that way. (To me, this unification is more a benefit than name hiding.) (b) proposes to add extra hidden code in and around the temporary function to partly undo the isolation. list comprehensions would no longer be equivalent to list(generator_expression), unless generator_expressions got the same treatment, in which case they would no longer be equivalent to calling the obvious generator function. Breaking either equivalence might break someone's code. --- How loop variables might be isolated without a nested scope: After a comprehension is parsed, so that names become strings, rename the loop variables to something otherwise illegal. For instance, i could become '<i>', just as lambda becomes '<lambda>' as the name of the resulting function. Expand the comprehension as in Python 2, except for deleting the loop names along with the temporary result name. Assignment expressions within a comprehension would become assignment expressions within the for loop expansion and would automatically add or replace values in the namespace containing the comprehension. In other words, I am suggesting that if we want name expressions in comprehensions to act as they would in Python 2, then we should consider reverting to an altered version of the Python 2 expansion. --- In any case, I think (b) should be a separate PEP linked to a PEP for (a). The decision for (a) could be reject (making (b) moot), accept with (b), or accept unconditionally (but still consider (b)). -- Terry Jan Reedy

On 26.06.2018 1:58, Greg Ewing wrote:
Chris Angelico wrote:
The wheel turns round and round, and the same spokes come up.
Unless there's a repository of prior discussion no-one can be bothered to gather scraps from around the Net. Wikis solve this by all the discussion being in one place, and even they struggle is there were multiple.
A discussion long past, and a discussion yet to come.
There are no beginnings or endings in the Wheel of Python...
-- Regards, Ivan

On Tue, Jun 26, 2018 at 05:42:43AM +1000, Chris Angelico wrote:
So..... sublocal scopes, like in the earliest versions of PEP 572?
The wheel turns round and round, and the same spokes come up.
It isn't as if comprehensions (and generator expressions) run in a proper separate scope. It is more half-and-half, sometimes it is seperate, sometimes it isn't: py> def show_subscope(): ... a, b = 1, 2 ... print("Comprehension scope, Part A") ... print(next(locals() for x in (1,))) ... print("Comprehension scope, Part B") ... print(next(obj for obj in (locals(),))) ... py> show_subscope() Comprehension scope, Part A {'x': 1, '.0': <tuple_iterator object at 0xb799cf8c>} Comprehension scope, Part B {'b': 2, 'a': 1} Comprehensions already run partly in the surrounding scope. I tried to take a survey of people on the Python-List mailing list, so see what their expectations of comprehension scope was. Disappointingly, not many people responded, but those who did, invariably think in terms of comprehensions running inside their enclosing scope, like any other expression: https://mail.python.org/pipermail/python-list/2018-June/734838.html (Please excuse the doubled-up posts, some misconfigured news server is periodically sending duplicate posts.) (Oh and ignore my comment about Python 2 -- I was thinking of something else.) Given the code shown: def test(): a = 1 b = 2 result = [value for key, value in locals().items()] return result nobody suggested that the result ought to be the empty list, which is what you should get if the comprehension ran in its own scope. Instead, they all expected some variation of [1, 2], which is what you would get if the comprehension ran in the enclosing scope. A decade or more since generator expressions started running in their own half-local-half-sublocal scope, people still think of scoping in terms of LEGB and don't think of comprehensions as running in their own scope *except* to the very limited degree that sometimes they are either surprised or pleased that "the loop variable doesn't leak". For example: http://nbviewer.jupyter.org/github/rasbt/python_reference/blob/master/tutori... doesn't mention comprehensions until the very end, almost in passing, and doesn't describe them as a separate scope at all. Rather, they are described as using closures "to prevent the for-loop variable to cut [sic] into the global namespace." This doesn't mention comprehension subscope at all: https://www.python-course.eu/python3_global_vs_local_variables.php Even the official documentation doesn't explicitly state that comprehensions are a separate scope: https://docs.python.org/3/reference/executionmodel.html#resolution-of-names rather leaving it to an after thought, to mention in passing almost as if it were an implementation-dependent accident, that comprehensions cannot see variables defined in any surrounding class scope. Aside from the loop variable (which PEP 572 will not change!) I see no evidence that the average non-core developer Python programmer considers comprehensions as a separate scope, or wants them to be a separate scope. Regardless of comprehensions being implicitly wrapped in a function or not, the average developer doesn't want the loop variable to "leak", and that's as far as their consideration has needed to go until now. But when pressed to explicitly consider the scope inside a comprehension, the evidence I have seen is that they consider it the same as the local scope surrounding it. Which is not wrong, as can be seen from the example above. Unlike the loop variable, I don't believe that assignment-expression bindings quote-unquote "leaking" from comprehensions will come as a surprise. On the contrary -- given that Nick et al have gone to great lengths to ensure that as a first approximation, comprehensions are equivalent to a simple for-loop running in the current scope: result = [expr for a in seq] # is almost the same as result = [] for a in seq: result.append(expr) I expect that people will be surprised if explicit, non-loop variable assignments *don't* occur in the current scope. If all that takes to implement is something like an implicit "nonlocal", that's hardly worse than the implicit functions already used. -- Steve

On 27 June 2018 at 07:54, Steven D'Aprano <steve@pearwood.info> wrote:
But test() returns [1, 2]. So does that say (as you claim above) that "the comprehension ran in the enclosing scope"? Doesn't it just say that the outermost iterable runs in the enclosing scope? So everybody expected the actual behaviour? (Disclaimer: in my response, I said that I had no clear expectation, which I stand by - locals() exposes implementation details that I don't normally feel that I need to know - but certainly the majority of respondents expected 1 and 2 to appear). On the other hand,
and I bet no-one would have expected that if you'd posed that question (I certainly wouldn't). Although some might have said [('v', 'a')]. I suspect some would have expected a and b to appear there too, but that's just a guess... So yes, it's likely that people would have found the current behaviour unexpected in respect of locals(). But I imagine most people only care about the effective results when referencing variables, and
i.e., thanks to scope nesting, you can still reference locals from the enclosing scope. The problem is that := allows you to *change* values in a scope, and at that point you need to know *which* scope. So to that extent, the locals() question is important. However, I still suspect that most people would answer that they would like := to assign values *as if* they were in the enclosing scope, which is not really something that I think people would express in answer to a question about locals(). This can be achieved with an implicit "nonlocal" (and some extra shenanigans if the enclosing scope has a nonlocal or global declaration itself). Which, AIUI, is what the current proposal tries to do. IMO, the big question over the current PEP 572 proposal is whether it goes too far in the direction of "do what I mean". Superficially, the semantics are pretty clearly "what people would expect", and indeed that's been the whole focus recently to capture and satisfy *expected* behaviour. But there are edge cases (there always are when you work from messy real-world requirements rather than nice clean mathematical definitions ;-)) and the question is essentially whether any of those are bad enough to be an issue. I'm starting to feel that they aren't, and I'm moving towards a cautious +0 (or even +1) on the proposal. Paul

On Wed, Jun 27, 2018 at 5:30 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Yes - because the *outermost iterable* runs in the enclosing scope. But suppose you worded it like this: def test(): a = 1 b = 2 vars = {key: locals()[key] for key in locals()} return vars What would your intuition say? Should this be equivalent to dict(locals()) ? ChrisA

On 27 June 2018 at 08:52, Chris Angelico <rosuav@gmail.com> wrote:
As I said on python-list, my intuition doesn't apply to locals() - I simply have no idea what I'd "expect" from that code, other than a request to go back and write it more clearly :-) *After* staring at it for a while and trying to interpret it base on the detailed knowledge I've gained from this thread, I'd say it does nothing remotely useful, and if you want dict(locals()) you should write it. (No, test() is not equivalent, because the two instances of locals() refer to different scopes, but I can't imagine why I'd ever need to know that outside of solving artificial puzzles like this). Paul

On Wed, Jun 27, 2018 at 7:19 PM, Steven D'Aprano <steve@pearwood.info> wrote:
It gets funnier with nested loops. Or scarier. I've lost the ability to distinguish those two. def test(): spam = 1 ham = 2 vars = [key1+key2 for key1 in locals() for key2 in locals()] return vars Wanna guess what that's gonna return? ChrisA

On Wed, Jun 27, 2018 at 08:00:20AM -0400, Eric V. Smith wrote:
You can just copy and paste the function into the interactive interpreter and run it :-) But where's the fun in that? The point of the exercise is to learn first hand just how complicated it is to try to predict the *current* scope behaviour of comprehensions. Without the ability to perform assignment inside them, aside from the loop variable, we've managed to avoid thinking too much about this until now. It also demonstrates the unrealisticness of treating comprehensions as a separate scope -- they're hybrid scope, with parts of the comprehension running in the surrounding local scope, and parts running in an sublocal scope. Earlier in this thread, Nick tried to justify the idea that comprehensions run in their own scope, no matter how people think of them -- but that's an over-simplification, as Chris' example above shows. Parts of the comprehension do in fact behave exactly as the naive model would suggest (even if Nick is right that other parts don't). As complicated and hairy as the above example is, (1) it is a pretty weird thing to do, so most of us will almost never need to consider it; and (2) backwards compatibility requires that we live with it now (at least unless we introduce a __future__ import). If we can't simplify the scope of comprehensions, we can at least simplify the parts that actually matters. What matters are the loop variables (already guaranteed to be sublocal and not "leak" out of the comprehension) and the behaviour of assignment expressions (open to discussion). Broadly speaking, there are two positions we can take: 1. Let the current implementation of comprehensions as an implicit hidden function drive the functionality; that means we duplicate the hairiness of the locals() behaviour seen above, although it won't be obvious at first glance. What this means in practice is that assignments will go to different scopes depending on *where* they are in the comprehension: [ expr for x in iter1 for y in iter2 if cond ...] [ BBBBBB for x in AAAAAA for y in BBBBBB if BBBBBB ...] Assignments in the section marked "AAAAAA" will be in the local scope; assignments in the BBBBBB sections will be in the sublocal scope. That's not too bad, up to the point you try to assign to the same name in AAAAAA and BBBBBB. And then you are likely to get confusing hard to debug UnboundLocalErrors. 2. Or we can keep the current behaviour for locals and the loop variables, but we can keep assignment expressions simple by ensuring they always bind to the enclosing scope. Compared to the complexity of the above, we have the relatively straight forward: [ AAAAAA for x in AAAAAA for y in AAAAAA if AAAAAA ...] The loop variables continue to be hidden away in the invisible, implicit comprehension function, where they can't leak out, while explicit assignments to variables (using := or given or however it is spelled) will always go into the surrounding local scope, like they do in every other expression. Does it matter that the implementation of this requires an implicit nonlocal declaration for each assignment? No more than it matters that comprehensions themselves require an implicit function. And what we get out of this is simpler semantics at the Python level: - Unless previous declared global, assignment expressions always bind to the current scope, even if they're inside a comprehension; - and we don't have to deal with the oddity that different bits of a comprehension run in different scopes (unless we go out of our way to use locals()); merely using assignment expressions will just work consistently and simply, and loop variables will still be confined to the comprehension as they are now. -- Steve

On 27.06.2018 16:49, Steven D'Aprano wrote:
This isn't as messy as you make it sound if you remember that the outermost iterable is evaluated only once at the start and all the others -- each iteration. Anyone using comprehensions has to know this fact. The very readable syntax also makes it rather straightforward (though admittedly requiring some hand-tracing) to figure out what is evaluated after what.
-- Regards, Ivan

On Wed, Jun 27, 2018 at 05:52:16PM +0300, Ivan Pozdeev via Python-Dev wrote:
The question isn't *how often* they are evaluated, or how many loops you have, but *what scope* they are evaluated in. Even in a single loop comprehension, parts of it are evaluated in the local scope and parts are evaluated in an implicit sublocal scope. The overlap between the two is the trap, if you try to assign to the same variable in the loop header and then update it in the loop body. Not to mention the inconsistency that some assignments are accessible from the surrounding code: [expr for a in (x := func(), ...) ] print(x) # works while the most useful ones, those in the body, will be locked up in an implicit sublocal scope where they are unreachable from outside of the comprehension: [x := something ... for a in sequence ] print(x) # fails -- Steve

On 28.06.2018 1:42, Steven D'Aprano wrote:
All expressions inside the comprehension other than the initial iterable have access to the loop variables generated by the previous parts. So they are necessarily evaluated in the internal scope for that to be possible. Since this is too an essential semantics that one has to know to use the construct sensibly, I kinda assumed you could make that connection... E.g.: [(x*y) for x in range(5) if x%2 for y in range(x,5) if not (x+y)%2] A B C D E C and D have access to the current x; E and A to both x and y.
-- Regards, Ivan

On 28.06.2018 2:31, Ivan Pozdeev via Python-Dev wrote:
This means btw that users cannot rely on there being a single internal scope, or a scope at all. The public guarantee is only the access to the loop variables (and, with the PEP, additional variables from assignments), of the current iteration, generated by the previous parts.
-- Regards, Ivan

On 28.06.2018 2:45, Ivan Pozdeev via Python-Dev wrote:
The expressions in the comprehension just somehow automagically determine which of the variables are internal and which are local. How they do that is an implementation detail. And the PEP doesn't need to (and probably shouldn't) make guarantees here other than where the variables from expressions are promised to be accessible.
-- Regards, Ivan

Ivan Pozdeev via Python-Dev wrote:
That fact alone doesn't imply anthing about the *scopes* in which those iterators are evaluated, however. Currently the only situation where the scoping makes a difference is a generator expression that isn't immediately used, and you can get a long way into your Python career without ever encountering that case. -- Greg

On Wed, Jun 27, 2018 at 08:30:00AM +0100, Paul Moore wrote:
I think I was careful enough to only say that this was the same result you would get *if* the comprehension ran in the outer scope. Not to specifically say it *did* run in the outer scope. (If I slipped up anywhere, sorry.) I did say that the comprehension runs *partly* in the surrounding scope, and the example shows that the local namespace in the "... in iterable" part is not the same as the (sub)local namespace in the "expr for x in ..." part. *Parts* of the comprehension run in the surrounding scope, and parts of it run in an implicit sublocal scope inside a hidden function, giving us a quite complicated semantics for "comprehension scope": [expression for a in first_sequence for b in second ... ] |------sublocal-----|----local-----|------sublocal------| Try fitting *that* in the LEGB (+class) acronym :-) This becomes quite relevant once we include assignment expressions. To make the point that this is not specific to := but applies equally to Nick's "given" syntax as well, I'm going to use his syntax: result = [a for a in (x given x = expensive_function(), x+1, 2*x, x**3)] Here, the assignment to x runs in the local part. I can simulate that right now, using locals, but only outside of a function due to CPython's namespace optimization inside functions. (For simplicity, I'm just going to replace the call to "expensive_function" with just a constant.) py> del x py> [a for a in (locals().__setitem__('x', 2) or x, x+1, 2*x, x**3)] [2, 3, 4, 8] py> x 2 This confirms that the first sequence part of the comprehension runs in the surrounding local scope. So far so good. What if we move that assignment one level deep? Unfortunately, I can no longer use locals for this simulation, due to a peculiarity of the CPython function implementation. But replacing the call to locals() with globals() does the trick: del x # simulate [b*a for b in (1,) for a in (x given x = 2, x+1, 2*x, x**3)] [b*a for b in (1,) for a in (globals().__setitem__('x', 2) or x, x+1, 2*x, x**3)] That also works. But the problem comes if the user tries to assign to x in both the local and a sublocal section: # no simulation here, sorry [b*a for b in (x given x = 2, x**2) for a in (x given x = x + 1, x**3)] That looks like it should work. You're assigning to the same x in two parts of the same expression. Where's the problem? But given the "implicit function" implementation of comprehensions, I expect that this ought to raise an UnboundLocalError. The local scope part is okay: # needs a fixed-width font for best results [b*a for b in (x given x = 2, x**2) for a in (x given x = x + 1, x**3)] ..............|-----local part----|.....|--------sublocal part--------| but the sublocal part defines x as a sublocal variable, shadowing the surrounding local x, then tries to get a value for that sublocal x before it is defined. If we had assignment expressions before generator expressions and comprehensions, I don't think this would have been the behaviour we desired. (We might, I guess, accept it as an acceptable cost of the implicit function implementation. But we surely wouldn't argue for this complicated scoping behaviour as a good thing in and of itself.) In any case, we can work around this (at some cost of clarity and unobviousness) by changing the name of the variable. Not a big burden when the variable is a single character x: [b*a for b in (x given x = 2, x**2) for a in (y given y = x + 1, y**3)] but if x is a more descriptive name, that becomes more annoying. Nevermind, it is a way around this. Or we could Just Make It Work by treating the entire comprehension as the same scope for assignment expressions. (I stress, not for the loop variable.) Instead of having to remember which bits of the comprehension run in which scope, we have a conceptually much simpler rule: - comprehensions are expressions, and assignments inside them bind to the enclosing local scope, just like other expressions: - except for the loop variables, which are intentionally encapsulated inside the comprehension and don't "leak". The *implementation details* of how that works are not conceptually relevant. We may or may not want to advertise the fact that comprehensions use an implicit hidden function to do the encapsulation, and implicit hidden nonlocal to undo the effects of that hidden function. Or whatever implementation we happen to use.
So everybody expected the actual behaviour?
More or less, if we ignore a few misapprehensions about how locals works.
I suspect not. To be honest, I didn't even think of asking that question until after I had asked the first.
That is my belief as well. But that was intentionally not the question I was asking. I was interested in seeing whether people thought of comprehensions as a separate scope, or part of the enclosing scope. -- Steve

On Mon, Jun 25, 2018 at 8:37 PM, Terry Reedy <tjreedy@udel.edu> wrote:
[...]
this email have ever deliberately taken advantage of the limited Python 3 scope in comprehensions and generator expressions to use what would otherwise be a conflicting local variable name? I appreciate that the scope limitation can sidestep accidental naming errors, which is a good thing. Unfortunately, unless we anticipate Python 4 (or whatever) also making for loops have an implicit scope, I am left wondering whether it's not too large a price to pay. After all, special cases aren't special enough to break the rules, and unless the language is headed towards implicit scope for all uses of "for" one could argue that the scope limitation is a special case too far. It certainly threatens to be yet another confusion for learners, and while that isn't the only consideration, it should be given due weight.

On Mon, Jun 25, 2018 at 2:16 PM Steve Holden <steve@holdenweb.com> wrote:
No, never, but the opposite has bitten me in production code (as I related several months back, a class-level variable was being used on the lhs of a comprehension and that failed when it was run in Py3). The caveat is that our code base is Py2+Py3, so we have the mindset that comprehension variables always leak.

On Mon, Jun 25, 2018 at 5:14 PM Steve Holden <steve@holdenweb.com> wrote:
I have never once *deliberately* utilized the Python 3 local scoping in comprehensions. There were a few times in Python 2 where I made an error of overwriting a surrounding name by using it in a comprehension, and probably Python 3 has saved me from that a handful of times. Where I ever made such an error, it was with names like 'x' and 'i' and 'n'. They are useful for quick use, but "more important" variables always get more distinctive names anyway. Had the Python 2 behavior remained, I would have been very little inconvenienced; and I suppose comprehensions would have been slightly less "magic" (but less functional-programming).
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

Terry Reedy wrote:
This doesn't change the situation conceptually, though, since the question arises of why not do the same mangling for names assigned within the comprehension. A decision still needs to be made about whether we *want* semantics that leak some things but not others. -- Greg

On Tue, Jun 26, 2018 at 10:54:12AM +1200, Greg Ewing wrote:
A decision still needs to be made about whether we *want* semantics that leak some things but not others.
My sense (or bias, if you prefer) is that the answer to that depends on how you word the question. If you talk about "leaking", or give examples with trivial 1-character names that look all too easy to accidentally clobber, people will say "No": # Given this: x = 999 [(x := i)*x for i in (1, 2)] # should print(x) afterwards result in 4? but if you show a useful example that doesn't look like an accident waiting to happen, but a deliberate feature: # Given this: previous = 0 [previous + (previous := i) for i in (1, 2, 3)] # what value would you expect previous to have # at the completion of the loop? they'll be more receptive to the idea. (If they're not opposed to assignment expressions at all.) Avoiding leading questions is *hard*, and I believe that in general people don't know what they want until they've got it. I say that from considering all the times I've made a radical about face, features which I was *sure* would be awful actually turned out to be not awful at all -- augmented assignment, for instance. -- Steve

[This is my one response today] On Mon, Jun 25, 2018 at 12:40 PM Terry Reedy <tjreedy@udel.edu> wrote:
Fair enough, and by itself this might not have been enough reason to make the change. But see below.
That depends on how you see it -- to me (b) just means that there's an implicit nonlocal[1] to make the assignment have the (desirable) side-effect. The key thing to consider here is whether that side-effect is in fact desirable. For me, the side-effect of the comprehension's loop control variable was never desirable -- it was just an implementation detail leaking out. (And that's different from leaking a regular for-loop's control variable -- since we have 'break' (and 'else') there are some legitimate use cases. But comprehensions try to be expressions, and here the side effect is at best useless and at worst a nasty surprise.)
Right, and this consistency convinced me that the change was worth it. I just really like to be able to say "[... for ...]" is equivalent to "list(... for ...)", and similar for set and dict.
(b) proposes to add extra hidden code in and around the temporary function to partly undo the isolation.
But it just adds a nonlocal declaration. There's always some hidden code ('def' and 'return' at the very least).
Ah, there's the rub! I should probably apologize for not clarifying my terminology more. In the context of PEP 572, when I say "comprehensions" I include generators! PEP 572 states this explicitly ( https://github.com/python/peps/blame/master/pep-0572.rst#L201-L202). Certainly PEP 572 intends to add that implicit nonlocal to both comprehensions and generator expressions. (I just got really tired of writing that phrase over and over, and at some point I forgot that this is only a parenthetical remark added in the PEP's latest revision, and not conventional terminology -- alas. :-) Part (b) of PEP 572 does several things of things to *retain* consistency: - The target of := lives in the same scope regardless of whether it occurs in a comprehension, a generator expression, or just in some other expression. - When it occurs in a comprehension or generator expression, the scope is the same regardless of whether it occurs in the "outermost iterable" or not. If we didn't have (b) the target would live in the comprehension/genexpr scope if it occurred in a comprehension/genexp but outside its "outermost iterable", and in the surrounding scope otherwise.
Possibly this is based on a misunderstanding of my use of "comprehensions". Also, since your trick can only be used for list/set/dict comprehensions, but not for generator expressions (at least I assume you don't want it there) it would actually *reduce* consistency between list/set/dict comprehensions and generator expressions.
For me personally, (b) makes the PEP more consistent, so I'm not in favor of breaking up the PEP. But we can certainly break up the discussion -- that's why I started using the labels (a) and (b). ---------- [1] Sometimes it's an implicit global instead of an implicit nonlocal -- when there's already a global for the same variable in the target scope. -- --Guido van Rossum (python.org/~guido)

On 6/26/2018 10:36 PM, Guido van Rossum wrote:
[This is my one response today]
Thank you for clearly presenting how you see 'comprehension', 'generator expression' and by implication 'equivalent code'. The latter can either be a definition or an explanation. The difference is subtle but real, and, I believe, the essence of the disagreement over iteration variables. If the code equivalent to a comprehension is its definition, like a macro expansion, then survival of the iteration variable is to be expected. If the equivalent code is an explanation of the *result* of evaluating a *self-contained expression*, then leakage is easily seen a wart, just as leakage of temporaries from any other expression would be. My interpretation of what you say below is that you always wanted, for instance, [i*i for i in iterable] == [j*j for j in iterable] to be true, and saw the leakage making this not quite true as a wart. In other words, within comprehensions (including generator expressions) iterations names should be regarded as dummy placeholders and not part of the value. If this is correct, the list comprehension syntax could have been [\0 * \0 for \0 in iterable] with \1, \2, ... used as needed. (I am using the regex back-reference notation in a way similar to the use of str.format forward reference notation.) I will stop here for now, as it is 1:30 am for me. Terry
-- Terry Jan Reedy

Guido van Rossum wrote:
Greg seem to be +0 or better for (a)
Actually, I'm closer to -1 on (a) as well. I don't like := as a way of getting assignment in an expression. The only thing I would give a non-negative rating is some form of "where" or "given". Brief summary of reasons for disliking ":=": * Cryptic use of punctuation * Too much overlap in functionality with "=" * Asymmetry between first and subsequent uses of the bound value * Makes expressions cluttered and hard to read to my eyes -- Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
+1 to this; ‘:=’ doesn't convey the meaning well. Python's syntax typically eschews cryptic punctuation in faviour of existing words that convey an appropriate meaning, and I agree with Greg that would be a better way to get this effect. -- \ “Self-respect: The secure feeling that no one, as yet, is | `\ suspicious.” —Henry L. Mencken | _o__) | Ben Finney

Not giving a vote, as I'm just a lurker, but: Le 25/06/2018 à 01:30, Greg Ewing a écrit :
This resonates with me for a yet different reason: expressing the feature with a new operator makes it feel very important and fundamental, so that beginners would feel compelled to learn it early, and old-timers tend to have a strong gut reaction to it. Using merely a keyword makes it less prominent. Baptiste

On 25.06.2018 2:30, Greg Ewing wrote:
"as" was suggested even before is became a keyword in `with'. ( if (re.match(regex,line) as m) is not None: <do smth> ) The only objective objection I've heard is it's already used in `import' and `with' -- but that's perfectly refutable.
-- Regards, Ivan

On 26.06.2018 1:34, Greg Ewing wrote:
What do you mean by "asymmetry"? The fact that the first time around, it's the expression and after that, the variable? If that, it's not a "problem". The whole idea is to assign the result of a subexpression to something. If you force any assignments to be outside, it won't be a subexpression anymore, but effectively a separate statement -- if not syntactically, then visually at least -- both of which are the things the feature's purpose is to avoid. If you seek to force assignments outside, you should've rather suggested inline code blocks e.g. like anonymous methods in C# ( { a=foo(); b=bar(); return a+b;} ). Using this assigned result elsewhere in the same expression (akin to regex backreferences) is not a part of the basic idea actually. It depends on the evaluation order (and whether something is evaluated at all), so I doubt it should even be allowed -- but even if it is, it's a side benefit at best. -- Regards, Ivan

Ivan Pozdeev via Python-Dev wrote:
Using this assigned result elsewhere in the same expression (akin to regex backreferences) is not a part of the basic idea actually.
If that's true, then the proposal has mutated into something that has *no* overlap whatsoever with the use case that started this whole discussion, which was about binding a temporary variable in a comprehension, for use *within* the comprehension.
It depends on the evaluation order (and whether something is evaluated at all),
Which to my mind is yet another reason not to like ":=". -- Greg

Why is this discussion talking about comprehensions at all? Is there a decent use case for using assignments in comprehensions (as opposed to language lawyering or deliberate obfuscation)? Regards Antoine. On Thu, 28 Jun 2018 01:25:14 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On Wed, Jun 27, 2018 at 03:41:23PM +0200, Antoine Pitrou wrote:
Yes. The *very first* motivating example for this proposal came from a comprehension. I think it is both unfortunate and inevitable that the discussion bogged down in comprehension-hell. Unfortunate because I don't think that the most compelling use-cases involve comprehensions at all. But inevitable because *comprehensions are the hard case*, thanks to the (justifiable!) decision to implement them as implicit hidden functions. In my opinion, the really two BIG wins for assignment expressions are while loops and cascades of if... blocks. Tim Peters has also given a couple of good examples of mathematical code that would benefit strongly from this feature. Going back a few months now, they were the examples that tipped me over from the opinion "Oh, just re-write the comprehension as a loop" to the opinion "You know, I think this feature actually is useful... and as a bonus, you can keep using the comprehension" But that requires that we get the comprehension scoping right. Not just leave it as an unspecified implementation detail. -- Steve

Steven D'Aprano wrote:
I think the unfortunateness started when we crossed over from talking about binding a temporary name for use *within* a comprehension or expression, to binding a name for use *outside* the comprehension or expression where it's bound. As long as it's for internal use, whether it's in a comprehension or not isn't an issue.
Well, I remain profoundly unconvinced that writing comprehensions with side effects is ever a good idea, and Tim's examples did nothing to change that. -- Greg

On 28.06.2018 2:31, Greg Ewing wrote:
I've shown in <05f368c2-3cd2-d7e0-9f91-27afb40d5b35@mail.mipt.ru> (27 Jun 2018 17:07:24 +0300) that assignment expressions are fine in most use cases without any changes to scoping whatsoever. So, as Guido suggested in <CAP7+vJ+xBAT4ZvAo4B7qSqxnnpj8jJ1VZ-Le7EwT8=n-UTjE3Q@mail.gmail.com> (26 Jun 2018 19:36:14 -0700), the scoping matter can be split into a separate PEP and discussion.
-- Regards, Ivan

Le 28/06/2018 à 01:31, Greg Ewing a écrit :
Comprehensions with side effects feel scary indeed. But I could see myself using some variant of the "cumsum" example (for scientific work at the command prompt):
x=0; [x:=x+i for i in range(5)]
Here the side effects are irrelevant, the "x" variable won't be reused. But it needs to be initialized at the start of the comprehension. I would happily get rid of the side-effects, but then what would be a non-cryptic alternative to the above example? Baptiste

On 6/28/2018 8:05 AM, Baptiste Carvello wrote:
Creating an unneeded list with a comprehension purely for side effects is considered a bad idea by many. x = 0 for i in range(5): x += i
Here the side effects are irrelevant, the "x" variable won't be reused.
If we ignore the side effect on x, the above is equivalent to 'pass' ;-) Perhaps you meant x = 0 cum = [x:=x+i for i in range(5)] which is equivalent to x, cum = 0, [] for i in range(5): x += i; cum.append(x)
The above as likely intended can also be written import itertools as it cum = list(it.accumulate(range(5))) We have two good existing alternatives to the proposed innovation. -- Terry Jan Reedy

On 27.06.2018 16:25, Greg Ewing wrote:
I don't know what and where "started" it (AFAIK the idea has been around for years) but for me, the primary use case for an assignment expression is to be able to "catch" a value into a variable in places where I can't put an assignment statement in, like the infamous `if re.match() is not None'.
which was about binding a temporary variable in a comprehension, for use *within* the comprehension.
Then I can't understand all the current fuss about scoping. AFAICS, it's already like I described in https://mail.python.org/pipermail/python-dev/2018-June/154067.html : the outermost iterable is evaluated in the local scope while others in the internal one: In [13]: [(l,i) for l in list(locals())[:5] for i in locals()] Out[13]: [('__name__', 'l'), ('__name__', '.0'), ('__builtin__', 'l'), ('__builtin__', '.0'), ('__builtin__', 'i'), ('__builtins__', 'l'), ('__builtins__', '.0'), ('__builtins__', 'i'), ('_ih', 'l'), ('_ih', '.0'), ('_ih', 'i'), ('_oh', 'l'), ('_oh', '.0'), ('_oh', 'i')] (note that `i' is bound after the first evaluation of internal `locals()' btw, as to be expected) If the "temporary variables" are for use inside the comprehension only, the assignment expression needs to bind in the current scope like the regular assignment statement, no changes are needed!
-- Regards, Ivan

Ivan Pozdeev via Python-Dev wrote:
This seems to be one of only about two uses for assignment expressions that gets regularly brought up. The other is the loop-and-a-half, which is already adequately addressed by iterators. So maybe instead of introducing an out-of-control sledgehammer in the form of ":=", we could think about addressing this particular case. Like maybe adding an "as" clause to if-statements: if pattern.match(s) as m: do_something_with(m) -- Greg

On 28.06.2018 2:44, Greg Ewing wrote:
I've skimmed for the origins of "as" (which I remember seeing maybe even before Py3 was a thing) and found this excellent analysis of modern languages which is too a part of the PEP 572 discussion: https://mail.python.org/pipermail/python-ideas/2018-May/050920.html It basically concludes that most recently-created languages do not have assignment expressions; they rather allow assignment statement(s?) before the tested expression in block statements (only if/while is mentioned. `for' is not applicable because its exit condition in Python is always the iterable's exhaustion, there's nothing in it that could be used as a variable). It, however, doesn't say anything about constructs that are not block statements but are equivalent to them, like the ternary operator. (In comprehensions, filter conditions are the bits equivalent to if/while statements.) -- Regards, Ivan

On Thu, Jun 28, 2018 at 11:29 AM, Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
Now read this response. https://mail.python.org/pipermail/python-ideas/2018-May/050938.html ChrisA

On Mon, Jun 25, 2018 at 11:30:24AM +1200, Greg Ewing wrote:
":=" is the second most common syntax used for assignment in common programming languages, not just Pascal. Even modern languages like Go use it. If that's "cryptic", what word would you use to describe @decorator syntax? *wink* Honestly Greg, can you put your hand on your heart and swear that if you came across "name := expression" in source code you wouldn't be able to hazard a guess as the meaning of the := operator?
* Too much overlap in functionality with "="
If you are willing to consider a non-negative rating under the "given/willing"spelling, presumably the "overlap in functionality" isn't that important. (Otherwise it would be an argument against the feature *regardless of spelling*.) So why should it be an argument against the := spelling?
* Asymmetry between first and subsequent uses of the bound value
I don't know what this means.
* Makes expressions cluttered and hard to read to my eyes
And Nick's more verbose "given" proposal makes expressions less cluttered? result = process(first=(spam := ham or eggs), second=spam*5) result = process(first=(spam given spam = ham or eggs), second=spam*5) The := spelling has three syntactic elements: the target name, the := operator itself, and the expression being assigned. The syntax you are willing to consider has five elements: an arbitrarily complex return expression, the keyword "given", the target name, the = operator, and the expression being assigned. It isn't rational to say that adding extra complexity and more syntactic elements *reduces* clutter. At the minimum, Nick's syntax requires: - an extra keyword ("given" or "where") - a repetitive, redundant, repeated use of the target name just to save one : character. That adds, not subtracts, clutter. Aside from the asymmetry issue (which I don't understand) it seems that most of your arguments against := apply equally, or even more strongly, to the "expr given name = expr" version. I know matters of taste are deeply subjective, but we ought to distinguish between *subjective* and *irrational* reasons for disliking proposed features, and try to resist the irrational ones: "We should change the spelling of set.add to set.append, as that will remove the troublesome double-letter, and reduce typing." *wink* -- Steve

On 25 June 2018 at 02:24, Guido van Rossum <guido@python.org> wrote:
Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or 'NAME := EXPR'?" bothers me a bit, but it's the implementation implications of parent local scoping that I fear will create a semantic tar pit we can't get out of later.
Unfortunately, I think the key rationale for (b) is that if you *don't* do something along those lines, then there's a different strange scoping discrepancy that arises between the non-comprehension forms of container displays and the comprehension forms: (NAME := EXPR,) # Binds a local tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local [NAME := EXPR] # Binds a local [NAME := EXPR for __ in range(1)] # Doesn't bind a local list(NAME := EXPR for __ in range(1)) # Doesn't bind a local {NAME := EXPR} # Binds a local {NAME := EXPR for __ in range(1)} # Doesn't bind a local set(NAME := EXPR for __ in range(1)) # Doesn't bind a local {NAME := EXPR : EXPR2} # Binds a local {NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local Those scoping inconsistencies aren't *new*, but provoking them currently involves either class scopes, or messing about with locals(). The one virtue that choosing this particular set of discrepancies has is that the explanation for why they happen is the same as the explanation for how the iteration variable gets hidden from the containing scope: because "(EXPR for ....)" et al create an implicitly nested scope, and that nested scope behaves the same way as an explicitly nested scope as far as name binding and name resolution is concerned. Parent local scoping tries to mitigate the surface inconsistency by changing how write semantics are defined for implicitly nested scopes, but that comes at the cost of making those semantics inconsistent with explicitly nested scopes and with the read semantics of implicitly nested scopes. The early iterations of PEP 572 tried to duck this whole realm of potential semantic inconsistencies by introducing sublocal scoping instead, such that the scoping for assignment expression targets would be unusual, but they'd be consistently unusual regardless of where they appeared, and their quirks would clearly be the result of how assignment expressions were defined, rather than only showing up in how they interacted with other scoping design decisions made years ago. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2018 at 12:44, Nick Coghlan <ncoghlan@gmail.com> wrote:
I've been mostly ignoring this proposal for a while now, so I'm going to respond here in the context of someone with a bit of an idea of the underlying complexities, but otherwise coming at it as a new proposal.
None of those "discrepancies" bother me in the slightest, when taken in isolation as you present them here. I suspect you could lead me through a chain of logic that left me understanding why you describe them as discrepancies, but without that explanation, I'm fine with all of them. I'd also say that they seem contrived (not just in the use of artificial names, but also in the sense that I'm not sure why I'd want to use this *pattern*) so I'd happily say "well, don't do that then" if things started behaving non-intuitively.
And to reinforce my point above, I already consider putting significant code in class scopes, or using locals() to be techniques that should only be used sparingly and with a clear understanding of the subtleties. I'm sure you could say "but the examples above would be much more common" in response to which I'd like to see real use cases that behave non-intuitively in the way you're concerned about.
But that's precisely why I find the behaviour intuitive - the nested scope is the *reason* things behave this way, not some sort of easily-overlooked way the "problem" can be explained away.
Those last two paragraphs made my head explode, as far as I can see by virtue of the fact that they try to over-analyze the fairly simple intuition I have that "there's a nested scope involved". Disclaimer: I may well have got a *lot* of subtleties wrong here, and it's quite likely that my impressions don't stand up to the harsh reality of how the implementation works. But my comments are on the basis of my *intuition*, whether that's right or wrong. And if the reality violates my intuition, it's *other* constructs that I find non-intuitive, not this one. (I'm perfectly happy to concede that it's not possible to avoid *any* non-intuitive behaviour - all I'm trying to say is that my intuition doesn't balk at this one, unlike yours). Paul

On 6/25/2018 8:25 AM, Paul Moore wrote:
Of course not, in local scopes where is it not executed. But it would, in the nested function where the assignment *is* executed. Ditto for all of the following.
Me neither. I pretty much agree with the rest of what Paul said. If we don't want comprehensions to execute in a nested scope, then we should not create one. See my response to Guido for a possible alternative. -- Terry Jan Reedy

On 25.06.2018 14:44, Nick Coghlan wrote:
I've got an idea about this. The fact is, assignments don't make much sense in an arbitrary part of a comprehension: `for' variables are assigned every iteration, so when the result is returned, only the final value will be seen. (And if you need a value every iteration, just go the explicit way and add it to the returned tuple.) Contrary to that, the "feeder" expression is only evaluated once at the start -- there, assignments do make sense. Effectively, it's equivalent to an additional line: seq = range(calculate_b() as bottom, calculate_t() as top) results = [calculate_r(bottom,r,top) for r in seq] So, I suggest to evaluate the "feeder" expression in a local scope but expressions that are evaluated every iteration in a private scope. -- Regards, Ivan

[This is my one reply in this thread today. I am trying to limit the amount of time I spend to avoid another overheated escalation.] On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Others have remarked this too, but it really bother me that you are focusing so much on the implementation of parent local scoping rather than on the "intuitive" behavior which is super easy to explain -- especially to someone who isn't all that familiar (or interested) with the implicit scope created for the loop control variable(s). According to Steven (who noticed that this is barely mentioned in most tutorials about comprehensions) that is most people, however very few of them read python-dev. It's not that much work for the compiler, since it just needs to do a little bit of (new) static analysis and then it can generate the bytecode to manipulate closure(s). The runtime proper doesn't need any new implementation effort. The fact that sometimes a closure must be introduced where no explicit initialization exists is irrelevant to the runtime -- this only affects the static analysis, at runtime it's no different than if the explicit initialization was inside `if 0`. Unfortunately, I think the key rationale for (b) is that if you
In what sense are they not new? This syntax doesn't exist yet.
Yeah, but most people don't think much about that explanation. You left out another discrepancy, which is more likely to hit people in the face: according to your doctrine, := used in the "outermost iterable" would create a local in the containing scope, since that's where the outermost iterable is evaluated. So in this example a = [x := i+1 for i in range(y := 2)] the scope of x would be the implicit function (i.e. it wouldn't leak) while the scope of y would be the same as that of a. (And there's an even more cryptic example, where the same name is assigned in both places.) This is another detail of comprehensions that I assume tutorials (rightly, IMO) gloss over because it's so rarely relevant. But it would make the explanation of how := works in comprehensions more cumbersome: you'd have to draw attention to the outermost iterable, otherwise "inline assignment in comprehensions has the same scope as the comprehension's loop control variable(s)" would lead one to believe that y's scope above would also be that of the implicit function.
Nobody thinks about write semantics though -- it's simply not the right abstraction to use here, you've introduced it because that's how *you* think about this.
There was also another variant in some iteration or PEP 572, after sublocal scopes were already eliminated -- a change to comprehensions that would evaluate the innermost iterable in the implicit function. This would make the explanation of inline assignment in comprehensions consistent again (they were always local to the comprehension in that iteration of the PEP), at the cost of a backward incompatibility that was ultimately withdrawn. -- --Guido van Rossum (python.org/~guido)

On 26 June 2018 at 02:27, Guido van Rossum <guido@python.org> wrote:
[This is my one reply in this thread today. I am trying to limit the amount of time I spend to avoid another overheated escalation.]
Aye, I'm trying to do the same, and deliberately spending some evenings entirely offline is helping with that :)
One of the things I prize about Python's current code generator is how many of the constructs can be formulated as simple content-and-context independent boilerplate removal, which is why parent local scoping (as currently defined in PEP 572) bothers me: rather than being a new primitive in its own right, the PEP instead makes the notion of "an assignment expression in a comprehension or generator expression" a construct that can't readily decomposed into lower level building blocks the way that both assignment expressions on their own and comprehensions and generator expressions on their own can be. Instead, completely new language semantics arise from the interaction between two otherwise independent features. Even changes as complicated as PEP 343's with statement, PEP 380's yield from, and PEP 492's native coroutines all include examples of how they could be written *without* the benefit of the new syntax. By contrast, PEP 572's parent local scoping can't currently be defined that way. Instead, to explain how the code generator is going to be expected to handle comprehensions, you have to take the current comprehension semantics and add two new loops to link up the bound names correctly:: [item := x for x in items] becomes: # Each bound name gets declared as local in the parent scope if 0: for item in (): pass def _list_comp(_outermost_iter): # Each bound name gets declared as: # - nonlocal if outer scope is a function scope # - global item if outer scope is a module scope # - an error, otherwise _result = [] for x in _outermost_iter: _result.append(x) return _result _expr_result = _list_comp(items) This is why my objections would be reduced significantly if the PEP explicitly admitted that it was defining a new kind of scoping semantics, and actually made those semantics available as an explicit "parentlocal NAME" declaration (behind a "from __future__ import parent_locals" guard), such that the translation of the above example to an explicitly nested scope could just be the visually straightforward:: def _list_comp(_outermost_iter): parentlocal item _result = [] for x in _outermost_iter: item = x _result.append(x) return _result _expr_result = _list_comp(items) That splits up the learning process for anyone trying to really understand how this particular aspect of Python's code generation works into two distinct pieces: - "assignment expressions inside comprehensions and generator expressions use parent local scoping" - "parent local scoping works <the way that PEP 572 defines it>" If the PEP did that, we could likely even make parent locals work sensibly for classes by saying that "parent local" for a method definition in a class body refers to the closure namespace where we already stash __class__ references for the benefit of zero-arg super (this would also be a far more robust way of defining private class variables than name mangling is able to offer). Having parent locals available as a language level concept (rather than solely as an interaction between assignment expressions and implicitly nested scopes) also gets us to a point where context-independent code thunks that work both at module level and inside another function can be built as nested functions which declare all their working variables as parentlocal (you still need to define the thunks inline in the scope you want them to affect, since this isn't dynamic scoping, but when describing the code, you don't need to say "as a module level function define it this way, as a nested function define it that way"). An explicit "parentlocal NAME" concept at the PEP 572 layer would also change the nature of the draft "given" proposal from competing with PEP 572, to instead being a follow-up proposal that focused on providing control of target name declarations in lambda expressions, comprehensions, and generator expressions such that: - (lambda arg: value := arg given parentlocal value) # Exports "value" to parent scope - any(x for x in items given parentlocal x) # Exports "x" to parent scope - [y for x in data if (y := f(x)) given y] # *Avoids* exporting "y" to parent scope With parent local scoping in the mix the proposed "given" syntax could also dispense with initialiser and type hinting support entirely and instead only allow: - "... given NAME" (always local, no matter the default scoping) - "... given parentlocal NAME" (always parent local, declaring if necessary) - "... given nonlocal NAME" (always nonlocal, error if not declared in outer scope) - "... given global NAME" (always global, no matter how nested the current scope is) - "... given (TARGET1, TARGET2, ...)" (declaring multiple assignment targets) If you want an initialiser or a type hint, then you'd use parentlocal semantics. If you want to keep names local (e.g. to avoid exporting them as part of a module's public API) then you can do that, too.
The simplest way to illustrate the scope distinction today is with "len(locals())": >>> [len(locals()) for i in range(1)] [2] >>> [len(locals())] [7] But essentially nobody ever does that, so the distinction doesn't currently matter. By contrast, where assignment expressions bind their targets matters a *lot*, so PEP 572 makes the existing scoping oddities a lot more significant.
Yeah, the fact it deals with this problem nicely is one aspect of the parent local scoping that I find genuinely attractive.
The truth of the last part of that paragraph means that the only way for the first part of it to be true is to decide that my way of thinking is *so* unusual that nobody else in the 10 years that Python 3 has worked the way it does now has used the language reference, the source code, the disassembler, or the debugger to formulate a similar mental model of how they expect comprehensions and generator expressions to behave. I'll grant that I may be unusual in thinking about comprehensions and generator expressions the way I do, and I definitely accept that most folks simply don't think about the subtleties of how they handle scopes in the first place, but I *don't* accept the assertion that I'm unique in thinking about them that way. There are simply too many edge cases in their current runtime behaviour where the "Aha!" moment at the end of a debugging effort is going to be the realisation that they're implemented as an implicitly nested scope, and we've had a decade of Python 3 use where folks prone towards writing overly clever comprehensions have been in a position to independently make that discovery.
The early iterations of PEP 572 tried to duck this whole realm of potential semantic inconsistencies by introducing sublocal scoping
Yeah, the current "given" draft has an open question around the idea of having the presence of a "given" clause pull the outermost iterable evaluation inside the nested scope. It still doesn't really solve the problem, though, so I think I'd actually consider PEP-572-with-explicit-parent-local-scoping-support the version of assignment expressions that most cleanly handles the interaction with comprehension scopes without making that interaction rely on opaque magic (instead, it would be relying on an implicit target scope declaration, the same as any other name binding - the only unusual aspect is that the implicit declaration would be "parentlocal NAME" rather than the more typical local variable declaration). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
I'm not sure that's possible. If I understand correctly, part of the definition of "parent local" is that "parent" refers to the nearest enclosing *non-comprehension* scope, to give the expected result for nested comprehensions. If that's so, then it's impossible to fully decouple its definition from comprehensions. -- Greg

On 27 June 2018 at 23:38, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I'm OK with a target scope declaration construct having lexical-scope-dependent behaviour - exactly what "nonlocal NAME" will do depends on both the nature of the current scope, and on which names are declared as local in which outer scopes, and that's also implicitly the case for all name lookups. However, PEP 572 in its current form takes the position "parent local scoping is sufficiently useful to make it a required pre-requisite for adding assignment expressions, but not useful enough to expose as a new scope declaration primitive", and I've come to the view that it really is the "A+B=MAGIC!" aspect of the current proposal that bothers me, whereas "A+B implies C for <pragmatic reasons>" doesn't bother me any more than the implicit non-local references introduced as part of the original lexical scoping changes bother me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jun 27, 2018 at 9:27 AM Paul Moore <p.f.moore@gmail.com> wrote:
So, my interpretation is that it will behave like this? x = 2 y = [x := 3 for i in range(1)] print(x) 3 def f(): x = 4 y = [x := 5 for i in range(1)] print(x) f() 5 class C: x = 6 y = [x := 7 for i in range(1)] print(x) C() 6 print(x) 7

[Nick Coghlan]
Of course the PEP doesn't take that position at all: it doesn't even contain the term "parent local scoping". That's your term, which nobody else uses unless they're replying to you ;-) What the PEP does say: """ an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as "comprehensions") binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope. """ It's a small collection of plainly stated rules for specifying the intended semantics. If you want to claim that this _is_ "useful enough to expose as a new scope declaration primitive", it's really on you to present use cases to justify that claim. I'd present some for you, but I don't have any (I don't care that "by hand" conversion of nested comprehensions to workalike Python nested functions may require a bit of thought to establish the intended scope of assignment expression target names - all of which is easily doable without adding any new statements). I don't _expect_ that other good use cases exist. The gimmick's purpose is to make code that visually _appears_ to belong to a block act as if embedded assignments do occur in that block. If there's an explicitly nested function, that fundamental motivation no longer applies.

Nick Coghlan wrote:
Yes, but my point is that having an explicit "parentlocal" scope declaration doesn't help to make anything more orthogonal, because there's no way it can have *exactly* the same effect as a comprehension's implicit parent-local scoping. In other words, taking a comprehension and manually expanding it into a function with parentlocal declarations wouldn't give you something exactly equivalent to the original. If that's the purpose of having an explicit parentlocal, then it fails at that purpose. If that's *not* the purpose, then I'm not really sure what the purpose is, because I can't think of a situation where I'd choose to use parentlocal instead of nonlocal with an explicit assignment in the outer scope. Except maybe for the class-scope situation, which seems like an extremely obscure reason to introduce a whole new scoping concept with its own keyword. -- Greg

[Nick Coghlan]
[Greg Ewing]
Yes, but my point is that having an explicit "parentlocal" scope
Sure it can - but I already explained that. This is the analogy to "nonlocal" Nick is making: neither "nonlocal" nor "parentlocal" tell you which scope a declared name _does_ belong to. Instead they both say "it's not this scope" and specify algorithms you can follow to determine the scope to which the name does belong. "parentlocal" isn't an accurate name because the owning scope may not be the parent block at all, and it may even be a synonym for "global". I think "by hand" translations of nested comprehensions into nested functions are clearer _without_ the "parentlocal" invention.- then you have to be explicit about what the context requires. Nick hates that because it isn't uniform. I like that because I don't want to pretend a non-uniform thing is uniform ;-) The only real use case here is synthesizing nested functions to implement comprehensions/genexps. In other words, taking a comprehension and manually expanding
You can add (a sufficient number of) parentlocal declarations to get the precise intended semantics. Then again, that can also be done today (without the "parentlocal" invention).
For example, if the name is declared "global" in the outer scope, you'll get a compile-time error if you try to declare it "nonlocal" in the contained scope. "parentlocal" adjusts its meaning accordingly, becoming a synonym for "global" in that specific case.

On Wed, Jun 27, 2018 at 07:29:52PM -0500, Tim Peters wrote: [...]
"Parentlocal" is only a thing if we buy into the paradigm that inside comprehensions is a separate "local". And *that* is only true under two circumstances: - if you are utterly immersed in the implementation of comprehensions as invisible, implicit functions; - or if you start from the premise that comprehensions ought to encapsulate not just the loop variable, but anything else as well. But experimenting with locals() inside comprehensions shows that comprehension-scope *isn't* a well-defined thing. It already bleeds out of the comprehension, and so would some (but only some!) assignment expressions. Instead, if we start from the premise that comprehensions (like any other expression) run in the current scope, then there is no need to invent a term "parentlocal". There's just the usual LEGB scopes, plus class (which people usually forget). With no sublocal scopes (a term we never even had prior to this PEP) assignments inside the comprehension are no more special than assignments inside any other expression. They bind in the current scope, same as always, and keep the sensible identity that these two expressions are exactly equivalent in their visible semantics: [x:=0, x:=1, x:=2] [x:=i for i in (0, 1, 2)] including assignments. What about the loop variable? They ARE special, which is completely justified by the Zen: Although practicality beats purity. We can take a series of ever-more-detailed explanations, starting from the highest "bird's eye" view and gradually dropping further into the murky details of the implementation when, and if, required: - assignment within comprehensions is no different from assignment in any other expression, it occurs in the local scope; - loop variables? they're a special case, for good reason, and are encapsulated inside the comprehension; - how? they're hidden in an implicit, invisible scope, same as .0 the implicit, invisible iterator object; - oh, you didn't know about the .0 variable? well forget about it, it's an undocumented implementation detail, just like the invisible, implicit function used by comprehensions; - oh, you didn't know about that either? read the source code. Only the first two levels of explanation are part of Python the language. The rest is CPython implementation. -- Steve

Thank you all. I will accept the PEP as is. I am happy to accept *clarification* updates to the PEP if people care to submit them as PRs to the peps repo (https://github.com/python/peps), and that could even (to some extent) include summaries of discussion we've had, or outright rejected ideas. But even without any of those I think the PEP is very clear so I will not wait very long (maybe a week). On Mon, Jul 2, 2018 at 8:38 AM Steven D'Aprano <steve@pearwood.info> wrote:
-- --Guido van Rossum (python.org/~guido)

On 02/07/2018 19:19, Guido van Rossum wrote:
It's late to raise this, but what exactly are the objections to the syntax expr -> name # or variations such as expr => name instead of name := expr The PEP mentions that this syntax does not have a problem that "as" does, but does not list any downsides of it. It conforms to left-to-right evaluation, where name:=expr does not. It (I would argue) reduces the asymmetry of the first use of a sub-expression in cases such as [ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ] vs [ ( (y := f(x))**2, y**3, y**4) for x in iterable ] because the first "y" is closer to the way it is used, viz "**2". Regards Rob Cliffe

[Rob Cliffe]
It's late to raise this,
By months, yes ;-)
but what exactly are the objections to the syntax
expr -> name # or variations such as expr => name
instead of
name := expr
The PEP mentions that this syntax does not have a problem that "as"
does, but does not list any downsides of it.
My guess: it probably strikes too many as "excessive novelty", These are assignment expressions. Python's assignment statements put the target at the left. Why change that? ":=" is used for assignment in many more other languages than "->" is. Why fight that?
It conforms to left-to-right evaluation, where name:=expr does not.
? Only "expr" is evaluated, so left-to-right seems irrelevant here. The "evaluation" of a simple name as a binding target is a no-op (no code is generated). If you really do see this as a wart anyway, then it's positively a Good Thing that it's exactly the same "wart" as in Python's assignment statements.
It (I would argue) reduces the asymmetry of the first use of a sub-expression in cases such as
[ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ]
vs
[ ( (y := f(x))**2, y**3, y**4) for x in iterable ]
because the first "y" is closer to the way it is used, viz "**2".
The first form reads a little better to me too, but not a lot better. The problem I have with variations of this example on its own (which comes up surprisingly often with minor changes) is that it's clearer spelled today via [(y**2, y**3, y**4) for y in map(f, iterable)] Spelling that with either form of assignment expression reads significantly worse than that to my eyes But more importantly, it's expected that assignment expressions will be used _most_ often to make some common `if` and `while` patterns briefer. Hardly all. Our eyes are already trained to "look at the far right end" for the value being tested, and, e.g., while data := sock.recv(): preserves that. Especially in code that doesn't _always_ use assignment expressions in such contexts (which is likely all significant blobs of code), it would be visually jarring to have to "sometimes look in the middle instead" to extract the important part of: while sock.recv() -> data: "Look to the left for the name, look to the right for the value" is the rule for assignment statements, assignment expressions, and `for` loop targets. But there's no "QED" here because this isn't a deductive science. The final answer is "because that's what Guido liked best" ;-)

2018-07-02 20:19 GMT+02:00 Guido van Rossum <guido@python.org>:
Thank you all. I will accept the PEP as is. (...)
I see more and more articles ("on the Internet") saying that Guido van Rossum already accepted the PEP. Is the PEP already accepted or will be accepted? Right now, https://www.python.org/dev/peps/pep-0572/ status is "Draft". Victor

On Tue, Jul 3, 2018 at 4:25 PM Victor Stinner <vstinner@redhat.com> wrote:
That's a rather philosophical question. I clearly said "I will" not "I might". And if you're asking whether it's likely that I'll change my mind, no. I would like help with updates to the PEP to summarize some of the discussions and rejected proposals. And I am giving myself a week to "cool off". But I am muting threads that bring up objections that I've heard before (e.g. "design principles"). So those articles aren't wrong. Your patience is appreciated. -- --Guido van Rossum (python.org/~guido)

On Jul 2, 2018, at 8:34 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Guido has decided — and despite my concerns, I’m going to enjoy my new loop-and-a half construct:-) But a comment on this:
Sure — and I don’t think that’s confusing. However, generator expressions ( why don’t we call them generator comprehensions?) are a different story, as they may be run at some arbitrary time in the future. This hasn’t been an issue (except for the loop variable, which has been addressed) because: 1) Much of the time, the gen_ex is run right away, in-line. 2) There aren’t many ways to manipulate the local namespace in a gen_ex. With assignment expressions, it will be much easier to manipulate the local namespace, so there is room for some real confusion here. So a real local namespace gen_exp (and comprehensions, for consistency) would be nice. However, that ship has pretty much sailed. Will it end up being a common problem? Probably not, because (a) is still the case, and := will be used infrequently, and hopefully with unlikely to clash names. And as for all the other languages that have assignment expressions? Do they have constructs like generator expressions? -CHB

[Chris Barker]
However, generator expressions ( why don’t we call them generator
comprehensions?)
Because nobody really liked the "iterator comprehensions" or "accumulator displays" they were variously called at the start. https://mail.python.org/pipermail/python-dev/2003-October/039186.html As that explains, "generator expressions" was an attempt to break away from that "comprehensions" was always a dubious term, carried over from set theory where the term focuses on the optional "if" part rather than the more fundamental iterator or computation parts. At the start, for some (forgotten by me) reason it seemed important to make a distinction between "things like this" that were evaluated at once (list, dict, and set comprehensions) and the new-fangled accumulator displays that got evaluated lazily. But the "generator" in "generator comprehensions" would really be enough all by itself to make that clear enough. So if we had it to do over again I'd sigh and accept "generator comprehensions" anyway. It's been an eternal PITA - and especially in the PEP 572 threads! - to keep typing "comprehensions or generator expressions". Then again, if I had the power of Guido's time machine, I'd go back more, and not use "comprehensions" for anything to begin with. Instead we'd have list, dict, set, and generator twizzlers, affectionately called listwiz, dictwiz, setwiz, and gentwiz by the cool kids :-)

On Mon, Jul 2, 2018 at 11:42 PM, Tim Peters <tim.peters@gmail.com> wrote:
I always wondered about that :-) -- I'd say for most of us that aren't familiar with set theory, it's kind of a "sounds something like putting thing together" word and I just left it at that, and learned what they are.
Well, too late to change the official name, but not too late to start using the term in threads like these -- and other documentation, etc.... I find there is a lot of confusion about the word "generator", as it implies a "thing that generates values on the fly" (like, say the range() object. But then, in Python, a generator is something that gets crated by a generator function, and CAN be an "thing (iterator) that generates things on the fly", but can also be a more generic coroutine, and can be used in nifty ways that really have nothing to do with generating a bunch of value. (like pytest fixtures, for example) So we have generators, iterators, and iterables, and generators can be iterators, but aren't always, and any number of iterators can generate values on the fly, and .... so it's all a bit of a mess to explain to a newbie. Then again, if I had the power of Guido's time machine, I'd go back more,
I'd like that! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 7/3/2018 2:42 AM, Tim Peters wrote:
Amen. I cannot make 'comprehension' in this context comprehensible without some linguistic twisting.
I learned the set notion, such as {n^2: n in N; 1 <= n < 100, n even} # math {n*n for n in range(1,100) if not n%2} # python as 'set builder' notation. If we had followed the math precedent, instead of <other computer language>, we would have set builders, list builders, dict builders, and generator builders. I half seriously think we should consider this for 3.8 for the benefit of future Python programmers as well as ourselves. Comprehensions that can contain assignment expressions are a slightly new thing. -- Terry Jan Reedy

Terry Reedy wrote:
I was intending to suggest something like that back when comprehensions were first being discussed, but people raced ahead and adopted the term "comprehension" before I got the chance. "List builder" and "dict builder" make a lot of sense, but "generator builder" not so much -- it *is* a generator, not something that builds a generator. In fact it doesn't build anything in the sense that the others do. So maybe "generator expression" is the best we could have done. -- Greg

On Wed, Jul 04, 2018 at 11:13:20AM +1200, Greg Ewing wrote:
But [expr for x in seq] is a list, just as (expr for ...) is a generator. If you don't believe me, try it: py> type([x for x in (1,)]) <class 'list'> py> type(x for x in (1,)) <class 'generator'> So I think the similarity is complete. Further, if we think of "list builder" as an abbreviation of "list builder syntax", we have: - list builder syntax is syntax which returns a list; - dict builder syntax is syntax which returns a dict; - set builder syntax is syntax which returns a set; - generator builder syntax is syntax which returns a generator. Of course, there are other ways to build lists, such as calling the constructor, or using a list display ("list literal", except it isn't always a literal). But they're not *builder syntax* :-) In hindsight, I think "spam builder (syntax)" would have been better than the rather mysterious technical word "comprehension" and the not very felicitous term "generator expression". -- Steve

Steven D'Aprano wrote:
You only get a list/dict/set from the first three after you've run the iterators within it, but with a generator expression, you already have a generator before you've run it. That makes it feel different to me. -- Greg

On 7/6/2018 11:20 AM, Brett Cannon wrote:
"Comprehension" was an incomprehensible term to me, when I first heard it. After reading the documentation, I comprehended it, but the term is meaningless. "generator expression" is actually a little more comprehensible, by analogy to a power (electricity) generator... values are generated. But comprehensions would have been more intuitively understood by me, had they been called "list generators" or "dict generators" or "set generators". The difference between comprehensions and generators seems to be one of timing and sequencing: the comprehensions are comprehended as fast as possible, while generators are lazy (unlike those electricity generators, unless you turn them on and off repeatedly). So neither term is very intuitive to "plain old programmers" unless perhaps they happen to know some of the advanced math concepts where the term "comprehension" is claimed to come from (obviously, in spite of a math degree from some years back, I never encountered that term). Focusing just on lists, at this time, "list builder" would be better than "list generator" because using "generator" to replace "comprehension" would be confusing because of the history of the terminology as used/documented currently. A generator seems to be a "lazy list builder". If the names are changed, for a time both terminologies would have to coexist, so it would be critical to use terms not already in use in Python terminology. I would find it an improvement to use terms like "list builder" rather than "comprehension".

On 7/6/2018 11:51 AM, Chris Barker - NOAA Federal via Python-Dev wrote: via phone...
Are we just having fun here?
I floated the idea as a trial balloon to see what response it got.
Or might we actually start using a new naming convention for the-syntax-formerly-known-as-generator-expressions?
Since Guido, the first respondent, did not immediately shoot the idea down, I intend to flesh it out and make it more concrete.
To expand on this: an iterable represents a collection of information objects. Some iterables are concrete collections. Others are more abstract, generating objects on demand in an order that may or may not be significant. Set builders in math define a set in terms of 1 set, 0 to 1 filters, and 1 transform: defined-set = map(tranform, filter(predicate, known-set)). (One could say that there is always a filter, which defaults to passing everything.) Python builders generalize the type 'set' to 'iterable' and the first and second numbers 1 to n and specify a particular nested order of iteration and filtration. For n left as 1, the type generalization is new_iterable = output_type(map(transform, filter(predicate, iter(iterable)). I omitted above the potential dependence of iterable, predicate, and transform pre-existing arguments. For generator builders, define output type 'generator' as a identity function when the input is a generator. The generalization to n > 1 is tricky to specify with functions call, as I did above, because the filtered iterations are nested rather than crossed. Consequently, each iterable and filter (as well as the tranform) could depend not only on values existing before the top call but also the current values of surrounding iterations.
What all 4 results have in common is that they are (mutable) iterables produced from iterators and other inputs with builder syntax. Aside from that, it is true that there are differences between concrete iterables like set, list, and dict versus generator iterators. But to me, this is secondary in this context. One could note that lists and dicts can be subscripted, sets and generators cannot. Or note that dict builders are 'different' because they use the funny dict item ':' notation instead of the (key,value) notation that would make {dict-builder} = dict(dict-builder) true without needing an asterisk. (But then something else would be needed to mark {(k,v) for k,v in it} as a dict rather than set. The use of ':' is quite clever.) -- Terry Jan Reedy

On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy <tjreedy@udel.edu> wrote:
Since Guido, the first respondent, did not immediately shoot the idea down, I intend to flesh it out and make it more concrete.
Maybe I should have shot it down. The term is entrenched in multiple languages by now (e.g. https://en.wikipedia.org/wiki/List_comprehension). Regarding "list builder" one could argue that it would just add more confusion, since there's already an unrelated Builder Pattern ( https://en.wikipedia.org/wiki/Builder_pattern) commonly used in Java. (Though I worry about the presence of a Python example in that Wikipedia page. :-) Also, "generator builder" is not much more expressive than "generator expression", and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions". Maybe it's not too late to start calling the latter "generator comprehensions" so that maybe by the year 2025 we can say "comprehensions" and everyone will understand we mean all four types? FWIW more people should start using "list display" etc. for things like [a, b, c]. -- --Guido van Rossum (python.org/~guido)

On 07.07.2018 2:31, Guido van Rossum wrote:
According to https://en.wikipedia.org/wiki/List_comprehension#History, the term's known from at least 1977 and comes from such influential languages as NPL, Miranda and Haskell. So it's not you to blame for it :-)
Also, "generator builder" is not much more expressive than "generator expression",
"generator builder" is simply incorrect. The GE doesn't "build" generators, it's a generator itself. It's a generator _and_ an expression. What could be a more obvious name? This suggestion looks like coming from someone who hasn't quite grasped generators yet.
and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions".
Since "X comprehensions" are advertised as and intended to be functionally equivalent to `X(generator expression)', I use just "generator expressions" to refer to all. That's accurate because the common part with the distinctive syntax -- which is the thing referred to when addressing them all -- effectively _is_ a generator expression (the syntax differences in the leading term are insignificant), what wraps it is of no concern. So, no new terms are necessary, but someone who cares may add a note to the docs to this effect.

On 07.07.2018 2:58, Ivan Pozdeev via Python-Dev wrote:
Oh, I see. So, "comprehension" is actually the official term for this "distinctive syntax", and the fact that "generator expressions" came to use it is but a coincidence. In that case, we can do a Solomon's decision: mention _both_ that "comprehension" is the official term for the syntax in GE's reference entry, _and_ the fact that "X comprehensions" are effectively wrapped GEs in their reference entries. Then everyone will learn both terminologies and could choose which is more convenient to use.

https://github.com/python/cpython/pull/8145 On 07.07.2018 3:33, Ivan Pozdeev via Python-Dev wrote:

On Sat, Jul 07, 2018 at 02:58:23AM +0300, Ivan Pozdeev via Python-Dev wrote:
Also, "generator builder" is not much more expressive than "generator expression",
I agree with Guido on that comment. The only advantage (such little as it is) is that we can refer to them all using the same terminology: [list | set | dict | generator] builder syntax but given how prevalent the comprehension terminology has become, maybe the best we can hope for is to start using "generator comprehension".
"generator builder" is simply incorrect. The GE doesn't "build" generators, it's a generator itself.
Nobody suggested that it was an thing that you call to build a generator. The name refers to the syntax, not the object. I did refer to it as *generator builder syntax* in my earlier post, and explicitly noted that "list/set/dict/generator builder" was the abbreviated form. But as Guido says, the possible confusion with the Builder design pattern makes this terminology confusing. If we were back in Python 2.2 days when neither the feature nor the terminology "comprehension" were so well established, perhaps we could have gone with "builder" instead, but I think that ship has sailed.
It's a generator _and_ an expression. What could be a more obvious name?
It's not about the obviousness, it is about it being a mouthful to say "comprehension or generator expression" to represent something which is conceptually a single kind of thing. We can use "comprehension" to group "list comprehension or dict comprehension or dict comprehension", but generator expressions are the odd one out.
This suggestion looks like coming from someone who hasn't quite grasped generators yet.
I assure you that both Greg and I understand generators quite well. -- Steve

In response to Guido's reply to my post fleshing out my idea to use 'generator|list|set|dict builder', On 7/6/2018 7:58 PM, Ivan Pozdeev via Python-Dev wrote:
"A list comprehension is a syntactic construct available in some programming languages for creating a list based on existing lists. It follows the form of the mathematical set-builder notation (set comprehension) as distinct from the use of map and filter functions." Mathematicians do not always agree on terminology and notation. I believe that 'set builder notatation' is both older and was and perhaps is more widespread than 'set comprehension'. I have read that it is at least a century old. But https://en.wikipedia.org/wiki/Set-builder_notation does not seem to be the place In any case, Python's comprehensions use an English-syntax version of extended set builder notation. "In Python, the set-builder's braces are replaced with square brackets, parentheses, or curly braces, giving list, generator, and set objects, respectively. Python uses an English-based syntax."
Also, "generator builder" is not much more expressive than "generator expression",
I looked for an alternative 'x' to 'comprehension' such that 'generator|list|set|dict x' works and is specific to the notation. 'Builder' is a reasonable choice. 'expression' is way too general. A 'list expression', for instance, is any expression that evaluated to a list. In this context, I consider that the specific term 'says more' than the general term. On the face of it, a generator expression is an expression that evaluates to a generator. In this sense, 'f(args)', where f is a generator function, is a generator expression. In any case, 'generator comprehension' is an awkward 8 syllable mouthful. -- Terry Jan Reedy

On 7/6/2018 9:01 PM, Terry Reedy wrote:
I'm not sure if your quote above was quoting documentation, or was a suggested quote to add to the documentation, I think the latter, as Google didn't find it. The conflict between the "Builder pattern" and "set-builder notation" can be disambiguated by consistently using the hyphenated "set-builder" (as wikipedia does). And happily, by using wikipedia terms, they would be easily found with explanations outside of python docs as well as (if this is done) inside. We do not need [ typ + ' builder' for typ in ('set', 'list', 'dict', 'generator')] only set-builder. The fencing and : determine the type of the result. We could use [ typ + ' form of set-builder' for typ in ('set', 'list', 'dict', 'generator')] in the few places where the type of the set-builder must be disambiguated, avoiding the need for the compound terms. The result of ( set-builder ) is a generator. We do not need the term "generator expression" or "generator comprehension". Use "generator form of set-builder"... yes, it is one or two syllables longer, but is clearer. A generator can be produced in one of two ways: either a function containing a yield, or a set-builder delimited by parentheses or used as an actual parameter to a function, both of which can be referred to as the "generator form of set-builder". Glenn

On 7/6/2018 7:31 PM, Guido van Rossum wrote:
I glad you did not do so immediately since some of what I worked out since applies to the alternative of consistently using 'comprehension'.
I was not aware of that. I read enough to see that as a relevant conflict.
If one views 'generator expression' as a 2-word phrase, as opposed to a defined compound word, it could mean either 'expression that contains a generator' or 'expression that evaluates to a generator. With the latter meaning, 'generator_func(*args)' is a generator expression. I intended 'generator builder' to be more clearly delimited. So is 'generator comprehension'.
and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions".
That was part of my motivation also.
Maybe it's not too late to start calling the latter "generator comprehensions"
Having proposed a more drastic change, I obviously think it is not too late to change the doc at least for 3.8. (If we do it now, I would consider 3.7 also.) Rename the Expressions section to just 'Comprehensions'. Define 'comprehension' perhaps as "an expression that defines an iterable using Python's adaptation and generalization of extended set builder notation". Comprehensions have to be fenced for use (except for gencomps in function calls) to determine the concrete type of iterable. The key:value syntax that separates dict from set displays separates dict from set comprehensions. Otherwise: Change to 'generator comprehension'. Do whatever to the doc grammar. Adjust glossary entries. If allowed in our reference format, perhaps link to Wikipedia articles on 'set builder notation' and 'list comprehension'. The 8 syllables of 'generator comprehension' is bit long for a compound word. Python uses '<genexpr>' as the pseudo-name for generators. Some people use 'genexp' as an abbreviation (do they pronounce the 'p'?), along with listcomp. 'Gencomp' should do as well.
Definitely. -- Terry Jan Reedy

TL;DR- +1 on advocating the term “generator comprehension” as a synonym for “generator expression” in future documentation and instructional materials. The long version: If we were starting from scratch, maybe it would makes sense to use “ builder” or something else, rather than comprehension. But we’re not starting from scratch. And “list comprehension” has become a pretty universal (or at least common) term in the python world. So we really can’t expect to re-name that. And despite the term having its roots in an esoteric corner of mathematics, I’ve never seen anyone confused by it — it’s just a new word for a new thing to most folks. Which is actually why “generator expression” is a bit confusing — it seems to have more in common with comprehensions than with generators (As in: the things functions with a yield in them produce) In fact, the term “generator” is a bit confusing itself — due to it being a common English word, it seems to refer to the generic idea of “a thing that generates values on the fly”, which, of course it can be in Python. But it actually is a specific type. And you can make a “thing that generates values in the fly” with a regular class that follows the iterator protocol. So re-using the word in “generator expression” just adds a bit to the confusion. (Even though it’s technically correct, it does “make” an actual generator object, doesn’t it? (On a phone, so I can’t check) but that’s kind of an implementation detail. Also: if you google: “python list comprehension” you get a LOT of hits to tutorials, etc. So it is WAY too late to change that name. When you google “python generator expression” the top hit is the python docs, and the next few are about comparing generator expressions and list comprehensions. So I think we could start a new naming scheme for those. ‘cause then we’d get lost of questions about the difference between generator expressions and generator comprehensions ;-) -CHB

On Mon, 9 Jul 2018 at 12:05 Guido van Rossum <guido@python.org> wrote:
I think this is worth a try.
How far do we want to go with this? Update the docs? Update the grammar and/or code? I think the former is probably good enough for now to see if it takes, and if it does then we can talk about updating code to not confuse people.

Definitely docs first. And we should keep references to generator expressions too, if only to explain that they've been renamed. Perhaps someone should try it out in a 3rd party tutorial to see how it goes? I'm CC'ing Raymond Hettinger. On Mon, Jul 9, 2018 at 12:57 PM Brett Cannon <brett@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)

On Mon, Jul 9, 2018 at 3:18 PM, Guido van Rossum <guido@python.org> wrote:
I'm not sure what "trying it out in a tutorial" would look like. I try to be pretty clear about terminology when I teach newbies -- so I don't want to tell anyone this new thing is called a "generator comprehension" if they aren't going to see that term anywhere else. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 13 July 2018 at 15:30, Chris Barker via Python-Dev <python-dev@python.org> wrote:
Nina Zakharenko made the "they're officially called generator expressions, but I find it more helpful to think of them as generator comprehensions" observation in her PyCon 2018 presentation on "Elegant Solutions for Everyday Python Problems": https://www.slideshare.net/nnja/elegant-solutions-for-everyday-python-proble... The article from Ned Batchelder mentioned in that slide is this one, which goes through the history of Raymond originally proposing the notion as generator comprehensions, them getting changed to generator expressions during the PEP 289 discussions, and then asking if it might be worth going back to the originally suggested name: https://nedbatchelder.com/blog/201605/generator_comprehensions.html And then in PEP 572, it was found that being able to group all 4 constructs (list/set/dict comps + generator expressions) under a single term was a genuinely useful shorthand: https://www.python.org/dev/peps/pep-0572/#scope-of-the-target So trying out the terminology in a tutorial context would be to do something similar to what Nina did in her talk: introduce the notion of list/set/dict/generator comprehensions, and then make a side note that the latter are officially referred to as "generator expressions". This wouldn't be the first time that terminology has differed between Python-as-commonly-taught and Python-as-formally-defined, as I've yet to hear anyone refer to container displays outside a language design discussion - everyone else calls them container literals (or, more typically, a literal for the specific builtin container type being discussed). In this case, though, we'd be considering eventually changing the language reference as well, and perhaps even some day the AST node name (from GeneratorExp to GeneratorComp). We wouldn't need to change anything in the grammar definition (since that already shares the comp_for and comp_if syntax definitions between container comprehensions and generator expressions), or the AST node structure (since GeneratorExp already uses a "comprehensions" attribute, the same as the ListComp/SetComp/DictComp nodes). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Thanks Nick, I'll adopt this approach when I update my teaching materials. If I think of it, I"ll post here when I do that -CHB On Sun, Jul 15, 2018 at 12:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

I've pretty much always taught using the "comprehension" term. It makes sense to introduce comprehensions on concrete collections first. Then I typically say something like "This special kind of comprehension is called a 'generator expression'". Usually that's accompanied by a motivation like creating a listcomp of a hundred million items, then showing that creating the generator expression is instantaneous. However, after the initial introduction, I consistently call it a generator expression. Albeit, for the students I teach—data scientists and quants, mostly—there's not a lot of time spent on that being the introduction (by then I'm talking about NumPy and Pandas, and scikit-learn, and Seaborn, abd Statsmodels, and so on. If things are "lazy" it's because they are Dask deferred or Dask DataFrame. On Sun, Jul 15, 2018, 1:02 PM Chris Barker via Python-Dev < python-dev@python.org> wrote:

[Nick Coghlan]>
actually made those semantics available as an explicit
[Greg Ewing] I'm not sure that's possible. If I understand correctly,
So a nested comprehension would declare its assignment expression targets as parentlocal in its synthesized function, and in all the containing synthesized functions generated for containing comprehensions. This appears in some strained ;-) way "natural" only because there is no explicit way to declare something "local" in Python. In just about any other language with closures and nested lexical scopes, comprehensions and generator expressions would have been implemented via nested functions that explicitly declared their "for" target names "local". and nothing else. The only change needed then for PEP 572 (b) semantics would be to declare assignment expression target names local (if their scope wasn't already known) in the closest containing non-synthesized block. None of which really matters. The real question is which semantics are desired.

[Tim]
If the parent has a matching parentlocal declaration for the same name then the original
really refers to the grandparent - and so on.
[Greg]
Ah, I missed that part, sorry -- I withdraw that particular objecttion.
Good! I have another reply that crossed in the mail.
I don't even think it makes "explaining" easier. It doesn't eliminate any corner cases, it just pushes them into the definition of what "parentllocal" means. What it would do is make writing synthetic functions "by hand" to implement comprehensions more uniform, because "parentlocal" would handle the corner cases by itself instead of making the programmer figure out when and where they need to type "nonlocal", "global", and/or cruft to establish a name as local to a block in which the name otherwise does't appear as a binding target. But to the extent that doing such translations by hand is meant to be "educational", it's more educational to learn how to do that stuff yourself.

So IIUC you are okay with the behavior described by the PEP but you want an explicit language feature to specify it? I don't particularly like adding a `parentlocal` statement to the language, because I don't think it'll be generally useful. (We don't have `goto` in the language even though it could be used in the formal specification of `if`, for example. :-) But as a descriptive mechanism to make the PEP's spec clearer I'm fine with it. Let's call it `__parentlocal` for now. It would work a bit like `nonlocal` but also different, since in the normal case (when there's no matching `nonlocal` in the parent scope) it would make the target a local in that scope rather than trying to look for a definition of the target name in surrounding (non-class, non-global) scopes. Also if there's a matching `global` in the parent scope, `__parentlocal` itself changes its meaning to `global`. If you want to push a target through several level of target scopes you can do that by having a `__parentlocal` in each scope that it should push through (this is needed for nested comprehensions, see below). Given that definition of `__parentlocal`, in first approximation the scoping rule proposed by PEP 572 would then be: In comprehensions (which in my use in the PEP 572 discussion includes generator expressions) the targets of inline assignments are automatically endowed with a `__parentlocal` declaration, except inside the "outermost iterable" (since that already runs in the parent scope). There would have to be additional words when comprehensions themselves are nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's intention is that inline assignments anywhere there end up targeting the scope containing the outermost comprehension. But this can all be expressed by adding `__parentlocal` for various variables in various places (including in the "outermost iterable" of inner comprehensions). I'd also like to keep the rule prohibiting use of the same name as a comprehension loop control variable and as an inline assignment target; this rule would also prohibit shenanigans with nested comprehensions (for any set of nested comprehensions, any name that's a loop control variable in any of them cannot be an inline assignment target in any of them). This would also apply to the "outermost iterable". Does this help at all, or did I miss something? --Guido On Wed, Jun 27, 2018 at 5:27 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

[Guido]
If this has to be done ;-) , I suggest removing that last exception. That is, "[all] targets of inline assignments in comprehensions are declared __parentlocal", period, should work fine for (b). In case one appears in the outermost iterable of the outermost comprehension, I believe such declaration is merely semantically redundant, not harmful. Where "redundant" means someone is so familiar with the implementation that the scope implications of "already runs in the parent scope" are immediately clear. For someone muddy about that, it would be a positive help to have the intent clarified by removing the exception. Plus 99% of the point of "parentlocal" seemed to be to allow mindless ("uniform") by-hand translation of nested comprehensions to nested Python functions, and an exception for the outermost iterable would work against that intent.

On 28 June 2018 at 08:31, Guido van Rossum <guido@python.org> wrote:
Yep, it does, and I don't think you missed anything. Using "__parentlocal" to indicate "parent local scoping semantics apply here" still gives the concept a name and descriptive shorthand for use in pseudo-code expansions of assignment expressions in comprehensions, without needing to give it an actually usable statement level syntax, similar to the way we use "_expr_result" and "_outermost_iter" to indicate references that in reality are entries in an interpreter's stack or register set, or else a pseudo-variable that doesn't have a normal attribute identifier. And if anyone does want to make the case for the syntax being generally available, they don't need to specify how it should work - they just need to provide evidence of cases where it would clarify code unrelated to the PEP 572 use case. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jun 24, 2018 at 04:33:38PM +1000, Nick Coghlan wrote: [...]
You seem to be talking about an implementation which could change in the future. I'm talking semantics of the proposed language feature. As a programmer writing Python code, I have no visibility into the implementation. The implementation could change ten times a day for all I care, so long as the semantics remain the same. I want the desired semantics to drive the implementation, not the other way around. You seem to want the implementation to drive the semantics, by eliminating the proposed feature because it doesn't match your deep understanding of the implementation as a nested function. I want this feature because its useful, and without it the use-cases for assignment expressions are significantly reduced. As far as "implicit", for the sake of the discussion, I'll grant you that one. Okay, the proposed behaviour will implicitly enable comprehensions to export their state. Now what? Is that a good thing or a bad thing? If "implicit" (with or without the scare quotes) is such a bad thing to be avoided, why are comprehensions implemented using an implicit function?
You talk about "nested comprehension scope", and that's a critical point, but I'm going to skip answering that for now. I have a draft email responding to another of your posts on that topic, which I hope to polish in the next day.
If it is okay for you to amend the list comprehension to behave as if it were wrapped in an implicit nested function, why shouldn't it be okay to behave as if assignments inside the comprehension included an implicit nonlocal declaration?
Obviously "everyone" is an exaggeration, but, yes, I stand by that -- most people don't even give comprehension scope a thought until they get bitten by it. Either because (Python 2) they don't realise the loop variable is local to their current scope: http://www.librador.com/2014/07/10/Variable-scope-in-list-comprehension-vs-g... or (Python 3) they get bitten by the change: https://old.reddit.com/r/Python/comments/425qmb/strange_python_27_34_differe... (As is so often the case, whatever behaviour we choose, we're going to surprise somebody.) It is hardly surprising that people don't think too hard about scoping of comprehensions. Without a way to perform assignments inside comprehensions, aside from the loop variables themselves, there's nothing going on inside a comprehension where it makes a visible difference whether it is a local scope or a sublocal scope. *IF* assignment expressions are introduced, that is going to change. We have some choices: 1. Keep assignment expressions encapsulated in their implicit function, and be prepared for people to be annoyed because (with no way to declare them global or non-local inside an expression), they can't use them to get data in and out of the comprehension. 2. Allow assignment expressions to be exported out of the comprehension, and be prepared for people to be annoyed because they clobbered a local. (But for the reasons Tim Peters has already set out, I doubt this will happen often.) 3. Allow some sort of extra comprehension syntax to allow global/nonlocal declarations inside comprehensions. x = 1 [nonlocal x := x+i for i in sequence] (Hmmm... I thought I would hate that more than I actually do.) 4. Have some sort of cunning plan whereby if the variable in question exists in the local scope, it is implicitly local inside the comprehension: x = 1 [x := i+1 for i in (1, 2)] assert x == 3 but if it doesn't, then the variable is implicitly sublocal inside the comprehension: del x [x := i+1 for i in (1, 2)] x # raises NameError Remember, the driving use-case which started this (ever-so-long) discussion was the ability to push data into a comprehension and then update it on each iteration, something like this: x = initial_value() results = [x := transform(x, i) for i in sequence] Please, Nick, take your implementor's hat off, forget everything you know about the implementation of comprehensions and their implicit nested function, and tell me that doesn't look like it should work. -- Steve

On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Which means there is another option. 5. Have the assignment be local to the comprehension, but the initial value of ANY variable is looked up from the surrounding scopes. That is: you will NEVER get UnboundLocalError from a comprehension/genexp; instead, you will look up the name as if it were in the surrounding scope, either getting a value or bombing with regular old NameError. Or possibly variations on this such as "the immediately surrounding scope only", rather than full name lookups. It'd have an awkward boundary somewhere, whichever way you do it. This isn't able to send information *out* of a comprehension, but it is able to send information *in*. ChrisA

On Sun, Jun 24, 2018 at 2:10 PM Chris Angelico <rosuav@gmail.com> wrote:
But this "horrifies" me for a slightly different reason: it effectively introduces a new case of dynamic scoping, which Python used to do everywhere but has long switched away from, with the exception of class scopes (whose difference with function scopes sometimes confuses people -- usually people who put too much code in their class scope). -- --Guido van Rossum (python.org/~guido)

This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected. On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <steve@pearwood.info> wrote:
If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner. results = accumulate(sequence, transform) The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section. While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first. Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases. On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev < python-dev@python.org> wrote:
again, not a huge deal, just a little bit more complexity
I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language. On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
On the other hand, an "expert" may be so steeped in a particular subculture that he no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.
On the contrary, I believe that "easy for beginners" should be a major concern. Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts. Mozilla's array comprehensions are almost identical to Python's, aside
from a couple of trivial differences:
I can't prove it, but I think the phrase ordering difference is not trivial.
Python: any(line.startswith('#') for line in file) English: Any line starts with "#" in the file?

On Sun, Jun 24, 2018 at 2:41 PM Michael Selik <mike@selik.org> wrote:
Sure. Since the target for the PEP is Python 3.8 I am in no particular hurry. It would be important to know how you present it to your students.
I think that's a misunderstanding. At the very least the typical use case is *not* using an existing transform function which is readily passed to accumulate -- instead, it's typically written as a simple expression (e.g. `total := total + v` in the PEP) which would require a lambda. Plus, I don't know what kind of students you are teaching, but for me, whenever the solution requires a higher-order function (like accumulate), this implies a significant speed bump -- both when writing and when reading code. (Honestly, whenever I read code that uses itertools, I end up making a trip to StackOverflow :-).
The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section.
(I think you mean first_true().)
I trust Tim's intuition here, he's written about this. Also, Python's predecessor, ABC, had quantifiers (essentially any() and all()) built into the language, and the semantics included making the first (counter-)example available ( https://homepages.cwi.nl/~steven/abc/qr.html#TESTS). Essentially IF SOME x IN values HAS x < 0: WRITE "Found a negative x:", x equivalently IF EACH x IN values HAS x >= 0: # ... ELSE: WRITE "Found a negative x:", x and even IF NO x IN values HAS x < 0: # ... ELSE: WRITE "Found a negative x:", x
Perhaps, but IMO the new itertool would be much less useful than the new syntax.
I don't think that appeal to beginners is Python's greatest strength. I'd rather say that it is its appeal to both beginners and experts (and everyone in between). If true appeal to beginners is needed, Scratch or Processing would probably be better.
Nor should we cater to them excessively though. While the user is indeed king, it's also well known that most users when they are asking for a feature don't know what they want (same for kings, actually, that's why they have advisors :-).
I'm sorry, but this offends me, and I don't believe it's true at all. Python is *not* a beginners language, and you are mixing ease of use and ease of learning. Python turns beginners into experts at an unprecedented rate, and that's the big difference with Scratch. -- --Guido van Rossum (python.org/~guido)

On Sun, Jun 24, 2018 at 4:57 PM Guido van Rossum <guido@python.org> wrote:
Absolutely. Since this has come up, I'll make an effort to be more systematic in data collection.
Plus, I don't know what kind of students you are teaching, but for me,
Mostly mid-career professionals, of highly varying backgrounds. The higher-order functions do require some cushioning getting into, but I have some tricks I've learned over the years to make it go over pretty well. On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
By saying "Scratch for adults" I meant that Python is a language that can be adopted by beginners and rapidly make them professionals, not that it's exclusively a beginner's language. Also, Scratch and similar languages, like NetLogo, have some interesting features that allow beginners to write some sophisticated parallelism. I don't mean "beginner's language" in that it's overly simplistic, but that it enables what would be complex in other languages. I realize that my phrasing was likely to be misunderstood without knowing the context that I teach working professionals who are asked to be immediately productive at high-value tasks.

Steven D'Aprano wrote:
You seem to be talking about an implementation which could change in the future. I'm talking semantics of the proposed language feature.
The way I see it, it's not about implementation details, it's about having a mental model that's easy to reason about. "Comprehensions run in their own scope, like a def or lambda" is a clear and simple mental model. It's easy to explain and keep in your head. The proposed semantics are much more complicated, and as far as I can see, are only motivated by use cases that you shouldn't really be doing in the first place. -- Greg

On Sun, Jun 24, 2018 at 03:56:47PM +1000, Steven D'Aprano wrote:
There is no consensus that the change to comprehensions was a good thing or justified.
On re-reading that, I think its wrong -- it wasn't really what I intended to say. What I intended to say was, in hindsight, more like: *Despite the consensus to change comprehension scope*, there's a contingent of people who are not convinced that the change was a good thing or justified. Sorry for the inaccurate comment. Mea culpa. -- Steve

On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote: [...]
Without knowing how you worded the question, and the reasons for this horrified reaction, I'm afraid that isn't really helpful. It is nothing more than an appeal to emotion: https://en.wikipedia.org/wiki/Wisdom_of_repugnance Such strong emotions as "horrified" are typically a sign of an immediate, emotional gut reaction, not careful thought. We often see those sorts of reactions attached to the most objectively trivial matters. Immediate gut reactions are rarely a good guide because they tend to over-value the status quo, exaggerate the difficulty and costs of change, and under-estimate the benefits. Speaking personally, I've learned to question my immediately gut reaction. (And I remember to do so at least half the time.) PEP 572 is an example: when the issue was first raised back in February, my gut reaction was "Not in MY Python!!!" but by taking it seriously and running through some examples over the course of the discussion, I realised that, actually, I cautiously favour the idea. Of course, matters of *personal taste* cannot be anything but gut reaction, but in those matters, what one person holds strongly another can legitimately reject strongly. We ought to try to look beyond personal taste, and try (even if only imperfectly) to consider rational reasons for and against a proposal. If we do, reactions like "horrified" are rarely justified. It's just a minor feature in a programming language, the world will go on one way or the other, and Python already has trickier gotchas.
While we certainly don't want to make "non-obvious" a virtue for its own sake, obviousness (obvious to who?) ought to take a distant second place to *useful*. Otherwise we'd have to give up an awful lot of existing Python, starting with the fundamental execution model. (Oh, the number and length of arguments about whether Python uses call by value or call by reference, why mutable defaults and [[]]*3 are "broken"... if you think Python's execution model is "obvious" you've been using Python too long ;-) But as Tim Peters has said on a number of occasions, nobody is suggesting changing the interpretation of current comprehension semantics. Comprehension loop variables will continue to remain isolated to the comprehension. (And for the record, that makes *comprehensions* a weird special case, not assignment expressions. All other expressions run in the current lexical scope. Comprehensions introduce an implicit, invisible, sub-local scope that doesn't match up with a change in indentation as class and def statements do.) The behaviour in question is a matter of *assignment expression* semantics, not comprehensions. And honestly, I don't see why the proposed behaviour is "horrifying". Here's the high-level overview: - at the top level of a module, assignment expressions assign in the global scope; - inside a class, assignment expressions assign in the class scope; - inside a function, assignment expressions assign in the function local scope (unless declared global or nonlocal); - inside a comprehension, assignment expressions assign in the surrounding lexical scope (the surrounding function, class or module). The first three are the same as ordinary statement assignment. The last one is what you would expect if you treat comprehensions as any other expression which run in the current lexical scope. (The current function or class or module.) Even if we treat it as a "weird special case" (I don't think it is, but for the sake of the argument let's say it is) its not hard to explain. As I discuss below, you can get a very long way indeed working with comprehensions without once thinking about the scope they run in. By the time you need to think about comprehension scope, it shouldn't be hard to deal with the rule: - loop variables are hidden in a comprehension private scope; - explicit assignment expression variables are not. This is not async, or metaclasses, or even Unicode. [...]
I can't say I've done a broad survey, but the third-party documentation I've read on comprehensions typically glosses over the scoping issues without mentioning them. To the extent that scoping is even hinted at, comprehensions are treated as expressions which are exactly equivalent to re-writing them as a for-loop in the current scope. This is a typical example, found as the top result on googling for "python comprehensions": https://www.google.com/search?q=python+comprehensions http://www.pythonforbeginners.com/basics/list-comprehensions-in-python Nothing is mentioned about scope, and it repeats the inaccurate but simple equivalency: for item in list: if conditional: expression But perhaps that tutorial is too old. Okay this recent one is only a little more than a year old: https://hackernoon.com/list-comprehension-in-python-8895a785550b Again, no mention of scoping issues, comprehensions are simply expressions which presumably run in the same scope as any other expression. I think you over-estimate how many newcomers to Python are even aware that the scope of comprehensions is something to consider.
I look forward to reading it, and I promise I won't go by my gut reaction :-) -- Steve

On Sat, Jun 23, 2018 at 1:48 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Even first-party documentation elides that distinction. The same inaccurate-but-simple equivalency - even using the word "equivalent" - comes up here: https://docs.python.org/3/howto/functional.html?highlight=equivalent#generat... So I'm very sympathetic to the desire to have assignment expressions inside comprehensions behave like assignment expressions outside comprehensions. The trouble is that they are then _not_ the same as other names inside comprehensions. One way or another, there's a confusing distinction, especially at class scope. Unless this comes with an actual semantic change that affects existing code, there is going to be a bizarre disconnect *somewhere*, and it's just a matter of where. ChrisA

On 23 June 2018 at 13:48, Steven D'Aprano <steve@pearwood.info> wrote:
I put quite a bit of work into making it possible for folks to gloss over the distinction and still come to mostly-correct conclusions about how particular code snippets would behave. I was only able to achieve it because the folks that designed lexical scoping before me had made *read* access to lexical scopes almost entirely transparent, and because generator expressions were designed to fail fast if there was a bug in the expression defining the outermost iterable (which meant that even at class scope, the outermost iterable expression still had access to class level variables, because it was evaluated *outside* the nested scope). *Write* access to lexically nested scopes, by contrast, was omitted entirely from the original lexical scoping design, and when it was later added by https://www.python.org/dev/peps/pep-3104/, it was done using an explicit "nonlocal" declaration statement (expressly akin to "global"), and PEP 3099 explicitly ruled out the use of ":=" to implicitly declare the target name as being non-local. PEP 572 is thus taking the position that: - we now want to make write access to outer scopes implicit (despite PEP 3099 explicitly ruling that out as desired design feature) - but only in comprehensions and generator expressions (not lambda expressions, and not full nested functions) - and only for assignment expressions, not for loop iteration variables - and we want it to implicitly choose between a "global NAME" declaration and a "nonlocal NAME" declaration based on where the comprehension is defined - and this is OK because "nobody" actually understands how comprehensions hide the iteration variable in practice, and "everybody" thinks they're still a simple for loop like they were in Python 2 - the fact that the language reference, the behaviour at class scopes, the disassembly output, and the behaviour in a debugger all indicate that comprehensions are full nested scopes isn't important This level of additional complication and complexity in the scoping semantics simply isn't warranted for such a minor readability enhancement as assignment expressions. Cheers, Nick. P.S. "You did such a good job of minimising the backwards compatibility breakage when we changed the semantics of scoping in comprehensions that we now consider your opinion on reasonable scoping semantics for comprehensions to be irrelevant, because everybody else still thinks they work the same way as they did in Python 2" is such a surreal position for folks to be taking that I'm having trouble working out how to effectively respond to it. Guido has complained that "I keep changing my mind about what I want", but that's not what's actually going on: what I want is to keep folks from taking our already complicated scoping semantics and making it close to impossible for anyone to ever infer how they work from experimentation at the interactive prompt. That goal has pretty much stayed consistent since the parent local scoping proposal was first put forward. What keeps changing is my tactics in pursuing that goal, based on my current perception of what the folks pushing that proposal are actually trying to achieve (which seems to be some combination of "We want to pretend that the Python 3 scoping changes in comprehensions never happened, but we still want to avoid leaking the iteration variables somehow" and "We want to enable new clever tricks with state export from comprehensions and generator expressions"), as well as repeatedly asking myself what *actually* bothers me about the proposal (which I've now distilled down to just the comprehension scoping issue, and the reliance on an arbitrary syntactic restriction against top level usage to avoid competing with traditional assignment statements). -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (25)
-
Antoine Pitrou
-
Baptiste Carvello
-
Ben Finney
-
Brett Cannon
-
Chris Angelico
-
Chris Barker
-
Chris Barker - NOAA Federal
-
David Mertz
-
Eric Fahlgren
-
Eric V. Smith
-
Glenn Linderman
-
Greg Ewing
-
Guido van Rossum
-
Ivan Pozdeev
-
Mark Dickinson
-
Michael Selik
-
Mike Miller
-
Nick Coghlan
-
Paul Moore
-
Rob Cliffe
-
Steve Holden
-
Steven D'Aprano
-
Terry Reedy
-
Tim Peters
-
Victor Stinner