Mailman 3 Syntax proposal of for..in..if in regular for loops - Python-ideas

Syntax proposal of for..in..if in regular for loops

older
Config file template motivation...

Svein Seldal

2 Mar 2022 2 Mar '22

1:28 a.m.

I'm sorry for reposting, but this message got stuck in moderation approval for 5 days so I figured I should try again. I'd like to propose extending the for statement to include conditionals akin to comprehensions in order to simplify for loop statements: `for .. in .. if ..:` E.g. for x in y if x in c: some_op(x) which is functionally equivalent as for x in y: if x not in c: continue some_op(x) The `for .. in .. if` syntax is a well-known construct from list comprehension syntax [1]. Other alternative ways to do this with list comprehension is: for x in (a for a in y if c): or it = (a for a in y if c) for x in it: Without having examined all use cases, I believe the same syntax should be applied this syntax as for asynchronous comprehensions. [1] PEP 202 - List Comprehensions [2] PEP 530 - Asynchronous Comprehensions Best regards, Svein Seldal

Show replies by date

Paul Bryan

2 Mar 2 Mar

1:42 a.m.

Very strong +1. On Wed, 2022-03-02 at 02:28 +0100, Svein Seldal wrote:

...

I'm sorry for reposting, but this message got stuck in moderation approval for 5 days so I figured I should try again.

I'd like to propose extending the for statement to include conditionals akin to comprehensions in order to simplify for loop statements:

     `for .. in .. if ..:`

E.g.

     for x in y if x in c:          some_op(x)

which is functionally equivalent as

     for x in y:          if x not in c:              continue          some_op(x)

The `for .. in .. if` syntax is a well-known construct from list comprehension syntax [1]. Other alternative ways to do this with list comprehension is:

     for x in (a for a in y if c):

or

     it = (a for a in y if c)      for x in it:

Without having examined all use cases, I believe the same syntax should be applied this syntax as for asynchronous comprehensions.

[1] PEP 202 - List Comprehensions [2] PEP 530 - Asynchronous Comprehensions

Best regards, Svein Seldal _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7IXPRO... Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

2:37 a.m.

On Wed, Mar 02, 2022 at 02:28:33AM +0100, Svein Seldal wrote:

...

for x in y if x in c: some_op(x)

What does this new syntax give us that we don't already have with this? for x in y if x in c: some_op(x) or the other multiple ways of writing the equivalent code? I see no new functionality here. Is the only advantage that you save one line and one indent level? Both are cheap. To be precise, one extra line in something which is already a multiline statement is essentially insignificant, and while an extra indent level *can* be important, if you have already used so many indents that it becomes important, you probably should be refactoring your code. All I see here is adding to the complexity of the parser and the language for no meaningful benefit. Its not even easier to read, it crams more on one line which in real code with meaningful variable names and a meanigful condition starts to get a bit long: # this line desperately needs to be split over two lines for student in students if mean(student.grade(subject) for subject in student.get_subjects()) > 0.8): ... When I write list comprehensions with an if condition, probably 90% of the time I end up moving the if condition to a second or even third line of the comprehension. I expect the same will apply here. To save one line for maybe 10% of my for...if constructs, I don't think it is worth the bother of adding yet another way to do it. -- Steve

Paul Bryan

2:51 a.m.

1. It aligns with existing syntax in list comprehensions and generator expressions. 2. In probably majority of cases it would be more readable to me; "iterate over iterable for items meeting condition:". 3. Could it be easier to optimize an explicit filter expression to improve iteration performance? On Wed, 2022-03-02 at 13:37 +1100, Steven D'Aprano wrote:

...

On Wed, Mar 02, 2022 at 02:28:33AM +0100, Svein Seldal wrote:

...
    for x in y if x in c:         some_op(x)

What does this new syntax give us that we don't already have with this?

    for x in y         if x in c:             some_op(x)

or the other multiple ways of writing the equivalent code?

I see no new functionality here. Is the only advantage that you save one line and one indent level? Both are cheap.

To be precise, one extra line in something which is already a multiline statement is essentially insignificant, and while an extra indent level *can* be important, if you have already used so many indents that it becomes important, you probably should be refactoring your code.

All I see here is adding to the complexity of the parser and the language for no meaningful benefit. Its not even easier to read, it crams more on one line which in real code with meaningful variable names and a meanigful condition starts to get a bit long:

    # this line desperately needs to be split over two lines     for student in students if mean(student.grade(subject) for subject in student.get_subjects()) > 0.8):         ...

When I write list comprehensions with an if condition, probably 90% of the time I end up moving the if condition to a second or even third line of the comprehension. I expect the same will apply here.

To save one line for maybe 10% of my for...if constructs, I don't think it is worth the bother of adding yet another way to do it.

Michael Smith

3:40 a.m.

+1 This is just a small improvement, but worthwhile. It's intuitive IMO to be able to use similar filtering expressions to comprehensions at the top of a for loop. Here's an example: # get the Hadoop version by scanning pyspark jars. # Vague attribution: https://stackoverflow.com/a/50242383 for path in Path("pyspark/jars")).glob("hadoop-*.jar") if not path.stem.startswith("hadoop-shaded-guava"): name, _, version = path.stem.rpartition("-") ... glob is already a filtering mechanism. It's just not *quite* flexible enough to give just the items I want. I'd like to be able to enter the loop with only file names of interest. –Michael A. Smith On Tue, Mar 1, 2022 at 21:51 Paul Bryan wrote:

...

1. It aligns with existing syntax in list comprehensions and generator expressions. 2. In probably majority of cases it would be more readable to me; "iterate over iterable for items meeting condition:". 3. Could it be easier to optimize an explicit filter expression to improve iteration performance?

On Wed, 2022-03-02 at 13:37 +1100, Steven D'Aprano wrote:

On Wed, Mar 02, 2022 at 02:28:33AM +0100, Svein Seldal wrote:

for x in y if x in c: some_op(x)

What does this new syntax give us that we don't already have with this?

for x in y if x in c: some_op(x)

or the other multiple ways of writing the equivalent code?

I see no new functionality here. Is the only advantage that you save one line and one indent level? Both are cheap.

To be precise, one extra line in something which is already a multiline statement is essentially insignificant, and while an extra indent level *can* be important, if you have already used so many indents that it becomes important, you probably should be refactoring your code.

All I see here is adding to the complexity of the parser and the language for no meaningful benefit. Its not even easier to read, it crams more on one line which in real code with meaningful variable names and a meanigful condition starts to get a bit long:

# this line desperately needs to be split over two lines for student in students if mean(student.grade(subject) for subject in student.get_subjects()) > 0.8): ...

When I write list comprehensions with an if condition, probably 90% of the time I end up moving the if condition to a second or even third line of the comprehension. I expect the same will apply here.

To save one line for maybe 10% of my for...if constructs, I don't think it is worth the bother of adding yet another way to do it.

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KTM6XD... Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

5:29 a.m.

On Tue, Mar 01, 2022 at 10:40:06PM -0500, Michael Smith wrote:

...

+1

This is just a small improvement, but worthwhile. It's intuitive IMO to be able to use similar filtering expressions to comprehensions at the top of a for loop.

Good news! We have been able to use filtering conditions at the top of a for loop since Python 1.5 or even earlier, back in the 1990s. And the excellent thing about the existing syntax is that we can split the for loop and the filter onto separate lines for enhanced readablity, without needing to add extraneous parentheses and indent. # Don't need this. for student in students if ( mean(student.grade(subject) for subject in student.get_subjects()) > 0.8) ): ... # Can do this. for student in students: if mean(student.grade(subject) for subject in student.get_subjects()) > 0.8): ... And if you are thinking right now "Huh, there is hardly any difference between the two, what's the big deal?" -- exactly. Outside of pretend code with one letter variable names, `for a in b if c`, the actual benefit in real code is extremely minor, if any. In most cases you're going to split the line over two lines anyway, and if you find yourself worried about saving an indent, almost surely your code needs refactoring.

...

Here's an example:

# get the Hadoop version by scanning pyspark jars. # Vague attribution: https://stackoverflow.com/a/50242383 for path in Path("pyspark/jars")).glob("hadoop-*.jar") if not path.stem.startswith("hadoop-shaded-guava"):

Ironically, your one line loop and filter was word-wrapped because the line was too long. Do you use black or PEP8? Do the projects you work on enforce 79 column or even 100 column coding standards? Then your example, with 106 columns, will have to be split over two lines. Which is my point. This suggestion sounds good, but in real code, you probably can't use it. And even if you can, it doesn't make it any easier to write loops with filters, or add the ability to do something new that we can't easily do now. It is purely a cosmetic change, and not even a very interesting cosmetic change. -- Steve

Stephen J. Turnbull

8:01 a.m.

Michael Smith writes:

...

This is just a small improvement, but worthwhile. It's intuitive IMO to be able to use similar filtering expressions to comprehensions at the top of a for loop.

Intuitive I guess, but in comprehensions it was "strictly from need" since comprehension syntax is an expression. There is no other way to explicitly filter in comprehensions. (One could write a generator function to do the filtering, but "explicit is better ...".)

...

Here's an example:

for path in Path("pyspark/jars")).glob("hadoop-*.jar") if not path.stem.startswith("hadoop-shaded-guava"):

"Bad MUA! Bad, bad MUA! You know you're not supposed to respect line length constraints when somebody's trying to make a point." My 63-year-old eyes struggle with 80-character lines, but in all fairness to those with better eyesight I'm not going to argue for Mom's 65[1]. But 106 (and that's assuming this is top-level "scripting" code) would get a "restructure and resubmit" from me. Footnotes: [1] The number of characters produced by my Mom's typewriter with 6.5" lines.

Steven D'Aprano

9:58 a.m.

On Wed, Mar 02, 2022 at 05:01:38PM +0900, Stephen J. Turnbull wrote:

...

My 63-year-old eyes struggle with 80-character lines, but in all fairness to those with better eyesight I'm not going to argue for Mom's 65[1].

This is a very good point that the "80 columns is too short" crowd forget. 80 columns is *already* significantly longer (20-25%) than the optimum column width for prose text. It is a compromise between optimimum reading width for prose (about 65 columns of monospaced text) and the needs of program code, which does not always work well with word-wrapping. But I digress. -- Steve

Christopher Barker

5:27 a.m.

On Tue, Mar 1, 2022 at 6:43 PM Steven D'Aprano wrote:

...

...
for x in y if x in c: some_op(x)

What does this new syntax give us that we don't already have with this?

for x in y if x in c: some_op(x)

I think it's really the equivalent of for x in y: if not x in c: break do_stuff which to me give the proposed syntax a bit more relative strength. I'm probably +0 -- but I do like comprehension syntax, and have often wanted a "side effect" comprehension: $ side_effect(item) for item in an_iterable if condition(item) $ (using $ 'cause there aren't any more brackets ...) rather than having to write: for item in an_iterable: if condition(item): side_effect(item) To the point where I sometimes write the list comp and ignore the generated list. NOt too huge a burden to cretae. list of None and throw it away, but it feels wrong ;-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Steven D'Aprano

10:22 a.m.

On Tue, Mar 01, 2022 at 09:27:05PM -0800, Christopher Barker wrote:

...

I think it's really the equivalent of

for x in y: if not x in c: break do_stuff

which to me give the proposed syntax a bit more relative strength.

Forgotten the difference between continue and break, have we? :-) Svein (the OP) did specify continue, not break, which matches the equivalent syntax in comprehensions. Semantically, there is no difference between if not condition: continue block and if condition: block but as of CPython 3.10, the byte-code from the first version is slightly longer, so I imagine (but haven't measured) it will be ever-so-slightly less efficient. (But unlikely to be meaningfully different.)

...

I'm probably +0 -- but I do like comprehension syntax, and have often wanted a "side effect" comprehension:

Me too! I've sometimes wanted something to call a bunch of functions, or a single function with different arguments, solely for the side-effects. Off-topic, but since you raised the issue... is there a standard functional programming term for a variant of map() that applies a single argument to a series of different functions? # regular map map(func, list_of_args) # (func(arg) for arg in list_of_args) # variant map? map(arg, list_of_funcs) # (func(arg) for func in list_of_funcs) -- Steve

Paul Moore

10:51 a.m.

On Wed, 2 Mar 2022 at 10:27, Steven D'Aprano wrote:

...

Off-topic, but since you raised the issue... is there a standard functional programming term for a variant of map() that applies a single argument to a series of different functions?

# regular map map(func, list_of_args) # (func(arg) for arg in list_of_args)

# variant map? map(arg, list_of_funcs) # (func(arg) for func in list_of_funcs)

That sounds like what I've heard referred to as "map apply". I don't think functional languages tend to have a particular name for it, because it falls naturally out of the syntax for mapping and functional application. (And as you show, Python is similar - the generator comprehension is easy enough that a named function is not often useful). Paul

Dennis Sweeney

12:12 p.m.

I've heard "evaluation map" for a related mathematical concept: the natural map from X to (X -> Y) -> Y in some cartesian closed category (whatever that means :-), like the natural embedding of a vector space into its double dual space, or like this sort of eval_at function that you can then plug into map: def eval_at(x): return lambda f: f(x) list(map(eval_at(arg), functions)) It also reminds me of a Clojure feature, where IIRC a key can be used as a function so that `(:key mymap)` and `(mymap :key)` both mean "the value in `mymap` corresponding to the key `:key`"

Ricky Teachey

2:25 p.m.

I think I'm -0.5 but I have a question for the people on here smarter than me (pretty much all): Is there some opportunity for some kind of compiler magic when the iterable of a for loop is fully contained in a place easily findable by the compiler, and not spread over multiple if and for statements? I am imagining that something like this could be magically "looked into" and made more efficient in some way, maybe by JIT complier or something: for x for y in range(11, 100, 3) if (y % 10) for x in range(y): frob(x) compared to this: for y in range(11, 100, 3): if (y % 10); for x in range(y): frob(x) Am I instilling too much faith in the power of the complier on this Ash Wednesday morning? ;) --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler On Wed, Mar 2, 2022 at 7:13 AM Dennis Sweeney wrote:

...

I've heard "evaluation map" for a related mathematical concept: the natural map from X to (X -> Y) -> Y in some cartesian closed category (whatever that means :-), like the natural embedding of a vector space into its double dual space, or like this sort of eval_at function that you can then plug into map:

def eval_at(x): return lambda f: f(x) list(map(eval_at(arg), functions))

It also reminds me of a Clojure feature, where IIRC a key can be used as a function so that `(:key mymap)` and `(mymap :key)` both mean "the value in `mymap` corresponding to the key `:key`" _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YFYX76... Code of Conduct: http://python.org/psf/codeofconduct/

Chris Angelico

6:11 p.m.

On Thu, 3 Mar 2022 at 01:28, Ricky Teachey wrote:

...

I think I'm -0.5 but I have a question for the people on here smarter than me (pretty much all):

Is there some opportunity for some kind of compiler magic when the iterable of a for loop is fully contained in a place easily findable by the compiler, and not spread over multiple if and for statements?

Yes, there absolutely is, and you can see some of that by disassembling:

...

...
...
def f1(): ... for x in [1,2,3]: print(x) ... def f2(): ... for x in (1,2,3): print(x) ...

In theory, the first one means "build a list with three values, then iterate over it", and the second one means "iterate over the constant tuple (1,2,3)", but CPython implements them both the same way. Try "dis.dis(f1)" and "dis.dis(f2)" to see this similarity.

...

I am imagining that something like this could be magically "looked into" and made more efficient in some way, maybe by JIT complier or something:

JIT compilation is orthogonal to this sort of optimization; what you're thinking of here is a form of constant folding.

...

for x for y in range(11, 100, 3) if (y % 10) for x in range(y): frob(x)

compared to this:

for y in range(11, 100, 3): if (y % 10); for x in range(y): frob(x)

Am I instilling too much faith in the power of the complier on this Ash Wednesday morning? ;)

I think you are, and here's why: It's far too easy for something to have disrupted the optimization (for instance, shadowing the name "range"). Where a use-case is common, the compiler can be taught to optimize it; where it's uncommon, the effort isn't worth it. Constant folding is an incredibly useful technique, but it's not quite able to do this :) JIT compilation is also incredibly powerful. I don't know whether it'd be able to optimize what you're thinking of, but if it did, what you'd end up with is a much more generic optimization that applies to all loops, or maybe to all calls to range(). There are some other possibilities. It wouldn't surprise me if PyPy has a special check to optimize "for VAR in range(...):" that consists of (a) verifying that range hasn't been shadowed, and then (b) iterating over the numbers directly, without constructing a range object. On the other hand, it also wouldn't greatly surprise me if it doesn't, on the basis that this isn't beneficial enough to have special code, and the generic handling is able to cover it. ChrisA

Jeremiah Paige

7:53 p.m.

I have on a few occasions wanted a for..in..if statement and if it existed would have used it. However, I agree that the level of change a new statement type brings to the language is probably too high for this feature. But what if python lifted the newline requirement for blocks that contain compound statements? That is, statements that end in a ':' can be followed by other statements that end in a ':' on the same line. AFAICT there would be no ambiguity (to the parser; to humans, depends). Doing so would add the OPs requested feature, though it would be two statements on one line with one extra character. It would also essentially bring the full comprehension syntax to for loops since fors and ifs could be chained arbitrarily. # for..if for x in y: if x in c: some_op(x # nested generator-like for for line in doc: for word in line.split(): spellcheck(word) # side effect one-liner for item in an_iterable: if condition(item): side_effect(item)) Regards, Jeremiah

David Mertz, Ph.D.

8:03 p.m.

I really am shocked by how many people seem to have broken ENTER keys on their keyboards. Let's just keep Python readable rather than see how much we can cram on a line. On Wed, Mar 2, 2022, 2:56 PM Jeremiah Paige wrote:

...

I have on a few occasions wanted a for..in..if statement and if it existed would have used it. However, I agree that the level of change a new statement type brings to the language is probably too high for this feature.

But what if python lifted the newline requirement for blocks that contain compound statements? That is, statements that end in a ':' can be followed by other statements that end in a ':' on the same line. AFAICT there would be no ambiguity (to the parser; to humans, depends). Doing so would add the OPs requested feature, though it would be two statements on one line with one extra character. It would also essentially bring the full comprehension syntax to for loops since fors and ifs could be chained arbitrarily.

# for..if for x in y: if x in c: some_op(x

# nested generator-like for for line in doc: for word in line.split(): spellcheck(word)

# side effect one-liner for item in an_iterable: if condition(item): side_effect(item))

Regards, Jeremiah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LWPJKA... Code of Conduct: http://python.org/psf/codeofconduct/

Rob Cliffe

3 Mar 3 Mar

2:32 a.m.

...

I really am shocked by how many people seem to have broken ENTER keys on their keyboards. You mock. (As far as I remember you are always opposed to new language features/changes.) But the proposal would give people the choice of Saving a level of indentation at the cost of having two suite-introductions on the same line. Keeping the two suit-introductions on separate lines (as now) at

On 02/03/2022 20:03, David Mertz, Ph.D. wrote: the cost of an extra level of indentation. Sometimes one will be better, sometimes the other. Usually the choice will be a subjective one. That's no reason not to have the choice. Of course, this has to be weighed against the cost of the change. Best wishes Rob Cliffe

...

Let's just keep Python readable rather than see how much we can cram on a line.

On Wed, Mar 2, 2022, 2:56 PM Jeremiah Paige wrote:

I have on a few occasions wanted a for..in..if statement and if it existed would have used it. However, I agree that the level of change a new statement type brings to the language is probably too high for this feature.

But what if python lifted the newline requirement for blocks that contain compound statements? That is, statements that end in a ':' can be followed by other statements that end in a ':' on the same line. AFAICT there would be no ambiguity (to the parser; to humans, depends). Doing so would add the OPs requested feature, though it would be two statements on one line with one extra character. It would also essentially bring the full comprehension syntax to for loops since fors and ifs could be chained arbitrarily.

# for..if for x in y: if x in c: some_op(x

# nested generator-like for for line in doc: for word in line.split(): spellcheck(word)

# side effect one-liner for item in an_iterable: if condition(item): side_effect(item))

Regards, Jeremiah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LWPJKA... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list --python-ideas@python.org To unsubscribe send an email topython-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived athttps://mail.python.org/archives/list/python-ideas@python.org/message/6COVUG... Code of Conduct:http://python.org/psf/codeofconduct/

Steven D'Aprano

8:50 a.m.

On Thu, Mar 03, 2022 at 02:32:25AM +0000, Rob Cliffe via Python-ideas wrote:

...

But the proposal would give people the choice of Saving a level of indentation at the cost of having two suite-introductions on the same line. Keeping the two suit-introductions on separate lines (as now) at the cost of an extra level of indentation.

Why only two? Why not more than two? for item in seq: if cond: while flag: with something as x: for y in x.thing(): if condition: block Many other languages allow developers to cram code into enormous one-liners. Should we do the same? This is not a rhetorical question. And I have a non-rhetorical answer. In my opinion, no we should not. Python, for good or ill, is an opinionated language regarding indentation and statements. The general rule for compound statements is that the statement that introduces the block must appear on its own line. While you can put the block on the same line as the colon using semicolons: if condition: print(1); print(2) # Legal. you can't introduce a new block: if condition: for x in loop: # Not legal. A compound statement that ends with a colon is either either followed by a newline and one indent, or a series of semicolon separated simple statements. Never by another compound statement. This is, I believe, partly to keep the parser simple, and partly to keep the code easy to read, write and reason about. Could we do something different? Of course we could. But would the result still feel like Python? I don't think so. Ultimately, the choice comes down to taste. Python is not Perl, or APL, or Forth, or Hyperscript, or Inform. Anyone who has been reading my emails over the years knows that I am particularly fond of Forth and Hyperscript, and yet I would not want Python to be more like either of them. You don't have to think that a syntactic feature is *bad* to think that it doesn't belong in Python. Fresh strawberries are great. Mushroom sauce is great. But strawberries with mushroom sauce is ... not. -- Steve

Chris Angelico

9:04 a.m.

On Thu, 3 Mar 2022 at 19:54, Steven D'Aprano wrote:

...

On Thu, Mar 03, 2022 at 02:32:25AM +0000, Rob Cliffe via Python-ideas wrote:

...
But the proposal would give people the choice of Saving a level of indentation at the cost of having two suite-introductions on the same line. Keeping the two suit-introductions on separate lines (as now) at the cost of an extra level of indentation.

Why only two? Why not more than two?

for item in seq: if cond: while flag: with something as x: for y in x.thing(): if condition: block

Many other languages allow developers to cram code into enormous one-liners. Should we do the same?

This is not a rhetorical question. And I have a non-rhetorical answer.

In my opinion, no we should not.

Python has a history of making conceptual actions shorter than the mere combination of their parts. For instance, we don't have this construct: for i in range(len(stuff)) using thing = stuff[i]: No, we have this: for i, thing in enumerate(stuff): No matter how much you personally pooh-pooh the idea, filtered iteration is a very real concept, and people do it as best they can with the idioms available. In Python, that usually means putting part of the iteration code into the body of the loop: for thing in stuff: if not isinteresting(thing): continue There's an inefficient and verbose option of the genexp, but I don't think many people like it: for thing in (thing for thing in stuff if isinteresting(thing)): although when it's actually a function like this, you have this option: for thing in filter(isinteresting, stuff): which actually looks good. I think this is a pretty clear indication that the idea makes sense: functional programming languages have an idiom that aligns perfectly with it, it's just that Python's lambda syntax is too clunky for inline expressions to look as good. for thing in filter(lambda n: n % 7 < 2, stuff): I do not think Python should have a syntax which is merely removing a newline from what can already be done. But a proper syntax for filtered iteration WOULD be of extreme value. In bracey languages, I'll often combine the 'if' and 'for' onto one line, because newline flexibility lets me write filtered iteration that way. It's not ideal but it's a lot better than nothing: for (....) if (...) { ... } What Python needs is not a way to cram more onto one line. What Python needs is a way to express an abstract concept: "iterate over the interesting parts of this collection".

...

A compound statement that ends with a colon is either either followed by a newline and one indent, or a series of semicolon separated simple statements. Never by another compound statement.

You're thinking FAR FAR too concretely about this. It's not about newlines. It's about expressing programmer concepts.

...

Fresh strawberries are great. Mushroom sauce is great. But strawberries with mushroom sauce is ... not.

You DO know that you just made several people think "hmm, maybe I should try strawberries with mushroom sauce", right? ChrisA

Steven D'Aprano

10:10 a.m.

On Thu, Mar 03, 2022 at 08:04:57PM +1100, Chris Angelico wrote:

...

Python has a history of making conceptual actions shorter than the mere combination of their parts. For instance, we don't have this construct:

for i in range(len(stuff)) using thing = stuff[i]:

Rather than inventing new (fake) syntax to illustrate your point, a better example would be: for i in range(len(stuff)): thing = stuff[i] which is real code that people have written (including me!), and two statements. Just like the for... if... compound statement being debated now. And we did take that two-line idiom from Python 1.x and turn it into a one-line *functional* style in Python 2, using enumerate().

...

No, we have this:

for i, thing in enumerate(stuff):

Indeed. What we *didn't* do is invent new syntax for i in range(len(stuff)) using stuff[i] as thing we just came up with a function, enumerate(). If only Python had a function that would allow us to combine iteration and filtering... *wink*

...

No matter how much you personally pooh-pooh the idea, filtered iteration is a very real concept,

o_O What did I say that made you think I denied the existence of filtered iteration? Was it the post where I pointed out we've been able to do filtered iteration going back to Python 1.x days? To be clear, there are lots of concepts in coding. Not all of them require their own specialised syntax. We don't have specialised syntax for a try...except block inside a loop, we use composition by putting a try...except block inside a for loop. Composition of statements is not a bug to be fixed. [...]

...

There's an inefficient and verbose option of the genexp, but I don't think many people like it:

for thing in (thing for thing in stuff if isinteresting(thing)):

Indeed. But if you combine map() and filter() with the iteration, it becomes very powerful: for thing in (expression for x in stuff if isinteresting(x)): And with the walrus operator, you can filter on the mapped value as well: for thing in (y:=expression for x in stuff if isinteresting(x or y)):

...

although when it's actually a function like this, you have this option:

for thing in filter(isinteresting, stuff):

which actually looks good. I think this is a pretty clear indication that the idea makes sense: functional programming languages have an idiom that aligns perfectly with it, it's just that Python's lambda syntax is too clunky for inline expressions to look as good.

Nobody said that the idea of filtered looping doesn't make sense. They're only questioning whether it needs its own syntax instead of composing existing syntax.

...

for thing in filter(lambda n: n % 7 < 2, stuff):

Looks fine to me. But you could also use: for thing in (n for n in stuff if n%7 < 2): which also looks fine to me.

...

I do not think Python should have a syntax which is merely removing a newline from what can already be done. But a proper syntax for filtered iteration WOULD be of extreme value.

We have proper syntax: compose the for loop with the filter. Or use the functional idiom with filter(func, items). Or various other options such as generators. These are all "proper" syntax. They're just not *dedicated specialist syntax* for filtered iteration. With so many options to choose from, I don't think we need dedicated syntax for this operation. What does it add, aside from bloat?

...

What Python needs is not a way to cram more onto one line. What Python needs is a way to express an abstract concept: "iterate over the interesting parts of this collection".

Something like composition of a for-loop and an if, or filter(). Yes, if only we were capable of doing filtered iteration in Python! Then at last Python would be Turing Complete and a Real Programming Language!!! *wink*

...

...
A compound statement that ends with a colon is either either followed by a newline and one indent, or a series of semicolon separated simple statements. Never by another compound statement.

You're thinking FAR FAR too concretely about this. It's not about newlines.

Of course it is. The whole point of the proposal is to move a two line statement into a single line. Earlier in this thread, I pointed out that this proposal adds no new functionality to Python. It doesn't allow us to do anything we can't already do, or even make it easier to do it. Literally all it saves is a newline and an indent.

...

It's about expressing programmer concepts.

Right. And composing a for-loop with a if statement expresses that concept perfectly. As does filter().

...

...
Fresh strawberries are great. Mushroom sauce is great. But strawberries with mushroom sauce is ... not.

You DO know that you just made several people think "hmm, maybe I should try strawberries with mushroom sauce", right?

:-) -- Steve

Chris Angelico

11:30 a.m.

On Thu, 3 Mar 2022 at 21:14, Steven D'Aprano wrote:

...

What did I say that made you think I denied the existence of filtered iteration? Was it the post where I pointed out we've been able to do filtered iteration going back to Python 1.x days?

ANYTHING can be done by composing concepts. We don't need anything more advanced than Brainf*. Why do we have better concepts? Because they do a better job of expressing abstract concepts.

...

To be clear, there are lots of concepts in coding. Not all of them require their own specialised syntax. We don't have specialised syntax for a try...except block inside a loop, we use composition by putting a try...except block inside a for loop.

Composition of statements is not a bug to be fixed.

Indeed, but I'm putting the viewpoint - which a number of other people have also put - that filtered iteration DOES deserve a better way of expressing it.

...

Nobody said that the idea of filtered looping doesn't make sense. They're only questioning whether it needs its own syntax instead of composing existing syntax.

Yes. And it keeps coming up, so I think you should probably acknowledge the fact that maybe, just maybe, this is more significant than "one newline".

...

...
You're thinking FAR FAR too concretely about this. It's not about newlines.

Of course it is. The whole point of the proposal is to move a two line statement into a single line. Earlier in this thread, I pointed out that this proposal adds no new functionality to Python. It doesn't allow us to do anything we can't already do, or even make it easier to do it.

Literally all it saves is a newline and an indent.

No, it is not. It is expressing the concept of filtered iteration. Do you, or don't you, accept that that is a concept? One moment you say that it is a concept but you think it shouldn't get dedicated syntax, then the next, you imply that it isn't even a concept, and all we're doing is reformatting code. That is simply not the case.

...

...
It's about expressing programmer concepts.

Right. And composing a for-loop with a if statement expresses that concept perfectly. As does filter().

No, it doesn't.

...

...
...
Fresh strawberries are great. Mushroom sauce is great. But strawberries with mushroom sauce is ... not.

You DO know that you just made several people think "hmm, maybe I should try strawberries with mushroom sauce", right?

:-)

I love how utterly unapologetic you are. And I really hope that someone actually tries it, and reports back to the list. Who knows? Maybe we'll start a new fad! ChrisA

Steven D'Aprano

3:44 p.m.

On Thu, Mar 03, 2022 at 10:30:32PM +1100, Chris Angelico wrote:

...

On Thu, 3 Mar 2022 at 21:14, Steven D'Aprano wrote:

...
What did I say that made you think I denied the existence of filtered iteration? Was it the post where I pointed out we've been able to do filtered iteration going back to Python 1.x days?

ANYTHING can be done by composing concepts. We don't need anything more advanced than Brainf*. Why do we have better concepts? Because they do a better job of expressing abstract concepts.

They also do a better job of expressing *concrete* concepts, like addition. I believe this is a BF program to read two single digit numbers, add them, and print the result: ,>,[<+>-]<------------------------------------------------. And if you think that is nasty, you should see Malbolge. I don't think that the difference between the status quo and the proposal: # status quo for a filtered loop for item in items: if condition: block # proposal for a filtered loop for item in items if condition: block is comparable to the difference between BF and Python. How about you?

...

...
To be clear, there are lots of concepts in coding. Not all of them require their own specialised syntax. We don't have specialised syntax for a try...except block inside a loop, we use composition by putting a try...except block inside a for loop.

Composition of statements is not a bug to be fixed.

Indeed, but I'm putting the viewpoint - which a number of other people have also put - that filtered iteration DOES deserve a better way of expressing it.

Better than the three or four ways we already have? Okay. To me, comprehensions were a significant improvement over the Python 1.x status quo, because they permitted us to write list builders as expressions, which could then be embedded directly in statements or other expressions without needing temporary variables. Generators and iterators were significant improvements, because they allow us to do things we couldn't do (easily, or at all) before. Context managers were another significant improvement. Aside from saving one line and one indent level, what *significant* improvement does the proposed change give us? This is not a rhetorical question.

...

...
Nobody said that the idea of filtered looping doesn't make sense. They're only questioning whether it needs its own syntax instead of composing existing syntax.

Yes. And it keeps coming up, so I think you should probably acknowledge the fact that maybe, just maybe, this is more significant than "one newline".

I have repeatedly said that it also saves an indent level. I never bothered to mention the saving of one colon, because I thought that even for Python-Ideas that would be too trivially unimportant to mention. What else does it give us? The one-line proposal and the two line status quo express exactly the same concept: a loop with a filter. The functional programming idiom using filter() is even better at expressing that concept, at least for the case where the predicate is a function, but many people have an unfortunate aversion to f.p. idioms and so won't use it.

...

...
...
You're thinking FAR FAR too concretely about this. It's not about newlines.

Of course it is. The whole point of the proposal is to move a two line statement into a single line. Earlier in this thread, I pointed out that this proposal adds no new functionality to Python. It doesn't allow us to do anything we can't already do, or even make it easier to do it.

Literally all it saves is a newline and an indent.

No, it is not. It is expressing the concept of filtered iteration.

Which the existing idioms already do. So we have three or four ways of expressing filtered iteration, and the proposal is for another way of expressing filtered iteration which differs in that it saves a line and an indent level, and let's not forget that vitally important colon. Is there any other difference that I have missed?

...

Do you, or don't you, accept that that is a concept? One moment you say that it is a concept but you think it shouldn't get dedicated syntax,

Yes. Many concepts don't have dedicated syntax. We don't have dedicated syntax for, say, recursion. (We just use a function call.) Or for getting the length of a sequence (another function call). Or sorting.

...

then the next, you imply that it isn't even a concept,

Citation required.

...

and all we're doing is reformatting code. That is simply not the case.

Okay, now we're getting somewhere! So there is a semantic difference between the status quo and the new proposal. I'm sorry, I missed that! Mea culpa. Please take pity on me and explain what I have missed. In the status quo, we have a filtered loop written as: for item in items: if condition: block but of course you know that already :-) So how does the proposal differ? for item in items if condition: block means what, if it isn't merely a reformatting of the status quo?

...

...
...
It's about expressing programmer concepts.

Right. And composing a for-loop with a if statement expresses that concept perfectly. As does filter().

No, it doesn't.

Wait, you are saying that for item in filter(predicate, items) *doesn't* express the concept of a loop with a filter? Then what does it express? And what do you say to the poster who wrote about "filtered iteration": [quote] ... you have this option: for thing in filter(isinteresting, stuff): which actually looks good. I think this is a pretty clear indication that the idea [filtered iteration] makes sense: functional programming languages have an idiom that aligns perfectly with it [/quote] The author of that post certainly sounds like he thinks that the f.p. idiom `for item in filter(...)` expresses filtered iteration "perfectly". (At least for the case where the predicate is a function. I don't think he is too fond of lambdas.) https://mail.python.org/archives/list/python-ideas@python.org/message/JQGDLV... -- Steve

Chris Angelico

5:39 p.m.

On Fri, 4 Mar 2022 at 02:48, Steven D'Aprano wrote:

...

On Thu, Mar 03, 2022 at 10:30:32PM +1100, Chris Angelico wrote:

...
On Thu, 3 Mar 2022 at 21:14, Steven D'Aprano wrote:

...
What did I say that made you think I denied the existence of filtered iteration? Was it the post where I pointed out we've been able to do filtered iteration going back to Python 1.x days?

ANYTHING can be done by composing concepts. We don't need anything more advanced than Brainf*. Why do we have better concepts? Because they do a better job of expressing abstract concepts.

They also do a better job of expressing *concrete* concepts, like addition.

I believe this is a BF program to read two single digit numbers, add them, and print the result:

,>,[<+>-]<------------------------------------------------.

This is a BF program that expresses an abstract concept of addition. On what basis do you consider addition to be a concrete concept? Is Python's idea of addition a single machine instruction? What, in your view, makes one thing abstract and another thing concrete? I put it to you that "add two numbers" is an abstract concept. Even more so, "add two things" is definitely an abstract concept, and we have a clean way of expressing this, regardless of whether it's string concatenation, list extension, or numeric arithmetic. The BF program you show clearly demonstrates that there are concrete actions that BF is capable of, and they can be composed into the abstract idea of integer addition. Composition can do anything! Why do we need better languages? Because they are better able to express abstract concepts like "add two integers". Calling addition "concrete" because it has a simple spelling *in Python* is the Blub Paradox at its best. You are denying features that don't exist because composition can create them, and dismissing the exact same argument about features that do exist, simply because... they do exist. Addition is no more or less abstract than iteration, or than filtered iteration, or than anything else. It's not a question of "this is abstract, this is concrete". It's that pretty much everything we ever think about is an abstract concept, to be implemented by the compiler/interpreter in its own concrete way. The real question is: Which abstract concepts deserve syntax? (In fact, even BF's fundamentals could be considered abstract concepts. But doing so would require that you be willing to consider "concrete" to be the level of electrical engineering. Or domino engineering. A half-adder can be implemented using a few square meters of racing dominoes, and once you design that abstract concept, you can build a four-bit adder with a full room of dominoes. When you think on THAT scale, even a simple 'or' gate is a very abstract concept!!)

...

I don't think that the difference between the status quo and the proposal:

# status quo for a filtered loop for item in items: if condition: block

# proposal for a filtered loop for item in items if condition: block

is comparable to the difference between BF and Python. How about you?

There is a difference of degree, to be sure. But in the status quo, the programmer has to put the condition into the body, since there's no way to express it in the header. A for loop can be implemented more concretely using a while loop and iter/next. We don't work that way. Why? Because it doesn't adequately express the *concept* that we're trying to get across. Programming is about expressing the ideas that we programmers have, in ways that other programmers AND a computer will be able to understand. Please can you stop blub-paradoxing yourself by assuming that what exists is completely different from what doesn't exist?

...

...
...
To be clear, there are lots of concepts in coding. Not all of them require their own specialised syntax. We don't have specialised syntax for a try...except block inside a loop, we use composition by putting a try...except block inside a for loop.

Composition of statements is not a bug to be fixed.

Indeed, but I'm putting the viewpoint - which a number of other people have also put - that filtered iteration DOES deserve a better way of expressing it.

Better than the three or four ways we already have? Okay.

To me, comprehensions were a significant improvement over the Python 1.x status quo, because they permitted us to write list builders as expressions, which could then be embedded directly in statements or other expressions without needing temporary variables.

Generators and iterators were significant improvements, because they allow us to do things we couldn't do (easily, or at all) before.

Context managers were another significant improvement.

Aside from saving one line and one indent level, what *significant* improvement does the proposed change give us? This is not a rhetorical question.

Once again, you keep on focusing on "one line and one indent level". It's not about that. I don't know how many more ways I can word this, but *filtered iteration is an abstract concept that is very useful to express*. Stop arguing against the strawman of the indentation level, because I have never ONCE argued that I want to save lines, I have never ONCE argued that I want to save indentation levels. Please. Drop that tired line of argument.

...

...
...
Nobody said that the idea of filtered looping doesn't make sense. They're only questioning whether it needs its own syntax instead of composing existing syntax.

Yes. And it keeps coming up, so I think you should probably acknowledge the fact that maybe, just maybe, this is more significant than "one newline".

I have repeatedly said that it also saves an indent level. I never bothered to mention the saving of one colon, because I thought that even for Python-Ideas that would be too trivially unimportant to mention.

And I never argued that it's worth saving the colon either, so you're still arguing against something that isn't the point.

...

What else does it give us? The one-line proposal and the two line status quo express exactly the same concept: a loop with a filter.

The two-line version is a loop, followed by a check, followed by skipping the rest of the loop body. The proposal is a loop over part of a list. This is not the same thing.

...

The functional programming idiom using filter() is even better at expressing that concept, at least for the case where the predicate is a function, but many people have an unfortunate aversion to f.p. idioms and so won't use it.

Yes, so where there predicate IS a function, great! But the versions where it's not a function are much uglier. There are several, on an ugliness spectrum, and the ones that aren't quite as ugly tend to be quite inefficient to run, so people will tend to shy away from them. I wouldn't mind that if they were really beautiful, but they aren't beautiful enough to ignore performance completely.

...

...
...
...
You're thinking FAR FAR too concretely about this. It's not about newlines.

Of course it is. The whole point of the proposal is to move a two line statement into a single line. Earlier in this thread, I pointed out that this proposal adds no new functionality to Python. It doesn't allow us to do anything we can't already do, or even make it easier to do it.

Literally all it saves is a newline and an indent.

No, it is not. It is expressing the concept of filtered iteration.

Which the existing idioms already do. So we have three or four ways of expressing filtered iteration, and the proposal is for another way of expressing filtered iteration which differs in that it saves a line and an indent level, and let's not forget that vitally important colon.

Is there any other difference that I have missed?

Filtered iteration is not "loop over everything, then inside the loop body, do a check, and possibly skip the rest of the body". Filtered iteration is "loop over some of the list".

...

...
Do you, or don't you, accept that that is a concept? One moment you say that it is a concept but you think it shouldn't get dedicated syntax,

Yes. Many concepts don't have dedicated syntax.

We don't have dedicated syntax for, say, recursion. (We just use a function call.) Or for getting the length of a sequence (another function call). Or sorting.

...
then the next, you imply that it isn't even a concept,

Citation required.

You're arguing that it is exactly the same as the composition of multiple statements.

...

...
and all we're doing is reformatting code. That is simply not the case.

Okay, now we're getting somewhere! So there is a semantic difference between the status quo and the new proposal. I'm sorry, I missed that! Mea culpa. Please take pity on me and explain what I have missed.

In the status quo, we have a filtered loop written as:

for item in items: if condition: block

but of course you know that already :-)

So how does the proposal differ?

for item in items if condition: block

means what, if it isn't merely a reformatting of the status quo?

Well, if "a + b" is just a reformatting of the BF code you posted, then yes, it's merely reformatting.

...

...
...
...
It's about expressing programmer concepts.

Right. And composing a for-loop with a if statement expresses that concept perfectly. As does filter().

No, it doesn't.

Wait, you are saying that

for item in filter(predicate, items)

*doesn't* express the concept of a loop with a filter?

Then what does it express?

This does. The trouble is that it ONLY expresses that cleanly in the case where the predicate is already a function. Otherwise it gets cluttered with the mess of lambda functions, and it's a lot less clear.

...

And what do you say to the poster who wrote about "filtered iteration":

[quote] ... you have this option:

for thing in filter(isinteresting, stuff):

which actually looks good. I think this is a pretty clear indication that the idea [filtered iteration] makes sense: functional programming languages have an idiom that aligns perfectly with it [/quote]

The author of that post certainly sounds like he thinks that the f.p. idiom `for item in filter(...)` expresses filtered iteration "perfectly".

(At least for the case where the predicate is a function. I don't think he is too fond of lambdas.)

Exactly. Try the versions where that isn't the case, and you'll see why. We have [x + 1 for x in stuff if x % 3] without lambda functions. We could have just used [x + 1 for x in filter(lambda x: x % 3, stuff)] but that's just ugly. There is a huge difference between filter() with an existing, and usefully-named, predicate, and filter() with a lambda function (or worse, a single-use global function whose entire purpose is that filtration). Lambda functions are extremely useful when they're needed. They are NOT arbitrary expressions that can be inserted anywhere. Blub Paradox has you thoroughly in its grip. ChrisA

Steven D'Aprano

4 Mar 4 Mar

12:54 p.m.

This post is rather off topic. If you don't want to get bogged down in philosophical arguments, you might want to skip this one and come back to my next reply, which I promise will be more on top. On Fri, Mar 04, 2022 at 04:39:12AM +1100, Chris Angelico wrote:

...

...
They also do a better job of expressing *concrete* concepts, like addition.

I believe this is a BF program to read two single digit numbers, add them, and print the result:

,>,[<+>-]<------------------------------------------------.

This is a BF program that expresses an abstract concept of addition.

No, it expresses the concept of addition of a concrete data type, integer, not just in the abstract but specifically of one digit decimal integers. Not numbers in general, or operator overloading, or addition of matrices, vectors. It is not the abstract commutative, associative binary operator represented by `+` in abstract algebra. It is the concrete integer addition. https://en.wikipedia.org/wiki/Algebraic_structure Anyway, let's not get too bogged down over semantic disagreements about what is "concrete" and "abstract", since to some degree these are both *relative* terms. Addition of integers is concrete compared to addition in the abstract, and abstract compared to addition in the sense of adding two apples to three apples and getting five apples.

...

On what basis do you consider addition to be a concrete concept?

Baby chickens of only a few days old, with no training, are instinctively capable of simple addition and subtraction. That's pretty far from the level of abstraction we find in abstract algebra, or in Monads (for example). https://www.quantamagazine.org/animals-can-count-and-use-zero-how-far-does-t... https://wiki.haskell.org/All_About_Monads

...

Is Python's idea of addition a single machine instruction?

Yes, it is the BINARY_ADD machine instruction in the Python virtual machine.

...

...
...
dis.dis('x+y') 1 0 LOAD_NAME 0 (x) 2 LOAD_NAME 1 (y) 4 BINARY_ADD 6 RETURN_VALUE

Maybe some day, someone will create a physical Python Machine, like people created physical Lisp Machines and Forth Machines.

...

What, in your view, makes one thing abstract and another thing concrete?

I think that this discussion is already getting too far off-topic, so I'm going to decline to answer beyond what I have already said above. I will respond to the rest of your post shortly. -- Steve

Christopher Barker

5:34 p.m.

...

...
In the status quo, we have a filtered loop written as:

for item in items: if condition: block

The thing is, if block is more than a couple lines, you need to look down to see if there’s an else or any number of elif blocks down there— so this isn’t a clear expression of a filtered iteration after all. Which is why I suggested that: for item in items: if not condition: continue block Would be the more direct expression of "filtered iteration" -- still not a whole lot of code, but a bit more awkward. The other thing that I think is relevant is that comprehensions already have a filtering option -- so while new syntax technically. this proposal would be familiar and create a bit more consistency in the language. Another half formed idea: Comprehension can express map-like behavior or map and filter behavior, but no way to filter without a do nothing map. e.g if you want to only filter, you need to do: (item for item in items if condition) I wonder if there's a way to not need to the "item for item" wasted text: (Item in items if condition) and [item in items if condition] maybe?? it's not legal syntax now. That would then allow: for item in (item in items if condition): block which is almost what's being asked for here. But then the repetition again. Which leads us back to the OP’s proposal. I’m actually liking that more now …. -CHB

...

...
but of course you know that already :-)

So how does the proposal differ?

for item in items if condition: block

means what, if it isn't merely a reformatting of the status quo?

Well, if "a + b" is just a reformatting of the BF code you posted, then yes, it's merely reformatting.

...
...
...
...
It's about expressing programmer concepts.

Right. And composing a for-loop with a if statement expresses that concept perfectly. As does filter().

No, it doesn't.

Wait, you are saying that

for item in filter(predicate, items)

*doesn't* express the concept of a loop with a filter?

Then what does it express?

This does. The trouble is that it ONLY expresses that cleanly in the case where the predicate is already a function. Otherwise it gets cluttered with the mess of lambda functions, and it's a lot less clear.

...
And what do you say to the poster who wrote about "filtered iteration":

[quote] ... you have this option:

for thing in filter(isinteresting, stuff):

which actually looks good. I think this is a pretty clear indication that the idea [filtered iteration] makes sense: functional programming languages have an idiom that aligns perfectly with it [/quote]

The author of that post certainly sounds like he thinks that the f.p. idiom `for item in filter(...)` expresses filtered iteration "perfectly".

(At least for the case where the predicate is a function. I don't think he is too fond of lambdas.)

Exactly. Try the versions where that isn't the case, and you'll see why.

We have [x + 1 for x in stuff if x % 3] without lambda functions. We could have just used [x + 1 for x in filter(lambda x: x % 3, stuff)] but that's just ugly. There is a huge difference between filter() with an existing, and usefully-named, predicate, and filter() with a lambda function (or worse, a single-use global function whose entire purpose is that filtration).

Lambda functions are extremely useful when they're needed. They are NOT arbitrary expressions that can be inserted anywhere.

Blub Paradox has you thoroughly in its grip.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SRW66Z... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Chris Angelico

5:53 p.m.

On Sat, 5 Mar 2022 at 04:34, Christopher Barker wrote:

...

...
...
In the status quo, we have a filtered loop written as:

for item in items: if condition: block

The thing is, if block is more than a couple lines, you need to look down to see if there’s an else or any number of elif blocks down there— so this isn’t a clear expression of a filtered iteration after all. Which is why I suggested that:

for item in items: if not condition: continue block

Would be the more direct expression of "filtered iteration" -- still not a whole lot of code, but a bit more awkward.

The other thing that I think is relevant is that comprehensions already have a filtering option -- so while new syntax technically. this proposal would be familiar and create a bit more consistency in the language.

Another half formed idea:

Comprehension can express map-like behavior or map and filter behavior, but no way to filter without a do nothing map.

e.g if you want to only filter, you need to do:

(item for item in items if condition)

I wonder if there's a way to not need to the "item for item" wasted text:

(Item in items if condition) and [item in items if condition] maybe??

it's not legal syntax now. That would then allow:

Tempting, very tempting. I've been in the situation of wanting this before. Without the word "for", though, it doesn't say that it's a loop, and "item in items" is a condition, "if" can be used as part of an if/else expression, and overall, this is dangerously close to valid syntax. What if omitting the value were to automatically use the loop iterator? [for item in items if condition] But at this point it's not really saving much. ChrisA

Stephen J. Turnbull

5 Mar 5 Mar

11:22 a.m.

Chris Angelico writes:

...

What Python needs is not a way to cram more onto one line. What Python needs is a way to express an abstract concept: "iterate over the interesting parts of this collection".

Python has one, and you've already mentioned it: for thing in (x for x in this_collection if is_interesting(x)): It's noticably verbose, but it's an exact translation of your statement of the abstract concept above. It has all the benefits of the proposed syntax except compactness[1]. I'd have to see a fair number of examples to decide whether I think saving len(" (x for x in)") == 13 characters allows enough additional filtered iterations to be comfortably expressed on one line to be worth new syntax. I suspect that the redundancy and grouping parenthesis aspects won't make much difference for anything more complicated than the above, but they might. Steve Footnotes: [1] By "compactness" I mean all of brevity, DRY, and absence of grouping parentheses. I don't know if that's a useful term to summarize all of those, but let me throw it out there.

Chris Angelico

12:10 p.m.

On Sat, 5 Mar 2022 at 22:22, Stephen J. Turnbull wrote:

...

Chris Angelico writes:

...
What Python needs is not a way to cram more onto one line. What Python needs is a way to express an abstract concept: "iterate over the interesting parts of this collection".

Python has one, and you've already mentioned it:

for thing in (x for x in this_collection if is_interesting(x)):

It's noticably verbose, but it's an exact translation of your statement of the abstract concept above. It has all the benefits of the proposed syntax except compactness[1].

It is extremely verbose, considering that the filtering part of a comprehension uses the same variable name as iteration, doesn't need anything to be repeated, and is just another clause. A simple representation of that in a statement loop would be "for thing in this_collection if is_interesting(thing):", which doesn't repeat itself at all (the variable name "thing" is kinda essential to the concept of filtration here, so I don't count that); and repetition isn't simply about number of characters on the line, it's about reducing the potential for errors. (Plus, the genexp adds a significant amount of run-time overhead.) So you're right, I stand (partly) corrected: there IS a very clunky way to spell this concept. What Python needs is a non-clunky way to express this. ChrisA

Paul Moore

12:32 p.m.

On Sat, 5 Mar 2022 at 12:12, Chris Angelico wrote:

...

On Sat, 5 Mar 2022 at 22:22, Stephen J. Turnbull wrote:

...
Python has one, and you've already mentioned it:

for thing in (x for x in this_collection if is_interesting(x)):

It's noticably verbose, but it's an exact translation of your statement of the abstract concept above. It has all the benefits of the proposed syntax except compactness[1].

It is extremely verbose, considering that the filtering part of a comprehension uses the same variable name as iteration, doesn't need anything to be repeated, and is just another clause. A simple representation of that in a statement loop would be "for thing in this_collection if is_interesting(thing):", which doesn't repeat itself at all (the variable name "thing" is kinda essential to the concept of filtration here, so I don't count that); and repetition isn't simply about number of characters on the line, it's about reducing the potential for errors. (Plus, the genexp adds a significant amount of run-time overhead.)

So you're right, I stand (partly) corrected: there IS a very clunky way to spell this concept.

What Python needs is a non-clunky way to express this.

for thing in filter(is_interesting, this_collection): ... That seems pretty non-clunky. Is the issue here that "filter" is not sufficiently well-known? Or that you don't want to name the is_interesting function, and lambdas are "too clunky"? This feels like another case where the general dislike of lambda results in people wanting special-case syntax so they can avoid either writing a throwaway function, or using lambda. If we had "placeholder" expressions (see, for example https://pypi.org/project/placeholder/) so we could do things like from placeholder import _ for thing in filter(_%3==0, the_list): ... would that be sufficiently "non-clunky"? There are many ways of achieving this sort of result. Clearly, from the fact that this request comes up repeatedly, there's *something* unsatisfying about all of them, but I'm not entirely clear what particular problem is uniquely solved by new "for x in collection if condition" syntax, and not by any of the other possibilities? Personally, I don't mind having the if on a separate line, I find the generator expression tolerable but a bit verbose, and I'm glad "filter" and the placeholder library exist in case I need them, but I've never really found the existing options sufficiently annoying that I've wanted a for...if statement. Paul

Chris Angelico

12:41 p.m.

On Sat, 5 Mar 2022 at 23:32, Paul Moore wrote:

...

On Sat, 5 Mar 2022 at 12:12, Chris Angelico wrote:

...
On Sat, 5 Mar 2022 at 22:22, Stephen J. Turnbull wrote:

...
Python has one, and you've already mentioned it:

for thing in (x for x in this_collection if is_interesting(x)):

It's noticably verbose, but it's an exact translation of your statement of the abstract concept above. It has all the benefits of the proposed syntax except compactness[1].

It is extremely verbose, considering that the filtering part of a comprehension uses the same variable name as iteration, doesn't need anything to be repeated, and is just another clause. A simple representation of that in a statement loop would be "for thing in this_collection if is_interesting(thing):", which doesn't repeat itself at all (the variable name "thing" is kinda essential to the concept of filtration here, so I don't count that); and repetition isn't simply about number of characters on the line, it's about reducing the potential for errors. (Plus, the genexp adds a significant amount of run-time overhead.)

So you're right, I stand (partly) corrected: there IS a very clunky way to spell this concept.

What Python needs is a non-clunky way to express this.

for thing in filter(is_interesting, this_collection): ...

That seems pretty non-clunky. Is the issue here that "filter" is not sufficiently well-known? Or that you don't want to name the is_interesting function, and lambdas are "too clunky"? This feels like another case where the general dislike of lambda results in people wanting special-case syntax so they can avoid either writing a throwaway function, or using lambda.

The throwaway function is a *terrible* idea for a lot of situations, because it puts the condition completely out-of-line. You have to go dig elsewhere to find out the meaning of the filter. A lambda function can work, but is about as clunky as the genexp. Using filter() is only non-clunky in the specific situation where a function already exists to do the filtration, and in my personal experience, that's been quite rare.

...

If we had "placeholder" expressions (see, for example https://pypi.org/project/placeholder/) so we could do things like

from placeholder import _ for thing in filter(_%3==0, the_list): ...

would that be sufficiently "non-clunky"?

That is something I could get behind. I don't like the underscore, since that usually means "meaningless", and would have to be carefully managed to avoid shadowing; but that is definitely a possibility. Does it work if you need to use the underscore twice? For instance, what if you want to find long-running jobs, where "_.end - _.start > 30" ? (This also still has the performance cost of filter and a lambda function, which is a lot more than just having a bit of extra code as part of the loop. But that's less significant.)

...

There are many ways of achieving this sort of result. Clearly, from the fact that this request comes up repeatedly, there's *something* unsatisfying about all of them, but I'm not entirely clear what particular problem is uniquely solved by new "for x in collection if condition" syntax, and not by any of the other possibilities?

They are all clunky except in very specific circumstances, like "iterate over the non-empty elements" or something. They don't generalize well.

...

Personally, I don't mind having the if on a separate line, I find the generator expression tolerable but a bit verbose, and I'm glad "filter" and the placeholder library exist in case I need them, but I've never really found the existing options sufficiently annoying that I've wanted a for...if statement.

And clearly a number of other people DO mind having it on a separate line, because it's putting code in the body that belongs in the header. However, as I found out by writing PEP 671, there are enough people who focus on the concrete that it's never going to happen. All you have to do is keep on arguing against straw-men and eventually people get weary of arguing the same unanswered arguments again and again. The Blub Paradox is strong on this list. ChrisA

Paul Moore

1:48 p.m.

On Sat, 5 Mar 2022 at 12:43, Chris Angelico wrote:

...

The throwaway function is a *terrible* idea for a lot of situations, because it puts the condition completely out-of-line. You have to go dig elsewhere to find out the meaning of the filter. A lambda function can work, but is about as clunky as the genexp. Using filter() is only non-clunky in the specific situation where a function already exists to do the filtration, and in my personal experience, that's been quite rare.

*shrug* I'm not sure I'd agree with you that throwaway functions are quite as terrible as you claim, but we're into the area of personal preference, so I'm not going to debate that one. But equally "let's have new syntax because some people find the current options unattractive" is a difficult argument to sell. So I'm still inclined to ask for something more objective.

...

...
If we had "placeholder" expressions (see, for example https://pypi.org/project/placeholder/) so we could do things like

from placeholder import _ for thing in filter(_%3==0, the_list): ...

would that be sufficiently "non-clunky"?

That is something I could get behind. I don't like the underscore, since that usually means "meaningless", and would have to be carefully managed to avoid shadowing; but that is definitely a possibility.

I've not used the library extensively, and I'd encourage you to read the docs if you want details, but you can of course do "from placeholder import _ as X" (or whatever placeholder you prefer).

...

Does it work if you need to use the underscore twice? For instance, what if you want to find long-running jobs, where "_.end - _.start > 30" ?

Yes. Again, check the docs for all the details, but this does work.

...

(This also still has the performance cost of filter and a lambda function, which is a lot more than just having a bit of extra code as part of the loop. But that's less significant.)

From the library docs, "Every effort is made to optimize the placeholder instance. It's 20-40x faster than similar libraries on PyPI" and "Performance should generally be comparable to inlined expressions, and faster than lambda". I've not done measurements myself, but I did do some investigation in the past, and the claims seem realistic.

...

...
There are many ways of achieving this sort of result. Clearly, from the fact that this request comes up repeatedly, there's *something* unsatisfying about all of them, but I'm not entirely clear what particular problem is uniquely solved by new "for x in collection if condition" syntax, and not by any of the other possibilities?

They are all clunky except in very specific circumstances, like "iterate over the non-empty elements" or something. They don't generalize well.

I'm not sure how a for-if statement generalises any better than for ...: if not (condition): break So "doesn't generalize well" isn't the whole story here, I suspect.

...

...
Personally, I don't mind having the if on a separate line, I find the generator expression tolerable but a bit verbose, and I'm glad "filter" and the placeholder library exist in case I need them, but I've never really found the existing options sufficiently annoying that I've wanted a for...if statement.

And clearly a number of other people DO mind having it on a separate line, because it's putting code in the body that belongs in the header.

I don't disagree, and if that's the argument, then I'm fine with that.

...

However, as I found out by writing PEP 671, there are enough people who focus on the concrete that it's never going to happen. All you have to do is keep on arguing against straw-men and eventually people get weary of arguing the same unanswered arguments again and again.

If someone were to implement this feature, I wouldn't object. In fact, I'd probably use it (not often, maybe, but I wouldn't actively avoid it). For comparison, though, I've yet to use an assignment expression, so don't put too much weight on "Paul would use it" as a measure of acceptability ;-) I think the problem here is that getting enthusiastic community support is always going to be hard, and as Python's user base gets bigger, any group of supporters looks smaller and smaller in comparison. "Everyone likes this, let's implement it" is probably the wrong way of getting language changes through these days, TBH (if indeed it ever was the right way...) But the PEP process is perceived (and presented) as a process of getting consensus, even though it isn't, really. Paul

Christopher Barker

6 Mar 6 Mar

1:55 a.m.

On Sat, Mar 5, 2022 at 4:33 AM Paul Moore wrote:

...

for thing in filter(is_interesting, this_collection): ...

That seems pretty non-clunky. Is the issue here that "filter" is not sufficiently well-known? Or that you don't want to name the is_interesting function, and lambdas are "too clunky"? This feels like another case where the general dislike of lambda results in people wanting special-case syntax so they can avoid either writing a throwaway function, or using lambda.

Speaking for me -- it's not a dislike of lambda -- it's a dislike of the map / filter approach vs comprehensions in general. I guess you could say I have a dislike for lambda -- but really it's a dislike for having to create a function when an expression will do ;-) -- and lamda only supports expressions, so it will always do. This entire conversation -- at least from the side of the skeptics -- seems to be ignoring comprehensions: If we all thought that map and filter were perfectly adequate, comprehensions never would have been added at all. And not only were comprehensions added, but both a looping and filtering mechanism was added at the same time. Let's imagine for the moment that the filtering was not yet added -- then say the way to run a filtered loop in a comprehension would be: [expr for thing in filter(lambda thing: expr, an_iterable)] Would anyone really largure that there would be no point in adding the filter explicitly to get to (what we do have now): [expr1 for thing in an_iterable if expr2] The other relevant point is that adding an `if` to the for loop is new syntax, yes, but it mirrors the comprehension syntax very closely -- so is not nearly the cognitive load that most syntax additions are. In fact, I imagine if newbies were to learn about comprehensions first, they'd be surprised that there was no if allowed in for loops. Yes, this is a fairly minor improvement, but to me it's not adding syntax to "save a line" "or not have to use lambdas" -- but rather it's adding syntax to make it possible to express a particular concept in a way that is clear, obvious, and more consistent with the rest of the language (i.e. comprehensions). I was +0, but after this discussion, now +1. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Paul Moore

11:53 a.m.

On Sun, 6 Mar 2022 at 01:55, Christopher Barker wrote:

...

On Sat, Mar 5, 2022 at 4:33 AM Paul Moore wrote:

...
for thing in filter(is_interesting, this_collection): ...

That seems pretty non-clunky. Is the issue here that "filter" is not sufficiently well-known? Or that you don't want to name the is_interesting function, and lambdas are "too clunky"? This feels like another case where the general dislike of lambda results in people wanting special-case syntax so they can avoid either writing a throwaway function, or using lambda.

Speaking for me -- it's not a dislike of lambda -- it's a dislike of the map / filter approach vs comprehensions in general. I guess you could say I have a dislike for lambda -- but really it's a dislike for having to create a function when an expression will do ;-) -- and lamda only supports expressions, so it will always do.

This entire conversation -- at least from the side of the skeptics -- seems to be ignoring comprehensions:

Personally, I'm certainly not ignoring comprehensions. "for thing in (x for x in collection if is_interesting(x))" uses comprehensions just fine, if you don't like the verbosity of "x for x in", then that's an issue with comprehensions, not a reason why comprehensions don't address this issue, surely? (Personally, I find the repetitiveness of "x for x in" mildly annoying, but not enough to put me off comprehensions[1]). I'm offering map/filter precisely *because* the people arguing for this construct don't seem to like the idea of using a comprehension. So IMO it's the supporters of the idea who are ignoring comprehensions (or don't want to use them for some unclear reason that seems to boil down to "it doesn't look nice"). Syntactically, the proposal is fundamentally just removing "x for x in" and a set of parentheses, in the special case where you don't also want to do a calculation on the item - you need to go back to the comprehension form if you need "for thing in (x.interesting_bit for x in collection if is_interesting(x))". Or, of course, you split it up: for x in collection: if not is_interesting(x): break thing = x.interesting_bit ... But maybe I just differ in how much I feel comfortable cramming into a single line...?

...

If we all thought that map and filter were perfectly adequate, comprehensions never would have been added at all. And not only were comprehensions added, but both a looping and filtering mechanism was added at the same time. Let's imagine for the moment that the filtering was not yet added -- then say the way to run a filtered loop in a comprehension would be:

[expr for thing in filter(lambda thing: expr, an_iterable)]

Would anyone really largure that there would be no point in adding the filter explicitly to get to (what we do have now):

[expr1 for thing in an_iterable if expr2]

The other relevant point is that adding an `if` to the for loop is new syntax, yes, but it mirrors the comprehension syntax very closely -- so is not nearly the cognitive load that most syntax additions are.

Just because comprehensions are good, doesn't mean *everything* needs to look like them. Comprehensions are good because they are *expressions*, not because they are one-liners. For statements aren't expressions, they are statements, so having the filter be an extra statement isn't the problem that it is for a comprehension.

...

In fact, I imagine if newbies were to learn about comprehensions first, they'd be surprised that there was no if allowed in for loops.

Maybe, if that were a compelling argument, why don't *other* languages with comprehensions also have for loops with an if clause?

...

Yes, this is a fairly minor improvement, but to me it's not adding syntax to "save a line" "or not have to use lambdas" -- but rather it's adding syntax to make it possible to express a particular concept in a way that is clear, obvious, and more consistent with the rest of the language (i.e. comprehensions).

I was +0, but after this discussion, now +1.

I was, and remain, indifferent to the idea. I'm not against, but I don't see the point of all the energy being spent trying to persuade people this should be added. If anything, trying to explain why I find the arguments given in favour of the proposal weak, is simply making me less interested in the proposal - so I think the main result of this discussion is likely to be just to push people to more entrenched positions, and not change many minds. Which in general, is quite likely why discussions like this on python-ideas are so frustrating. IMO, if someone is interested enough, they can write a PEP. If the arguments are strong enough, there's no need for everyone on python-ideas to agree with the proposal. And conversely, if the key argument is just that everyone on python-ideas thought it was a good idea, it's still probably going to get rejected because PEPs aren't simply popularity contests... Of course, if people just want to debate, that's fine but it won't result in anything happening (except the previously noted entrenching of positions) ;-) Paul [1] Straw man proposal for removing this redundancy: allow leaving off the initial expression if it's the same as the for-variable, so (for x in collection if condition) is the same as (x for x in collection if condition). It saves almost nothing in terms of typing, so I'm unconvinced it's worth it, but maybe it'll satisfy the "it's too verbose" crowd...

Christopher Barker

9:40 p.m.

...

Personally, I'm certainly not ignoring comprehensions. "for thing in (x for x in collection if is_interesting(x))" uses comprehensions just fine, if you don't like the verbosity of "x for x in", then that's an issue with comprehensions, not a reason why comprehensions don't address this issue, surely? (Personally, I find the repetitiveness of "x for x in" mildly annoying, but not enough to put me off comprehensions[1]).

Earlier on the thread, I made a similar point that it would be nice to have a way to filter without the redundant for x in x. Though I can’t think of a really good way to express it. But as for filtered for loops: "for thing in (x for x in collection if is_interesting(x))" It not so much the extraneous “x for x” as the duplicated “for thing in” that bugs me. I’m curious— to the skeptics: do you think that The above (or a later if block) is just as good or better than for thing in collection if isinteresting(thing): Or just that it’s not worth it to make a change. But the other part of why I think comprehensions are relevant is that introducing anf if to the for statement is not brand new syntax - it would be allowing existing syntax in a new context that is highly related. And as for documentation and all that, it’s hard to imagine very many people not understanding what it means. Do I care enough to write a PEP? No. So this, like many other small ideas, will probably die on the vine. Oh well. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Paul Moore

10:33 p.m.

On Sun, 6 Mar 2022 at 21:41, Christopher Barker wrote:

...

...
Personally, I'm certainly not ignoring comprehensions. "for thing in (x for x in collection if is_interesting(x))" uses comprehensions just fine, if you don't like the verbosity of "x for x in", then that's an issue with comprehensions, not a reason why comprehensions don't address this issue, surely? (Personally, I find the repetitiveness of "x for x in" mildly annoying, but not enough to put me off comprehensions[1]).

Earlier on the thread, I made a similar point that it would be nice to have a way to filter without the redundant for x in x. Though I can’t think of a really good way to express it. But as for filtered for loops:

"for thing in (x for x in collection if is_interesting(x))"

It not so much the extraneous “x for x” as the duplicated “for thing in” that bugs me.

I’m curious— to the skeptics: do you think that The above (or a later if block) is just as good or better than

for thing in collection if isinteresting(thing):

Or just that it’s not worth it to make a change.

I think that for thing in collection: if not isinteresting(thing): break ... is at least as good as for thing in collection if isinteresting(thing): ... and better in many ways (most importantly for me, it works in older versions of Python, which is important for libraries that support multiple versions). I think that for thing in (x for x in collection if isinteresting(x)): ... is only marginally worse, and not sufficiently worse to justify new syntax.

...

But the other part of why I think comprehensions are relevant is that introducing anf if to the for statement is not brand new syntax - it would be allowing existing syntax in a new context that is highly related.

Agreed, this makes the impact of new syntax smaller. But I'm not really worried about that. It's not that it's a big change, just that it *is* a change, and there's a minimal cost for *any* change to the language definition. So someone has to care enough to pay that cost (in terms of writing a PEP, an implementation, and documentation, and in terms of going through the process of getting the change accepted).

...

And as for documentation and all that, it’s hard to imagine very many people not understanding what it means.

Do I care enough to write a PEP? No. So this, like many other small ideas, will probably die on the vine.

Yes, this is the real problem. It's simply not compelling enough, even for supporters of the idea, for them to do the necessary work to make it happen.

...

Oh well.

Indeed. Paul

Chris Angelico

10:43 p.m.

On Mon, 7 Mar 2022 at 09:33, Paul Moore wrote:

...

...
Do I care enough to write a PEP? No. So this, like many other small ideas, will probably die on the vine.

Yes, this is the real problem. It's simply not compelling enough, even for supporters of the idea, for them to do the necessary work to make it happen.

It is indeed a problem. When a good idea dies because it is simply too hard to get through the hassles of pushing it to a decision, there is a fundamental problem. It's good to have a bit of inertia, so that status quo gets maintained, but at the moment, the extent to which ideas get shot down makes it look as if this list is python-idea-killing. This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig. ChrisA

Paul Moore

7 Mar 7 Mar

10:32 a.m.

On Sun, 6 Mar 2022 at 22:43, Chris Angelico wrote:

...

On Mon, 7 Mar 2022 at 09:33, Paul Moore wrote:

...
...
Do I care enough to write a PEP? No. So this, like many other small ideas, will probably die on the vine.

Yes, this is the real problem. It's simply not compelling enough, even for supporters of the idea, for them to do the necessary work to make it happen.

It is indeed a problem. When a good idea dies because it is simply too hard to get through the hassles of pushing it to a decision, there is a fundamental problem. It's good to have a bit of inertia, so that status quo gets maintained, but at the moment, the extent to which ideas get shot down makes it look as if this list is python-idea-killing.

I agree - there are a lot of people here who will strongly defend the status quo, as well as a lot of people who simply like to argue for arguing's sake. But it's worth remembering that "survive python-ideas" is *not* part of the PEP process. And it's *also* worth remembering that Python is now so big, and so popular, that change is far more costly than it used to be.

...

This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

What I take away from this is that if you have a good idea, you don't *need* to put it through python-ideas, and if you do, getting shot down on python-ideas doesn't mean your idea is dead. But it *does* mean that if you take your idea to python-ideas, you have to accept that not everyone will like it, and be prepared to defend/justify it in *spite* of that. That's (in theory) OK, because it encourages you to focus on writing a PEP that contains objective reasons why your proposal is worthwhile. But it's also bad, because it makes python-ideas a hostile environment that puts new people off, and burns long-time contributors out. The ideas that I see failing here are often "I think X would be neat", or "We should do Y because it's like X which already exists". Those may not be bad ideas, but they need fleshing out. A PEP that says "a bunch of people on python-ideas things X would be neat" is not going to get accepted any more than one that just says that you think it's a neat idea. The *tone* on python-ideas is (unnecessarily) hostile, but I'd like to assume that the *intent* is helpful. For example, "why do you need X, we can already do it in this way" can be read as "your idea isn't needed", but it can also be read as "you need to think about why your idea is better than the current way, and be able to articulate that clearly and persuasively (for the PEP at least)". Personally, I *always* intend my responses to be taken as helpful, even if I sometimes get derailed into side-arguments about details where I'm not focused on the main proposal, and I then become more confrontational (I apologise that this happens, but I'm only human ;-)). Maybe things would be better if python-ideas were more positive, encouraging, and supportive. But I don't know how to make that happen - people didn't sign up here to mentor potential PEP authors, after all. Paul

Stephen J. Turnbull

12:53 p.m.

Chris Angelico writes:

...

It's good to have a bit of inertia, so that status quo gets maintained, but at the moment, the extent to which ideas get shot down makes it look as if this list is python-idea-killing. This keeps happening.

I don't see lots of *good* ideas for syntax changes here. Most of the ideas that show up, like filtered iteration, are *really* marginal. They're marginal in the number of people who think they're really an improvement, and even for those folks they're typically "nice to haves" at best. And then somebody comes along and says "ok, I'll overcome my anti-syntax inertia if you show me that this could actually be used somewhat frequently", but that's apparently too much effort. Sorry, but I'm perfectly happy to see those ideas go away at that point. But they don't have to if the proponent doesn't want them to. As Paul points out this list can't approve *or* kill an idea. This list is supposed to be for *improving* ideas before submitting them for a decision. If the discussion leads to impasse, and the proponent still thinks it's a good idea, they can, and should, either request judgment from the SC or directly submit a merge request if it's the kind of thing that only needs one core dev. Personally, I'm pretty conservative. I'm pretty happy with the ideas that get through the process, and with those that don't. Since you're not -- you think more improvements should be accepted -- I think it's mostly on you and those who feel as you do to help people who have good ideas to recognize when they've reached impasse and more to a forum where a decision will be made. (Note that I'm not asking anyone to do this for their own ideas; that's really hard.) I don't know if this is the kind of thing that can be documented, but if so that might help.

...

All the successful ideas seem to happen elsewhere, notably on typing-sig.

Typing is a poor example. typing-sig has done a good job of keeping its promise to use only existing syntax. The only typing-inspired syntax change I can remember is variable annotations, and that was a really big deal for typing. So typing can basically make decisions within its own community, independently of this list, and often the Steering Council as well. It's also a much younger than Python itself; of course it's going to progress faster. Steve

Brendan Barnwell

6:48 p.m.

On 2022-03-06 14:43, Chris Angelico wrote:

...

This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

You seem to see that as a positive thing, but I would be happier if fewer typing-related changes made it in. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

Chris Angelico

11:32 p.m.

On Tue, 8 Mar 2022 at 10:20, Brendan Barnwell wrote:

...

On 2022-03-06 14:43, Chris Angelico wrote:

...
This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

You seem to see that as a positive thing, but I would be happier if fewer typing-related changes made it in.

I'm not sure whether it's positive or not. All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames, no matter how good or bad. I do NOT believe that every idea on typing-sig is good and that every idea on python-ideas is bad, so what is it that makes typing-sig actually successful in refining ideas into usable form that python-ideas is failing at? ChrisA

Jelle Zijlstra

11:39 p.m.

El lun, 7 mar 2022 a las 15:35, Chris Angelico () escribió:

...

On Tue, 8 Mar 2022 at 10:20, Brendan Barnwell wrote:

...
On 2022-03-06 14:43, Chris Angelico wrote:

...
This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

You seem to see that as a positive thing, but I would be happier

if

...
fewer typing-related changes made it in.

I'm not sure whether it's positive or not. All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames, no matter how good or bad. I do NOT believe that every idea on typing-sig is good and that every idea on python-ideas is bad, so what is it that makes typing-sig actually successful in refining ideas into usable form that python-ideas is failing at?

An important point here is that most typing changes aren't language changes, but just new things added to typing.py. Getting a library function changed or added is a lot easier than adding new syntax. Besides, typing is much younger than the language, so there is more remaining opportunity for useful additions. And there are plenty of ideas that get proposed around typing that don't make it to PEPs.

...

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OP6NSG... Code of Conduct: http://python.org/psf/codeofconduct/

Chris Angelico

11:43 p.m.

On Tue, 8 Mar 2022 at 10:39, Jelle Zijlstra wrote:

...

El lun, 7 mar 2022 a las 15:35, Chris Angelico () escribió:

...
On Tue, 8 Mar 2022 at 10:20, Brendan Barnwell wrote:

...
On 2022-03-06 14:43, Chris Angelico wrote:

...
This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

You seem to see that as a positive thing, but I would be happier if fewer typing-related changes made it in.

I'm not sure whether it's positive or not. All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames, no matter how good or bad. I do NOT believe that every idea on typing-sig is good and that every idea on python-ideas is bad, so what is it that makes typing-sig actually successful in refining ideas into usable form that python-ideas is failing at?

An important point here is that most typing changes aren't language changes, but just new things added to typing.py. Getting a library function changed or added is a lot easier than adding new syntax. Besides, typing is much younger than the language, so there is more remaining opportunity for useful additions. And there are plenty of ideas that get proposed around typing that don't make it to PEPs.

Not ALL typing changes are just new things in typing.py, so that doesn't cover everything. And yes, I am sure that a lot of things get proposed and not implemented - my point is that typing-sig is successfully finding the good ideas and refining them into actual code, but python-ideas is 100% shooting ideas to pieces. ChrisA

Christopher Barker

8 Mar 8 Mar

3:55 a.m.

On Mon, Mar 7, 2022 at 3:43 PM Chris Angelico wrote:

...

On Tue, 8 Mar 2022 at 10:39, Jelle Zijlstra wrote:

...
El lun, 7 mar 2022 a las 15:35, Chris Angelico ()

escribió:

...
but python-ideas is 100% shooting ideas to pieces.

100% really? Maybe my sense of time is blurred, but some ideas do make it through. And There are a couple that got at least modest support, but none of us found the time/energy to write a PEP — which is a pretty darn heavy lift. -CHB But one difference is that Typing is much less mature - far more room for new additions.

...

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UL4XNZ... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Jelle Zijlstra

4:09 a.m.

...

On Mon, Mar 7, 2022 at 3:43 PM Chris Angelico wrote:

...
On Tue, 8 Mar 2022 at 10:39, Jelle Zijlstra wrote:

...
El lun, 7 mar 2022 a las 15:35, Chris Angelico ()

escribió:

...
but python-ideas is 100% shooting ideas to pieces.

100% really? Maybe my sense of time is blurred, but some ideas do make it through.

PEP 618 (zip(strict=True)) and PEP 616 (str.removeprefix) originated on

El lun, 7 mar 2022 a las 19:57, Christopher Barker () escribió: python-ideas and were accepted.

Chris Angelico

4:13 a.m.

On Tue, 8 Mar 2022 at 15:10, Jelle Zijlstra wrote:

...

El lun, 7 mar 2022 a las 19:57, Christopher Barker () escribió:

...
On Mon, Mar 7, 2022 at 3:43 PM Chris Angelico wrote:

...
On Tue, 8 Mar 2022 at 10:39, Jelle Zijlstra wrote:

...
El lun, 7 mar 2022 a las 15:35, Chris Angelico () escribió: but python-ideas is 100% shooting ideas to pieces.

100% really? Maybe my sense of time is blurred, but some ideas do make it through.

PEP 618 (zip(strict=True)) and PEP 616 (str.removeprefix) originated on python-ideas and were accepted.

Okay, I exaggerated slightly for effect in saying 100%, but still - look at the discussions on python-ideas about those. I'm not sure about 618, but certainly 616 got a ridiculous level of negativity until it moved to python-dev, where it actually got some support, which is how it came to be implemented. ChrisA

Finn Mason

4:33 a.m.

On Mon, Mar 7, 2022, 9:12 PM Jelle Zijlstra wrote:

...

El lun, 7 mar 2022 a las 19:57, Christopher Barker () escribió:

...
On Mon, Mar 7, 2022 at 3:43 PM Chris Angelico wrote:

...
On Tue, 8 Mar 2022 at 10:39, Jelle Zijlstra wrote:

...
El lun, 7 mar 2022 a las 15:35, Chris Angelico ()

escribió:

...
but python-ideas is 100% shooting ideas to pieces.

100% really? Maybe my sense of time is blurred, but some ideas do make it through.

PEP 618 (zip(strict=True)) and PEP 616 (str.removeprefix) originated on python-ideas and were accepted.

Even more recently, PEP 680 (tomllib in the standard library). The python-ideas discussion was short and very positive. Then the PEP was written and the "meat" of the discussion was done on discuss.python.org Python-ideas: https://mail.python.org/archives/list/python-ideas@python.org/thread/IWJ3I32... Discuss: https://discuss.python.org/t/pep-680-tomllib-support-for-parsing-toml-in-the... _______________________________________________

...

Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QIW26K... Code of Conduct: http://python.org/psf/codeofconduct/

-- Finn (Mobile)

David Mertz, Ph.D.

5:09 a.m.

As someone else noted, few of the ideas originating on typing-sig are for syntax changes. One might argue that some are "abuse of notation." Generally they are of the sort "let's support indexing or bitwise operators on some more types" or let's let more builtin things act as types in annotations. Whether good or bad, that's not syntax The last time typing-sig proposed a syntax change was PEP 677 which was rejected. Finn notes several non-syntax changes that were accepted from python-ideas. Those are new parameters or methods for builtins, again not syntax. He notes another which is "add a library to stdlib", again not syntax. Most ideas SHOULD get rejected, simply because most are poorly or partially thought through. But even many that are *reasonable* in and of themselves are not worth adding the complication. I'm sure if Guido had put `for x in stuff if cond` I'd probably use it most days. But tens of millions of users, and billions of lines of production code later, it's not worth the disruption. It's not worth "breaking" hundreds of Python books (a few of which I've written). Saving a newline—or even very slightly changing the emphasis of code as read—are not close to worth including the extra way to do the same thing. Of course the balance might be different for something really cumbersome to do with existing syntax. This isn't that. On Mon, Mar 7, 2022, 11:34 PM Finn Mason

...

PEP 618 (zip(strict=True)) and PEP 616 (str.removeprefix) originated on

...
python-ideas and were accepted.

Even more recently, PEP 680 (tomllib in the standard library). The python-ideas discussion was short and very positive. Then the PEP was written and the "meat" of the discussion was done on discuss.python.org

Paul Moore

6:33 a.m.

On Mon, 7 Mar 2022 at 23:44, Chris Angelico wrote:

...

Not ALL typing changes are just new things in typing.py, so that doesn't cover everything. And yes, I am sure that a lot of things get proposed and not implemented - my point is that typing-sig is successfully finding the good ideas and refining them into actual code, but python-ideas is 100% shooting ideas to pieces.

Bikeshed problem. I barely understand most of the proposals on typing-sig, whereas *everyone* knows what "for..in..if" means (and hence has an opinion). Add to that the fact that the framing of many proposals on python-ideas feels like "here's a neat idea, why doesn't someone (not me) implement it", whereas proposals on typing-sig generally seem to involve an interested party doing a *lot* of work up front, and then summarising that in a proposal, and it's not hard to see why python-ideas is both more accessible for people proposing incompletely thought through ideas, and more hostile towards them... To put it another way, if the culture on python-ideas expected people to do as much up-front work on a proposal as typing-sig seems to (from my experience, at least), there would be far fewer ideas on here, but they would be a lot better, and would be much more likely to be successful. Paul

Brendan Barnwell

4:16 a.m.

On 2022-03-07 15:32, Chris Angelico wrote:

...

On Tue, 8 Mar 2022 at 10:20, Brendan Barnwell wrote:

...
On 2022-03-06 14:43, Chris Angelico wrote:

...
This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

You seem to see that as a positive thing, but I would be happier if fewer typing-related changes made it in.

I'm not sure whether it's positive or not. All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames, no matter how good or bad. I do NOT believe that every idea on typing-sig is good and that every idea on python-ideas is bad, so what is it that makes typing-sig actually successful in refining ideas into usable form that python-ideas is failing at?

I still don't agree with that. I think that many of the changes proposed here actually are bad (or at least not good enough), and that's why they don't make it in. When you say "no matter how good or bad" do you just mean "no matter how good or bad I personally think they are", or are you using some other metric for how good or bad the proposals "actually" are? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

Chris Angelico

5:42 a.m.

On Tue, 8 Mar 2022 at 15:37, Brendan Barnwell wrote:

...

On 2022-03-07 15:32, Chris Angelico wrote:

...
On Tue, 8 Mar 2022 at 10:20, Brendan Barnwell wrote:

...
On 2022-03-06 14:43, Chris Angelico wrote:

...
This keeps happening. All the successful ideas seem to happen elsewhere, notably on typing-sig.

You seem to see that as a positive thing, but I would be happier if fewer typing-related changes made it in.

I'm not sure whether it's positive or not. All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames, no matter how good or bad. I do NOT believe that every idea on typing-sig is good and that every idea on python-ideas is bad, so what is it that makes typing-sig actually successful in refining ideas into usable form that python-ideas is failing at?

I still don't agree with that. I think that many of the changes proposed here actually are bad (or at least not good enough), and that's why they don't make it in. When you say "no matter how good or bad" do you just mean "no matter how good or bad I personally think they are", or are you using some other metric for how good or bad the proposals "actually" are?

Use any metric you like, including whether the features end up getting implemented, and then compare that to the sentiment on this list, which is clearly summarized by this exact post. Basically you're saying that ideas here ARE mostly bad, and therefore should not be implemented, which is quite the opposite of what seems to happen elsewhere. The ideas that eventually succeed do so by getting support from places other than python-ideas, because of people whose attitudes are "the status quo is fine, all you're doing is saving <X>" (eg "a newline" from this thread). When you consider that a number of typing-based proposals offer, in a concrete sense, no more than that, it leaves you wondering whether (a) it's worth making type annotations better in ways that aren't applicable to the rest of the language, or (b) maybe it isn't about keystrokes after all, and there's something else going on. Look at the python-ideas sentiment about the match statement. Compare to the way that python-dev discussed it. ChrisA

Greg Ewing

9 Mar 9 Mar

11:09 p.m.

On 8/03/22 12:32 pm, Chris Angelico wrote:

...

All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames,

Typing is an area of Python that is under active development, so it is somewhat malleable, and there are still real problems that actually need to be solved. On the other hand, the core syntax of Python has been well established for a long time. Most of the proposals for adding to it nowadays tend to be of the "wouldn't it nice if..." variety that are not addressing anything that's widely seen as a problem. It's not surprising that such suggestions rarely succeed.

...

no matter how good or bad.

If an idea can be shot down in flames that easily, it wasn't really such a good idea when seen in the wider context. It's not just the merits of the idea itself that need to be considered, but whether it brings enough benefit to be worth the disruption of introducing it. As Python matures and solidifies, that bar becomes harder and harder to clear. -- Greg

Jeff Edwards

30 Mar 30 Mar

1:50 a.m.

As someone who has read the entirety of those thread and is not a core-dev but having some experience with breaking and non-breaking changes at scale, from my perspective: 1. This is a common syntax for an existing idea that is intuitive given other existing syntax (comprehensions) 2. This would introduce a change that usage breaks backwards compatibility with older interpreters, which means if a core open source project uses it, their consumers will either need to update their interpreter to keep up to date or risk a fork in one of their code bases to stay long-term maintained I agree that point 2 should never be taken lightly, so realistically it may need a PEP to justify its appearance, but I’ve also never seen a syntactical addition to Python quite as idiom-shifting as pattern-matching in 3.10, so it’s not like it’s never been done. From my viewpoint, the discussion did have a lot of adversarial tone which I understood not as bike-shedding (contributing only to easily-understood problems/suggestions), but similarly to the “symbols versus words” balance in that shorter syntax isn’t necessarily more readable or easier to read about and trying to keep the tenet that there should be preferably only one way to express the concept (or at least preferentially fewer). Also, I’m sure that there is a lot of bike-shedding that can happen in such an open forum (I mean, I’m doing it right now in a way) and that it can cause reviewers to build callouses and may cause some good ideas to get overlooked in favor of stability of interfaces. However, I think the myriad of existing expression possibilities that were suggested for this situation actually is an argument that if there is a clearly more expressive and readable way of representing it, then it could actually be an improvement that could help reduce unnecessary plurality. Given it doesn’t diverge from extant pythonic patterns (e.g. comprehensions), is intuitive to use and read (IMHO, given any code can be made unintelligible) and that there are multiple instances in my own code where it would have been the most readable way to express this idea, I’d be for it, especially if it were simply being rolled into a minor version already containing a syntactic addition. Cheers, Jeff On Wed, Mar 9, 2022 at 3:11 PM Greg Ewing wrote:

...

On 8/03/22 12:32 pm, Chris Angelico wrote:

...
All I can see is that changes get proposed on typing-sig and actually make it into the language, but changes that get proposed on python-ideas are invariably shot down in flames,

Typing is an area of Python that is under active development, so it is somewhat malleable, and there are still real problems that actually need to be solved.

On the other hand, the core syntax of Python has been well established for a long time. Most of the proposals for adding to it nowadays tend to be of the "wouldn't it nice if..." variety that are not addressing anything that's widely seen as a problem. It's not surprising that such suggestions rarely succeed.

...
no matter how good or bad.

If an idea can be shot down in flames that easily, it wasn't really such a good idea when seen in the wider context.

It's not just the merits of the idea itself that need to be considered, but whether it brings enough benefit to be worth the disruption of introducing it. As Python matures and solidifies, that bar becomes harder and harder to clear.

-- Greg _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TBFFQ2... Code of Conduct: http://python.org/psf/codeofconduct/

Stephen J. Turnbull

8 Mar 8 Mar

5:32 a.m.

Brendan Barnwell writes:

...

I would be happier if fewer typing-related changes made it in.

I'm curious: do you use type annotations yourself? If yes, do you just prefer minimal hints to a precise typing system, or have there been changes that are actually harmful in your opinion? Examples? If no, are there examples of typing changes that actually cause problems for your code? (My definition of "actual problems" includes problems you know how to work around but are easy to miss until something goes wrong, aka bug magnets.) Typing changes that are annoying because you need to work around them (but you pretty reliably DTRT with them)? Other issues? I have no axe to grind here. I want to be informed about arguments for and against. Steve

Brendan Barnwell

5:46 a.m.

On 2022-03-07 21:32, Stephen J. Turnbull wrote:

...

Brendan Barnwell writes:

...
I would be happier if fewer typing-related changes made it in.

I'm curious: do you use type annotations yourself? If yes, do you just prefer minimal hints to a precise typing system, or have there been changes that are actually harmful in your opinion? Examples? If no, are there examples of typing changes that actually cause problems for your code? (My definition of "actual problems" includes problems you know how to work around but are easy to miss until something goes wrong, aka bug magnets.) Typing changes that are annoying because you need to work around them (but you pretty reliably DTRT with them)?

No, I don't use typing annotations. They are one of the changes I wish hadn't made it in. :-) Of course, because of that they don't impact me much directly, but I do notice a lot of discussion about them in various places where people ask for help with Python problems (where I sometimes hang out to help). My perception based on that is that typing annotation take up air in the room, so to speak. They result in people wrestling with the type annotation system and the type annotation tools rather than writing code to do things, which is what I think Python is good at. (I should note that to the limited extent that I have tried to use type annotations myself, my experience has been similar. They are easy enough for simple things, but as soon as I want to do anything more, I quickly find the effort of dealing with them outweighs the benefit of using them.) So basically the reason I don't like typing-related changes is that I perceive them as being a bunch of churning motion in an area that really does not deserve that much attention or effort and is not worth it. Which is, in many cases, the same reason I oppose various proposals on this list. There is also a bit of slippery-slope concern, in that as more and more people adopt these changes, more and more other people who don't want to worry about typing wind up having to worry about it in order to work on projects. And that's also something that leads me to oppose many proposals on here: even if they're something I wouldn't have to use myself, they increase the surface area that I have to know in order to be conversant with Python.

...

I have no axe to grind here. I want to be informed about arguments for and against.

Sure, understood. I don't particularly have an axe to grind either, since as I said I'm mostly able to ignore type annotations. :-) But there are those lurking problems around the edges that I mentioned. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

David Mertz, Ph.D.

7 Mar 7 Mar

3:09 a.m.

...

Earlier on the thread, I made a similar point that it would be nice to have a way to filter without the redundant for x in x. Though I can’t think of a really good way to express it. But as for filtered for loops:

"for thing in (x for x in collection if is_interesting(x))"

It's pretty important that comprehensions can express transformations (i.e. map) along with filters. A lot of the examples and argument for extension of `filter()` to general loops miss this point. Yes, `x for x in stuff if pred(x)` looks kinda redundant. But `transform(x) for x in stuff if pred(x)` really doesn't. And transformation in my mind is at least as common and at least as important as filtering. Happily, I can write this in Python right now: items = (transform(x) for x in stuff if pred(x)) for item in items: foo = something_with(item) bar = something_else(item) if result(foo, bar): ... I'm not making a slippery slope argument, I do know one thing is possible without the other. But if you really want an "entire comprehensions in a loop statement" we wind up with something like: for item := transform(x) for x in stuff if pred(x): foo = something_with(item) ... Which is obviously a *possible* language, but I'd be -100 on making loop statements that complicated needlessly. The way we have now is great, and clear, and already exists. Other spellings are available as well, of course, such as `if not transform(x): continue`. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

Stephen J. Turnbull

5 Mar 5 Mar

1:27 p.m.

Chris Angelico writes:

...

What Python needs is a non-clunky way to express [filtered iteration].

I don't. In my own code there aren't any else-less cases of for: if:. All the filtered iterations have intervening code at the top of the loop before the if. Most could be refactored to use filtered iteration, but it would come at a cost in readability because the filters are moderately complex expressions, the auxiliaries used in the filter condition have meaningful names in the problem domain. Also, the auxiliaries are sometimes used in the loop body. But if somebody's willing to do the spade work to show that filtered iteration would clarify a fair number of loops in a substantial existing corpus, I'll be happy to overcome my aversion to new syntax and get behind it for those who do need it. Steve

Greg Ewing

3 Mar 3 Mar

10:30 a.m.

On 3/03/22 9:50 pm, Steven D'Aprano wrote:

...

While you can put the block on the same line as the colon using semicolons:

if condition: print(1); print(2) # Legal.

and there's nothing to stop you from making arbitrarily long lines using this feature... if condition: print(1); print(2); print(buckle); print(my_shoe); print("Is this line long enough yet?") So if Python is opinionated about line lengths, it's rather selectively opinionated. -- Greg

Steven D'Aprano

1:56 p.m.

What I said: "Python, for good or ill, is an opinionated language regarding indentation and statements." And then Greg refuted something I never said:

...

So if Python is opinionated about line lengths, it's rather selectively opinionated.

The Python language isn't opinionated about line lengths, except as a coding standard. So whatever point you think you are making about line lengths, it is not refuting anything I have said. PEP 8 isn't mandatory, but lots of people follow it. Even those who don't still try to avoid writing 800-column lines. Greg, I trust that you will agree that even if the interpreter allows them, long one-liners with semicolons are not considered Pythonic. -- Steve

Rob Cliffe

2:34 a.m.

...

I really am shocked by how many people seem to have broken ENTER keys on their keyboards. You mock. (As far as I remember you have always opposed new language features/changes. Correct me if I am wrong.) But the proposal would give people the choice of Saving a level of indentation at the cost of having two suite-introductions on the same line. Keeping the two suit-introductions on separate lines (as now) at

...

Let's just keep Python readable rather than see how much we can cram on a line.

On Wed, Mar 2, 2022, 2:56 PM Jeremiah Paige wrote:

I have on a few occasions wanted a for..in..if statement and if it existed would have used it. However, I agree that the level of change a new statement type brings to the language is probably too high for this feature.

But what if python lifted the newline requirement for blocks that contain compound statements? That is, statements that end in a ':' can be followed by other statements that end in a ':' on the same line. AFAICT there would be no ambiguity (to the parser; to humans, depends). Doing so would add the OPs requested feature, though it would be two statements on one line with one extra character. It would also essentially bring the full comprehension syntax to for loops since fors and ifs could be chained arbitrarily.

# for..if for x in y: if x in c: some_op(x

# nested generator-like for for line in doc: for word in line.split(): spellcheck(word)

# side effect one-liner for item in an_iterable: if condition(item): side_effect(item))

Regards, Jeremiah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LWPJKA... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list --python-ideas@python.org To unsubscribe send an email topython-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived athttps://mail.python.org/archives/list/python-ideas@python.org/message/6COVUG... Code of Conduct:http://python.org/psf/codeofconduct/

Rob Cliffe

2:27 a.m.

I have had precisely the same idea. It feels better to make this feature general (if introduced at all) than make it specific to 'for' + 'if'. I think there would have to be a rule that any 'if' that appeared on a line with other suite-introducing-statements could not have a corresponding 'else'. Best wishes Rob Cliffe On 02/03/2022 19:53, Jeremiah Paige wrote:

...

I have on a few occasions wanted a for..in..if statement and if it existed would have used it. However, I agree that the level of change a new statement type brings to the language is probably too high for this feature.

But what if python lifted the newline requirement for blocks that contain compound statements? That is, statements that end in a ':' can be followed by other statements that end in a ':' on the same line. AFAICT there would be no ambiguity (to the parser; to humans, depends). Doing so would add the OPs requested feature, though it would be two statements on one line with one extra character. It would also essentially bring the full comprehension syntax to for loops since fors and ifs could be chained arbitrarily.

# for..if for x in y: if x in c: some_op(x

# nested generator-like for for line in doc: for word in line.split(): spellcheck(word)

# side effect one-liner for item in an_iterable: if condition(item): side_effect(item))

Regards, Jeremiah

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LWPJKA... Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

2 Mar 2 Mar

8:27 p.m.

On Wed, Mar 02, 2022 at 09:25:01AM -0500, Ricky Teachey wrote:

...

Is there some opportunity for some kind of compiler magic when the iterable of a for loop is fully contained in a place easily findable by the compiler, and not spread over multiple if and for statements?

I am not an expert on compiler optimization techniques, but I would be very surprised if something as minor as a computation being split over two statements instead of one would make a difference. What would make a difference would be statements *between* the for and the if: # some hypothetical optimization may apply for spam in eggs: if cheese: block # may block the optimization for spam in eggs: block if cheese: block but even there, a *sufficiently smart* optimizer may be able to do some pretty wild things. This is why, for instance, optimizing C compilers can often completely reorder your code. https://stackoverflow.com/questions/26190364/is-it-legal-for-a-c-optimizer-t... In a practical sense, given the relatively limited human and financial resources available to Python compared to C, it might be slightly easier for *people* to program an optimizer to spot the opportunity in the single statement case compared to the two statement case. But even there, I doubt it -- optimization is unlikely to occur at the source code level, more likely at the AST or even bytecode level, where any such difference between 1 vs 2 statements is likely to disappear. -- Steve

Christopher Barker

5:42 p.m.

On Wed, Mar 2, 2022 at 2:25 AM Steven D'Aprano wrote:

...

...
I think it's really the equivalent of

for x in y: if not x in c: break do_stuff

which to me give the proposed syntax a bit more relative strength.

Forgotten the difference between continue and break, have we? :-)

Wow! Now THAT was a brain fart! Untested code and all. I often tell me students I rarely write more than two lines of code without a mistake…. Semantically, there is no difference between

...

if not condition: continue block

and

if condition: block

Sure — and as you said, there are any number of other ways to accomplish this. But my point (that was likely weakened by posting completely incorrect code) was that “block” here is often a lot more than one line, in which case I, at least, would prefer to filter out the unwanted items first, and then leave the rest of the contents of the loop the same. If nothing else, that allows readers of the code to know that there is not an else: down there somewhere. but as of CPython 3.10, the byte-code from the first version is slightly

...

longer, so I imagine (but haven't measured) it will be ever-so-slightly less efficient. (But unlikely to be meaningfully different.)

And I certainly wouldn’t make this choice one way or another due to a micro optimization. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Stephen J. Turnbull

6:28 a.m.

Steven D'Aprano writes:

...

On Wed, Mar 02, 2022 at 02:28:33AM +0100, Svein Seldal wrote:

...
for x in y if x in c: some_op(x)

What does this new syntax give us that we don't already have with this?

for x in y if x in c: some_op(x)

It would actually work! (missing colon ;-)

...

I see no new functionality here. Is the only advantage that you save one line and one indent level? Both are cheap.

It's arguably consistent with comprehension syntax, which one could argue makes learning either simpler if you already know the other. But I'm -0.5 on this change as needless churn, and -0.5 on it because I really don't want to invite the full comprehension syntax (with nested "in ... if" clauses) and without it it's still not the same, for a net "vote" of math.nextafter(-1,0).

David Mertz, Ph.D.

3:31 a.m.

-1. Extra complexity for no benefit. Attractive nuisance to making overly complex statement lines. On Tue, Mar 1, 2022, 8:31 PM Svein Seldal wrote:

...

I'm sorry for reposting, but this message got stuck in moderation approval for 5 days so I figured I should try again.

I'd like to propose extending the for statement to include conditionals akin to comprehensions in order to simplify for loop statements:

`for .. in .. if ..:`

E.g.

for x in y if x in c: some_op(x)

which is functionally equivalent as

for x in y: if x not in c: continue some_op(x)

The `for .. in .. if` syntax is a well-known construct from list comprehension syntax [1]. Other alternative ways to do this with list comprehension is:

for x in (a for a in y if c):

or

it = (a for a in y if c) for x in it:

Without having examined all use cases, I believe the same syntax should be applied this syntax as for asynchronous comprehensions.

[1] PEP 202 - List Comprehensions [2] PEP 530 - Asynchronous Comprehensions

Best regards, Svein Seldal _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7IXPRO... Code of Conduct: http://python.org/psf/codeofconduct/

brian.patrick.mccall＠gmail.com

1 Apr 1 Apr

6:40 a.m.

+1 Python has always seemed to me to allow one to completely describe exactly what you are iterating over in a for loop in the one line containing the for statement. And that is more or less the principle I tend to follow, unless the for statement simply gets to be too long. I have even used the `for item in (item for item in items if ... ):`, although this is sometimes too much of a brainflick to be worth it. I have also used (on separate lines) `subset = (item for item in items if ...); for item in subset:`, but if for...in...if syntax ever became available, I'd wear it out.

Hasan Diwan

4 Apr 4 Apr

1:21 a.m.

On Tue, 1 Mar 2022 at 17:31, Svein Seldal wrote:

...

for x in (a for a in y if c):

-1 on this form, I know it's supported, but I can see it getting ugly, i.e. for x in (a for x in (a for x in a)) # add a few more "for x in (a" and it will get messy fast

...

or

it = (a for a in y if c) for x in it:

This seems more explicit, but does this not give us syntactic sugar for already supported use cases? -- H -- OpenPGP: https://hasan.d8u.us/openpgp.asc https://t.mailpgn.com/l/?u=fbe9f93f-fd80-43c5-9866-4f515788cc99&fl=https%3A%2F%2Fhasan.d8u.us%2Fopenpgp.asc If you wish to request my time, please do so using *bit.ly/hd1AppointmentRequest https://t.mailpgn.com/l/?u=ef691519-2071-46a7-ad52-381462e771f4&fl=http%3A%2F%2Fbit.ly%2Fhd1AppointmentRequest* . Si vous voudrais faire connnaisance, allez a *bit.ly/hd1AppointmentRequest https://t.mailpgn.com/l/?u=07d0ad86-434b-4af5-a4a1-4a036a47cd02&fl=http%3A%2F%2Fbit.ly%2Fhd1AppointmentRequest* . https://t.mailpgn.com/l/?u=b5bad592-b977-4c0e-a77f-f17d51c6ee46&fl=https%3A%2F%2Fsks-keyservers.net%2Fpks%2Flookup%3Fop%3Dget%26amp%3Bsearch%3D0xFEBAD7FFD041BBA1Sent from my mobile device Envoye de mon portable

752

Age (days ago)

785

Last active (days ago)

List overview

Download

65 comments

20 participants

participants (20)

Brendan Barnwell
brian.patrick.mccall＠gmail.com
Chris Angelico
Christopher Barker
David Mertz, Ph.D.
Dennis Sweeney
Finn Mason
Greg Ewing
Hasan Diwan
Jeff Edwards
Jelle Zijlstra
Jeremiah Paige
Michael Smith
Paul Bryan
Paul Moore
Ricky Teachey
Rob Cliffe
Stephen J. Turnbull
Steven D'Aprano
Svein Seldal

Syntax proposal of for..in..if in regular for loops

tags

participants (20)