Filtered "for" loop with list-comprehension-like syntax
One common pattern I run across when parsing plain text data files is that I want to skip over blank lines when processing. If I wanted to build a list of all non-blank lines in the file, I could simply do: lines = [line for line in input_file if line.strip()] But as a loop, it almost invariably gets written as: for line in input_file: if not line.strip(): continue # do the real processing It seems odd that "for x in y if z" is allowed in comprehensions but not in a regular for loop. Why not let for x in y if z: do_stuff(x) be a shorthand for for x in y: if not z: continue do_stuff(x) Similarly, I occasionally have multiple sections that need to be handled differently. One way to write this is: for line in input_file: if is_section_delimiter(line): break do_stuff_1(line) for line in input_file: # this picks up where the last one left off if is_section_delimiter(line): break do_stuff_2(line) etc. (This is a little bit of a weird idiom with files since repeated iteration over them remembers where it left off, at least in 2.7.) It would be nice to have this shorthand for it: for line in input_file while not is_section_delimiter(line): do_stuff_1(line) for line in input_file while not is_section_delimiter(line): do_stuff_2(line) etc. This makes it more immediately clear (to me, at least) that it stops at the end of the section. This could also be added to comprehensions; it's somewhat tricky to emulate in comprehensions now. I think the easiest way to do the equivalent of [f(x) for x in y while z] with a comprehension is a = [(f(x) if z else None) for x in y] try: idx = a.index(None) except ValueError: # no None found pass else: # truncate before first None a = a[:idx] but even that fails if None is a potentially valid result of f(x) (or if you forget to use the try/except block and z was always True), and it processes the entire list even though it may throw out a sizable chunk of it immediately after. The only totally safe way I can think of to do it now is by unpacking it into a loop: a = [] for x in y: if not z: break a.append(f(x)) I think adding these would make such idioms a little more readable, but it might not be enough of a gain to justify a syntax addition. Thoughts? Dan Baker
Dan Baker <dbaker3448@gmail.com> writes:
It seems odd that "for x in y if z" is allowed in comprehensions but not in a regular for loop. Why not let
for x in y if z: do_stuff(x)
be a shorthand for
for x in y: if not z: continue do_stuff(x)
This can already be spelled: for x in (w for w in y if z): do_stuff(x) Which is not to forestall discussion of the proposed language change, but only to point out that there is an existing idiom for this.
Similarly, I occasionally have multiple sections that need to be handled differently. One way to write this is: for line in input_file: if is_section_delimiter(line): break do_stuff_1(line) for line in input_file: # this picks up where the last one left off if is_section_delimiter(line): break do_stuff_2(line) etc.
That looks like it would be better modelled with an explicit state transition when the condition is encountered, without stopping the iteration: handlers = [do_stuff_1, do_stuff_2, do_stuff_3] handle_line = handlers.pop(0) for line in input_file: if is_section_delimiter(line): handle_line = handlers.pop(0) handle_line(line) -- \ “Philosophy is questions that may never be answered. Religion | `\ is answers that may never be questioned.” —anonymous | _o__) | Ben Finney
On Fri, May 20, 2011 at 5:57 PM, Dan Baker <dbaker3448@gmail.com> wrote:
One common pattern I run across when parsing plain text data files is that I want to skip over blank lines when processing. If I wanted to build a list of all non-blank lines in the file, I could simply do:
lines = [line for line in input_file if line.strip()]
But as a loop, it almost invariably gets written as:
for line in input_file: if not line.strip(): continue # do the real processing
It seems odd that "for x in y if z" is allowed in comprehensions but not in a regular for loop. Why not let
for x in y if z: do_stuff(x)
be a shorthand for
for x in y: if not z: continue do_stuff(x)
Yes, "why not" indeed. Because you can already do that in any number of different ways; you showed one (two if you count the comprehension), another is for x in y: if z: do_stuff(x) Do we really need more ways to spell the same thing? (Hint: this is a rhetorical question. I recommend you study the zen of Python before replying.)
Similarly, I occasionally have multiple sections that need to be handled differently. One way to write this is: for line in input_file: if is_section_delimiter(line): break do_stuff_1(line) for line in input_file: # this picks up where the last one left off if is_section_delimiter(line): break do_stuff_2(line) etc. (This is a little bit of a weird idiom with files since repeated iteration over them remembers where it left off, at least in 2.7.)
It would be nice to have this shorthand for it: for line in input_file while not is_section_delimiter(line): do_stuff_1(line) for line in input_file while not is_section_delimiter(line): do_stuff_2(line) etc.
This makes it more immediately clear (to me, at least)
Ay, there's the rub. More syntactical options means more things to learn for every single Python user. One of the attractions of Python is that it is relatively small and simple. Let's keep it that way!
that it stops at the end of the section. This could also be added to comprehensions; it's somewhat tricky to emulate in comprehensions now. I think the easiest way to do the equivalent of [f(x) for x in y while z] with a comprehension is a = [(f(x) if z else None) for x in y] try: idx = a.index(None) except ValueError: # no None found pass else: # truncate before first None a = a[:idx] but even that fails if None is a potentially valid result of f(x) (or if you forget to use the try/except block and z was always True), and it processes the entire list even though it may throw out a sizable chunk of it immediately after. The only totally safe way I can think of to do it now is by unpacking it into a loop: a = [] for x in y: if not z: break a.append(f(x))
I think adding these would make such idioms a little more readable, but it might not be enough of a gain to justify a syntax addition. Thoughts?
Indeed it is not enough. -- --Guido van Rossum (python.org/~guido)
I had a feeling that might be the answer. Sometimes a little syntactic sugar isn't bad, but even reasonable people won't always agree on which kinds - and too far down that road lies Perl. Thanks anyway. Dan
On 5/20/2011 8:57 PM, Dan Baker wrote:
It seems odd that "for x in y if z" is allowed in comprehensions but not in a regular for loop.
Comprehensions are expressions and therefore need everything packed into them that is needed. For statements and if statement are statements and both can be followed in there suite with an many statements as needed, so there is no *need* to pack more than is necessary into the header line. Even doc strings, which are conceptually part of the header, were put down into the suite. -- Terry Jan Reedy
participants (4)
-
Ben Finney -
Dan Baker -
Guido van Rossum -
Terry Reedy