> One of the points of using a function is to not define it in the local
scope, but in some other namespace, so it can be reused and tested.
I consider the "tested" part to be of particular importance. Especially if it were to become multiple layers deep, this can become a real issue.
This might be a bit subjective, but I've always considered list comprehensions to have a similar purpose as lambda expressions: to be used for small and easy to read operations, such as for single mapping, substitution and/or filtering. I.E. "[f(item) if condition else 0 for item in sequence]", "[func(item) for item in sequence if conditional]", or "[item for index, item in enumerate(sequence) if not index % 10]". As such, I tend to avoid and discourage the usage of complex nested comprehensions [1].
The proposed syntax helps to address the readability issue of nested comprehensions, in addition to allowing the usage of compound statements such as "try-except" within them. But that doesn't address the core issue of using "anonymous function"-like features [2] for complex operations: they are difficult to debug and have to be transformed in order to be directly testable [3]. This is especially problematic when they contain compound statements.
Also, to provide an alternative perspective on the "existing syntax" referenced in the OP:
def create_matrix(matrix):
# Emphasis on not having the function definition inside of the for loop.
# gen_row() can be defined inside of the function if you're certain you won't
# need it again, but this makes it easier to move gen_row() to its own definition
# later if it becomes more complex and you want to test create_matrix() and
# gen_row() separately.
def gen_row(row):
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0
new_matrix = []
for row in matrix:
new_row = list(gen_row(row))
new_matrix.append(new_row)
return new_matrix
But the above could also be made more succinct using the existing list comp syntax. Something like this is probably along the lines of what I'd use in my own code:
def create_matrix(matrix):
def gen_row(row):
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0
# "tuple(gen_row(row))" would be a bit better if the rows have a fixed # of cells, which is common.
return [list(gen_row(row)) for row in matrix]
This is a much more fair comparison, and I personally prefer the above over the proposed syntax:
new_matrix = [
for row in matrix: [
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0
]
]
As for the the argument of making complex comprehensions less confusing, I would consider the potentially confusing usage of complex comprehensions referenced in the OP to be a misuse of the feature in the first place. If a comprehension is starting to look like it might confuse readers, it's a sign that it should be refactored into it's own dedicated function. Of course, this isn't to say that any code which isn't immediately clear to Python users of all experience should be refactored. But, if the current version might be confusing to many readers and it can made more clear without any loss in functionality, performance, or design quality, refactoring is often the right move. Something like this:
[
f(x)
for y in z
for x in y
if g(x)
]
Can easily be refactored into something like this:
def gen_foo(z):
for y in z:
for x in y:
if g(x):
yield f(x)
return list(gen_foo(z))
Similar to the previous set of examples, I think the above is a much more fair comparison to the existing syntax, and I personally prefer it over the proposed syntax.
IMO, there's pretty much zero practical benefit in using the former version compared to the above. As far as development time goes, you may spend a tiny bit more time thinking of a useful name for "gen_foo", but a good name can help greatly in making the purpose of the code more clear. So, I can't really see that as a downside.
With all of the above in mind, I'm -1 on the proposal; I don't see a substantial enough benefit from using the proposed syntax compared to the already existing syntax. I'll admit that it has some appeal regarding convenience, but it would very likely encourage anti-patterns (especially for those with beginner and intermediate level experience with Python).
---
[1] - Although I'm not an advocate, there are certainly some valid use cases for 2-layer deep nested comprehensions, such as "[(f(cell) for cell in row) for row in table]". But, any deeper than that can quickly become convoluted and a mess to debug; even 2-layer nested comps can be massive pain at times. For an example of what I'd consider to be an overly complex 2-layer comp, I just recently encountered the following: "[[vj, [i for i, vi in enumerate(L) if vj == L[i] ]] for j, vj in enumerate(L) if L[j] not in L[:j]]".
[2] - By "anonymous-function like features", I'm referring to features like comprehensions and lambda expressions; as well as equivalent features in other languages. In C#, this might look like "items.Select(x => f(x));", in JS "items.map(x => f(x));", in Java "items.stream().map(x -> f(x));", etc. It's a bit difficult to determine exactly when an operation or series of operations becomes complex enough to justify avoiding these features in favor of a separate, dedicated function, but I think it should always be considered.
[3] - By "have to be transformed in order to be directly testable", I mean changing the comprehension into a dedicated function that can be called from unit tests. You could technically "test" it by copying and pasting the entire comprehension into a unit test, but that would be terrible in practice (for fairly obvious reasons). Compare this to a nested function, which can be directly moved into its own separate function (assuming it was structured and named well).