Temporary variables in comprehensions
As far as I can see, a comprehension like alist = [f(x) for x in range(10)] is better than a for-loop for x in range(10): alist.append(f(x)) because the previous one shows every element of the list explicitly so that we don't need to handle `append` mentally. But when it comes to something like [f(x) + g(f(x)) for x in range(10)] you find you have to sacrifice some readableness if you don't want two f(x) which might slow down your code. Someone may argue that one can write [y + g(y) for y in [f(x) for x in range(10)]] but it's not as clear as to show what `y` is in a subsequent clause, not to say there'll be another temporary list built in the process. We can even replace every comprehension with map and filter, but that would face the same problems. In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension. In my opinion, code like [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**] is more natural than any solution we now have. And that's why I pro the new syntax, it's clear, explicit and readable, and is nothing beyond the functionality of the present comprehensions so it's not complicated. And I hope the discussion could focus more on whether we should allow assigning temporary variables in comprehensions rather than how to solve the specific example I mentioned above.
I +1 this at surface level; Both Haskell list comprehensions and Scala for comprehensions have variable assignment in them, even between iterating and this is often very useful. Perhaps syntax can be generalised as: [expr_using_x_and_y for i in is x = expr_using_i for j in is y = expr_using_j_and_x] This demonstrates the scope of each assignment; available in main result and then every clause that follows it. Sorry to op who will receive twice, forgot reply to all On 15 Feb 2018 7:03 am, "fhsxfhsx" <fhsxfhsx@126.com> wrote:
For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`, I don't really see what the issue is, if you really want to make it shorter, you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare case where I like `map` more than comprehensions. For more complex case, just define a intermediate generator along the lines ``` f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] ``` Which does exactly the same thing but - Is more readable and explicit - Has no memory overhead thanks to lazy evaluation (btw, you should consider generators for your nested comprenshions) While I am sometimes in the same state of mind, wishing for variables in comprehensions seems to me like a good indicator that your code needs refactoring. Best, E On 15 February 2018 at 10:32, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote:
I +1 this at surface level; Both Haskell list comprehensions and Scala
for comprehensions have variable assignment in them, even between iterating and this is often very useful. Perhaps syntax can be generalised as:
and then every clause that follows it.
that we don't need to handle `append` mentally. the specific example I mentioned above.
Note that you can already do: [y + g(y) for x in range(10) for y in [f(x)]] i.e. for y in [expr] does exactly what the OP wants. No new syntax needed. If you hang out on python-list , you'll soon notice that many newbies struggle already with the list comprehension syntax. It's a mini-language which is almost, but not entirely, exactly unlike normal Python code. Let's not complicate it further. Stephan 2018-02-15 10:53 GMT+01:00 Evpok Padding <evpok.padding@gmail.com>:
You are right and actually I sometimes did the same thing in a temporary script such as in ipython. Because in my opinion, it's not really elegant code as one may be puzzled for the list `[f(x)]`. Well, although that's quite subjective. And also I test the code and find another `for` clause can make time cost about 1.5 times in my computer, even when I optimize `[f(x)]` to a generator `(f(x), )`. Though that's not very big issue if you don't care about efficiency that much. But a temporary list or generator is redundant here. `[f(x)]` can be an alternative, but I think it is worth a new syntax. At 2018-02-15 18:11:47, "Stephan Houben" <stephanh42@gmail.com> wrote: Note that you can already do: [y + g(y) for x in range(10) for y in [f(x)]] i.e. for y in [expr] does exactly what the OP wants. No new syntax needed. If you hang out on python-list , you'll soon notice that many newbies struggle already with the list comprehension syntax. It's a mini-language which is almost, but not entirely, exactly unlike normal Python code. Let's not complicate it further. Stephan 2018-02-15 10:53 GMT+01:00 Evpok Padding <evpok.padding@gmail.com>: For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`, I don't really see what the issue is, if you really want to make it shorter, you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare case where I like `map` more than comprehensions. For more complex case, just define a intermediate generator along the lines ``` f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] ``` Which does exactly the same thing but - Is more readable and explicit - Has no memory overhead thanks to lazy evaluation (btw, you should consider generators for your nested comprenshions) While I am sometimes in the same state of mind, wishing for variables in comprehensions seems to me like a good indicator that your code needs refactoring. Best, E On 15 February 2018 at 10:32, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote:
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
I'm not sure it does indicate a need for refactoring, I'd argue it's quite a common pattern, at least in functional languages from which this construct arises. In fact, in those languages, there are laws that govern interactions with the comprehensions (though this comes from monads and monads perhaps don't quite apply to pythons model). These laws define behaviour that is expected equivalent by users; [x for x in xs] = xs [f(x) for x in [x]] = f(x) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)] Even though that last law isn't completely analogous to the given example from the OP, the transformation he wants to be able to do does arise from the laws. So it could be argued that not being able to flatten the comprehension down via law 3 is unexpected behaviour and in order to achieve this you'd need a form of assignment in the comprehension or suffer inefficiencies. But that's probably just my functional brain talking... On 15 Feb 2018 9:53 am, "Evpok Padding" <evpok.padding@gmail.com> wrote:
On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw14896.2014@my.bristol.ac.uk
wrote:
These laws define behaviour that is expected equivalent by users;
[x for x in xs] = xs
OK -- that's the definition...
[f(x) for x in [x]] = f(x)
well, not quite: [f(x) for x in [x]] = [f(x)] Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)]
well, no. using two for expressions yields the outer product -- all combinations: In [*14*]: xs = range(3) In [*15*]: [(x,y) *for* x *in* xs *for* y *in* xs] Out[*15*]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2. But nesting the comps applies one expression, and then the other, yielding a length len(xs) list. but you wrote: [g(y) for x in xs for y in f(x)] which I'm not sure what you were expecting, as f(x) is not a sequence (probably)... To play with your examples: Define some functions that make it clear what's been applied: In [*16*]: *def* f(x): ...: *return* "f(*{}*)".format(x) ...: In [*17*]: *def* g(x): ...: *return* "g(*{}*)".format(x) and a simple sequence to use: In [*18*]: xs = range(3) Now your examples: In [*19*]: [x *for* x *in* xs] Out[*19*]: [0, 1, 2] In [*20*]: [f(x) *for* x *in* [x]] Out[*20*]: ['f(5)'] In [*21*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*21*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] OK -- all good f applied, then g, but then the last one: In [*27*]: [g(y) *for* x *in* xs *for* y *in* f(x)] Out[*27*]: ['g(f)', 'g(()', 'g(0)', 'g())', 'g(f)', 'g(()', 'g(1)', 'g())', 'g(f)', 'g(()', 'g(2)', 'g())'] in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function: In [*29*]: *def* f(x): ...: *return* 2*x Then you just get an error: In [*30*]: [g(y) *for* x *in* xs *for* y *in* f(x)] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-30-82ba3864654b> in <module>() ----> 1 [g(y) for x in xs for y in f(x)] <ipython-input-30-82ba3864654b> in <listcomp>(.0) ----> 1 [g(y) for x in xs for y in f(x)] TypeError: 'int' object is not iterable The the nested comp is what is desired here: In [*31*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*31*]: ['g(0)', 'g(2)', 'g(4)'] Except you probably want a generator expression in the inner loop to avoid bulding an extra list: In [*33*]: [g(y) *for* y *in* (f(x) *for* x *in* xs)] Out[*33*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] So back to the OP's example: In [*34*]: [f(x) + g(f(x)) *for* x *in* range(10)] Out[*34*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] that is best done with comps as: In [*36*]: [fx + g(fx) *for* fx *in* (f(x) *for* x *in* range(10))] Out[*36*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier: In [*41*]: fx = (f(x) *for* x *in* range(10)) In [*42*]: [x + g(x) *for* x *in* fx] Out[*42*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy: In [*46*]: *import* *numpy* *as* *np* In [*47*]: *def* f(x): ...: *return* x * 2 ...: In [*48*]: *def* g(x): ...: *return* x * 3 ...: ...: In [*49*]: xs = np.arange(3) In [*50*]: f(xs) + g(f(xs)) Out[*50*]: array([ 0, 8, 16]) is pretty compact, and can be "optimized with a temp: In [*51*]: fx = f(xs) ...: fx + g(fx) ...: Out[*51*]: array([ 0, 8, 16]) pretty simple isn't it? So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact? I'm guessing not :-) (the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-) (I'm still trying to figure out why folks think map() or filter() help here...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I’d like to clarify that f(x) was indeed meant to be a sequence. As per monad law: do { y <- do { x <- m; f x } g y } === do { x <- m; y <- f x; g y } I think you might have misunderstood the types of things; f: Function[a, List[b]] and g: Function[b, List[c]]. m: List[a] But I DO think my old the fly write up went wrong, converting do-notation to list comprehensions isn’t completely straightforward The above is equivalent to [g y | x <- m, y <- f x] in Haskell and the top is [g y | y <- [z |x <- m, z <- f x]] These have analogous structures in python; [g(y) for x in m for y in f(x)] and [g(y) for y in [z for x in m for z in f(x)]] (I think?) And yes the left identity law I posted was missing the [f(x)] brackets. If I’ve not made another mistake, that *should* now work? From: Chris Barker [mailto:chris.barker@noaa.gov] Sent: 15 February 2018 23:34 To: jw14896.2014@my.bristol.ac.uk Cc: Evpok Padding <evpok.padding@gmail.com>; Python-Ideas <python-ideas@python.org> Subject: Re: [Python-ideas] Temporary variables in comprehensions On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw14896.2014@my.bristol.ac.uk <mailto:jw14896.2014@my.bristol.ac.uk> > wrote: These laws define behaviour that is expected equivalent by users; [x for x in xs] = xs OK -- that's the definition... [f(x) for x in [x]] = f(x) well, not quite: [f(x) for x in [x]] = [f(x)] Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)] well, no. using two for expressions yields the outer product -- all combinations: In [14]: xs = range(3) In [15]: [(x,y) for x in xs for y in xs] Out[15]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2. But nesting the comps applies one expression, and then the other, yielding a length len(xs) list. but you wrote: [g(y) for x in xs for y in f(x)] which I'm not sure what you were expecting, as f(x) is not a sequence (probably)... To play with your examples: Define some functions that make it clear what's been applied: In [16]: def f(x): ...: return "f({})".format(x) ...: In [17]: def g(x): ...: return "g({})".format(x) and a simple sequence to use: In [18]: xs = range(3) Now your examples: In [19]: [x for x in xs] Out[19]: [0, 1, 2] In [20]: [f(x) for x in [x]] Out[20]: ['f(5)'] In [21]: [g(y) for y in [f(x) for x in xs]] Out[21]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] OK -- all good f applied, then g, but then the last one: In [27]: [g(y) for x in xs for y in f(x)] Out[27]: ['g(f)', 'g(()', 'g(0)', 'g())', 'g(f)', 'g(()', 'g(1)', 'g())', 'g(f)', 'g(()', 'g(2)', 'g())'] in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function: In [29]: def f(x): ...: return 2*x Then you just get an error: In [30]: [g(y) for x in xs for y in f(x)] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-30-82ba3864654b> in <module>() ----> 1 [g(y) for x in xs for y in f(x)] <ipython-input-30-82ba3864654b> in <listcomp>(.0) ----> 1 [g(y) for x in xs for y in f(x)] TypeError: 'int' object is not iterable The the nested comp is what is desired here: In [31]: [g(y) for y in [f(x) for x in xs]] Out[31]: ['g(0)', 'g(2)', 'g(4)'] Except you probably want a generator expression in the inner loop to avoid bulding an extra list: In [33]: [g(y) for y in (f(x) for x in xs)] Out[33]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] So back to the OP's example: In [34]: [f(x) + g(f(x)) for x in range(10)] Out[34]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] that is best done with comps as: In [36]: [fx + g(fx) for fx in (f(x) for x in range(10))] Out[36]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier: In [41]: fx = (f(x) for x in range(10)) In [42]: [x + g(x) for x in fx] Out[42]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy: In [46]: import numpy as np In [47]: def f(x): ...: return x * 2 ...: In [48]: def g(x): ...: return x * 3 ...: ...: In [49]: xs = np.arange(3) In [50]: f(xs) + g(f(xs)) Out[50]: array([ 0, 8, 16]) is pretty compact, and can be "optimized with a temp: In [51]: fx = f(xs) ...: fx + g(fx) ...: Out[51]: array([ 0, 8, 16]) pretty simple isn't it? So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact? I'm guessing not :-) (the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-) (I'm still trying to figure out why folks think map() or filter() help here...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
I’d like to clarify that f(x) was indeed meant to be a sequence. As per monad law: *do* { y *<-* *do* { x *<-* m; f x } g y } === *do* { x *<-* m; y *<-* f x; g y } I think you might have misunderstood the types of things; f: Function[a, List[b]] and g: Function[b, List[c]]. m: List[a] I’m still confused — f and g take a scale and return a list? Or take two parameters, a scalar and a list??? Either way, doesn’t really fit with python comprehensions, which are more or less expecting functions that except and return a single element (which could be a triple, but I digress) And the key point is that in python: for x in seq1 for y in seq2 Does all combinations, like a nested for loop would. But I DO think my old the fly write up went wrong, converting do-notation to list comprehensions isn’t completely straightforward Well, I don’t think I get the do notation at all.... The above is equivalent to [g y | x <- m, y <- f x] in Haskell and the top is [g y | y <- [z |x <- m, z <- f x]] These have analogous structures in python; [g(y) for x in m for y in f(x)] and [g(y) for y in [z for x in m for z in f(x)]] (I think?) Is this current Python or a new proposed syntax — ‘cause I can’t make any sense of it in python. Maybe define some f,g,m that behave as you expect, and see what python does.... If I’ve not made another mistake, that **should* *now work? Now on a phone, so can’t test, but I also can’t tell what you are expecting the result to be. Back to one of your examples: [f(x) for x in [x]] What does that mean??? for x in seq Means iterate through seq, and assign each item to the name x. If that seq has x in it — I’m not sure that is even legal python — the scope in a comprehension confuses me. But that is the equivalent is something like: it= iter(seq) while True: Try: x = next(it) Except StopIteration: Break (Excuse the caps — hard to write code on a phone) So not sure how x gets into that sequence before the loop starts. -CHB *From:* Chris Barker [mailto:chris.barker@noaa.gov <chris.barker@noaa.gov>] *Sent:* 15 February 2018 23:34 *To:* jw14896.2014@my.bristol.ac.uk *Cc:* Evpok Padding <evpok.padding@gmail.com>; Python-Ideas < python-ideas@python.org> *Subject:* Re: [Python-ideas] Temporary variables in comprehensions On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote: These laws define behaviour that is expected equivalent by users; [x for x in xs] = xs OK -- that's the definition... [f(x) for x in [x]] = f(x) well, not quite: [f(x) for x in [x]] = [f(x)] Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)] well, no. using two for expressions yields the outer product -- all combinations: In [*14*]: xs = range(3) In [*15*]: [(x,y) *for* x *in* xs *for* y *in* xs] Out[*15*]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2. But nesting the comps applies one expression, and then the other, yielding a length len(xs) list. but you wrote: [g(y) for x in xs for y in f(x)] which I'm not sure what you were expecting, as f(x) is not a sequence (probably)... To play with your examples: Define some functions that make it clear what's been applied: In [*16*]: *def* f(x): ...: *return* "f(*{}*)".format(x) ...: In [*17*]: *def* g(x): ...: *return* "g(*{}*)".format(x) and a simple sequence to use: In [*18*]: xs = range(3) Now your examples: In [*19*]: [x *for* x *in* xs] Out[*19*]: [0, 1, 2] In [*20*]: [f(x) *for* x *in* [x]] Out[*20*]: ['f(5)'] In [*21*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*21*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] OK -- all good f applied, then g, but then the last one: In [*27*]: [g(y) *for* x *in* xs *for* y *in* f(x)] Out[*27*]: ['g(f)', 'g(()', 'g(0)', 'g())', 'g(f)', 'g(()', 'g(1)', 'g())', 'g(f)', 'g(()', 'g(2)', 'g())'] in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function: In [*29*]: *def* f(x): ...: *return* 2*x Then you just get an error: In [*30*]: [g(y) *for* x *in* xs *for* y *in* f(x)] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-30-82ba3864654b> in <module>() ----> 1 [g(y) for x in xs for y in f(x)] <ipython-input-30-82ba3864654b> in <listcomp>(.0) ----> 1 [g(y) for x in xs for y in f(x)] TypeError: 'int' object is not iterable The the nested comp is what is desired here: In [*31*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*31*]: ['g(0)', 'g(2)', 'g(4)'] Except you probably want a generator expression in the inner loop to avoid bulding an extra list: In [*33*]: [g(y) *for* y *in* (f(x) *for* x *in* xs)] Out[*33*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] So back to the OP's example: In [*34*]: [f(x) + g(f(x)) *for* x *in* range(10)] Out[*34*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] that is best done with comps as: In [*36*]: [fx + g(fx) *for* fx *in* (f(x) *for* x *in* range(10))] Out[*36*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier: In [*41*]: fx = (f(x) *for* x *in* range(10)) In [*42*]: [x + g(x) *for* x *in* fx] Out[*42*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy: In [*46*]: *import* *numpy* *as* *np* In [*47*]: *def* f(x): ...: *return* x * 2 ...: In [*48*]: *def* g(x): ...: *return* x * 3 ...: ...: In [*49*]: xs = np.arange(3) In [*50*]: f(xs) + g(f(xs)) Out[*50*]: array([ 0, 8, 16]) is pretty compact, and can be "optimized with a temp: In [*51*]: fx = f(xs) ...: fx + g(fx) ...: Out[*51*]: array([ 0, 8, 16]) pretty simple isn't it? So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact? I'm guessing not :-) (the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-) (I'm still trying to figure out why folks think map() or filter() help here...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 2/15/2018 8:37 PM, Chris Barker - NOAA Federal wrote:
Reusing a previously bound name as an iteration variable is a bad idea. It works in 3.x because the outermost iterable, but only the outermost iterable, is pre-calculated before executing the comprehension. Hence 'x in [x]' sometimes works, and sometimes not. ('Outermost' is topmost in nested loops, left most in comprehension.)
To put it another way, l = [x for x in [x]] is actually calculated as _temp = [x]; l = [x for x in _temp]. In general, other iterables cannot be precalculated since they might depend on prior iteration variables. -- Terry Jan Reedy
On Thu, Feb 15, 2018 at 10:13:37AM +0000, Jamie Willis wrote:
Python is not a functional language, and the usual Pythonic solution to an overly complex or inefficient functional expression is to refactor into non-functional style. In my experience, most Python users have never even heard of monads, and those who have, few grok them. (I know I don't.)
I think these should be: [x for x in xs] = list(xs) # not to be confused with [x] [f(x) for x in [x]] = [f(x)] [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in [f(x)]]
Even though that last law isn't completely analogous to the given example from the OP,
It can be used though. He has: [f(x) + g(f(x)) for x in xs] which can be written as [y + g(y) for x in xs for y in [f(x)]] Here's an example: # calls ord() twice for each x py> [ord(x) + ord(x)**2 for x in "abc"] [9506, 9702, 9900] # calls ord() once for each x py> [y + y**2 for x in "abc" for y in [ord(x)]] [9506, 9702, 9900] And one last version: py> [y + y**2 for y in [ord(x) for x in "abc"]] [9506, 9702, 9900] In production, I'd change the last example to use a generator comprehension.
Law 3 does apply, and I'm not sure what you mean by the statement that we can't flatten the comprehension down. -- Steve
A generator can be a good idea, however, I wonder if it's really readable to have a `f_samples`. And the structure is the same as in [y + g(y) for y in [f(x) for x in range(10)]] if you replace the list with a generator. And it's similar for other solutions you mentioned. Well, I know that it can be quite different for everyone to determine whether a piece of code is readable or not, maybe it's wise to wait for more opinions. There's another problem that, as you distinguished the `simple` and `more complex` case, the offered solutions seem very specific, I'm not sure if there's a universal solution for every case of temporary variables in comprehensions. If not, I think it's too high cost to reject a new syntax if you need to work out a new solution every time. At 2018-02-15 17:53:21, "Evpok Padding" <evpok.padding@gmail.com> wrote: For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`, I don't really see what the issue is, if you really want to make it shorter, you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare case where I like `map` more than comprehensions. For more complex case, just define a intermediate generator along the lines ``` f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] ``` Which does exactly the same thing but - Is more readable and explicit - Has no memory overhead thanks to lazy evaluation (btw, you should consider generators for your nested comprenshions) While I am sometimes in the same state of mind, wishing for variables in comprehensions seems to me like a good indicator that your code needs refactoring. Best, E On 15 February 2018 at 10:32, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote:
On 15 February 2018 at 05:56, fhsxfhsx <fhsxfhsx@126.com> wrote:
... as long as the code inside the comprehension remains relatively simple. It's easy to abuse comprehensions to the point where they are less readable than a for loop, but that's true of a lot of things, so isn't a specific problem with comprehensions.
Agreed. I hit that quite often.
That is a workaround (and one I'd not thought of before) but I agree it's ugly, and reduces readability. Actually, factoring out the inner comprehension like Evpok Padding suggests: f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] is very readable and effective, IMO, so it's not *that* obvious that local names are beneficial.
In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
"We need" is a bit strong here. "It would be useful to have" is probably true for some situations.
The problem is that you haven't proposed an actual syntax here, just that one should be invented. There have been discussions on this in the past (a quick search found https://mail.python.org/pipermail/python-ideas/2011-April/009863.html and https://mail.python.org/pipermail/python-ideas/2012-January/013468.html, for example).
The problem isn't so much "whether we should allow it" as "can we find a syntax that is acceptable", and only then "does the new syntax give sufficient benefit to be worth adding". New syntax has a pretty high cost, and proposals that don't suggest explicit syntax will get stuck because you can't judge whether adding the capability is "worth it" without being clear on what the cost is - particularly when the benefit is relatively small (which this is). Agreed that it's important to focus on the general problem, but part of the discussion *will* include arguing as to why the existing workarounds and alternatives are less acceptable than new syntax. And that will need to include discussion of specific cases. Generally, in that sort of discussion, artificial examples like "y=f(x)" don't fare well because it's too easy to end up just debating subjective views on "readability". If you can provide examples from real-world code that clearly demonstrate the cost in terms of maintainability of the existing workarounds, that will help your argument a lot. Although you'll need to be prepared for questions like "would you be willing to drop support for versions of Python older than 3.8 in order to get this improvement?" - it's surprisingly hard to justify language (as opposed to library) changes when you really stop and think about it. Which is not to say that it can't be done, just that it's easy to underestimate the effort needed. Paul
Thank you Paul, what you said is enlightening and I agree on most part of it. I'll propose two candidate syntaxs. 1. `with ... as ...` This syntax is more paralles as there would be `for` and `with` clause as well as `for` and `with` statement. However, the existing `with` statement is semantically different from this one, although similar. 2. `for ... is ...` This syntax is more uniform as the existing `for` clause make an iterator which is a special kind of variable. However, I'm afraid this syntax might be confused with `for ... in ...` as they differ only on one letter. I like the latter one better. Other proposes are absolutely welcome. And here is an example which appears quite often in my code where I think a new syntax can help a lot: Suppose I have an list of goods showing by their ids in database, and I need to transform the ids into json including information from two tables, `Goods` and `GoodsCategory`, where the first table recording `id`, `name` and `category_id` indicating which category the goods belongs to, the second table recording `id`, `name` and `type` to the categories. With the new syntax, I can write [ { 'id': goods.id, 'name': goods.name, 'category': gc.name, 'category_type': gc.type, } for goods_id in goods_id_list for goods is Goods.get_by_id(goods_id) for gc is GoodsCategory.get_by_id(goods.category_id) ] And I cannot think of any good solutions as this one without it. To generalize this case, for each element of the list, I need two temporary variables (`goods` and `gc` in my case), and each one was used twice. And reply to the two past discussions you mentioned, 1.https://mail.python.org/pipermail/python-ideas/2011-April/009863.html This mail gave a solution to modify function `f` to keep the result. The weak point is obvious, you must modify the function `f`. 2.https://mail.python.org/pipermail/python-ideas/2012-January/013468.html This mail wrote
At 2018-02-15 18:08:46, "Paul Moore" <p.f.moore@gmail.com> wrote:
On Sun, Feb 18, 2018 at 12:23:28AM +0800, fhsxfhsx wrote:
I don't think they are even a little bit similar. The existing `with` statement is for controlling cleanup code (using a context manager). This proposal doesn't have anything to do with context managers or cleanup code. It's just a different way to spell "name = value" inside comprehensions.
Indeed. And frankly, treated as English grammar, "for value is name" doesn't make sense and is horribly ugly to my eyes: result = [x for value in sequence for value+1 is x]
I can of a few, starting with the most simple: write a helper function. Not every problem needs to be solved with new syntax. def dict_from_id(goods_id): goods = Goods.get_by_id(goods_id) gc = GoodsCategory.get_by_id(goods.category_id) return {'id': goods.id, 'name': goods.name, 'category': gc.name, 'category_type': gc.type } result = [dict_from_id(goods_id) for goods_id in goods_id_list] That's much nicer to read, you can document and test the dict_from_id() function, no new systax is required, it is easy to refactor, and I very much doubt that adding one extra function call is going to be a significant slowdown compared to the cost of two calls to get_by_id() methods and constructing a dict. (And if as you add more fields to the dict, the overhead of the function call becomes an even smaller proportion.) Or you can make this a method of the goods object, which is arguably a better OO design. Let the goods object be responsible for creating the dict. result = [Goods.get_by_id(goods_id).make_dict() for goods_id in goods_id_list] # or if you prefer result = [goods.make_dict() for goods in map(Goods.get_by_id, goods_id_list)] Here is a third solution: use a for-loop iterating over a single-item tuple to get the effect of a local assignment to a temporary variable: result = [{ # dict display truncated for brevity... } for goods_id in goods_id_list for goods in (Goods.get_by_id(goods_id),) for gc in (GoodsCategory.get_by_id(goods.category_id),) ] If you don't like the look of single-item tuples (foo,) you can use single-item lists instead [x] but they are a tiny bit slower to create. Serhiy has suggested that the interpreter can optimize the single-item loop to make it as fast as a bare assignment: https://bugs.python.org/issue32856 I think this is a neat trick, although Yuri thinks it is an ugly hack and doesn't want to encourage it. Neat or ugly, I think it is better than "for value is name". -- Steve
Hi fhsxfhsx, and welcome. My comments below, interleaved with yours. On Thu, Feb 15, 2018 at 01:56:44PM +0800, fhsxfhsx wrote: [quoted out of order]
Whether or not to allow this proposal will depend on what alternate solutions to the problem already exist, so your specific example is very relevant. Any proposed change has to compete with existing solutions.
While I personally agree with you, many others disagree. I know quite a few experienced, competent Python programmers who avoid list comprehensions because they consider them harder to read and reason about. They consider a regular for-loop better precisely because you do see the explicit call to append. (In my experience, those of us who get functional-programming idioms often forget that others find them tricky.) The point is that list comprehensions are already complex enough that they are difficult for many people to learn, and some people never come to grips with them. Adding even more features comes with a cost. The bottom line is that it isn't clear to me that allowing local variables inside comprehensions will make them more readable.
The usual comments about premature optimisation apply here. Setting a new comprehension variable is not likely to be free, and may even be more costly than calling f(x) twice if f() is a cheap expression: [x+1 + some_func(x+1) for x in range(10)] could be faster than [y + some_func(y) for x in range(10) let y = x + 1] or whatever syntax we come up with.
Someone may argue that one can write [y + g(y) for y in [f(x) for x in range(10)]]
Indeed. This would be the functional-programming solution, and I personally think it is an excellent one. The only changes are that I'd use a generator expression for the intermediate value, avoiding the need to make a full list, and I would lay it out more nicely, using whitespace to make the structure more clear: result = [y + g(y) for y in (f(x) for x in range(10)) ]
but it's not as clear as to show what `y` is in a subsequent clause, not to say there'll be another temporary list built in the process.
There's no need to build the temporary list. Use a generator comprehension. And I disagree that the value of y isn't as clear. An alternative is simply to refactor your list comprehension. Move the calls to f() and g() into a helper function: def func(x): y = f(x) return y + g(y) and now you can write the extremely clear comprehension [func(x) for x in range(10)] that needs no extra variable. [...]
In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
"Need" is very strong. I think that the two alternatives I mention above cover 95% of the cases where might use a local variable in a comprehension. And of the remaining cases, many of them will be so complex that they should be re-written as an explicit for-loop. So in my opinion, we're only talking about a "need" to solve the problem for a small proportion of cases: - most comprehensions don't need a local variable (apart from the loop variable) at all; - of those which do need a local variable, most can be easily solved using a nested comprehension or a helper function; - of those which cannot be solved that way, most are complicated enough that they should use a regular for-loop; - leaving only a small number of cases which are complicated enough to genuinely benefit from local variables but not too complicated. So this is very much a borderline feature. Occasionally it would be "nice to have", but on the negative side: - it adds complexity to the language; - makes comprehensions harder to read; - and people will use it unnecessarily where there is no readability or speed benefit (premature optimization again). It is not clear to me that we should burden *all* Python programmers with additional syntax and complexity of an *already complex* feature for such a marginal improvement.
How can you say that the new syntax is "clear, explicit and readable" when you haven't proposed any new syntax yet? For lack of anything better, I'm going to suggest "let y = f(x)" as the syntax, although personally I don't like it even a bit. Where should the assignment go? [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] [(y, y**2) for x in (1, 2, 3, 4) let y = x+1] I think they're both pretty ugly, but I can't think of anything else. Can we rename the loop variable, or is that an error? [(x, x**2) let x = x+1 for x in (1, 2, 3, 4)] How do they interact when you have multiple loops and if-clauses? [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 for a in range(y) let z = a+1 if z > 2 for b in range(z) let w = z+1] For simplicity, perhaps we should limit any such local assignments to the very end of the comprehension: [expression for name in sequence <zero or more for-loops and if-clauses> <zero or more let-clauses> ] but that means we can't optimise this sort of comprehension: [expression for x in sequence for y in (something_expensive(x) + function(something_expensive(x)) ] Or these: [expression for x in sequence if something_expensive(x) or condition(something_expensive(x)) ] I think these are very hard questions to answer. -- Steve
A bit of a nit — function call overhead is substantial in python, so if that is an actual function, rather than a simple expression, it’ll likely be slower to call it twice for any but trivially small iterables.
[(y, y**2) let y = x+1 for x in (1, 2, 3, 4)]
Do we need the let? [ g(y) for y = f(x) for c in seq] Or, with expressions: [y + y**2 for y = x+1 for x in (1,2,3)] Maybe that would be ambiguous— I haven’t thought carefully about it. -CHB
Hi Steve, Thank you for so detailed comments. My comments also below interleaved with yours. At 2018-02-16 08:57:40, "Steven D'Aprano" <steve@pearwood.info> wrote:
To be frank, I had not thought of this before. However, in my opinion, when considering adding a new syntax, we care more about the marginal cost. I mean, I think it is the functional-programming way which is tricky, but allowing a new syntax would not make things worse. Well, that's just a guess, maybe only those who are puzzled with comprehensions can give us an answer.
Someone may argue that one can write >> [y + g(y) for y in [f(x) for x in range(10)]] > >Indeed. This would be the functional-programming solution, and I >personally think it is an excellent one. The only changes are that I'd >use a generator expression for the intermediate value, avoiding the need >to make a full list, and I would lay it out more nicely, using >whitespace to make the structure more clear: > > result = [y + g(y) for y in > (f(x) for x in range(10)) > ] >
In my opinion, [ y + g(y) for x in range(10) let y = f(x) ] is better because it's more corresponding to a for-loop for x in range(10): y = f(x) result.append(y + g(y)) In my opinion, comprehensions are not real functional-programming because there is not even a function. Though there're similarities, map and filter are real functional-programming. Since the similarity between for-clause in comprehensions and the for-loop, I think it's better to write comprehensions more close to for-loop. I don't know but I guess maybe it can also help those who fear comprehensions better embrace them?
I think it can be a goods idea if there's a name to `func` which is easy to understand, or `func` is put close to the comprehension and is in a obvious place. But I feel it's not for the case I gave in another mail to Paul, https://mail.python.org/pipermail/python-ideas/2018-February/048997.html, (I'm sorry that the example is quite long, and I don't hope to copy it here) To me, it can be confusing to have several `func` when I have several lists at the same time and have to transform them each in a similar but different way.
[...] >> In a word, what I'm arguing is that we need a way to assign temporary >> variables in a comprehension. > >"Need" is very strong. I think that the two alternatives I mention above >cover 95% of the cases where might use a local variable in a >comprehension. And of the remaining cases, many of them will be so >complex that they should be re-written as an explicit for-loop. So in my >opinion, we're only talking about a "need" to solve the problem for a >small proportion of cases: > >- most comprehensions don't need a local variable (apart from > the loop variable) at all; > >- of those which do need a local variable, most can be easily > solved using a nested comprehension or a helper function; > >- of those which cannot be solved that way, most are complicated > enough that they should use a regular for-loop; > >- leaving only a small number of cases which are complicated enough > to genuinely benefit from local variables but not too complicated. > >So this is very much a borderline feature. Occasionally it would be >"nice to have", but on the negative side: > >- it adds complexity to the language; > >- makes comprehensions harder to read; > >- and people will use it unnecessarily where there is no readability > or speed benefit (premature optimization again). > >It is not clear to me that we should burden *all* Python programmers >with additional syntax and complexity of an *already complex* feature >for such a marginal improvement. > > >> In my opinion, code like >> [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**] >> is more natural than any solution we now have. >> And that's why I pro the new syntax, it's clear, explicit and readable > >How can you say that the new syntax is "clear, explicit and readable" >when you haven't proposed any new syntax yet? > >For lack of anything better, I'm going to suggest "let y = f(x)" as the >syntax, although personally I don't like it even a bit. > >Where should the assignment go? > > [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] > > [(y, y**2) for x in (1, 2, 3, 4) let y = x+1] > >I think they're both pretty ugly, but I can't think of anything else. > >Can we rename the loop variable, or is that an error? > > [(x, x**2) let x = x+1 for x in (1, 2, 3, 4)] > >How do they interact when you have multiple loops and if-clauses? > > [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 > for a in range(y) let z = a+1 if z > 2 > for b in range(z) let w = z+1] > > >For simplicity, perhaps we should limit any such local assignments to >the very end of the comprehension: > > [expression for name in sequence > <zero or more for-loops and if-clauses> > <zero or more let-clauses> > ] > >but that means we can't optimise this sort of comprehension: > > [expression for x in sequence > for y in (something_expensive(x) + function(something_expensive(x)) > ] > >Or these: > > [expression for x in sequence > if something_expensive(x) or condition(something_expensive(x)) > ] > > >I think these are very hard questions to answer. > I think the assignment should be treated equally as for-clause and if-clause which means
[(y, y**2) for x in (1, 2, 3, 4) let y = x+1] would be a right syntax. And [(x, x**2) for x in (1, 2, 3, 4) let x = x+1] would not cause an error because we can also write [(x, x**2) for x in (1, 2, 3, 4) for x in (4, 3, 2, 1)] now. I didn't see any problem in [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 for a in range(y) let z = a+1 if z > 2 for b in range(z) let w = z+1] In my opinion, it would behave the same as for x in (1, 2, 3, 4): y = x+1 for a in range(y): z = a+1 if z > 2: for b in range(z): w = z+1 mylist.append((w, w**2)) According to my understanding, the present for-clause and if-clause does everything quite similar to this nested way,
-- >Steve >_______________________________________________ >Python-ideas mailing list >Python-ideas@python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/
I believe list comprehensions are difficult to read because they are not formatted properly. For me, list comprehension clauses are an expression, followed by clauses executed in the order. Any list comprehension with more than one clause should be one-line-per clause. Examples inline: On 2018-02-15 19:57, Steven D'Aprano wrote:
Where should the assignment go? [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] [(y, y**2) for x in (1, 2, 3, 4) let y = x+1]
Since y is a function of x, it must follow the for clause:
They are applied in order:
which is a short form for:
On 2018-02-23 12:44, Neil Girdhar wrote:
Saving the indentation? Oh yes, for sure! This code reads like a story, the indentation is superfluous to that story. Should we add it to Python? I don't know; I quick scan through my own code, and I do not see much opportunity for list comprehensions of this complexity. Either my data structures are not that complicated, or I have try/except blocks inside a loop, or I am using a real query language (like SQL). pythonql seems to solve all these problems well enough (https://github.com/pythonql/pythonql).
On Fri, Feb 23, 2018 at 1:42 PM Kyle Lahnakoski <klahnakoski@mozilla.com> wrote:
That's a good thing that you are not writing code like this. I don't agree that the indentation is "superfluous". It makes the code easy to read. Anyway, Google's Python style guide also agrees that very long comprehensions like that are not worth it. ( https://google.github.io/styleguide/pyguide.html?showone=List_Comprehensions... ) Either my data structures are not that complicated, or I have try/except
Hi list,
personally I think that the biggest problem readability-wise is that "for" is a post-fix operator, which makes generators much harder to read. It's also very different from normal for loops, which have the "for" at the top. IMHO generators would be much easier to read with a prefix for, as in [for x in range(10): f(x) + g(f(x))] also nested generators get nicer like that: [for y in (for x in range(10): f(x)): y + g(y)] one could critique here that we shouldn't use colons in expressions, but that boat has sailed: we use them for lambdas. We do not write sq = x**2 lambda x and I am not proposing that. Also if/else could be written with colons, but I am not sure whether that's actually nicer: val = (if attr is None: 5 else: attr + 3) but it certainly is in case of ifs in generators: [for x in range(10): if x % 3 != 2: x] which could be problematic to parse if you compare that to [for x in range(10): if x % 3 == 2: x - 1 else: x + 1] one could even dig out the often-proposed always-rejected except-in-expression: [for x in range(10): try: f(x) except WhateverError: None] or even a with: [for x in file_names: with open(x) as f: f.read()] Greetings Martin
You should give an actual motivating example. I think none of these suggestions are more readable than just writing things out as a for loop. You argue that you want to avoid appending to a result list. In that case, I suggest writing your pattern as a generator function. Best, Neil On Thursday, February 15, 2018 at 2:03:31 AM UTC-5, fhsxfhsx wrote:
I +1 this at surface level; Both Haskell list comprehensions and Scala for comprehensions have variable assignment in them, even between iterating and this is often very useful. Perhaps syntax can be generalised as: [expr_using_x_and_y for i in is x = expr_using_i for j in is y = expr_using_j_and_x] This demonstrates the scope of each assignment; available in main result and then every clause that follows it. Sorry to op who will receive twice, forgot reply to all On 15 Feb 2018 7:03 am, "fhsxfhsx" <fhsxfhsx@126.com> wrote:
For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`, I don't really see what the issue is, if you really want to make it shorter, you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare case where I like `map` more than comprehensions. For more complex case, just define a intermediate generator along the lines ``` f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] ``` Which does exactly the same thing but - Is more readable and explicit - Has no memory overhead thanks to lazy evaluation (btw, you should consider generators for your nested comprenshions) While I am sometimes in the same state of mind, wishing for variables in comprehensions seems to me like a good indicator that your code needs refactoring. Best, E On 15 February 2018 at 10:32, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote:
I +1 this at surface level; Both Haskell list comprehensions and Scala
for comprehensions have variable assignment in them, even between iterating and this is often very useful. Perhaps syntax can be generalised as:
and then every clause that follows it.
that we don't need to handle `append` mentally. the specific example I mentioned above.
Note that you can already do: [y + g(y) for x in range(10) for y in [f(x)]] i.e. for y in [expr] does exactly what the OP wants. No new syntax needed. If you hang out on python-list , you'll soon notice that many newbies struggle already with the list comprehension syntax. It's a mini-language which is almost, but not entirely, exactly unlike normal Python code. Let's not complicate it further. Stephan 2018-02-15 10:53 GMT+01:00 Evpok Padding <evpok.padding@gmail.com>:
You are right and actually I sometimes did the same thing in a temporary script such as in ipython. Because in my opinion, it's not really elegant code as one may be puzzled for the list `[f(x)]`. Well, although that's quite subjective. And also I test the code and find another `for` clause can make time cost about 1.5 times in my computer, even when I optimize `[f(x)]` to a generator `(f(x), )`. Though that's not very big issue if you don't care about efficiency that much. But a temporary list or generator is redundant here. `[f(x)]` can be an alternative, but I think it is worth a new syntax. At 2018-02-15 18:11:47, "Stephan Houben" <stephanh42@gmail.com> wrote: Note that you can already do: [y + g(y) for x in range(10) for y in [f(x)]] i.e. for y in [expr] does exactly what the OP wants. No new syntax needed. If you hang out on python-list , you'll soon notice that many newbies struggle already with the list comprehension syntax. It's a mini-language which is almost, but not entirely, exactly unlike normal Python code. Let's not complicate it further. Stephan 2018-02-15 10:53 GMT+01:00 Evpok Padding <evpok.padding@gmail.com>: For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`, I don't really see what the issue is, if you really want to make it shorter, you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare case where I like `map` more than comprehensions. For more complex case, just define a intermediate generator along the lines ``` f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] ``` Which does exactly the same thing but - Is more readable and explicit - Has no memory overhead thanks to lazy evaluation (btw, you should consider generators for your nested comprenshions) While I am sometimes in the same state of mind, wishing for variables in comprehensions seems to me like a good indicator that your code needs refactoring. Best, E On 15 February 2018 at 10:32, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote:
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
I'm not sure it does indicate a need for refactoring, I'd argue it's quite a common pattern, at least in functional languages from which this construct arises. In fact, in those languages, there are laws that govern interactions with the comprehensions (though this comes from monads and monads perhaps don't quite apply to pythons model). These laws define behaviour that is expected equivalent by users; [x for x in xs] = xs [f(x) for x in [x]] = f(x) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)] Even though that last law isn't completely analogous to the given example from the OP, the transformation he wants to be able to do does arise from the laws. So it could be argued that not being able to flatten the comprehension down via law 3 is unexpected behaviour and in order to achieve this you'd need a form of assignment in the comprehension or suffer inefficiencies. But that's probably just my functional brain talking... On 15 Feb 2018 9:53 am, "Evpok Padding" <evpok.padding@gmail.com> wrote:
On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw14896.2014@my.bristol.ac.uk
wrote:
These laws define behaviour that is expected equivalent by users;
[x for x in xs] = xs
OK -- that's the definition...
[f(x) for x in [x]] = f(x)
well, not quite: [f(x) for x in [x]] = [f(x)] Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)]
well, no. using two for expressions yields the outer product -- all combinations: In [*14*]: xs = range(3) In [*15*]: [(x,y) *for* x *in* xs *for* y *in* xs] Out[*15*]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2. But nesting the comps applies one expression, and then the other, yielding a length len(xs) list. but you wrote: [g(y) for x in xs for y in f(x)] which I'm not sure what you were expecting, as f(x) is not a sequence (probably)... To play with your examples: Define some functions that make it clear what's been applied: In [*16*]: *def* f(x): ...: *return* "f(*{}*)".format(x) ...: In [*17*]: *def* g(x): ...: *return* "g(*{}*)".format(x) and a simple sequence to use: In [*18*]: xs = range(3) Now your examples: In [*19*]: [x *for* x *in* xs] Out[*19*]: [0, 1, 2] In [*20*]: [f(x) *for* x *in* [x]] Out[*20*]: ['f(5)'] In [*21*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*21*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] OK -- all good f applied, then g, but then the last one: In [*27*]: [g(y) *for* x *in* xs *for* y *in* f(x)] Out[*27*]: ['g(f)', 'g(()', 'g(0)', 'g())', 'g(f)', 'g(()', 'g(1)', 'g())', 'g(f)', 'g(()', 'g(2)', 'g())'] in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function: In [*29*]: *def* f(x): ...: *return* 2*x Then you just get an error: In [*30*]: [g(y) *for* x *in* xs *for* y *in* f(x)] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-30-82ba3864654b> in <module>() ----> 1 [g(y) for x in xs for y in f(x)] <ipython-input-30-82ba3864654b> in <listcomp>(.0) ----> 1 [g(y) for x in xs for y in f(x)] TypeError: 'int' object is not iterable The the nested comp is what is desired here: In [*31*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*31*]: ['g(0)', 'g(2)', 'g(4)'] Except you probably want a generator expression in the inner loop to avoid bulding an extra list: In [*33*]: [g(y) *for* y *in* (f(x) *for* x *in* xs)] Out[*33*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] So back to the OP's example: In [*34*]: [f(x) + g(f(x)) *for* x *in* range(10)] Out[*34*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] that is best done with comps as: In [*36*]: [fx + g(fx) *for* fx *in* (f(x) *for* x *in* range(10))] Out[*36*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier: In [*41*]: fx = (f(x) *for* x *in* range(10)) In [*42*]: [x + g(x) *for* x *in* fx] Out[*42*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy: In [*46*]: *import* *numpy* *as* *np* In [*47*]: *def* f(x): ...: *return* x * 2 ...: In [*48*]: *def* g(x): ...: *return* x * 3 ...: ...: In [*49*]: xs = np.arange(3) In [*50*]: f(xs) + g(f(xs)) Out[*50*]: array([ 0, 8, 16]) is pretty compact, and can be "optimized with a temp: In [*51*]: fx = f(xs) ...: fx + g(fx) ...: Out[*51*]: array([ 0, 8, 16]) pretty simple isn't it? So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact? I'm guessing not :-) (the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-) (I'm still trying to figure out why folks think map() or filter() help here...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I’d like to clarify that f(x) was indeed meant to be a sequence. As per monad law: do { y <- do { x <- m; f x } g y } === do { x <- m; y <- f x; g y } I think you might have misunderstood the types of things; f: Function[a, List[b]] and g: Function[b, List[c]]. m: List[a] But I DO think my old the fly write up went wrong, converting do-notation to list comprehensions isn’t completely straightforward The above is equivalent to [g y | x <- m, y <- f x] in Haskell and the top is [g y | y <- [z |x <- m, z <- f x]] These have analogous structures in python; [g(y) for x in m for y in f(x)] and [g(y) for y in [z for x in m for z in f(x)]] (I think?) And yes the left identity law I posted was missing the [f(x)] brackets. If I’ve not made another mistake, that *should* now work? From: Chris Barker [mailto:chris.barker@noaa.gov] Sent: 15 February 2018 23:34 To: jw14896.2014@my.bristol.ac.uk Cc: Evpok Padding <evpok.padding@gmail.com>; Python-Ideas <python-ideas@python.org> Subject: Re: [Python-ideas] Temporary variables in comprehensions On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw14896.2014@my.bristol.ac.uk <mailto:jw14896.2014@my.bristol.ac.uk> > wrote: These laws define behaviour that is expected equivalent by users; [x for x in xs] = xs OK -- that's the definition... [f(x) for x in [x]] = f(x) well, not quite: [f(x) for x in [x]] = [f(x)] Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)] well, no. using two for expressions yields the outer product -- all combinations: In [14]: xs = range(3) In [15]: [(x,y) for x in xs for y in xs] Out[15]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2. But nesting the comps applies one expression, and then the other, yielding a length len(xs) list. but you wrote: [g(y) for x in xs for y in f(x)] which I'm not sure what you were expecting, as f(x) is not a sequence (probably)... To play with your examples: Define some functions that make it clear what's been applied: In [16]: def f(x): ...: return "f({})".format(x) ...: In [17]: def g(x): ...: return "g({})".format(x) and a simple sequence to use: In [18]: xs = range(3) Now your examples: In [19]: [x for x in xs] Out[19]: [0, 1, 2] In [20]: [f(x) for x in [x]] Out[20]: ['f(5)'] In [21]: [g(y) for y in [f(x) for x in xs]] Out[21]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] OK -- all good f applied, then g, but then the last one: In [27]: [g(y) for x in xs for y in f(x)] Out[27]: ['g(f)', 'g(()', 'g(0)', 'g())', 'g(f)', 'g(()', 'g(1)', 'g())', 'g(f)', 'g(()', 'g(2)', 'g())'] in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function: In [29]: def f(x): ...: return 2*x Then you just get an error: In [30]: [g(y) for x in xs for y in f(x)] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-30-82ba3864654b> in <module>() ----> 1 [g(y) for x in xs for y in f(x)] <ipython-input-30-82ba3864654b> in <listcomp>(.0) ----> 1 [g(y) for x in xs for y in f(x)] TypeError: 'int' object is not iterable The the nested comp is what is desired here: In [31]: [g(y) for y in [f(x) for x in xs]] Out[31]: ['g(0)', 'g(2)', 'g(4)'] Except you probably want a generator expression in the inner loop to avoid bulding an extra list: In [33]: [g(y) for y in (f(x) for x in xs)] Out[33]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] So back to the OP's example: In [34]: [f(x) + g(f(x)) for x in range(10)] Out[34]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] that is best done with comps as: In [36]: [fx + g(fx) for fx in (f(x) for x in range(10))] Out[36]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier: In [41]: fx = (f(x) for x in range(10)) In [42]: [x + g(x) for x in fx] Out[42]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy: In [46]: import numpy as np In [47]: def f(x): ...: return x * 2 ...: In [48]: def g(x): ...: return x * 3 ...: ...: In [49]: xs = np.arange(3) In [50]: f(xs) + g(f(xs)) Out[50]: array([ 0, 8, 16]) is pretty compact, and can be "optimized with a temp: In [51]: fx = f(xs) ...: fx + g(fx) ...: Out[51]: array([ 0, 8, 16]) pretty simple isn't it? So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact? I'm guessing not :-) (the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-) (I'm still trying to figure out why folks think map() or filter() help here...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
I’d like to clarify that f(x) was indeed meant to be a sequence. As per monad law: *do* { y *<-* *do* { x *<-* m; f x } g y } === *do* { x *<-* m; y *<-* f x; g y } I think you might have misunderstood the types of things; f: Function[a, List[b]] and g: Function[b, List[c]]. m: List[a] I’m still confused — f and g take a scale and return a list? Or take two parameters, a scalar and a list??? Either way, doesn’t really fit with python comprehensions, which are more or less expecting functions that except and return a single element (which could be a triple, but I digress) And the key point is that in python: for x in seq1 for y in seq2 Does all combinations, like a nested for loop would. But I DO think my old the fly write up went wrong, converting do-notation to list comprehensions isn’t completely straightforward Well, I don’t think I get the do notation at all.... The above is equivalent to [g y | x <- m, y <- f x] in Haskell and the top is [g y | y <- [z |x <- m, z <- f x]] These have analogous structures in python; [g(y) for x in m for y in f(x)] and [g(y) for y in [z for x in m for z in f(x)]] (I think?) Is this current Python or a new proposed syntax — ‘cause I can’t make any sense of it in python. Maybe define some f,g,m that behave as you expect, and see what python does.... If I’ve not made another mistake, that **should* *now work? Now on a phone, so can’t test, but I also can’t tell what you are expecting the result to be. Back to one of your examples: [f(x) for x in [x]] What does that mean??? for x in seq Means iterate through seq, and assign each item to the name x. If that seq has x in it — I’m not sure that is even legal python — the scope in a comprehension confuses me. But that is the equivalent is something like: it= iter(seq) while True: Try: x = next(it) Except StopIteration: Break (Excuse the caps — hard to write code on a phone) So not sure how x gets into that sequence before the loop starts. -CHB *From:* Chris Barker [mailto:chris.barker@noaa.gov <chris.barker@noaa.gov>] *Sent:* 15 February 2018 23:34 *To:* jw14896.2014@my.bristol.ac.uk *Cc:* Evpok Padding <evpok.padding@gmail.com>; Python-Ideas < python-ideas@python.org> *Subject:* Re: [Python-ideas] Temporary variables in comprehensions On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote: These laws define behaviour that is expected equivalent by users; [x for x in xs] = xs OK -- that's the definition... [f(x) for x in [x]] = f(x) well, not quite: [f(x) for x in [x]] = [f(x)] Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence) [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)] well, no. using two for expressions yields the outer product -- all combinations: In [*14*]: xs = range(3) In [*15*]: [(x,y) *for* x *in* xs *for* y *in* xs] Out[*15*]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2. But nesting the comps applies one expression, and then the other, yielding a length len(xs) list. but you wrote: [g(y) for x in xs for y in f(x)] which I'm not sure what you were expecting, as f(x) is not a sequence (probably)... To play with your examples: Define some functions that make it clear what's been applied: In [*16*]: *def* f(x): ...: *return* "f(*{}*)".format(x) ...: In [*17*]: *def* g(x): ...: *return* "g(*{}*)".format(x) and a simple sequence to use: In [*18*]: xs = range(3) Now your examples: In [*19*]: [x *for* x *in* xs] Out[*19*]: [0, 1, 2] In [*20*]: [f(x) *for* x *in* [x]] Out[*20*]: ['f(5)'] In [*21*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*21*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] OK -- all good f applied, then g, but then the last one: In [*27*]: [g(y) *for* x *in* xs *for* y *in* f(x)] Out[*27*]: ['g(f)', 'g(()', 'g(0)', 'g())', 'g(f)', 'g(()', 'g(1)', 'g())', 'g(f)', 'g(()', 'g(2)', 'g())'] in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function: In [*29*]: *def* f(x): ...: *return* 2*x Then you just get an error: In [*30*]: [g(y) *for* x *in* xs *for* y *in* f(x)] --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-30-82ba3864654b> in <module>() ----> 1 [g(y) for x in xs for y in f(x)] <ipython-input-30-82ba3864654b> in <listcomp>(.0) ----> 1 [g(y) for x in xs for y in f(x)] TypeError: 'int' object is not iterable The the nested comp is what is desired here: In [*31*]: [g(y) *for* y *in* [f(x) *for* x *in* xs]] Out[*31*]: ['g(0)', 'g(2)', 'g(4)'] Except you probably want a generator expression in the inner loop to avoid bulding an extra list: In [*33*]: [g(y) *for* y *in* (f(x) *for* x *in* xs)] Out[*33*]: ['g(f(0))', 'g(f(1))', 'g(f(2))'] So back to the OP's example: In [*34*]: [f(x) + g(f(x)) *for* x *in* range(10)] Out[*34*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] that is best done with comps as: In [*36*]: [fx + g(fx) *for* fx *in* (f(x) *for* x *in* range(10))] Out[*36*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier: In [*41*]: fx = (f(x) *for* x *in* range(10)) In [*42*]: [x + g(x) *for* x *in* fx] Out[*42*]: ['f(0)g(f(0))', 'f(1)g(f(1))', 'f(2)g(f(2))', 'f(3)g(f(3))', 'f(4)g(f(4))', 'f(5)g(f(5))', 'f(6)g(f(6))', 'f(7)g(f(7))', 'f(8)g(f(8))', 'f(9)g(f(9))'] The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy: In [*46*]: *import* *numpy* *as* *np* In [*47*]: *def* f(x): ...: *return* x * 2 ...: In [*48*]: *def* g(x): ...: *return* x * 3 ...: ...: In [*49*]: xs = np.arange(3) In [*50*]: f(xs) + g(f(xs)) Out[*50*]: array([ 0, 8, 16]) is pretty compact, and can be "optimized with a temp: In [*51*]: fx = f(xs) ...: fx + g(fx) ...: Out[*51*]: array([ 0, 8, 16]) pretty simple isn't it? So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact? I'm guessing not :-) (the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-) (I'm still trying to figure out why folks think map() or filter() help here...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 2/15/2018 8:37 PM, Chris Barker - NOAA Federal wrote:
Reusing a previously bound name as an iteration variable is a bad idea. It works in 3.x because the outermost iterable, but only the outermost iterable, is pre-calculated before executing the comprehension. Hence 'x in [x]' sometimes works, and sometimes not. ('Outermost' is topmost in nested loops, left most in comprehension.)
To put it another way, l = [x for x in [x]] is actually calculated as _temp = [x]; l = [x for x in _temp]. In general, other iterables cannot be precalculated since they might depend on prior iteration variables. -- Terry Jan Reedy
On Thu, Feb 15, 2018 at 10:13:37AM +0000, Jamie Willis wrote:
Python is not a functional language, and the usual Pythonic solution to an overly complex or inefficient functional expression is to refactor into non-functional style. In my experience, most Python users have never even heard of monads, and those who have, few grok them. (I know I don't.)
I think these should be: [x for x in xs] = list(xs) # not to be confused with [x] [f(x) for x in [x]] = [f(x)] [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in [f(x)]]
Even though that last law isn't completely analogous to the given example from the OP,
It can be used though. He has: [f(x) + g(f(x)) for x in xs] which can be written as [y + g(y) for x in xs for y in [f(x)]] Here's an example: # calls ord() twice for each x py> [ord(x) + ord(x)**2 for x in "abc"] [9506, 9702, 9900] # calls ord() once for each x py> [y + y**2 for x in "abc" for y in [ord(x)]] [9506, 9702, 9900] And one last version: py> [y + y**2 for y in [ord(x) for x in "abc"]] [9506, 9702, 9900] In production, I'd change the last example to use a generator comprehension.
Law 3 does apply, and I'm not sure what you mean by the statement that we can't flatten the comprehension down. -- Steve
A generator can be a good idea, however, I wonder if it's really readable to have a `f_samples`. And the structure is the same as in [y + g(y) for y in [f(x) for x in range(10)]] if you replace the list with a generator. And it's similar for other solutions you mentioned. Well, I know that it can be quite different for everyone to determine whether a piece of code is readable or not, maybe it's wise to wait for more opinions. There's another problem that, as you distinguished the `simple` and `more complex` case, the offered solutions seem very specific, I'm not sure if there's a universal solution for every case of temporary variables in comprehensions. If not, I think it's too high cost to reject a new syntax if you need to work out a new solution every time. At 2018-02-15 17:53:21, "Evpok Padding" <evpok.padding@gmail.com> wrote: For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`, I don't really see what the issue is, if you really want to make it shorter, you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare case where I like `map` more than comprehensions. For more complex case, just define a intermediate generator along the lines ``` f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] ``` Which does exactly the same thing but - Is more readable and explicit - Has no memory overhead thanks to lazy evaluation (btw, you should consider generators for your nested comprenshions) While I am sometimes in the same state of mind, wishing for variables in comprehensions seems to me like a good indicator that your code needs refactoring. Best, E On 15 February 2018 at 10:32, Jamie Willis <jw14896.2014@my.bristol.ac.uk> wrote:
On 15 February 2018 at 05:56, fhsxfhsx <fhsxfhsx@126.com> wrote:
... as long as the code inside the comprehension remains relatively simple. It's easy to abuse comprehensions to the point where they are less readable than a for loop, but that's true of a lot of things, so isn't a specific problem with comprehensions.
Agreed. I hit that quite often.
That is a workaround (and one I'd not thought of before) but I agree it's ugly, and reduces readability. Actually, factoring out the inner comprehension like Evpok Padding suggests: f_samples = (f(x) for x in range(10)) [y+g(y) for y in f_samples] is very readable and effective, IMO, so it's not *that* obvious that local names are beneficial.
In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
"We need" is a bit strong here. "It would be useful to have" is probably true for some situations.
The problem is that you haven't proposed an actual syntax here, just that one should be invented. There have been discussions on this in the past (a quick search found https://mail.python.org/pipermail/python-ideas/2011-April/009863.html and https://mail.python.org/pipermail/python-ideas/2012-January/013468.html, for example).
The problem isn't so much "whether we should allow it" as "can we find a syntax that is acceptable", and only then "does the new syntax give sufficient benefit to be worth adding". New syntax has a pretty high cost, and proposals that don't suggest explicit syntax will get stuck because you can't judge whether adding the capability is "worth it" without being clear on what the cost is - particularly when the benefit is relatively small (which this is). Agreed that it's important to focus on the general problem, but part of the discussion *will* include arguing as to why the existing workarounds and alternatives are less acceptable than new syntax. And that will need to include discussion of specific cases. Generally, in that sort of discussion, artificial examples like "y=f(x)" don't fare well because it's too easy to end up just debating subjective views on "readability". If you can provide examples from real-world code that clearly demonstrate the cost in terms of maintainability of the existing workarounds, that will help your argument a lot. Although you'll need to be prepared for questions like "would you be willing to drop support for versions of Python older than 3.8 in order to get this improvement?" - it's surprisingly hard to justify language (as opposed to library) changes when you really stop and think about it. Which is not to say that it can't be done, just that it's easy to underestimate the effort needed. Paul
Thank you Paul, what you said is enlightening and I agree on most part of it. I'll propose two candidate syntaxs. 1. `with ... as ...` This syntax is more paralles as there would be `for` and `with` clause as well as `for` and `with` statement. However, the existing `with` statement is semantically different from this one, although similar. 2. `for ... is ...` This syntax is more uniform as the existing `for` clause make an iterator which is a special kind of variable. However, I'm afraid this syntax might be confused with `for ... in ...` as they differ only on one letter. I like the latter one better. Other proposes are absolutely welcome. And here is an example which appears quite often in my code where I think a new syntax can help a lot: Suppose I have an list of goods showing by their ids in database, and I need to transform the ids into json including information from two tables, `Goods` and `GoodsCategory`, where the first table recording `id`, `name` and `category_id` indicating which category the goods belongs to, the second table recording `id`, `name` and `type` to the categories. With the new syntax, I can write [ { 'id': goods.id, 'name': goods.name, 'category': gc.name, 'category_type': gc.type, } for goods_id in goods_id_list for goods is Goods.get_by_id(goods_id) for gc is GoodsCategory.get_by_id(goods.category_id) ] And I cannot think of any good solutions as this one without it. To generalize this case, for each element of the list, I need two temporary variables (`goods` and `gc` in my case), and each one was used twice. And reply to the two past discussions you mentioned, 1.https://mail.python.org/pipermail/python-ideas/2011-April/009863.html This mail gave a solution to modify function `f` to keep the result. The weak point is obvious, you must modify the function `f`. 2.https://mail.python.org/pipermail/python-ideas/2012-January/013468.html This mail wrote
At 2018-02-15 18:08:46, "Paul Moore" <p.f.moore@gmail.com> wrote:
On Sun, Feb 18, 2018 at 12:23:28AM +0800, fhsxfhsx wrote:
I don't think they are even a little bit similar. The existing `with` statement is for controlling cleanup code (using a context manager). This proposal doesn't have anything to do with context managers or cleanup code. It's just a different way to spell "name = value" inside comprehensions.
Indeed. And frankly, treated as English grammar, "for value is name" doesn't make sense and is horribly ugly to my eyes: result = [x for value in sequence for value+1 is x]
I can of a few, starting with the most simple: write a helper function. Not every problem needs to be solved with new syntax. def dict_from_id(goods_id): goods = Goods.get_by_id(goods_id) gc = GoodsCategory.get_by_id(goods.category_id) return {'id': goods.id, 'name': goods.name, 'category': gc.name, 'category_type': gc.type } result = [dict_from_id(goods_id) for goods_id in goods_id_list] That's much nicer to read, you can document and test the dict_from_id() function, no new systax is required, it is easy to refactor, and I very much doubt that adding one extra function call is going to be a significant slowdown compared to the cost of two calls to get_by_id() methods and constructing a dict. (And if as you add more fields to the dict, the overhead of the function call becomes an even smaller proportion.) Or you can make this a method of the goods object, which is arguably a better OO design. Let the goods object be responsible for creating the dict. result = [Goods.get_by_id(goods_id).make_dict() for goods_id in goods_id_list] # or if you prefer result = [goods.make_dict() for goods in map(Goods.get_by_id, goods_id_list)] Here is a third solution: use a for-loop iterating over a single-item tuple to get the effect of a local assignment to a temporary variable: result = [{ # dict display truncated for brevity... } for goods_id in goods_id_list for goods in (Goods.get_by_id(goods_id),) for gc in (GoodsCategory.get_by_id(goods.category_id),) ] If you don't like the look of single-item tuples (foo,) you can use single-item lists instead [x] but they are a tiny bit slower to create. Serhiy has suggested that the interpreter can optimize the single-item loop to make it as fast as a bare assignment: https://bugs.python.org/issue32856 I think this is a neat trick, although Yuri thinks it is an ugly hack and doesn't want to encourage it. Neat or ugly, I think it is better than "for value is name". -- Steve
Hi fhsxfhsx, and welcome. My comments below, interleaved with yours. On Thu, Feb 15, 2018 at 01:56:44PM +0800, fhsxfhsx wrote: [quoted out of order]
Whether or not to allow this proposal will depend on what alternate solutions to the problem already exist, so your specific example is very relevant. Any proposed change has to compete with existing solutions.
While I personally agree with you, many others disagree. I know quite a few experienced, competent Python programmers who avoid list comprehensions because they consider them harder to read and reason about. They consider a regular for-loop better precisely because you do see the explicit call to append. (In my experience, those of us who get functional-programming idioms often forget that others find them tricky.) The point is that list comprehensions are already complex enough that they are difficult for many people to learn, and some people never come to grips with them. Adding even more features comes with a cost. The bottom line is that it isn't clear to me that allowing local variables inside comprehensions will make them more readable.
The usual comments about premature optimisation apply here. Setting a new comprehension variable is not likely to be free, and may even be more costly than calling f(x) twice if f() is a cheap expression: [x+1 + some_func(x+1) for x in range(10)] could be faster than [y + some_func(y) for x in range(10) let y = x + 1] or whatever syntax we come up with.
Someone may argue that one can write [y + g(y) for y in [f(x) for x in range(10)]]
Indeed. This would be the functional-programming solution, and I personally think it is an excellent one. The only changes are that I'd use a generator expression for the intermediate value, avoiding the need to make a full list, and I would lay it out more nicely, using whitespace to make the structure more clear: result = [y + g(y) for y in (f(x) for x in range(10)) ]
but it's not as clear as to show what `y` is in a subsequent clause, not to say there'll be another temporary list built in the process.
There's no need to build the temporary list. Use a generator comprehension. And I disagree that the value of y isn't as clear. An alternative is simply to refactor your list comprehension. Move the calls to f() and g() into a helper function: def func(x): y = f(x) return y + g(y) and now you can write the extremely clear comprehension [func(x) for x in range(10)] that needs no extra variable. [...]
In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
"Need" is very strong. I think that the two alternatives I mention above cover 95% of the cases where might use a local variable in a comprehension. And of the remaining cases, many of them will be so complex that they should be re-written as an explicit for-loop. So in my opinion, we're only talking about a "need" to solve the problem for a small proportion of cases: - most comprehensions don't need a local variable (apart from the loop variable) at all; - of those which do need a local variable, most can be easily solved using a nested comprehension or a helper function; - of those which cannot be solved that way, most are complicated enough that they should use a regular for-loop; - leaving only a small number of cases which are complicated enough to genuinely benefit from local variables but not too complicated. So this is very much a borderline feature. Occasionally it would be "nice to have", but on the negative side: - it adds complexity to the language; - makes comprehensions harder to read; - and people will use it unnecessarily where there is no readability or speed benefit (premature optimization again). It is not clear to me that we should burden *all* Python programmers with additional syntax and complexity of an *already complex* feature for such a marginal improvement.
How can you say that the new syntax is "clear, explicit and readable" when you haven't proposed any new syntax yet? For lack of anything better, I'm going to suggest "let y = f(x)" as the syntax, although personally I don't like it even a bit. Where should the assignment go? [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] [(y, y**2) for x in (1, 2, 3, 4) let y = x+1] I think they're both pretty ugly, but I can't think of anything else. Can we rename the loop variable, or is that an error? [(x, x**2) let x = x+1 for x in (1, 2, 3, 4)] How do they interact when you have multiple loops and if-clauses? [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 for a in range(y) let z = a+1 if z > 2 for b in range(z) let w = z+1] For simplicity, perhaps we should limit any such local assignments to the very end of the comprehension: [expression for name in sequence <zero or more for-loops and if-clauses> <zero or more let-clauses> ] but that means we can't optimise this sort of comprehension: [expression for x in sequence for y in (something_expensive(x) + function(something_expensive(x)) ] Or these: [expression for x in sequence if something_expensive(x) or condition(something_expensive(x)) ] I think these are very hard questions to answer. -- Steve
A bit of a nit — function call overhead is substantial in python, so if that is an actual function, rather than a simple expression, it’ll likely be slower to call it twice for any but trivially small iterables.
[(y, y**2) let y = x+1 for x in (1, 2, 3, 4)]
Do we need the let? [ g(y) for y = f(x) for c in seq] Or, with expressions: [y + y**2 for y = x+1 for x in (1,2,3)] Maybe that would be ambiguous— I haven’t thought carefully about it. -CHB
Hi Steve, Thank you for so detailed comments. My comments also below interleaved with yours. At 2018-02-16 08:57:40, "Steven D'Aprano" <steve@pearwood.info> wrote:
To be frank, I had not thought of this before. However, in my opinion, when considering adding a new syntax, we care more about the marginal cost. I mean, I think it is the functional-programming way which is tricky, but allowing a new syntax would not make things worse. Well, that's just a guess, maybe only those who are puzzled with comprehensions can give us an answer.
Someone may argue that one can write >> [y + g(y) for y in [f(x) for x in range(10)]] > >Indeed. This would be the functional-programming solution, and I >personally think it is an excellent one. The only changes are that I'd >use a generator expression for the intermediate value, avoiding the need >to make a full list, and I would lay it out more nicely, using >whitespace to make the structure more clear: > > result = [y + g(y) for y in > (f(x) for x in range(10)) > ] >
In my opinion, [ y + g(y) for x in range(10) let y = f(x) ] is better because it's more corresponding to a for-loop for x in range(10): y = f(x) result.append(y + g(y)) In my opinion, comprehensions are not real functional-programming because there is not even a function. Though there're similarities, map and filter are real functional-programming. Since the similarity between for-clause in comprehensions and the for-loop, I think it's better to write comprehensions more close to for-loop. I don't know but I guess maybe it can also help those who fear comprehensions better embrace them?
I think it can be a goods idea if there's a name to `func` which is easy to understand, or `func` is put close to the comprehension and is in a obvious place. But I feel it's not for the case I gave in another mail to Paul, https://mail.python.org/pipermail/python-ideas/2018-February/048997.html, (I'm sorry that the example is quite long, and I don't hope to copy it here) To me, it can be confusing to have several `func` when I have several lists at the same time and have to transform them each in a similar but different way.
[...] >> In a word, what I'm arguing is that we need a way to assign temporary >> variables in a comprehension. > >"Need" is very strong. I think that the two alternatives I mention above >cover 95% of the cases where might use a local variable in a >comprehension. And of the remaining cases, many of them will be so >complex that they should be re-written as an explicit for-loop. So in my >opinion, we're only talking about a "need" to solve the problem for a >small proportion of cases: > >- most comprehensions don't need a local variable (apart from > the loop variable) at all; > >- of those which do need a local variable, most can be easily > solved using a nested comprehension or a helper function; > >- of those which cannot be solved that way, most are complicated > enough that they should use a regular for-loop; > >- leaving only a small number of cases which are complicated enough > to genuinely benefit from local variables but not too complicated. > >So this is very much a borderline feature. Occasionally it would be >"nice to have", but on the negative side: > >- it adds complexity to the language; > >- makes comprehensions harder to read; > >- and people will use it unnecessarily where there is no readability > or speed benefit (premature optimization again). > >It is not clear to me that we should burden *all* Python programmers >with additional syntax and complexity of an *already complex* feature >for such a marginal improvement. > > >> In my opinion, code like >> [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**] >> is more natural than any solution we now have. >> And that's why I pro the new syntax, it's clear, explicit and readable > >How can you say that the new syntax is "clear, explicit and readable" >when you haven't proposed any new syntax yet? > >For lack of anything better, I'm going to suggest "let y = f(x)" as the >syntax, although personally I don't like it even a bit. > >Where should the assignment go? > > [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] > > [(y, y**2) for x in (1, 2, 3, 4) let y = x+1] > >I think they're both pretty ugly, but I can't think of anything else. > >Can we rename the loop variable, or is that an error? > > [(x, x**2) let x = x+1 for x in (1, 2, 3, 4)] > >How do they interact when you have multiple loops and if-clauses? > > [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 > for a in range(y) let z = a+1 if z > 2 > for b in range(z) let w = z+1] > > >For simplicity, perhaps we should limit any such local assignments to >the very end of the comprehension: > > [expression for name in sequence > <zero or more for-loops and if-clauses> > <zero or more let-clauses> > ] > >but that means we can't optimise this sort of comprehension: > > [expression for x in sequence > for y in (something_expensive(x) + function(something_expensive(x)) > ] > >Or these: > > [expression for x in sequence > if something_expensive(x) or condition(something_expensive(x)) > ] > > >I think these are very hard questions to answer. > I think the assignment should be treated equally as for-clause and if-clause which means
[(y, y**2) for x in (1, 2, 3, 4) let y = x+1] would be a right syntax. And [(x, x**2) for x in (1, 2, 3, 4) let x = x+1] would not cause an error because we can also write [(x, x**2) for x in (1, 2, 3, 4) for x in (4, 3, 2, 1)] now. I didn't see any problem in [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 for a in range(y) let z = a+1 if z > 2 for b in range(z) let w = z+1] In my opinion, it would behave the same as for x in (1, 2, 3, 4): y = x+1 for a in range(y): z = a+1 if z > 2: for b in range(z): w = z+1 mylist.append((w, w**2)) According to my understanding, the present for-clause and if-clause does everything quite similar to this nested way,
-- >Steve >_______________________________________________ >Python-ideas mailing list >Python-ideas@python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/
I believe list comprehensions are difficult to read because they are not formatted properly. For me, list comprehension clauses are an expression, followed by clauses executed in the order. Any list comprehension with more than one clause should be one-line-per clause. Examples inline: On 2018-02-15 19:57, Steven D'Aprano wrote:
Where should the assignment go? [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] [(y, y**2) for x in (1, 2, 3, 4) let y = x+1]
Since y is a function of x, it must follow the for clause:
They are applied in order:
which is a short form for:
On 2018-02-23 12:44, Neil Girdhar wrote:
Saving the indentation? Oh yes, for sure! This code reads like a story, the indentation is superfluous to that story. Should we add it to Python? I don't know; I quick scan through my own code, and I do not see much opportunity for list comprehensions of this complexity. Either my data structures are not that complicated, or I have try/except blocks inside a loop, or I am using a real query language (like SQL). pythonql seems to solve all these problems well enough (https://github.com/pythonql/pythonql).
On Fri, Feb 23, 2018 at 1:42 PM Kyle Lahnakoski <klahnakoski@mozilla.com> wrote:
That's a good thing that you are not writing code like this. I don't agree that the indentation is "superfluous". It makes the code easy to read. Anyway, Google's Python style guide also agrees that very long comprehensions like that are not worth it. ( https://google.github.io/styleguide/pyguide.html?showone=List_Comprehensions... ) Either my data structures are not that complicated, or I have try/except
Hi list,
personally I think that the biggest problem readability-wise is that "for" is a post-fix operator, which makes generators much harder to read. It's also very different from normal for loops, which have the "for" at the top. IMHO generators would be much easier to read with a prefix for, as in [for x in range(10): f(x) + g(f(x))] also nested generators get nicer like that: [for y in (for x in range(10): f(x)): y + g(y)] one could critique here that we shouldn't use colons in expressions, but that boat has sailed: we use them for lambdas. We do not write sq = x**2 lambda x and I am not proposing that. Also if/else could be written with colons, but I am not sure whether that's actually nicer: val = (if attr is None: 5 else: attr + 3) but it certainly is in case of ifs in generators: [for x in range(10): if x % 3 != 2: x] which could be problematic to parse if you compare that to [for x in range(10): if x % 3 == 2: x - 1 else: x + 1] one could even dig out the often-proposed always-rejected except-in-expression: [for x in range(10): try: f(x) except WhateverError: None] or even a with: [for x in file_names: with open(x) as f: f.read()] Greetings Martin
You should give an actual motivating example. I think none of these suggestions are more readable than just writing things out as a for loop. You argue that you want to avoid appending to a result list. In that case, I suggest writing your pattern as a generator function. Best, Neil On Thursday, February 15, 2018 at 2:03:31 AM UTC-5, fhsxfhsx wrote:
participants (13)
-
Chris Barker -
Chris Barker - NOAA Federal
-
Evpok Padding -
fhsxfhsx -
Jamie Willis -
jw14896＠my.bristol.ac.uk -
Kyle Lahnakoski -
Martin Teichmann -
Neil Girdhar -
Paul Moore -
Stephan Houben -
Steven D'Aprano -
Terry Reedy