[Python-ideas] Temporary variables in comprehensions
fhsxfhsx
fhsxfhsx at 126.com
Sat Feb 17 13:54:42 EST 2018
Hi Steve, Thank you for so detailed comments.
My comments also below interleaved with yours.
At 2018-02-16 08:57:40, "Steven D'Aprano" <steve at pearwood.info> wrote:
>Hi fhsxfhsx, and welcome.
>
>My comments below, interleaved with yours.
>
>
>On Thu, Feb 15, 2018 at 01:56:44PM +0800, fhsxfhsx wrote:
>
>[quoted out of order]
>> And I hope the discussion could focus more on whether we should allow
>> assigning temporary variables in comprehensions rather than how to
>> solve the specific example I mentioned above.
>
>Whether or not to allow this proposal will depend on what alternate
>solutions to the problem already exist, so your specific example is very
>relevant. Any proposed change has to compete with existing solutions.
>>
>> As far as I can see, a comprehension like
>> alist = [f(x) for x in range(10)]
>> is better than a for-loop
>> for x in range(10):
>> alist.append(f(x))
>> because the previous one shows every element of the list explicitly so
>> that we don't need to handle `append` mentally.
>
>While I personally agree with you, many others disagree. I know quite a
>few experienced, competent Python programmers who avoid list
>comprehensions because they consider them harder to read and reason
>about. They consider a regular for-loop better precisely because you do
>see the explicit call to append.
>
>(In my experience, those of us who get functional-programming idioms
>often forget that others find them tricky.)
>
>The point is that list comprehensions are already complex enough that
>they are difficult for many people to learn, and some people never come
>to grips with them. Adding even more features comes with a cost.
>
>The bottom line is that it isn't clear to me that allowing local
>variables inside comprehensions will make them more readable.
>
To be frank, I had not thought of this before.
However, in my opinion, when considering adding a new syntax, we care more about the marginal cost.
I mean, I think it is the functional-programming way which is tricky, but allowing a new syntax would not make things worse.
Well, that's just a guess, maybe only those who are puzzled with comprehensions can give us an answer.
> >> But when it comes to something like >> [f(x) + g(f(x)) for x in range(10)] >> you find you have to sacrifice some readableness if you don't want two >> f(x) which might slow down your code. > >The usual comments about premature optimisation apply here. > >Setting a new comprehension variable is not likely to be free, and may even be >more costly than calling f(x) twice if f() is a cheap expression: > > [x+1 + some_func(x+1) for x in range(10)] > >could be faster than > > [y + some_func(y) for x in range(10) let y = x + 1] > >or whatever syntax we come up with. >
It is true. But since there are still so many cases where a temporary variable is faster.
Also, even without let-clause, one can write a for-loop with a temporary variable which slow down the code.
So, it seems that "setting a new comprehension variable may even be more costly" does not show any uselessness of temporary variables in comprehensions.
> >> Someone may argue that one can write >> [y + g(y) for y in [f(x) for x in range(10)]] > >Indeed. This would be the functional-programming solution, and I >personally think it is an excellent one. The only changes are that I'd >use a generator expression for the intermediate value, avoiding the need >to make a full list, and I would lay it out more nicely, using >whitespace to make the structure more clear: > > result = [y + g(y) for y in > (f(x) for x in range(10)) > ] >
In my opinion,
[
y + g(y)
for x in range(10)
let y = f(x)
]
is better because it's more corresponding to a for-loop
for x in range(10):
y = f(x)
result.append(y + g(y))
In my opinion, comprehensions are not real functional-programming because there is not even a function. Though there're similarities, map and filter are real functional-programming. Since the similarity between for-clause in comprehensions and the for-loop, I think it's better to write comprehensions more close to for-loop.
I don't know but I guess maybe it can also help those who fear comprehensions better embrace them?
>
>> but it's not as clear as to show what `y` is in a subsequent clause,
>> not to say there'll be another temporary list built in the process.
>
>There's no need to build the temporary list. Use a generator
>comprehension. And I disagree that the value of y isn't as clear.
>
>An alternative is simply to refactor your list comprehension. Move the
>calls to f() and g() into a helper function:
>
>def func(x):
> y = f(x)
> return y + g(y)
>
>and now you can write the extremely clear comprehension
>
>[func(x) for x in range(10)]
>
>that needs no extra variable.
>
I think it can be a goods idea if there's a name to `func` which is easy to understand, or `func` is put close to the comprehension and is in a obvious place.
But I feel it's not for the case I gave in another mail to Paul, https://mail.python.org/pipermail/python-ideas/2018-February/048997.html,
(I'm sorry that the example is quite long, and I don't hope to copy it here)
To me, it can be confusing to have several `func` when I have several lists at the same time and have to transform them each in a similar but different way.
> > >[...] >> In a word, what I'm arguing is that we need a way to assign temporary >> variables in a comprehension. > >"Need" is very strong. I think that the two alternatives I mention above >cover 95% of the cases where might use a local variable in a >comprehension. And of the remaining cases, many of them will be so >complex that they should be re-written as an explicit for-loop. So in my >opinion, we're only talking about a "need" to solve the problem for a >small proportion of cases: > >- most comprehensions don't need a local variable (apart from > the loop variable) at all; > >- of those which do need a local variable, most can be easily > solved using a nested comprehension or a helper function; > >- of those which cannot be solved that way, most are complicated > enough that they should use a regular for-loop; > >- leaving only a small number of cases which are complicated enough > to genuinely benefit from local variables but not too complicated. > >So this is very much a borderline feature. Occasionally it would be >"nice to have", but on the negative side: > >- it adds complexity to the language; > >- makes comprehensions harder to read; > >- and people will use it unnecessarily where there is no readability > or speed benefit (premature optimization again). > >It is not clear to me that we should burden *all* Python programmers >with additional syntax and complexity of an *already complex* feature >for such a marginal improvement. > > >> In my opinion, code like >> [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**] >> is more natural than any solution we now have. >> And that's why I pro the new syntax, it's clear, explicit and readable > >How can you say that the new syntax is "clear, explicit and readable" >when you haven't proposed any new syntax yet? > >For lack of anything better, I'm going to suggest "let y = f(x)" as the >syntax, although personally I don't like it even a bit. > >Where should the assignment go? > > [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)] > > [(y, y**2) for x in (1, 2, 3, 4) let y = x+1] > >I think they're both pretty ugly, but I can't think of anything else. > >Can we rename the loop variable, or is that an error? > > [(x, x**2) let x = x+1 for x in (1, 2, 3, 4)] > >How do they interact when you have multiple loops and if-clauses? > > [(w, w**2) for x in (1, 2, 3, 4) let y = x+1 > for a in range(y) let z = a+1 if z > 2 > for b in range(z) let w = z+1] > > >For simplicity, perhaps we should limit any such local assignments to >the very end of the comprehension: > > [expression for name in sequence > <zero or more for-loops and if-clauses> > <zero or more let-clauses> > ] > >but that means we can't optimise this sort of comprehension: > > [expression for x in sequence > for y in (something_expensive(x) + function(something_expensive(x)) > ] > >Or these: > > [expression for x in sequence > if something_expensive(x) or condition(something_expensive(x)) > ] > > >I think these are very hard questions to answer. >
I think the assignment should be treated equally as for-clause and if-clause which means
[(y, y**2) for x in (1, 2, 3, 4) let y = x+1]
would be a right syntax.
And
[(x, x**2) for x in (1, 2, 3, 4) let x = x+1]
would not cause an error because we can also write
[(x, x**2) for x in (1, 2, 3, 4) for x in (4, 3, 2, 1)]
now.
I didn't see any problem in
[(w, w**2) for x in (1, 2, 3, 4) let y = x+1
for a in range(y) let z = a+1 if z > 2
for b in range(z) let w = z+1]
In my opinion, it would behave the same as
for x in (1, 2, 3, 4):
y = x+1
for a in range(y):
z = a+1
if z > 2:
for b in range(z):
w = z+1
mylist.append((w, w**2))
According to my understanding, the present for-clause and if-clause does everything quite similar to this nested way,
> >-- >Steve >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180218/ea4336d7/attachment-0001.html>
More information about the Python-ideas
mailing list