[Python-ideas] Temporary variables in comprehensions

[Paul Moore]

On 15 February 2018 at 05:56, fhsxfhsx <fhsxfhsx at 126.com> wrote:
> As far as I can see, a comprehension like
> alist = [f(x) for x in range(10)]
> is better than a for-loop
> for x in range(10):
>   alist.append(f(x))
> because the previous one shows every element of the list explicitly so that
> we don't need to handle `append` mentally.

... as long as the code inside the comprehension remains relatively
simple. It's easy to abuse comprehensions to the point where they are
less readable than a for loop, but that's true of a lot of things, so
isn't a specific problem with comprehensions.

> But when it comes to something like
> [f(x) + g(f(x)) for x in range(10)]
> you find you have to sacrifice some readableness if you don't want two f(x)
> which might slow down your code.

Agreed. I hit that quite often.

> Someone may argue that one can write
> [y + g(y) for y in [f(x) for x in range(10)]]
> but it's not as clear as to show what `y` is in a subsequent clause, not to
> say there'll be another temporary list built in the process.
> We can even replace every comprehension with map and filter, but that would
> face the same problems.

That is a workaround (and one I'd not thought of before) but I agree
it's ugly, and reduces readability. Actually, factoring out the inner
comprehension like Evpok Padding suggests:

    f_samples = (f(x) for x in range(10))
    [y+g(y) for y in f_samples]

is very readable and effective, IMO, so it's not *that* obvious that
local names are beneficial.

> In a word, what I'm arguing is that we need a way to assign temporary
> variables in a comprehension.

"We need" is a bit strong here. "It would be useful to have" is
probably true for some situations.

> In my opinion, code like
> [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**]
> is more natural than any solution we now have.
> And that's why I pro the new syntax, it's clear, explicit and readable, and
> is nothing beyond the functionality of the present comprehensions so it's
> not complicated.

The problem is that you haven't proposed an actual syntax here, just
that one should be invented. There have been discussions on this in
the past (a quick search found
https://mail.python.org/pipermail/python-ideas/2011-April/009863.html
and https://mail.python.org/pipermail/python-ideas/2012-January/013468.html,
for example).

> And I hope the discussion could focus more on whether we should allow
> assigning temporary variables in comprehensions rather than how to solve the
> specific example I mentioned above.

The problem isn't so much "whether we should allow it" as "can we find
a syntax that is acceptable", and only then "does the new syntax give
sufficient benefit to be worth adding". New syntax has a pretty high
cost, and proposals that don't suggest explicit syntax will get stuck
because you can't judge whether adding the capability is "worth it"
without being clear on what the cost is - particularly when the
benefit is relatively small (which this is).

Agreed that it's important to focus on the general problem, but part
of the discussion *will* include arguing as to why the existing
workarounds and alternatives are less acceptable than new syntax. And
that will need to include discussion of specific cases. Generally, in
that sort of discussion, artificial examples like "y=f(x)" don't fare
well because it's too easy to end up just debating subjective views on
"readability". If you can provide examples from real-world code that
clearly demonstrate the cost in terms of maintainability of the
existing workarounds, that will help your argument a lot. Although
you'll need to be prepared for questions like "would you be willing to
drop support for versions of Python older than 3.8 in order to get
this improvement?" - it's surprisingly hard to justify language (as
opposed to library) changes when you really stop and think about it.
Which is not to say that it can't be done, just that it's easy to
underestimate the effort needed.

Paul