[Python-ideas] Accessing the result of comprehension's expression from the conditional

[Lie Ryan]

Aahz wrote:
> On Fri, Jun 19, 2009, Lie Ryan wrote:
>> In list/generator comprehension, currently we have no way to access the
>> result of the expression and have to write something like this:
>>
>> [f(x) for x in l if f(x) > 0]
>>
>> if f() is heavy or non-pure (i.e. have side effects), calling f() twice
>> might be undesirable.
> 
> Listcomps and genexps are like lambdas: run up against their limits and
> you should switch to a regular for loop or generator.

I think of it not as limitation but as an odd gap in functionality. I
think having the semantics that the filtering is done after the
expression part would be much more useful than the current behavior
(filtering before expression).

If filtering is done after expression, we can access both the original
and the transformed objects (it may also be possible to optimize the
cases where the filter does not use the transformed objects, although
this complex behavior wouldn't be pythonic)

Try rewriting this:

res = [x**x as F for x in nums if F < 100]
(note: this is my new preferred syntax)

Attempts:

- Using a regular for-loop

res = []
for x in nums:
    F = x**x
    if F < 100:
        res.append(F)

remarks: five lines that's much more difficult to understand than a
single, concise expression in standard form.

- Using nested comprehension

res = [F for F in (x**x for x in nums) if F < 100]

remarks: using nested list comprehension is DRY (on a loose definition
of DRY principle). I have to repeat the comprehension body twice, when I
only need one expression and one filtering.

- Using map()

res = [F for F in map(lambda x: x**x, nums) if F < 100]

remarks: when the expression part is just a simple expression, you have
to use lambda and that's plain ugly. Not to mention when you want to
filter based on both x and F.


Advantages of the proposal:
- shorter
- faster, as looping is done in C
- more readable. The main advantage of comprehension is that it have
standardized form, which is easier to understand, unlike a for-loop
which can have an infinite number of variations.
- (unnecessary) nested comprehension is an abuse.
- with `as` keyword, no new keyword and no ambiguity since currently
`as` cannot exist inside comprehension.

Disadvantages:
- reverses the current semantic of filtering-then-expression. This
shouldn't be too much problem since side-effect on the expression part
is a cardinal sin and...
- if the expression part is heavy, it might be possible to do
optimization by filtering first when the filter part does not require
the result (i.e. when there is no "as" clause). A good side effect of
this optimization is codes that relies on filtering being done before
expression will just work as they cannot contain an `as` keyword.
(As "simple is better than complex", I actually don't really like `as`
can change evaluation order; I much prefer to keep everything simple and
consistent, i.e. always evaluate expression then filter or otherwise)

possible syntaxes:
- [x**x as F for x in nums if F < 100]
  the as keyword is already often used to rename things
  (in with, import, etc) I like this one much better than @. The as
  part, of course, is optional
- [x**x for x in nums if @ < 100]
  the initial proposed syntax, ugly as hell.