[Python-ideas] Accessing the result of comprehension's expression from the conditional

[Lie Ryan]

Paul Moore wrote:
> 2009/6/19 Lie Ryan <lie.1296 at gmail.com>:
>> res = [x**x as F for x in nums if F < 100]
>> (note: this is my new preferred syntax)
> [...]
>> Advantages of the proposal:
>> - shorter
>> - faster, as looping is done in C
>> - more readable. The main advantage of comprehension is that it have
>> standardized form, which is easier to understand, unlike a for-loop
>> which can have an infinite number of variations.
>> - (unnecessary) nested comprehension is an abuse.
>> - with `as` keyword, no new keyword and no ambiguity since currently
>> `as` cannot exist inside comprehension.
>>
>> Disadvantages:
>> - reverses the current semantic of filtering-then-expression. This
>> shouldn't be too much problem since side-effect on the expression part
>> is a cardinal sin and...
>> - if the expression part is heavy, it might be possible to do
>> optimization by filtering first when the filter part does not require
>> the result (i.e. when there is no "as" clause). A good side effect of
>> this optimization is codes that relies on filtering being done before
>> expression will just work as they cannot contain an `as` keyword.
>> (As "simple is better than complex", I actually don't really like `as`
>> can change evaluation order; I much prefer to keep everything simple and
>> consistent, i.e. always evaluate expression then filter or otherwise)
>>
>> possible syntaxes:
>> - [x**x as F for x in nums if F < 100]
>>  the as keyword is already often used to rename things
>>  (in with, import, etc) I like this one much better than @. The as
>>  part, of course, is optional
>> - [x**x for x in nums if @ < 100]
>>  the initial proposed syntax, ugly as hell.
> 
> OK, with this explanation (and the new syntax) I see what you're
> getting at better.
> 
> However, changing the order of evaluate vs filter is a huge
> compatibility problem. There's no way this will be possible. Even with
> syntax triggering the change (so that it's one way with the "as", the
> other without), that's a disaster waiting to happen.

How about this syntax which would solve your concern for the semantic
change:

[x**x as F for x in lst if F() < 100]

it's similar to original `as` proposal, except that F is a callable
instead of direct value.

The advantage of F being callable is that it does not need semantic
change, the filtering part will be done before expression just like it
is right now. However, we can explicitly request for the expression to
be evaluated by calling F(); the return value of F() will be saved and
reused for the final result and other calls to F().

A diagrammatic explanation:

        +---------------------------
        | this is the part that name
        | the expression's callable
        |
      --+-
[x**x as F for x in lst if x and F() < 100 and isvalid(F())]
 -+--                      ---+----------------------------
  |                           |  -+-                   -+-
  |    the filtering part is  |   |                     |
  |    evaluated before       |   |                     |
  |    expression just like   |   |                     |
  |    current behavior       |   |                     |
  |    -----------------------+   |                     |
  |                               |                     |
  |    then when F gets called;   |                     |
  |    expression is evaluated,   |                     |
  |    cached, and returned       |                     |
  |    ---------------------------+                     |
  |                                                     |
  |             F is called again, return cached result |
  |             ----------------------------------------+
  |
  | at the end of the day, if F is called,
  | return the cached result, else evaluate
  | the expression and use that
  +-----------------------------------------


using the as-callable syntax, the semantic of this:

[f(x) as F for x in lst if g(F())]

would be similar to:

result = []
for x in lst:
    # F() ensures f(x) will be only ever be called once
    def F():
        nonlocal _cache
        if not _cache:
            _cache = f(x)
        return _cache

    _cache = None
    if g(F()):
        result.append(F())

the only disadvantage of this as-callable is if you forgot to call F.

> You have at least 3 explicit ways of stating your intent (genexp
> inside listcomp, map inside listcomp, explicit loop). None is as
> clean-looking as your (amended) proposal, but they work now, and they
> don't have the semantic issues of your proposal.
> 
> (For a more general, more radical, equally certain to be shot down,
> option, which at least doesn't introduce the change in semantics, you
> could try proposing "as" as an assignment-as-expression operator. So
> you could have
> 
>     [y for x in l if (f(x) as y) < 100]
> 
> Hmm, on second thoughts - no, don't bother... :-))

It took me several minutes to understand that one... and no, that syntax
as makes it way too easy to be too creative in list comprehension,
devaluing the "standard form" which IMO is the strongest point of list
comprehension. Not to mention that that syntax moved the expression part
to be inside the filtering part... which is quite... disturbing...