[Python-ideas] Map-then-filter in comprehensions

[Sjoerd Job Postmus]

On Tue, Mar 08, 2016 at 10:58:16PM +0000, Paul Moore wrote:
> On 8 March 2016 at 22:21, Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> > As for yet another syntax suggestion (if we want to introduce
> > something).
> >
> >     [y for x in numbers with abs(x) as y if y > 5]
> >
> > The benefit is that it reads quite natural: "with expr as name".
> > Another benefit is that keywords get reused.
> 
> Agreed, this is a plausible suggestion. But rather than just providing
> an example usage, it would be helpful to see the full syntax of
> comprehension with the proposed addition. See
> https://docs.python.org/3/reference/expressions.html#grammar-token-comprehension

It took me a while to figure this out (battling with the compiler over
this), but I settled on the following grammar.

comprehension ::=  expression comp_for
comp_for      ::=  "for" target_list "in" or_test [comp_iter]
comp_iter     ::=  comp_for | comp_if | comp_with
comp_if       ::=  "if" expression_nocond [comp_iter]
comp_with     ::=  "with" or_test "as" target
 
(For comp_with you probably want target_list instead of a single target.
However, I felt like double-checking that I got my actual assumptions
right by first implementing it (I've got it working now), and seeing as
I'm not that experienced with modifying the Python
parse/ast/symtable/compile phases, I decided to cop out and take the
easy route. It should be fairly trivial to extend it to a target_list
instead of a target.)

> - note that a comprehension can have arbitrarily many for and if
> clauses, interspersed in any order as long as the first one is a
> "for". I'm guessing you'd add "with <some_sort_of_expression> as
> <target_list>" as simply a third option. But what would the semantics
> be? I'm guessing that "with xxx as yyy" translates basically as a
> statement "yyy = xxx" in the notional expansion described in that
> section ("considering each of the for or if clauses a block...") 

Trying to word it in such a way:

"... considering each of the `for` or `if` clauses a block, nesting from
left to right, and evaluating the expression to produce an element each
time the innermost block is reached. The `with expr as target` should be
considered equivalent to `target = expr`.

(And here we already see the downside of this idea). Normally a
comprehension of the form

    (expr1 for target1 in expr2 for target2 in expr3 if expr4)

boils down to

    for target1 in expr2:
        for target2 in expr3:
            if expr4:
                yield expr1

The natural extension would be for 

    (expr1 for target1 in expr2 with expr3 as target2 if expr4)

to reduce as follows.

    for target1 in expr2:
        with expr3 as target2:
            if expr4:
                yield expr1
                
Instead, it becomes

    for target1 in expr2:
        target2 = expr3:
        if expr4:
            yield expr1

But of course we're not going to have context managers in
comprehensions, are we? So this inconsistency is somewhat forgiveable.

> Assuming that is the proposed definition, a few questions arise:
> 
> 1. Is the behaviour this would assign to "with <foo> as x with <bar>
> as x" (i.e., repeated bindings of the same name) what we'd want? Is it
> likely to cause confusion in practice?

You mean similar to

    >>> [x for x in range(5) for x in range(x)]
    [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

I've never been confused by that in practice, as most people use good
names for stuff. So I would see no ultra-important reason to prevent
blocking it for this case and not in others.

> 2. Do we need to restrict the <target_list>? Consider that "with
> something as global_var[0]" is allowed by the definition - that would
> leak values out of the comprehension. This isn't new - the "for"
> variable works like this already - but is it something that's more
> likely to be abused than currently? Should we care? (Python doesn't
> tend to prohibit people from writing bad code).

I'd suggest not to bother with resolving naming conflicts or assignments
to things other than names.

    [__ for foo.bar in range(10)]

is already valid Python. No need to limit new features in ways which
their 'friends' are not.

> > However, that already has semantics outside a comprehension for
> > context-managers.
> 
> This is indeed a concern, albeit not a huge one - the definition has
> to point out that in the expansion "with" should be read as an
> assignment (written backwards) not as a with statement. It's not an
> impossible burden, but it is mildly inconsistent.
> 
> This is probably the syntax I prefer of the ones suggested so far. But
> I still haven't seen any *really* convincing arguments that we need
> anything new in the first place.

I agree on not having seen a really convincing argument yet. Especially
since `for target in [expr]` works as well as `with expr as target`.

The one thing I can think of is common subexpression elimination.

    [cheap_process(expensive_thingy(x)) for x in items if expensive_thingy(x) > 0]

or

    [cheap_process(y) for x in items with expensive_thingy(x) as y if y > 0]


But in Python3, you could just as well write

    [cheap_process(x) for x in map(expensive_thingy, items) if x > 0]

> Paul