[Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Thu, Aug 10, 2017 at 01:25:24PM -0700, Chris Barker wrote:
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> 
> >  Also, there's a potential issue
> > here - consider
> >
> >     [expr for var in even_numbers() if is_odd(var) while var < 100]
> >
> > This is an infinite loop, even though it has a finite termination
> > condition (var < 100), because we only test the termination condition
> > if var is odd, which it never will be.

I'm not sure why Paul thinks this is an issue. There are plenty of ways 
to accidentally write an infinite loop in a comprehension, or a for 
loop, already:

[expr for var in even_numbers()]

will do it, if even_numbers is unexpectedly an infinite iterator. Or you 
could write:

for num in even_numbers():
    if is_odd(num) and num > 100:
        break

No loop syntax, whether it is functional style (takewhile, map, etc.), 
comprehension, or traditional style for loops, enables the programmer 
to avoid thinking about what they write.


> why is the termination only tested if teh if clause is True? Could then not
> be processed in parallel? or the while first....

Because we're following the standard Python rule of left-to-right 
execution. The while clause is tested only if the if clause is true 
because it follows the if clause.

I think that there's an argument to be made for the rule:

    We can have `if` in a comprehension, or `while`, but not both

in order to limit complexity. Analogy: 

(1) we intentionally limit the decorator @ syntax to a subset of 
expressions;

(2) likewise we intentionally allow (but don't encourage) monkey-
patching of Python classes only, not built-ins.

Just because we *can* allow arbitrary code combinations doesn't mean we 
*must*. We have a choice to say:

    "No, you cannot mix `if` and `when` in the same comprehension. 
    Why? Because we say so. Because it is confusing if you do."

I'd be okay with that rule.

But if we decide to allow arbitrary combinations of for/if/while in 
comprehensions, then I think we must keep the same left-to-right rule we 
have now. Currently we process multiple for/if clauses left-to-right:

    [expr for x in a if cond for y in b]

is equivalent to:

    for x in a:
        if cond:
            for y in b:
                expr

rather than moving the `if` to the end. If you want it at the end, put 
it there yourself. Adding `while` shouldn't change that. It would be 
crazy-complicated to have a rule:

    "the presence of a while means the comprehension is 
    processed in parallel"

or

    "all the while clauses are processed before (after?)
    the if clauses, regardless of their order of appearance."


> so maybe better to do:
> 
> [expr for var in even_numbers() while var < 100 if is_odd(var)]

Well sure, that's the *correct* way to write the code:

for var in even_numbers():
    if not (var < 100): break
    if is_odd(var):
        results.append(expr)

(for some definition of "correct" -- this is clearly an expensive way to 
generate an empty list.)

But in general one might wish to test the if or the while in either 
order.


> Maybe it's just me, but I would certainly expect the while to have
> precedence.

Does that apply to these idioms as well?

    while cond:
        if flag:
            ...

versus:

    if flag:
        while cond:
            ...


I would not expect them to be the same, and nor would I expect these to 
be the same:

    [expr for x in seq if flag while cond]

    [expr for x in seq while cond if flag]


> I guess I think of it like this:
> 
> "if" is providing a filtering mechanism
> 
> "while" is providing a termination mechanism
> 
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?

[process(n) for n in numbers while n > 0 if is_odd(n)] 

Halt on the first zero or negative number, regardless of whether it is 
even or odd, but process only odd numbers.



Paul: 
> > Obviously, this is a contrived example. And certainly "don't do 
> > that, then" is a valid response. But my instinct is that people are 
> > going to get this wrong - *especially* in a maintenance environment.

That's the argument for limiting comprehensions to either `if` or 
`while` but not both. And I actually would be okay with that -- 
especially if we leave open the possibility of relaxing the prohibition 
in the future.

But personally, I think that's under-estimating the ability of 
programmers to reason about loops. Of course a comprehension with 
multiple for/if/while clauses is hard to reason about, and we shouldn't 
*encourage* them, but we don't prohibit multiple for/if clauses. Why 
should `while` be held to a higher standard? If we allow people to shoot 
themselves in the foot by writing complex list comprehensions with ten 
`for` loops and seven `if` clauses, why should we baulk at allowing them 
a `while` clause as well?



-- 
Steve