[Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-09 11:54 AM, Nick Coghlan wrote:
> On 9 August 2017 at 15:38, Guido van Rossum <guido at python.org> wrote:
>> On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> The OP's proposal doesn't fit into that category though: rather it's
>>> asking about the case where we have an infinite iterator (e.g.
>>> itertools.count(0)), and want to drop items until they start meeting
>>> some condition (i.e. itertools.dropwhile) and then terminate the
>>> iterator as soon as another condition is no longer met (i.e.
>>> itertools.takewhile).
>> I don't think that's what the OP meant. The original proposal seemed to
>> assume that it would be somehow reasonable for the input ("integers" in the
>> example) to be able to see and parse the condition in the generator
>> expression ("1000 <= x < 100000" in the example, with "x" somehow known to
>> be bound to the iteration value). That's at least what I think the remark "I
>> like mathy syntax" referred to.
> Right, I was separating the original request to make "{x for x in
> integers if 1000 <= x < 1000000}" work into the concrete proposal to
> make exactly *that* syntax work (which I don't think is feasible), and
> the slightly more general notion of offering a more math-like syntax
> that allows finite sets to be built from infinite iterators by
> defining a termination condition in addition to a filter condition.

Ok. A concrete proposal would give a read-only 'filter' argument to the 
iterator somehow, which represents some form of simplified AST of the 
condition.

So e.g. {x for x in integers if (lambda v: 1000 <= v < 1000000)(x)} 
would never complete, but {x for x in integers if 1000 <= x < 1000000} 
would. (But perhaps lambda objects should include an AST attribute... 
Having it for normal functions would introduce too much overhead tho, 
and then it would no longer be a simplified AST, but rather a complete 
python AST, which we don't want.)

>
>>> There aren't any technical barriers I'm aware of to implementing that,
>>> with the main historical objection being that instead of the
>>> comprehension level while clause mapping to a while loop directly the
>>> way the for and if clauses map to their statement level counterparts,
>>> it would instead map to the conditional break in the expanded
>>> loop-and-a-half form:
>>>
>>>      while True:
>>>          if not condition:
>>>              break
>>>
>>> While it's taken me a long time to come around to the idea, "Make
>>> subtle infinite loops in mathematical code easier to avoid" *is* a
>>> pretty compelling user-focused justification for incurring that extra
>>> complexity at the language design level.
>> I haven't come around to this yet. It looks like it will make explaining
>> comprehensions more complex, since the translation of "while X" into "if not
>> X: break" feels less direct than the translations of "for x in xs" or "if
>> pred(x)". (In particular, your proposal seems to require more experience
>> with mentally translating loops and conditions into jumps -- most regulars
>> of this forum do that for a living, but I doubt it's second nature for the
>> OP.)
> Yeah, if we ever did add something like this, I suspect a translation
> using takewhile would potentially be easier for at least some users to
> understand than the one to a break condition:
>
>      {x for x in itertools.count(0) if 1000 <= x while x < 1000000}
>
>      <=>
>
>      x = set()
>      for x in itertools.count(0):
>          if 1000 <= x:
>              set.add(x)
>          # If you've never used the loop-and-a-half idiom, it's
>          # not obvious why "while <expr>" means "if not <expr>: break"
>          if not x < 1000000:
>              break
>
>      is roughly
>
>      {x for x in itertools.takewhile(itertools.count(0), lambda x: x <
> 1000000) if 1000 <= x}
>
>      <=>
>
>      x = set()
>      for x in takewhile(itertools.count(0), lambda x: x < 1000000):
>          if 1000 <= x:
>              set.add(x)
>
> However, the break condition is the translation that would make sense
> at a language *implementation* level (and would hence be the one that
> determined the relative location of the while clause in the expression
> form).
>
> That discrepancy *still* sets off alarm bells for me (since it's a
> clear sign that "how people would think this works" and "how it would
> actually work" probably wouldn't match), I'm also conscious of the
> amount of syntactic noise that "takewhile" introduces vs the "while"
> keyword.
>
> The counter-argument (which remains valid even against my own change
> of heart) is that adding a new comprehension clause doesn't actually
> fix the "accidental infinite loop" problem: "{x for x in
> itertools.count(0) if 1000 <= x < 1000000}" will still loop forever,
> it would just have a nicer fix to get it to terminate (adding " while
> x" to turn the second filter condition into a termination condition).
>
> So while I'm +0 where I used to be a firm -1, it's still only a +0 :)
>
> Cheers,
> Nick.
>