[Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Nick Coghlan]

On 21 October 2016 at 05:09, Random832 <random832 at fastmail.com> wrote:
> On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
>> Hi, I contributed the current list comprehension implementation (when
>> refactoring it for Python 3 to avoid leaking the iteration variable,
>> as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar
>> for a series of nested for and if statements" is precisely my
>> understanding of how they work, and what they mean.
>
> But it's simply not true. It has never been true and it will never be
> true. Something is not "syntactic sugar" if it doesn't compile to the
> exact same sequence of operations as the thing it is supposedly
> syntactic sugar for.

We don't need to guess about this, since we can consult the language
reference and see how comprehension semantics are specified for
language implementors:
https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries

Firstly, container displays are broken out into two distinct kinds:

    For constructing a list, a set or a dictionary Python provides
special syntax called “displays”, each of them in two flavors:

    - either the container contents are listed explicitly, or
    - they are computed via a set of looping and filtering
instructions, called a comprehension.

Secondly, the meaning of the clauses in comprehensions is spelled out
a little further down:

    The comprehension consists of a single expression followed by at
least one for clause and zero or more for or if clauses. In this case,
the elements of the new container are those that would be produced by
considering each of the for or if clauses a block, nesting from left
to right, and evaluating the expression to produce an element each
time the innermost block is reached.

We can also go back and read the design PEPs that added these features
to the language:

* List comprehensions: https://www.python.org/dev/peps/pep-0202/
* Generator expressions: https://www.python.org/dev/peps/pep-0289/

PEP 202 defined the syntax in terms of its proposed behaviour rather
than a syntactic, with the only reference to the nesting equivalence
being this BDFL pronouncement:

    - The form [... for x... for y...] nests, with the last index
      varying fastest, just like nested for loops.

PEP 289, by contrast, fully spells out the implied generator
definition that was used to guide the implementation of generator
expressions in the code generator:

    g = (tgtexp  for var1 in exp1 if exp2 for var2 in exp3 if exp4)

    is equivalent to:

    def __gen(bound_exp):
        for var1 in bound_exp:
            if exp2:
                for var2 in exp3:
                    if exp4:
                        yield tgtexp
    g = __gen(iter(exp1))
    del __gen

When I implemented the comprehension index variable hiding for Python
3.0, the final version was the one where I blended those two
definitions to end up with the situation where:

    data = [(var1, var2)  for var1 in exp1 if exp2 for var2 in exp3 if exp4)]

    is now equivalent to:

    def __comprehension(bound_exp):
        __hidden_var = []
        for var1 in bound_exp:
            if exp2:
                for var2 in exp3:
                    if exp4:
                        __hidden_var.append((var1, var2))
        return __hidden_var
    data = __comprehension(iter(exp1))
    del __comprehension

While it's pretty dated now (I wrote it circa 2.5 as part of a draft
book manuscript that was never published), if you'd like to learn more
about this, you may want to check out the section on "Initialising
standard containers" in
http://svn.python.org/view/sandbox/trunk/userref/ODF/Chapter02_StatementsAndExpressions.odt

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia