[Python-ideas] With clauses for generator expressions

[Nick Coghlan]

On Thu, Nov 15, 2012 at 9:11 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> The clauses have *always* been out of order. In the function, the "if"
> comes
> between the "for" and the yield expression. In the expression, the "for"
> comes
> in between. If the clause order implies the statement order (I would have
> put it
> in terms of the clause structure implying the scoping, but they're
> effectively
> the same idea), then our syntax has been wrong since list comprehensions
> were
> added in 2.0. So, I think (and hope!) that implication was never intended.
>

One, and only one, clause in a comprehension or generator expression is
written out of sequence: the innermost clause is lifted out and written
first. The rest of the expression is just written in the normal statement
order with the colons and newlines omitted. This is why you can chain
comprehensions to arbitrary depths without any ambiguity from the
compiler's point of view:

>>> seq = [0, [0, [0, 1]]]
>>> [z for x in seq if x for y in x if y for z in y if z]
[1]

(Note: even though you *can* chain the clauses like this, please don't, as
it's thoroughly unreadable for humans, even though it makes sense to the
compiler)

So *if* a context management clause was added to comprehension syntax, it
couldn't reasonably be added using the same design as was used to determine
the placement of the current iteration and filtering clauses.

If the determination is "place it at the end, and affect the whole
comprehension/generator expression regardless of the number of clauses",
then you're now very close to the point of it making more sense to propose
allowing context managers on arbitrary expressions, as it would be
impossible to explain why this was allowed:

    lines = list(line for line in f with open(name) as f)

But this was illegal:

    lines = (f.readlines() with open(name) as f)

And if arbitrary subexpressions are allowed, *then* you're immediately
going to have people wanting a "bind" builtin:

    class bind:
        def __init__(self, value):
            self.value = value
        def __enter__(self):
            return self.value
        def __exit__(self, *args):
            pass

    if (m is None with bind(pattern.match(data) as m):
        raise ValueError("{} does not match {}".format(data, pattern))
    # Do things with m...

This is not an endorsement of the above concepts, just making it clear that
I don't believe that attempting to restrict this to generator expressions
is a viable proposal, as the restriction is far too arbitrary (from a user
perspective) to form part of a coherent language design.

Generator expressions, like lambda expressions, are deliberately limited.
If you want to avoid those limits, it's time to upgrade to a named
generator or function. If you feel that puts things in the wrong order in
your code then please, send me your use cases so I can considering adding
them as examples in PEP 403 and PEP 3150. If you really want to enhance the
capabilities of expressions, then the more general proposal is the only one
with even a remote chance, and that's contingent on proving that the
compiler can be set up to give generator expressions the semantics you
propose (Off the top of my head, I suspect it should be possible, since the
compiler will know it's in a comprehension or generator expression by the
time it hits the with token, but there may be other technical limitations
that ultimately rule it out).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121115/15bfd2c1/attachment.html>