[Python-ideas] Fixing class scope brainstorm

[Nick Coghlan]

On 26 March 2018 at 01:50, Guido van Rossum <guido at python.org> wrote:
> In the PEP 572 threads there's some grumbling about class scopes.
>
> Here's a random brainstorm. How about we change the scoping rules so that
> only functions defined with 'def' (or inside one of those) cannot see the
> class scope, and comprehensions and lambdas treat the class scope as an
> outer scope so they can close over class variables?
>
> Not sure what should happen to nested classes, they should probably be
> lumped in with the 'def' scopes.

I think it's mainly in comprehensions that folks may find the current
behaviour surprising, as that's the case where we define a function
and immediately call it, so it mostly looks like regular inline code
execution.

Once explicitly delayed invocation is involved (lambdas, def
statements), folks seem to be more comfortable with the idea "Oh, of
course that's going to behave like a method at class scope". Generator
expressions also mostly get a pass, since the *iteration* is delayed,
even though the generator-iterator itself is created immediately.

One possibility that has occurred to me (but I've never investigated
to check the technical feasibility) is to see whether we might be able
to define a notion of "implied arguments" for the implicit
comprehension and generator expression function definitions.

The gist of that idea would be:

- when walking the symbol table for a comprehension or genexp, keep
track of every variable name mentioned in that subtree which is not
resolved in that subtree (which we already have the machinery to do,
as it's needed for compiling functions in general)
- treat all those names as *implied arguments* to the implicit
function, and call it with the right references in the right positions

While we'd likely still want to keep evaluation of the outermost
iterator outside the nested scope for the sake of better genexp
tracebacks, that would still be enough to enable cases like:

    class C:
        y = 1
        values = [x+y for x in range(10)]

The trick would be to make it so that that comprehension gets expanded as:

    def _listcomp(_outermost_iter, y):
        result = []
        for x in _outermost_iter:
            result.append(x+y)
        return result

    _listcomp_result = _listcomp(range(10), y)

One of the more attractive aspects of this possibility (assuming it
can be made to work) is that it isn't even a special case in the
runtime name lookup rules - it's just changing the definition of the
implicit functions that we create for comprehensions and generator
expressions to make sure that they can refer to *any* name that
resolves in the namespace where they're called (even
invisible-to-the-compiler names like those injected via __prepare__
methods, or assignments via locals() at class scope).

Cheers,
Nick.

P.S. I suspect we'd also find that this served as a performance
optimisation, since it wouldn't only prebind class locals, it would
prebind references to builtins and module globals as well.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia