[Python-ideas] Tweaking closures and lexical scoping to include the function being defined

[Nick Coghlan]

On Tue, Sep 27, 2011 at 12:12 AM, Guido van Rossum <guido at python.org> wrote:
> Somehow you triggered a thought in my head: maybe se should stop
> focusing on the syntax for a while and focus on the code we want to be
> generated for it. If we can design the bytecode, perhaps that would
> help us come up with the right keyword.

Heh, doing exactly that (i.e. thinking in terms of how it would *work*
and how I would *explain* those semantics to someone in terms of
existing language concepts) is where my nonlocal suggestion came from
in the first place :)

As I see it, there are 3 possible ways to approach this feature, and
they vary mainly in terms of when the initialisation expressions get
evaluated:
1. Prior to function definition (i.e. the same as default argument values)
2. After function definition but before decorators are applied
3. After decorator application

I don't like number 3, since it means the closure references haven't
been populated yet when the decorators are executed, so decorators
that needed to call the function wouldn't work any more and I think
number 2 would just be plain confusing. Hence, I prefer number 1, with
a decorator based idiom if anyone wants to play recursion tricks.

The semantics for a function with a function scoped variable declared
would then be along the lines of:

    @example
    def global_accumulator(x):
        nonlocal tally from 0 # Insert preferred spelling here
        tally += x
        return tally

meaning:

    def _outer():
        tally = 0
        @example
        def global_accumulator(x):
            tally += x
            return tally
        return global_accumulator

    global_accumulator = _outer()

Of course, the compiler wouldn't have to generate the bytecode like
that - it could easily do the expression evaluations, stash them on
the stack, and then create the inner function. The MAKE_CLOSURE opcode
could be adjusted to look at the code object and pop the
initialisation values for the function scoped variables and create the
additional cell references in the __closure__ attribute.

However, one advantage of actually going the implicit outer scope
route is that the function scoped variables would naturally be able to
refer to each other, so you could easily cache an initial value and
then some further derived values that depended on the first one.

For the recursion use cases, I'd suggest actually offering a
'functools.rebind_closure' function that provided an officially
supported way to modify a closure reference in a function. You could
then write a function with an early binding reference to itself as
follows:

    def recursive(f):
        f.rebind_closure('recurse', f)

    @recursive
    def factorial(x):
        nonlocal recurse from recursive # This value gets replaced by
the decorator
        if x = 0:
            return 1
        return x * recurse(x-1)

Yes, it would be rather clumsy, but it is also incredibly uncommon for
the simple 'recurse by name' approach to actually fail in practice.

If we did decide to support recursive references directly, as in:

    def factorial(x):
        nonlocal recurse from factorial # Early binding - and nonlocal
suggesting evaluation in outer scope ;)
        if x = 0:
            return 1
        return x * recurse(x-1)

Then the way to go would be the second option above where the function
scope variables can see the undecorated version of the function:

    def _outer():
        # First we define our function
        def global_accumulator(x):
            tally += x
            return tally
        # Then evaluate the function scope variables
        tally = 0
        # Then we apply our decorators
        return example(global_accumulator)

    global_accumulator = _outer()

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia