[Python-ideas] Tweaking closures and lexical scoping to include the function being defined

[Eric Snow]

On Wed, Sep 28, 2011 at 9:38 AM, Guido van Rossum <guido at python.org> wrote:
> On Wed, Sep 28, 2011 at 3:45 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Actually, there are some additional aspects that annoy me:
>> - the repetition of the variable name 'n'
>> - the scope of the variable name is wrong (since we only need it in
>> the inner function)
>> - indentation matters (cf try/except/finally)
>> - hidden signature for the actual function
>> - hoops to jump through to get decent introspection values (e.g. __name__)
>>
>> I find the following 5 line toy example:
>>
>> from threading import Lock
>>
>> def global_counter() [n=1, lock=Lock()]:
>>    with lock:
>>        print(n)
>>        n += 1
>
> Hm, this syntax is *too* concise. It's lack of keywords makes it open
> to misinterpretation.
>
>> Far more readable than the 10-line closure equivalent:
>>
>> from threading import Lock
>>
>> @apply # I sometimes wonder if we should bring this back in functools...
>> def global_counter():
>>    n = 1
>>    local = Lock()
>>    def global_counter():
>>        with lock:
>>            print(n)
>>            n += 1
>>    return global_counter
>
> True, the outer function completely obscures what's going on.
>
>> Or the 10-line 7-self class equivalent:
>>
>> from threading import Lock
>>
>> @apply
>> class global_counter:
>>    def __init__(self):
>>        self.n = 1
>>        self.lock = Lock()
>>
>>    def __call__(self):
>>        with self.lock:
>>            print(self.n)
>>            self.n += 1
>
> Yeah, anything involving __call__ is hard to figure out.
>
> So after reading some previous messages, I almost wish we could
> implement the "decorator + nonlocal statement" proposal:
>
> @init(n=1, lock=Lock())
> def global_counter():
>  nonlocal n, lock
>  with lock:
>    print(n)
>    n += 1
>
> Alas, the problem is that you can't actually implement such a
> decorator, not even using an extension module, because the compiler,
> upon seeing the "nonlocal n, lock", ignoring the decorator (since it
> should be applied at run time, not at compile time), will complain
> that there isn't actually an intermediary outer scope defining either
> n or lock.
>
> Some ways out of this:
>
> - let the compiler special-case a certain built-in decorator name at
> compile-time (unorthodox, not impossible)
>
> - different syntax, e.g.
>  @[n=1, lock=Lock()]
> or
>  @in(n=1, lock=Lock())
> or even
>  in n=1, lock=Lock()
>
> (all followed by "def global_counter() etc.")
>
> Of course once there's different syntax, the nonlocal declaration in
> the function is redundant. And clearly I'm back-peddling. :-)

I like the idea of a decorator approach too, though just normal
decorators.  Here's another alternative.  It could require exposing
function internals a little (impacting the language definition?).
There are two distinct outcomes we could pursue: def-time values (like
default arguments), and def-time variables (static-like).


The Def-time Value Approach
------------------------------------

For def-time values, the values would have to be stored in a function
attribute similar to __defaults__ and __kwdefaults__.  Let's call it
__spam__.  A keyword (not "nonlocal" <wink>) in the function body
would indicate that the name should be initialized with each call to
the appropriate value in __spam__.  Then we would have a (builtin?)
decorator factory that would update values in __spam__:

#<example>

def init_deftime(**kwargs):
    def decorator(f):
        for name in kwargs:
            index = figure_out_index(f, name)
            f.__spam__[index] = kwargs[name]
            #functools.update_spam(f, name, kwargs[name])
        return f
    return decorator

@init_deftime(x=5):
def f():
    some_keyword x
    print(x)

# prints 5

# Currently using default arguments:
def f(x=5):
    print(x)

#</example>

As suggested already, the keyword simply indicates to the compiler
that x should be a local and it should be initialized from __spam__.

Also, if the decorator is not used to initialize a name marked by
some_keyword, then it would either just use some default value or
raise some exception (just after all decorators have been applied or
at call time).


The Def-time Variable Approach
---------------------------------------

Perhaps we want static-like variables in functions instead.  Unless I
misread the more recent posts, such variables are a no-go.  Just in
case, I'll summarize a decorator solution.  Basically, all we do is
indicate through a keyword that a name will be found in the "function
definition scope".  Then closure happens like normal.

The trick is to reuse nonlocal and add the function definition scope
as the implicit default scope.  Currently you get a syntax error when
nonlocal refers to a name that is not found by the compiler in any
enclosing function scope.  This would change that.  The decorator
would be the mechanism for initializing the variable:

#<example>

def init_deftime(**kwargs):
    def decorator(f):
        for name in kwargs:
            functools.update_closure(f.__closure__, name, kwargs[name])
        return f
    return decorator

@init_deftime(x=5):
def f():
    nonlocal x
    print(x)

# prints 5

#</example>

Again, if the nonlocal name is never initialized (and not bound-to in
the body), either a default would have to be used or an exception
raised.


Wrap-up
----------

So there's a decorator approach.  For me it's a toss-up between
decorators and the bracket syntax in the signature that Nick
suggested.  I find the deftime-value-in-the-body proposals not nearly
as clear.  I haven't had much use for static-like variables so I
prefer def-time value approaches.

Of course, the simplest solution is to do like Terry (et al.)
suggested and just encourage the use of "private" default args.

-eric

>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>