[Python-ideas] Tweaking closures and lexical scoping to include the function being defined

[Eric Snow]

On Thu, Sep 29, 2011 at 8:05 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 29 September 2011 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>    @nonlocal(n=0, lock=threading.Lock())
>>    def global_counter():
>>        with lock:
>>            n += 1
>>            return n
>>
>> It's presence in the decorator list hints that this is something
>> happening outside the functions ordinary local scope, as well as
>> indicating when the initialisation happens. The 'nonlocal' then ties
>> in with how they behave from the point of view of the code in the
>> function body.
>
> I agree, this is a nice option. My biggest discomfort with it is the
> idea of a magic decorator name handled specially by the compiler (and
> worse still, it's a keyword so it's syntactically very weird indeed).
> I had similar discomfort over the new super, but I could get over that
> by simply assuming that super was a normal function that was just more
> magical than I understood. "@keyword" decorators don't even fit into
> my model of valid syntax :-(

I feel the same way.

The alternative is to leave nonlocal as just a simple statement, but
change its behavior when the name is not found inside a containing
function scope.  Currently that is a syntax error.

Instead it could imply that the name should fall back to the
definition-time "scope" of the current function.  In other words, the
compiler would treat the name like a closed variable anyway, but with
an empty cell (or some default initial value, like None)[1].  A
decorator would still be used to explicitly initialize the value:

   @init_nonlocal(n=0, lock=threading.Lock())
   def global_counter():
       nonlocal n
       with lock:
           n += 1
           return n

This approach has some advantages:

* doesn't require the compiler to do any special handling of decorators,
* fewer changes to the compiler (and less substantial),
* keeps the advantages of using a decorator for initialization,
* the nonlocal nature of the variable is clear in the function body
(where it is used),
* allows for "uninitialized" def-time names (see [1]),
* more intuitively consistent use of the keyword (says my gut),
* decorator could be applied to any function at any time (not just def-time);

...and some disadvantages:

* the statement may be redundant for def-time nonlocals (if it will
always be accompanied by the decorator),
* implicit initialization (if used) is...implicit (EIBTI),
* implicit default def-time scope is as well,
* (maybe) makes it too easy to modify closure cells, if that's a bad thing,
* decorator could be applied to any function at any time (not just def-time).

The init_nonlocal decorator [factory] would change the value of the
appropriate cell in __closure__.  Either the decorator would be the
function that exposed the ability to modify cells or it would leverage
another function that did so.  I would think the latter, so it could
look like this:

  def init_nonlocal(**kwargs):
      def decorator(f):
          freevars = f.__code__.co_freevars
          indices = {}
          for name in kwargs:
              try:
                  index = freevars.index(name)
              except ValueError:
                  raise NameError(...)
              indices[name] = index
          for name, index in indices.items():
              #f.__closure__[index].cell_contents = kwargs[name]
              functools.update_cell(f, index, kwargs[name]
          return f
      return decorator

or

  def init_nonlocal(**kwargs):
      def decorator(f):
          functools.update_freevars(**kwargs)
          return f
      return decorator

The cell-modifying capabilty would have to apply to any cell in
__closure__ and not just the ones associated with def-time nonlocals.
If the nonlocal statement gets associated with a name in an enclosing
function scope, then init_nonlocal would be used to explicitly force
the name to be a def-time nonlocal.  Maybe that's an argument for
syntax to explicitly identify a def-time nonlocal rather than relying
on any implicit behavior of the nonlocal keyword.

-eric


[1] explicitly uninitialized variables in some languages (like C) get
implicitly initialized to some default value.  I'm not sure what the
use case would be for an uninitialized nonlocal (or even if it makes
sense for a cell to be uninitialized).  In that situation I would
expect a NameError when the name is accessed.

>
> That said, I like it otherwise. The above says to me "the following
> values are nonlocal to this function", which I can read exactly the
> way it actually works. Whether that's a result of a week's immersion
> in Nick's propaganda, I can't say, though :-)
>
> Paul.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>