[Python-ideas] Tweaking closures and lexical scoping to include the function being defined

[Arnaud Delobelle]

On 2 October 2011 21:24, Ron Adam <ron3200 at gmail.com> wrote:
> On Sun, 2011-10-02 at 17:16 +0900, Stephen J. Turnbull wrote:
>> Nick Coghlan writes:
>>
>>  > It isn't quite - the name binding doesn't happen until *after* the
>>  > decorator chain has been invoked, so the function is anonymous while
>>  > the decorators are executing.
>>
>> As I understand the issue here, as far as the decorators are
>> concerned, the reference passed by the decorator syntax should be
>> enough to do any namespace manipulations that are possible in a
>> (non-magic) decorator.  Am I missing something?
>
> I've managed to do it with a function, but it isn't pretty and isn't
> complete.  It does work for simple cases.
>
>
> But it isn't easy to do...
>
>
> 1. Creating a dummy function with a __closure__, and taking the parts of
> interst from it.  (Requires exec to do it.)
>
> 2. Creating a new byte code object with the needed changes. (Hard to get
> right)
>
> 3. Create a new code object with the altered pieces.
>
> 4. Make a new function with the new code object and __closure__
> attribute.  Use the original function to supply all the other parts.

And you've got to take care of nested functions.  That is, look for
code objects in the co_consts attribute of the function's code objects
and apply the same transformation (recursively).  Moreover, you have
to modify (recursively) all the code objects so that any MAKE_FUNCTION
is changed to MAKE_CLOSURE, which involves inserting into the bytecode
a code sequence to build the tuple of free variables of the closure on
the stack.  At this point it may be almost easier to write a general
purpose code object to source code translator, stick a nonlocal
declaration at the start of the source of the function, wrap it in an
outer def and recompile the whole thing!

A simple example to illustrate:

def foo():
   def bar():
      return x + y
   return bar

This compiles to:

              0 LOAD_CONST               1 (<code object bar>)
              3 MAKE_FUNCTION            0
              6 STORE_FAST               0 (bar)

              9 LOAD_FAST                0 (bar)
             12 RETURN_VALUE

Now imagine we have the non-magic "nonlocal" decorator:

@nonlocal(x=27, y=15)
def foo():
   def bar():
      return x + y
   return bar

That should compile to something like this:

              0 LOAD_CLOSURE             0 (y)
              3 LOAD_CLOSURE             1 (x)
              6 BUILD_TUPLE              2
              9 LOAD_CONST               3 (<code object bar>)
             12 MAKE_CLOSURE             0
             15 STORE_FAST               0 (bar)
             18 LOAD_CONST               0 (None)
             21 RETURN_VALUE

And obvioulsy the "bar" code object need to be adjusted (LOAD_GLOBAL
-> LOAD_DEREF).

-- 
Arnaud