[Python-ideas] Tweaking closures and lexical scoping to include the function being defined

[Eric Snow]

On Fri, Sep 30, 2011 at 5:07 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Sep 30, 2011 at 1:25 AM, Ron Adam <ron3200 at gmail.com> wrote:
>> In otherwords the nonlocal feature can be moved into the Function class
>> constructor.  It doesn't need to actually be in the function it self.
>
> It's *really* important to remember the difference between compilation
> time, definition time and call time in this discussion, since name
> binding affects all of them.

+1

>
> Compilation time (usually at module import):
>    Compiler decides between STORE_FAST (function local), STORE_DEREF
> (function closure/nonlocal), STORE_GLOBAL (global declaration) and
> STORE_NAME (class/module scope).

Good point.  And it's important to reiterate that compilation only
results in code objects.  When a module is compiled, the bodies of all
function defs (and class defs and comprehensions) therein are compiled
separately (but relative to the module's code).  The resulting code
objects are put into the co_consts of the module's code object.
You'll find this if you look at a pyc file[1].

Likewise, if you have a nested function, the inner function's code
object will be stored in the co_consts of the outer function's code
object.  All this was a mystery to me until recently.

>
> Definition time (i.e. when the def statement itself is executed to
> create the function object):
>    Compiler creates any decorators, any default arguments, loads
> references to any outer cells on to the stack, loads the code object
> for the function body, creates the function (passing in the default
> arguments, cell references and code object), and only then applies the
> decorators.
>
> Call time (i.e. when the function is called):
>    Initialises the execution frame with space for the local variables
> and references to the cells for variables that may persist beyond this
> call, binds the arguments to the appropriate locals (potentially
> filling some in from the default arguments) and then passes the code
> object to the main eval loop to be executed.
>
> The reason nonlocal and global declarations need to exist in the first
> place is precisely to override the compiler's default assumption that
> all name bindings in function scope are references to local variables.

Exactly! That's a critical point to the discussion that isn't so obvious.

Compilation pre-computes a bunch of things that are used at call-time
and these are stored on the code object.  Two examples are the "space
for the local variables" and the size of the frame's stack, which are
partially dependent on the local/nonlocal nature of the various names
used in the function body.

Also, all the names in the signature or used in the body were mapped
to indices at compile-time (by virtue of being stored in various
tuples).  The actual compiled code (in co_code) refers to these
indices, which the eval loop uses when pulling values from the
appropriate appropriate tuples.  Those tuples were pulled from
function attributes like __closure__ and __defaults__.

So, any solution here must reflect that the compiler needs a way to
calculate these things.  On the flip side, any solution that relies on
def-time hacks of the function object must take into account the
subsequent changes that will be necessary on the function's code
object.  Particularly important is the relationship between
__closure__ and co_freevars[2].  It's part of why turning a local into
a closed variable is not a trivial thing.

<disclaimer>
I'll admit that I may have gotten some of this wrong, not having years
of experience to back me up here.  So I definitely defer to knowledge
of Nick, et al.  I have been pretty immersed in this stuff for several
months and the above is my resultant perspective, that of someone who
has dived in without a lot of background in it. :)
</disclaimer>

-eric

[1] See http://code.activestate.com/recipes/577880/
[2] http://hg.python.org/cpython/file/default/Include/funcobject.h#l34
     http://hg.python.org/cpython/file/default/Objects/funcobject.c#l454

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>