closures and dynamic binding

[Aaron "Castironpi" Brady]

On Sep 29, 9:14 am, Paul Boddie <p... at boddie.org.uk> wrote:
> On 29 Sep, 05:56, Terry Reedy <tjre... at udel.edu> wrote:
>
>
>
> > As I understand it, partly from postings here years ago...
>
> > Lexical: The namespace scope of 'n' in inner is determined by where
> > inner is located in the code -- where is is compiled.  This is Python
> > (and nearly all modern languages).  Even without closures, the global
> > scope of a function is the module it is defined in.
>
> This is how I understand it, too. The confusing part involves the
> definition of any inner function and how any "external" names are
> bound or defined. As we've seen...
>
>   def f(x):
>     def g():
>       return x
>     x += 1 # added for illustration
>     return g
>
> ...it might look at first glance like the function known as g (within
> f) should return the initial value of x (as known within f), since
> that was the value x had when g was defined. The following is an
> example execution trace based on that mental model:
>
> fn = f(1)
> -> def f(1):
> ->   def g(): # g defined with x as 1
> ->     return x # would be 1
> ->   x += 1 # x becomes 2
> ->   return g
> fn()
> -> def g():
> ->   return x # would still be 1
>
> However, as we know, this isn't the case in real Python since g isn't
> initialised with the value of x at the time of its definition - it
> instead maintains access to the namespace providing x. We must
> therefore revise the example:
>
> fn = f(1)
> -> def f(1):
> ->   def g(): # g refers to x within f(1)
> ->     return x # would be the current value of x within f(1)
> ->   x += 1 # x becomes 2
> ->   return g
> fn()
> -> def g(): # g refers to x within f(1)
> ->   return x # would be the current value of x within f(1), which is
> 2
>
> This is the dynamic aspect of closures: values aren't used to
> initialise inner functions; names are looked up from their origin.
>
> > Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
> > determined by where inner is called from. This is what you seemed to be
> > suggesting -- look up 'n' based on the scope it is *used* in.
>
> Indeed. Dynamic scoping is confusing in that one has to set up an
> appropriate "environment" for the closure to access so that references
> to names can be meaningfully satisfied; obviously, this creates all
> sorts of encapsulation problems. The confusing aspect of lexical
> scoping, however, especially if non-local names can be changed, is
> that the effects of closures are potentially distant - one doesn't
> always invoke inner functions in the place where they were defined, as
> we see above - and such effects may thus happen within "abandoned
> namespaces" - that is, namespaces which can no longer be revisited and
> used in their original context. So, in the above, once f has been
> invoked, the namespace for that invocation effectively lives on, but
> that namespace is not a general one for f - if we invoke f again, we
> get another namespace as one should reasonably expect.
>
> A somewhat separate issue is illustrated by the modification of x
> within f. Although for most statements, we would expect the value of x
> to evolve following from a top-to-bottom traversal of the code within
> a unit, function definition statements do not behave like, say, "for",
> "if" or "while" statements. Now although this should be obvious at the
> module global level, I feel that it can be easy to overlook within a
> function where one normally expects to find plain old control-flow
> constructs, expressions, assignments and so on. It is pertinent to
> note, with respect to the original inquiry, that lambda functions are
> subject to the same caveats, and I believe that lexical scoping was
> introduced precisely to make lambda functions less cumbersome to
> employ, eliminating the need to explicitly initialise them using the
> "identity" default parameters trick, but obviously introducing the
> consequences and potential misunderstandings described above.
>
> Paul

I'm thinking of a comparison to try to see an example.

I tried this which still surprised me at first.

>>> g= [0]
>>> f= lambda: g
>>> del g
>>> f()
NameError: global name 'g' is not defined

It took a little thinking.  The namespace in 'f' is the global
namespace.  They are one in the same dictionary.  id( f.namespace ) ==
id( main.namespace ).  f.namespace is main.namespace.

When f gets a parameter, its namespace is different by one (or two
when two parameters, etc).

>>> g=[0]
>>> f= lambda h: lambda: h
>>> i= f( g )
>>> del g
>>> i()
[0]

The statement 'del g' removes 'g' from the module/global namespace,
but it lives on in the namespace of 'f', or more accurately, 'i'.

>>> g=[0]
>>> f= lambda h: lambda: h
>>> i= f( g )
>>> j= f( g )
>>> del g
>>> i().append(1)
>>> j()
[0, 1]

Here, the namespaces of 'main', 'i', and 'j', all have references to
'g'.  'i' and 'j' still have them when 'main' loses its.

The lambda might be confusing.  Take a simpler example of a namespace:

>>> g= [0]
>>> def h():
...     class A:
...             a= g
...     return A
...
>>> i= h()
>>> del g
>>> i.a
[0]

Or even:

>>> g=[0]
>>> class A:
...     a= g
...
>>> del g
>>> A.a
[0]

The class is executed immediately so 'A.a' gets a hold of 'g'.
Function definitions keep a reference to the -identity-, not value, of
the namespace they were defined in, and parameters are part of that
namespace.

>>> def h( a ):
...     #new namespace in here, new on each call
...     def i():
...             return a
...     return i

'i' is not defined in the global namespace, it's defined in a brand
new one that is created on every call of 'h'.  'a' is defined in that,
so that's what 'a' when 'i' is called refers to.

To be specific, the namespace 'h' defines only includes the names that
are different from its enclosing scope.  Regardless of what's defined
below 'h', 'h' only defines one new variable, 'a'.  Its value is
passed in at call-time.  'i' needs to know what that namespace is--
that's how closures work-- so it saves a reference.  That saved
reference is to a namespace, not a variable though, which
distinguishes it from the 'class A: a= g' statement that's executes
immediately.  There, 'a' stores the value of g.  By contrast, 'i'
stores 'h's entire namespace.

In 'class A: a= g', the name 'a' is assigned to the contents of 'g'.
In 'def i(): return a', 'a' is the value of a look up in a namespace
by its name.

>>> def h( a ):
...     #new namespace in here
...     def i():
...             return a
...     return i
...
>>> g=[0]
>>> j= h(g)
>>> hex(id(g))
'0x9ff440'
>>> del g
>>> hex(id(j()))
'0x9ff440'
>>> j.func_closure
(<cell at 0x009FDF50: list object at 0x009FF440>,)

By the time 'g' is deleted, 'j' has already hijacked a reference to
it, which lives in the namespace of the 'i' it defined that time
through.  'j()', originally 'g', and 'j's namespace all refer to the
same object.

Variables are keys in a namespace, even if it's an "abandoned
namespace", adopting the term.