One function calling another defined in the same file being exec'd

[Mitchell L Model]

On Jan 7, 2010, at 10:45 PM, Steven D'Aprano <steve at REMOVE-THIS-cybersource.com.au 
 > wrote an extensive answer to my questions about one function  
calling another in the same file being exec'd. His suggestion about  
printing out locals() and globals() in the various possible places  
provided the clues to explain what was going on. I would like to  
summarize what I have learned from this, because although I have known  
all the relevant pieces for many years I never put them together in a  
way that explains the odd behavior I observed.

Statements that bind new names -- assignment, def, and class -- do so  
in the local scope. While exec'ing a file the local scope is  
determined by the arguments passed to exec; in my case, I passed an  
explicit local scope. It was particularly obtuse of me not to notice  
the effects of this because I was intentionally using it so that an  
assignment to 'result' in the exec'd script would enable the exec'ing  
code to retrieve the value of result. However, although the purity of  
Python with respect to the binding actions of def and class statements  
is wonderful and powerful, it is very difficult cognitively to view a  
def on a page and think "aha! that's just like an assignment of a  
newly created function to a name", even though that is precisely the  
documented behavior of def. So mentally I was making an incorrect  
distinction between what was getting bound locally and what was  
getting bound globally in the exec'd script.

Moreover, the normal behavior of imported code, in which any function  
in the module can refer to any other function in the module, seduced  
me into this inappropriate distinction. To my eye I was just defining  
and using function definitions the way they are in modules. There is a  
key difference between module import and exec: as Steven pointed out,  
inside a module locals() is globals(). On further reflection, I will  
add that what appears to be happening is that during import both the  
global and local dictionaries are set to a copy of the globals() from  
the importing scope and that copy becomes the value of the module's  
__dict__ once import has completed successfully. Top-level statements  
bind names in locals(), as always, but because locals() and globals()  
are the same dictionary, they are also binding them in globals(), so  
that every function defined in the module uses the modified copy of  
globals -- the value of the module's __dict__ -- as its globals() when  
it executes. Because exec leaves locals() and globals() distinct,  
functions defined at the top level of a string being exec'd don't see  
other assignments and definitions that are also in the string.

Another misleading detail is that top-level expressions in the exec  
can use other top-level names (assigned, def'd, etc.), which they will  
find in the exec string's local scope, but function bodies do not see  
the string's local scope. The problem I encountered arises because the  
function definitions need to access each other through the global  
scope, not the local scope. In fact, the problem would arise if one of  
the functions tried to call itself recursively, since its own name  
would not be in the global scope. So we have a combination of two  
distinctions: the different ways module import and exec use globals  
and locals and the difference between top-level statements finding  
other top-level names in locals but functions looking for them in  
globals.

Sorry for the long post. These distinctions go deep into the semantics  
of Python namespaces, which though they are lean, pure, and beautiful,  
have some consequences that can be surprising -- more so the more  
familiar you are with other languages that do things differently.

Oh, and as far as using import instead of exec for my scripts, I don't  
think that's appropriate, if only because I don't want my  
application's namespace polluted by what could be many of these pseudo- 
modules users might load during a session. (Yes, I could remove the  
name once the import is finished, but importing solely for side- 
effects rather than to use the imported module is offensive. Well, I  
would be using one module name -- result -- but that doesn't seem to  
justify the complexities of setting up the import and accessing the  
module when exec does in principle just what I need.)

Finally, once all of this is really understood, there is a simple way  
to change an exec string's def's to bind globally instead of locally:  
simply begin the exec with a global declaration for any function  
called by one of the others. In my example, adding a "global fn1" at  
the beginning of the file fixes it so exec works.

################################
global fn1        # enable fn1 to be called from fn2!
def fn1(val):
    return sum(range(val))

def fn2(arg):
    return fn1(arg)

result = fn2(5)
################################