Bizarre additional calling overhead.

[Chris Mellon]

On Sat, 2007-11-03 at 01:08 -0300, Gabriel Genellina wrote:
> En Fri, 02 Nov 2007 21:07:19 -0300, Matimus <mccredie at gmail.com> escribió:
> 
> > On Nov 2, 3:08 pm, "Chris Mellon" <arka... at gmail.com> wrote:
> >> >>> def test_func():
> >>
> >> ...     pass
> >> ...>>> import new
> >> >>> test_func2 = new.function(test_func.func_code, {}, "test_func2")
> >> >>> test_func2
> >>
> >> <function test_func2 at 0x01B8C2F0>>>> test_func
> >>
> >> <function test_func at 0x01B8C270>>>> import timeit
> >> >>> tf = timeit.Timer("test_func()", "from __main__ import test_func")
> >> >>> tf.repeat()
> >>
> >> [0.2183461704377247, 0.18068215314489791, 0.17978585841498085]>>> tf2 =  
> >> timeit.Timer("test_func2()", "from __main__ import test_func2")
> >> >>> tf2.repeat()
> >>
> >> [0.40015390239890891, 0.35893452879396648, 0.36034628133737456]
> >>
> >> Why almost twice the calling overhead for a dynamic function?
> >
> > So, I don't have an official explanation for why it takes twice as
> > long, but the only difference between the two functions I could find
> > was that test_func.func_globals was set to globals() and
> > test_func2.func_globals was an empty dict. When I re-created
> > test_func2 with globals set to globals() it ran just as fast as
> > test_func.
> 
> Yes - and that's a very important difference. Not because it's empty, nor  
> because it's not the same as globals(), but because the builtins as seen  
> by the function are not from the original __builtin__ module.
> 
>  From the Python Reference Manual, section 4.1:
> 	The built-in namespace associated with the execution of a code block is  
> actually found by looking up the name __builtins__ in its global  
> namespace; [...] __builtins__ can be set to a user-created dictionary to  
> create a weak form of restricted execution.
> 
>  From the Library Reference, section 28, Restricted Execution:
> 	The Python run-time determines whether a particular code block is  
> executing in restricted execution mode based on the identity of the  
> __builtins__ object in its global variables: if this is (the dictionary  
> of) the standard __builtin__ module, the code is deemed to be  
> unrestricted, else it is deemed to be restricted.
> 
> Section 3.2, when describing frame objects:
> 	f_restricted is a flag indicating whether the function is executing in  
> restricted execution mode
> 
> Let's try to see if this is the case. Getting the f_restricted flag is a  
> bit hard with an empty globals(), so let's pass sys as an argument:
> 
> py> def test_func(sys):
> ...     print "restricted", sys._getframe().f_restricted
> ...
> py> import sys, new
> py> test_func2 = new.function(test_func.func_code, {}, "test_fun
> c2")
> py> test_func(sys)
> restricted False
> py> test_func2(sys)
> restricted True
> 
> So test_func2 is running in restricted mode. That't the reason it is so  
> slow. Even if the restricted mode implementation is now considered unsafe  
> -because new style classes provide many holes to "escape" from the "jail"-  
> the checks are still being done.
> 
> Ok, now let's try to avoid entering restricted mode:
> 
> py> import __builtin__
> py> test_func3 = new.function(test_func.func_code, {'__builtins_
> _':__builtin__}, "test_func3")
> py> test_func3(sys)
> restricted False
> 
> And now, repeating the timings when __builtins__ is correctly set to the  
> builtin module __builtin__ (!), shows no difference with the original  
> function. So this was the cause of the slow down.
> 
> The rule is: unless you *actually* want to execute code in restricted  
> mode, pass globals() when building a new function object, or at least, set  
> '__builtins__' to the __builtin__ module

Thanks for the very informative and detailed information. After posting
I, like the other responders, figured out that it was related to the
empty globals dict but I had no idea how that could be happening.