Fast access to __builtins__
On Thu, Mar 27, 2003 at 04:37:59PM -0500, Raymond Hettinger wrote:
From past rumblings, I gather that Python is moving towards preventing __builtins__ from being shadowed.
I would like to know what you guys think about going ahead with that idea whenever the -O optimization flag is set.
The behavior of a program under -O should be as similar as possible to normal operation. This would break that for some programs. A per-file pragma directive would work. The downside of a pragma module or keyword would be what people try to add to it later. -jackdied
Raymond> From past rumblings, I gather that Python is moving towards Raymond> preventing __builtins__ from being shadowed. Raymond> I would like to know what you guys think about going ahead with Raymond> that idea whenever the -O optimization flag is set. Interesting idea, but I think of shadowing builtins being somewhat orthogonal to optimization in the usual sense (speed things up without changing the program's semantics). This is clearly a semantic change, so I'd like to see a different command line flag control this behavior. What happens if you run one program with builtin shadowing enabled, it writes some .pyo files with your suggested change, then later you run another program without it? Seems like there should be some memory in the file of how it was generated so the importer can raise an exception if it finds a mismatch between the shadowing command line flag and a previously generated bytecode file. Skip
Raymond Hettinger wrote:
From past rumblings, I gather that Python is moving towards preventing __builtins__ from being shadowed.
I would like to know what you guys think about going ahead with that idea whenever the -O optimization flag is set.
The idea is to scan the code for lines like:
LOAD_GLOBAL 2 (range)
and, if the name is found in __builtins__, then lookup the name, add the reference to the constants table and, replace the code with something like:
LOAD_CONST 5 (<type 'range'>)
Using the -O for this is not a working possibility. -OO is reserved for optimizations which can change semantics, but even there, I'd rather like a per-module switch than a command line switch. BTW, why not have a new opcode for symbols in the builtins and then only tweak the opcode implementation instead of having the compiler generate different code ?
The opcode replacement bypasses module level shadowing but leaves local shadowing intact. For example:
modglob = 1 range = xrange def f(list): for i in list: # local shadowing of 'list' is unaffected print ord(i) # access to 'ord' is optimized j = modglob # non-shadowed globals are unaffected k = range(j) # shadowing of globals is ignored
I've already tried out a pure python proof-of-concept and it is straightforward to recode it in C and attach it to PyCode_New().
Raymond Hettinger
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev
-- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Mar 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 5 days left EuroPython 2003, Charleroi, Belgium: 89 days left
[Jack Diederich]
The behavior of a program under -O should be as similar as possible to normal operation. This would break that for some programs. A per-file pragma directive would work.
[Skip Montanero]
This is clearly a semantic change, so I'd like to see a different command line flag control this behavior
[M.-A. Lemburg]
Using the -O for this is not a working possibility. -OO is reserved for optimizations which can change semantics, but even there, I'd rather like a per-module switch than a command line switch.
That makes good sense. Are you guys thinking of something like this: __fastbuiltins__ = True # optimize all subsequent defs in the module [Jack Diederich]
The downside of a pragma module or keyword would be what people try to add to it later.
Ideally, enabling the pragma would also trigger warnings when the module shadows a builtin. [M.-A. Lemburg]
BTW, why not have a new opcode for symbols in the builtins and then only tweak the opcode implementation instead of having the compiler generate different code ?
Either way results in changing one opcode/oparg pair, so I don't see how having a new opcode helps. At some point, the name has to be looked-up and a reference to it stored. Afterwards, LOAD_CONST is all that is needed to fetch the reference. Raymond Hettinger
On Thu, Mar 27, 2003 at 07:35:20PM -0500, Raymond Hettinger wrote: [Raymond proposed this pythonic version of a pragma]
__fastbuiltins__ = True # optimize all subsequent defs in the module
[Jack Diederich]
The downside of a pragma module or keyword would be what people try to add to it later.
Ideally, enabling the pragma would also trigger warnings when the module shadows a builtin.
I was thinking that a first-class keyword like pragma no_shadow_builtins would give people a hook to suggest all kinds of nastiness in the future. In your pythonness your suggestion avoided this entirely. We are talking about a very per-module thing, and author's intent. As a progression, how about a subclass of dictionaries that implement warn-on-assign and error-on-assign properties. Having a subclass of dicts specifically for symbol tables has been suggested before and has a wide variety of benefits[1]. This is a good example. -jackdied [1] benefits of a specific 'symtab' type that derives from dict (all variations on 'one stop shop for optimizations') * string-only * assign-once (builtins) * cached lookups
Raymond Hettinger wrote:
[per module switch] That makes good sense. Are you guys thinking of something like this:
__fastbuiltins__ = True # optimize all subsequent defs in the module
[M.-A. Lemburg]
BTW, why not have a new opcode for symbols in the builtins and then only tweak the opcode implementation instead of having the compiler generate different code ?
Either way results in changing one opcode/oparg pair, so I don't see how having a new opcode helps. At some point, the name has to be looked-up and a reference to it stored. Afterwards, LOAD_CONST is all that is needed to fetch the reference.
Right, but with the new opcode you could have the interpreter decide whether to optimize or not without recompiling the code. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Mar 28 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 4 days left EuroPython 2003, Charleroi, Belgium: 88 days left
M.-A. Lemburg wrote:
Using the -O for this is not a working possibility. -OO is reserved for optimizations which can change semantics, but even there, I'd rather like a per-module switch than a command line switch.
Optimization options that globally change semantics seem like a bad idea. How would you know some module you are using will not break? I agree with Mark that a per-module switch would be better. In this case, I'm not sure either option is necessary. If I understand Guido correctly, eventually programs may not be allowed to stick names into other modules that override builtins used by that module. If that is disallowed then the compiler knows if a name is a builtin or a global. We could introduce a warning for code that breaks the new rules and have a __future__ statement that implements the optimization. Neil
Neil Schemenauer <nas@python.ca>:
Optimization options that globally change semantics seem like a bad idea. How would you know some module you are using will not break? I agree with Mark that a per-module switch would be better.
There's something a bit strange about this situation, though. The compiler knows whether a module shadows any of its *own* builtins, and can avoid applying the optimisation to those names. So the optimisation doesn't change the semantics of the module itself, provided some conditions are met. But those conditions depend on things *outside* the module -- namely, whether any *other* module assigns to one of this module's globals so as to shadow a builtin. This makes me think that having a flag inside the module is not the right thing to do, or at least it's not the only thing that's needed. There needs to be a way to turn the optimisation *off* from outside the affected module. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Hi Raymond. Too bad you couldn't make it to the conference! We're all having a great time on and off the GWU premises. I used your "more zen" on a slide in my keynote.
From past rumblings, I gather that Python is moving towards preventing __builtins__ from being shadowed.
You must be misunderstanding. The only thing I want to forbid is to stick a name in *another* module's globals that would shadow a builtin. E.g. suppose module A contains: def f(a): return len(a) and module B contains: import A A.len = lambda a: len(a) or 1 # evil len() The assignment to A.len would be forbidden. OTOH this: import random if random.random() >= 0.5: len = 42 def f(): return len will always be allowed and mean what it currently means. The difference is that in the first module, analysis of module A does not reveal that len is shadowed; OTOH in the second example, analyzing just the module's code shows that len may be a global built-in. This is important because a programmer shouldn't have to know the names of built-in objects she doesn't use (also important because in a future version of the language, a name you've picked for a global may become a builtin). The idea of forbidding module B in the first example is that the optimizer is allowed to replace len(a) with a bytecode that calls PyOject_Size() rather than looking up "len" in globals and builtins. The optimizer should only be allowed to make this assumption if careful analysis of an entire module doesn't reveal any possibility that "len" can be shadowed. But it cannot be required to look at all other modules (since those other modules may not even have been written!). Hope this helps. BTW this idea is quite old; I've described it a few years ago under a subject something like "low-hanging fruit". --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
BTW this idea is quite old; I've described it a few years ago under a subject something like "low-hanging fruit".
I really like this idea. If a patch appeared on SF soon, do you think 2.3 could include a warning for code that violates the rule? If so, how about also including a flag to allowed optimizations based on the rule? For example, I think we could have the equivalent of LOAD_FAST for builtin names. Implementing the optimizations could be a bit of work, especially with the existing compiler, but I think the warning should be fairly easy. Neil
Guido van Rossum wrote:
BTW this idea is quite old; I've described it a few years ago under a subject something like "low-hanging fruit".
I really like this idea. If a patch appeared on SF soon, do you think 2.3 could include a warning for code that violates the rule?
Maybe. Though you probably would only want to warn when this is done to a .py module -- C extensions should be exempt. And the warning should only warn about inserting names that are actually builtins.
If so, how about also including a flag to allowed optimizations based on the rule? For example, I think we could have the equivalent of LOAD_FAST for builtin names. Implementing the optimizations could be a bit of work, especially with the existing compiler, but I think the warning should be fairly easy.
Sure, let's experiment! --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thu, 2003-03-27 at 23:15, Guido van Rossum wrote:
I really like this idea. If a patch appeared on SF soon, do you think 2.3 could include a warning for code that violates the rule?
Maybe. Though you probably would only want to warn when this is done to a .py module -- C extensions should be exempt. And the warning should only warn about inserting names that are actually builtins.
It seems like C extensions pose thorny problems that need to be solved. In particular, the C API says that module's have a dictionary and that adding a key creates global variable in the module. We'll have to break this one way or another, because we don't want to allow C extensions to add globals that shadow builtins. Right? There's a similar problem for Python code, but I imagine it's easy to come up with a dict proxy with the necessary restrictions along the lines of a new-style class dict proxy. How do we break the C API? There's lots of extension code that relies on getting the dict. My first guess is to add an exception that says setting a name that shadows a builtin has no effect. Then extend the getattr code and the module-dict-proxy to ignore those names. Jeremy
It seems like C extensions pose thorny problems that need to be solved. In particular, the C API says that module's have a dictionary and that adding a key creates global variable in the module. We'll have to break this one way or another, because we don't want to allow C extensions to add globals that shadow builtins. Right?
I don't see the problem. Typically, C extension modules don't have Python code that runs in their globals, so messing with a C extension's globals from the outside has no bad effect on Python code. The problem is more that once a module is loaded, you can't tell from the module whether it was loaded from a .py module or a C extension.
There's a similar problem for Python code, but I imagine it's easy to come up with a dict proxy with the necessary restrictions along the lines of a new-style class dict proxy.
I'd be happy to proclaim that doing something like import X d = X.__dict__ d["spam"] = 42 # or exec "spam = 42" in d is always prohibited.
How do we break the C API? There's lots of extension code that relies on getting the dict. My first guess is to add an exception that says setting a name that shadows a builtin has no effect. Then extend the getattr code and the module-dict-proxy to ignore those names.
The C code can continue to access the real dict. This is what happens for new-style classes: in Python, C.__dict__ is a read-only proxy, but in C, C->tp_dict is a real dict. Then the setattr operation can do as it pleases. For new-style classes, it doesn't forbid anything but updates the type struct when an operator was modified; for modules, it could issue a warning when a name is set that didn't exist before and that shadows a built-in. (Ideally, it should only warn about built-ins that are actually used by the module's code, but that requires the parser to make the list of such built-ins available somehow.) Anyway, the C code that accesses the dict usually lives in the extension module's init function. Frankly, I'm a bit confused by your post. Maybe I don't understand what you're proposing? --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I'd be happy to proclaim that doing something like
import X d = X.__dict__ d["spam"] = 42 # or exec "spam = 42" in d
is always prohibited.
That would break lazy module imports such as the one I'm using in mx.Misc.LazyModule.py. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Mar 28 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 4 days left EuroPython 2003, Charleroi, Belgium: 88 days left
I'd be happy to proclaim that doing something like
import X d = X.__dict__ d["spam"] = 42 # or exec "spam = 42" in d
is always prohibited.
That would break lazy module imports such as the one I'm using in mx.Misc.LazyModule.py.
But you could rewrite LazyModule.py to use setattr(X, "spam", 42), right? I don't think it's worth it to have a dict proxy that allows certain keys to be set but not others. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I'd be happy to proclaim that doing something like
import X d = X.__dict__ d["spam"] = 42 # or exec "spam = 42" in d
is always prohibited.
That would break lazy module imports such as the one I'm using in mx.Misc.LazyModule.py.
But you could rewrite LazyModule.py to use setattr(X, "spam", 42), right?
Sure.
I don't think it's worth it to have a dict proxy that allows certain keys to be set but not others.
The question is: why make this complicated ? If the programmer enables __fast_builtins__ (or similar) in the module scope, she should be aware that tweaking the module globals from the outside won't have the desired effect. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Mar 28 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 4 days left EuroPython 2003, Charleroi, Belgium: 88 days left
The question is: why make this complicated ?
If the programmer enables __fast_builtins__ (or similar) in the module scope, she should be aware that tweaking the module globals from the outside won't have the desired effect.
I don't want programmers to have to add all sorts of magical incantations to their top to guide the optimizer. Today it's __fast_builtins__, tomorrow it's a promise that a class won't be poked. Poking a module from the outside is frequent enough, but poking names that shadow builtins is extremely rare. So almost all modules would need __fast_builtins__, because it would almost always help. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Friday 28 March 2003 05:49 am, Guido van Rossum wrote: ...
I don't see the problem. Typically, C extension modules don't have Python code that runs in their globals, so messing with a C extension's globals from the outside has no bad effect on Python code.
It happens, though -- for code whose performance is not important, e.g. initialization and "resetting" kind of stuff, a PyRun_String can be SO much more concise and handier than meticulous expansion of basically the same things into tens of lines of C code... since "messing from the outside" happens after initialization, and the use cases I can easily find are all specifically DURING initialization, it may be that this problem is too rare to worry about, but, I'm not so sure. Alex
I don't see the problem. Typically, C extension modules don't have Python code that runs in their globals, so messing with a C extension's globals from the outside has no bad effect on Python code.
It happens, though -- for code whose performance is not important, e.g. initialization and "resetting" kind of stuff, a PyRun_String can be SO much more concise and handier than meticulous expansion of basically the same things into tens of lines of C code... since "messing from the outside" happens after initialization, and the use cases I can easily find are all specifically DURING initialization, it may be that this problem is too rare to worry about, but, I'm not so sure.
I think this use case won't have a problem. The C code has access to the real dict, so PyRun_String() never knows that it's poking into a module's globals. Also this is done during module initialization. --Guido van Rossum (home page: http://www.python.org/~guido/)
Alex Martelli <aleax@aleax.it>:
It happens, though -- for code whose performance is not important, e.g. initialization and "resetting" kind of stuff, a PyRun_String can be SO much more concise and handier than meticulous expansion of basically the same things into tens of lines of C code...
Nowadays you can let Pyrex do the expansion for you...:-) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
On Thu, 2003-03-27 at 23:49, Guido van Rossum wrote:
Frankly, I'm a bit confused by your post. Maybe I don't understand what you're proposing?
Modules are modules, right? That is, pickle.py and cPickle.so are both represented as module objects at runtime. A C extension can call PyModule_GetDict() on any module. If so, then any extension module can add names to the __dict__ of any Python module. The problem is that modules expose their representation at the C API level (namespace implemented as PyDictObject), so it's difficult to forbid things at the C level. Jeremy
Frankly, I'm a bit confused by your post. Maybe I don't understand what you're proposing?
Modules are modules, right? That is, pickle.py and cPickle.so are both represented as module objects at runtime. A C extension can call PyModule_GetDict() on any module. If so, then any extension module can add names to the __dict__ of any Python module. The problem is that modules expose their representation at the C API level (namespace implemented as PyDictObject), so it's difficult to forbid things at the C level.
Oh sure. I don't think it's necessary to forbid things at the C API level in the sense of making it impossible to do. We'll just document that C code shouldn't do that. There's plenty that C code could do but shouldn't because it breaks the world. I don't expect there will be much C in violation of this prohibition. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I don't think it's necessary to forbid things at the C API level in the sense of making it impossible to do. We'll just document that C code shouldn't do that.
What about Python code that modifies that module __dict__ directly? For example, using vars() or globals() to get a reference to it and doing __setitem__ on it. My warning code only catches assignments that go through the module tp_setattro slot. I suppose warning about direct __dict__ poking would require a proxy object to wrap the module dict. Neil
I don't think it's necessary to forbid things at the C API level in the sense of making it impossible to do. We'll just document that C code shouldn't do that.
What about Python code that modifies that module __dict__ directly? For example, using vars() or globals() to get a reference to it and doing __setitem__ on it. My warning code only catches assignments that go through the module tp_setattro slot. I suppose warning about direct __dict__ poking would require a proxy object to wrap the module dict.
Yeah, that's another niggling issue. It would be a shame if using globals() or vars() anywhere in a module would disable this optimization. But we can't make these return a proxy either, because they are frequently used with e.g. "exec ... in globals()". --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Though you probably would only want to warn when this is done to a .py module -- C extensions should be exempt.
Exempt from poking or being poked?
And the warning should only warn about inserting names that are actually builtins.
I have rough patch. The idea is to have the tp_setattro slot of modules check if the name being set is a builtin. It seems to work but perhaps there are cases that make that approach invalid. Time for bed now. :-) Neil
On Thu, Mar 27, 2003 at 09:51:13PM -0800, Neil Schemenauer wrote:
Guido van Rossum wrote:
Though you probably would only want to warn when this is done to a .py module -- C extensions should be exempt.
Exempt from poking or being poked?
And the warning should only warn about inserting names that are actually builtins.
I have rough patch. The idea is to have the tp_setattro slot of modules check if the name being set is a builtin.
Does it check if it's one of the standard __builtin__ module or whether it is an attribute of whatever object is currently set as the module's __builtins__ attribute? Oren
Does it check if it's one of the standard __builtin__ module or whether it is an attribute of whatever object is currently set as the module's __builtins__ attribute?
Only standard builtins need to be exempt, because the compiler isn't going to optimize non-standard builtins. That's because (a) there won't be special opcodes that implement those builtins directly, and (b) the bytecode compiler doesn't know the contents of __builtins__ so it can't possibly know about nonstandard builtins anyway to generate a LOAD_BUILTIN opcode. BTW, I expect that nonstandard builtins will be ruled out in some future version of the language, or will have to be declared differently. They are too confusing for the human reader of the code. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Friday, March 28, 2003, at 07:33 AM, Guido van Rossum wrote:
BTW, I expect that nonstandard builtins will be ruled out in some future version of the language, or will have to be declared differently. They are too confusing for the human reader of the code.
When you say "nonstandard builtins", do you mean nonstandard names or nonstandard values, or both? E.g. assigning gettext.ugettext() to builtin _() or setting open() to some debugging func. I wouldn't want to completely disallow these, but I'd be happy if you had to do something special and/or (more) explicit to make them work. -Barry
Barry> ... or setting open() to some debugging func. Barry> I wouldn't want to completely disallow these, but I'd be happy if Barry> you had to do something special and/or (more) explicit to make Barry> them work. Like a compiler flag to disable the run-time optimization so your debugging open() would be seen everywhere? Sort of like Guido's observation about __fastbuiltins__ = True, the frequent case (regular, optimized version of open()) should be the default, while the exception requires programmer or user action. Skip
On Fri, 2003-03-28 at 08:30, Skip Montanaro wrote:
Like a compiler flag to disable the run-time optimization so your debugging open() would be seen everywhere?
Sure, that would work. I'm still thinking about "from __builtins__ import open". Part of the issue there is that you might not be sure /which/ open is causing the problems. But I agree that this is not a common case; I don't even think it would be common programming practice (i.e. my use case is primarily debugging). -Barry P.S. I don't actually poke _() into builtins :)
BTW, I expect that nonstandard builtins will be ruled out in some future version of the language, or will have to be declared differently. They are too confusing for the human reader of the code.
When you say "nonstandard builtins", do you mean nonstandard names or nonstandard values, or both? E.g. assigning gettext.ugettext() to builtin _() or setting open() to some debugging func.
Nonstandard names. The compiler can't know what's in __builtin__, but it can know the names of the official built-ins.
I wouldn't want to completely disallow these, but I'd be happy if you had to do something special and/or (more) explicit to make them work.
"from __builtin__ import open" should do it. --Guido van Rossum (home page: http://www.python.org/~guido/)
[GvR]
Hi Raymond. Too bad you couldn't make it to the conference! We're all having a great time on and off the GWU premises.
Glad you guys are having a great time. I wish I could be there.
I used your "more zen" on a slide in my keynote.
Cool. Any chance of getting your keynote slides on the net?
From past rumblings, I gather that Python is moving towards preventing __builtins__ from being shadowed.
You must be misunderstanding.
The only thing I want to forbid is to stick a name in *another* module's globals that would shadow a builtin.
Yes, that *is* different. Allowing shadows means having to watch out for trees.
The idea of forbidding module B in the first example is that the optimizer is allowed to replace len(a) with a bytecode that calls PyOject_Size() rather than looking up "len" in globals and builtins. The optimizer should only be allowed to make this assumption if careful analysis of an entire module doesn't reveal any possibility that "len" can be shadowed . . . BTW this idea is quite old; I've described it a few years ago under a subject something like "low-hanging fruit".
The fruit is a bit high. Doing a full module analysis means deferring the optimization for a second pass after all the code has already been generated. It's doable, but much harder. def f(x): return len(x) + 10 # knowing whether to optimize this def g(): global len # when this is allowed len = lambda x: 5 # is a bear The task is much simpler if it can be known in advance that the substitution is allowed (i.e. a module level switch like: __fastbuiltins__ = True). Raymond Hettinger
Cool. Any chance of getting your keynote slides on the net?
Yes, after the conference.
From past rumblings, I gather that Python is moving towards preventing __builtins__ from being shadowed.
You must be misunderstanding.
The only thing I want to forbid is to stick a name in *another* module's globals that would shadow a builtin.
Yes, that *is* different. Allowing shadows means having to watch out for trees.
Being poetic?
The idea of forbidding module B in the first example is that the optimizer is allowed to replace len(a) with a bytecode that calls PyOject_Size() rather than looking up "len" in globals and builtins. The optimizer should only be allowed to make this assumption if careful analysis of an entire module doesn't reveal any possibility that "len" can be shadowed . . . BTW this idea is quite old; I've described it a few years ago under a subject something like "low-hanging fruit".
The fruit is a bit high. Doing a full module analysis means deferring the optimization for a second pass after all the code has already been generated. It's doable, but much harder.
You're stuck in a one-pass compiler mindset. We build a parse tree for the entire module before we start generating bytecode. We already have tools to do namespace analysis for the entire tree (Jeremy added these to implement nested scopes).
def f(x): return len(x) + 10 # knowing whether to optimize this
def g(): global len # when this is allowed len = lambda x: 5 # is a bear
The task is much simpler if it can be known in advance that the substitution is allowed (i.e. a module level switch like: __fastbuiltins__ = True).
-1000. --Guido van Rossum (home page: http://www.python.org/~guido/)
The fruit is a bit high. Doing a full module analysis means deferring the optimization for a second pass after all the code has already been generated. It's doable, but much harder.
You're stuck in a one-pass compiler mindset. We build a parse tree for the entire module before we start generating bytecode. We already have tools to do namespace analysis for the entire tree (Jeremy added these to implement nested scopes). . . .
The task is much simpler if it can be known in advance that the substitution is allowed (i.e. a module level switch like: __fastbuiltins__ = True).
-1000.
Having ruled out a module level switch, the -O flag, and the -OO flag, that leaves the namespace analysis of the entire tree or taking an approach that doesn't change the bytecode. Taking the second approach, I've loaded a small patch for caching lookups into the __builtins__ namespace: www.python.org/sf/711722 It's not as fast as using LOAD_CONST, but is safe in all but one extreme case: calling the function, having an intervening poke into the __builtins__ module, and then calling the function again. I put the cache lookup in the safest possible place. It can be made twice as fast by putting it before the func_globals() lookup. That works in all cases except: calling the function, having an intervening shadowing global assignment, and then calling the function again. This doesn't come-up anywhere in the test suite, my own apps, or apps I've downloaded. Note, regular shadowing (before the first function call) continues to work fine. The bad news is that I've made many timings and found only modest speed-ups in real code. It turns out that access time for builtins is less significant than the time to call and execute those builtins. But, every little bit helps. Raymond Hettinger
Raymond Hettinger wrote:
The bad news is that I've made many timings and found only modest speed-ups in real code. It turns out that access time for builtins is less significant than the time to call and execute those builtins. But, every little bit helps.
Perhaps you ought to look into special casing calling builtins, e.g. by adding a byte code CALL_BUILTIN ?! Since the signatures of the builtins are known in advance, the calling overhead could be reduced, though I'm not sure how much more can be gained since the function call code was refactored. Another idea which might be worth looking into is that of speeding up parsing of C function call arguments, e.g. by caching the results or adding fast paths for common combinations. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Mar 29 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 3 days left EuroPython 2003, Charleroi, Belgium: 87 days left
participants (12)
-
Alex Martelli -
Barry Warsaw -
Greg Ewing -
Guido van Rossum -
Jack Diederich -
Jeremy Hylton -
M.-A. Lemburg -
Neil Schemenauer -
Oren Tirosh -
Raymond Hettinger -
Raymond Hettinger -
Skip Montanaro