Using dict subclasses in eval() and exec ---------------------------------------- With the attached patch, you can now use a dict subclass which implements __getitem__ as the global or builtin namespace for eval() and exec. If you do not use the feature, the performance impact is low (<2%). The fast case adds only one test+branch to each eval_code() call (to set 'fastglobals' and 'slowbuiltins'), each lookup in globals (to check 'fastglobals'), and each lookup in builtins (to check 'slowbuiltins'). If you do use the feature, the performance impact is unfortunately quite substantial (400%). If you use subclasses of dict with eval()/exec, but do not define __getitem__, the performance penalty is modest (~10%). The smaller penalty of using a dict subclass without __getitem__ could probably be erased if the 'fastglobals' test can check that __getitem__ is not defined Included are a patch and a test suite. Note that the patch causes a failure in test_descr which is testing that __getitem__ is *not* called on a dictionary subclass. Pystone timings on a 1GHz Duron system running RedHat Linux 8.0 (gcc 3.2) follow. CVS +patch +subclass +__getattr__ 1 2.74 2.76 3 14.05 2 2.74 2.78 3.02 14.07 3 2.74 2.78 3.03 14.07 4 2.76 2.78 3.04 14.08 5 2.76 2.79 3.04 14.1 avg 2.748 2.778 3.026 14.074 % 100 101.1 110.1 512.2 * 2.3a0 CVS (28 Oct 2002) Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.74 This machine benchmarks at 18248.2 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second * 2.3a0 CVS + patch Pystone(1.1) time for 50000 passes = 2.78 This machine benchmarks at 17985.6 pystones/second Pystone(1.1) time for 50000 passes = 2.78 This machine benchmarks at 17985.6 pystones/second Pystone(1.1) time for 50000 passes = 2.79 This machine benchmarks at 17921.1 pystones/second Pystone(1.1) time for 50000 passes = 2.79 This machine benchmarks at 17921.1 pystones/second Pystone(1.1) time for 50000 passes = 2.76 This machine benchmarks at 18115.9 pystones/second * 2.3a0 CVS + patch + dict subclass w/o __getitem__ (commandline: ./python -c 'class D(dict): pass d = D(globals()); execfile("Lib/test/pystone.py", d)') Pystone(1.1) time for 50000 passes = 3.03 This machine benchmarks at 16501.7 pystones/second Pystone(1.1) time for 50000 passes = 3.04 This machine benchmarks at 16447.4 pystones/second Pystone(1.1) time for 50000 passes = 3 This machine benchmarks at 16666.7 pystones/second Pystone(1.1) time for 50000 passes = 3.04 This machine benchmarks at 16447.4 pystones/second Pystone(1.1) time for 50000 passes = 3.02 This machine benchmarks at 16556.3 pystones/second * 2.3a0 CVS + patch + dict subclass w/__getitem__ (commandline: ./python -c 'class D(dict): __getitem__ = lambda s, i: dict.__getitem__(s, i) d = D(globals()); execfile("Lib/test/pystone.py", d)') Pystone(1.1) time for 50000 passes = 14.05 This machine benchmarks at 3558.72 pystones/second Pystone(1.1) time for 50000 passes = 14.08 This machine benchmarks at 3551.14 pystones/second Pystone(1.1) time for 50000 passes = 14.07 This machine benchmarks at 3553.66 pystones/second Pystone(1.1) time for 50000 passes = 14.07 This machine benchmarks at 3553.66 pystones/second Pystone(1.1) time for 50000 passes = 14.1 This machine benchmarks at 3546.1 pystones/second
Jeff Epler wrote:
If all you want is a way to define a default value for a failing dictionary lookup, wouldn't it be better to simply add this feature to the standard dictionary implementation ? I don't think that many people will use this feature, yet everybody would pay for the performance hit. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
I don't think that many people will use this feature, yet everybody would pay for the performance hit.
I wonder how many reasonable real-life scenarios actually exist for this feature? My personal use would be for COM, to implement case-insensitive namespaces. In some cases, case-insensitivity is actually part of the implementation specification, and I can imagine other scenarios where case insensitive namespaces are "reasonable" even in the face of Python's excellent decision not to go this route. I wonder is this is actually the root of many requests to see this feature implemented? If not, it would be interesting to see what the competing desires really are. Mark.
Excellent observation. Perhaps the people who want this can get together and write a PEP that explains the motivation and why the alternatives don't work? (E.g. when you need case-insensitive variable names, perhaps it's just as easy to preprocess the source code?) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I can think of two 'real world' use cases for this patch that I remember. 1) When I was consulting for a finite element modeling company that was using Python to manipulate _very_ large objects through a database. I don't recall the details, but basically they wanted to be able to control database connection lifetimes in a way that wasn't possible at the time (Python 1.4 or so, I think =). These days I think that proxies would probably be leveraged to provide the capability, but namespace lookup would still be useful -- one objection they had to using proxies is that they didn't want their users to think in terms of objects with attributes -- they wanted their users to say: a = b + c and have the system figure out at runtime whether to evaluate b, a DB connection needed to be established, or not. None of the alternatives I thought of at the time "worked" for various reasons. Sorry I'm so vague, it's been a long time. (I don't even remember how the problem was resolved). 2) The other case where I would have used this is in building a high-power interactive shell for scientific users doing modeling (think Matlab on steroids), where I wanted the interpreter to be able to resolve names at runtime from pickled states -- that way the memory load was dependent on actual usage, but the "impression" to the user would have been that the system "never forgot". In both of these cases, Guido would be right in saying that that's not Python, but that's somewhat irrelevant -- the end-users would have had a "better" user experience with the feature on, and other aspects of the system would have made it quite clear that this wasn't vanilla Python. I like the feature, not necessarily for inclusion in core Python but as a way of experimenting with alternate universes in a research mode: "What would a PHP-like 'isset()' feel like?" "Could we do something akin to Tcl's TclObj dual representations?" Name resolution has always felt like the least "hookable" of Python's mechanisms (as opposed to attribute lookup, exception handling, etc.), and I wonder what good things could come of making it more open to experimentation. --david PS: Thinking of PHP, the fact that I can look up variables defined in sessions or cookies or whatnot really really easily is one of the reasons I'm liking PHP more than Python for web programming these days. I don't know enough of the technical details of the web to know automatically where what data is stored and in what format -- and PHP doesn't make me. Maybe web folks would use the dict-subclass feature to emulate this aspect of PHP.
I think (2) could be resolved using "ghosts" -- this is what Zope's persistency mechanism uses. The names must be present, but their content is loaded on demand. I expect that (1) probably could also be solved this way.
That's a good question.
But why not make it an attribute lookup? That's how Zope (again :-) deals with this. Remember acquisition? :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
That assumes that you can "preload" all of the names at initialization, correct? IOW, that you know the contents of the namespace in advance. Not a huge limitation, but not the same as lazy ghost-creation on lookup.
Yah, I remember. I'm not saying that one can't come up with a pythonic yet low-impact way of presenting web constructs to users. I guess I'm just expressing frustration with the fact that in PHP I don't really need to understand the web, whereas in Python I feel I have to. Seems wrong. =) --da
Correct. I don't see this as a huge limitation -- all "autoload" features that I'm familiar with (Emacs, Tcl; how about Perl?) require you to know at least the auto-loaded names in advance. (Also in my defence you only mentioned the cost of unpickling everything as an argument. :-)
I guess this depends on which Python application you use. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 2:11 PM -0500 10/29/02, Guido van Rossum wrote:
Perl doesn't require it. If a method call is made but that method name doesn't appear in the object's inheritance hierarchy, perl walks up the tree looking for a method named AUTOLOAD and if it finds one calls that. The AUTOLOAD method then has the opportunity to do something--satisfy the call, dispatch to a parent class, or pitch a fit. It's valid for an AUTOLOAD method to satisfy the request without instantiating a method for it. (In which case the next invocation of that method will end up back in AUTOLOAD) Works for subroutines too, though the hierarchy isn't supposed to be walked for those. (Older versions of perl would do it, but that's been deprecated) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk
Um, as long we're doing *method* lookup, Python's __getattr__ can do all that.
OK, so if there's a global AUTOLOAD and no global x, looking for x will call AUTOLOAD to satisfy x, right? Python doesn't do this, and this is what the "hookable namespace" feature is aiming at. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 2:26 PM -0500 10/29/02, Guido van Rossum wrote:
Sorry--it's a twisty conversation of little passages. I got somewhat turned around.
If there's an AUTOLOAD in a package, no X in that package, and you call X, then that package's AUTOLOAD will get called instead, yes. So if you had: print(Foo::Bar()); to print the output of the subroutine Bar in the package Foo, and there was no subroutine Bar in the package Foo but there *is* a subroutine AUTOLOAD in the package Foo, then Foo's AUTOLOAD will get called instead. Appropriate variables are filled in so that AUTOLOAD method can tell what sub it was called for. Only works for subs, not for global variables. (That's for perl 6, I expect) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk
If at some point it's decided that this patch is a good idea I think we can get rid of the performance hit in the normal case by duplicating eval_code2() with some preprocessor magic. eval_code2() would then call out to eval_code2_non_dict() if the globals weren't in a dict. Hmm, I wonder whether there are other situations where this idea would work too, i.e. situations where eval_code2() could decide at startup that it could take a fast path... -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -
I don't see the extra DECREF here (and below) needed to compensate for the fact that PyObject_GetItem() returns the object with incremented refcount but PyDict_GetItem() doesn't. What are you going to do with all the *other* places where PyDict_GetItem() is used? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tue, Oct 29, 2002 at 07:09:22AM -0500, Guido van Rossum wrote:
oops. I believe this can be fixed, if only by a DECREF in all paths and an INCREF in the fast path. However, this will worsen the "fast case" performance figures at least a tiny bit. [guido again]
What are you going to do with all the *other* places where PyDict_GetItem() is used?
Which places do you have in mind? The common feature request seems to be for globals() only, and I can't immediately think of a way of accessing globals() that my patch doesn't (attempt to) cover. Unfortunately, I'm not particularly excited about this capability myself. What I think this does show, however, is that using a sequence like (not quite C code) inline PyObject *PyObject_GetItem_FastForDicts(o, k) { if(PyDict_CheckExact(o)) { ret = PyDict_GetItem(o, k); Py_INCREF(ret); return ret; } return PyObject_GetItem(o, k); } will impose in-the-noise penalties for 'dict' instances, modest penalties for 'dict' subclass instances without __getitem__, and provide the full power for general objects. On Tue, Oct 29, 2002 at 08:57:01AM +0100, M.-A. Lemburg wrote:
It doesn't seem to be quite this simple, because you almost certainly want to have items in the builtin namespace appear even if you want to give a default on what would eventually be a NameError for a LOAD_GLOBAL/LOAD_NAME. You end up with something like class D(dict): def __getitem__(self, item): try: return dict.__getitem__(self, item) except KeyError: try: return getattr(__builtins__, item) except AttributeError: return "XX" which is more complicated that a "simple" defaulting dictionary. Unfortunately, this is another time when I am not a strong proponent of a feature, but merely wrote the patch to see if the "common wisdom" on which the rejection was based is valid. (common wisdom's been right in earlier cases, but I'm tempted to call the speed penalties for this change entirely tolerable if the purposes of people who do want this patch are good ones) On Tue, Oct 29, 2002 at 12:11:12AM -0500, Raymond Hettinger wrote [in a private message]:
It took me awhile to figure this out... LOAD_NAME first tries to load from the locals dict, where I did not change the way that name resolution functions. eval() uses the same dict for locals and globals if the locals (third arg) are not directly specified. The following works: class SpreadSheet(dict): def __getitem__(self, key): return eval(dict.__getitem__(self,key), self, {}) Jeff
I'm already unhappy about the performance hit of your patch. We're trying very hard to make global and built-in access as fast as possible. This patch adds extra local variable references to the "fast path". More INCREFS and DECREFS will slow it down even more. Not just because there's more executed code, but also because there's more code, period, which may reduce the I-cache effectiveness.
There are lots of other places where PyDict_*Item is used for an object that could be a dict subclass. Unless you've tracked all these down I think it wouldn't be wise to apply this patch. We either say (as we say now) "you can't override __getitem__ in a dict passed into the Python interpreter and expect it to work" or we say "you can override __getitem__ and it will work". It would be really bad to say "you can override __getitem__ and it will usually work but not in these twelve situations, because we didn't bother to change legacy code."
Unfortunately, I'm not particularly excited about this capability myself.
Me neither, to say the least. I still feel hostile, both because of the performance hit, and because I worry that it will be used to create obfuscated code that *looks* like Python but isn't Python.
I'd like to see more benchmarking of this (as well as more motivation).
Another deficiency of the patch.
--Guido van Rossum (home page: http://www.python.org/~guido/)
Still working from the version which is missing DECREFs, I get these timings from a program intended to more directly measure the speed of local, global, and builtin name lookups: Vanilla Python CVS. In effect these are three runs of the same thing, so you get some idea of the timing variations on my system. $ python-cvs/python lookup-speed.py Timing 4000000 name lookups type builtin global local overhead dict 2.74015 1.99215 1.21323 0.17610 D 2.69901 1.99129 1.24662 0.17832 E 2.73839 2.03171 1.25233 0.17843 CVS with my original patch (leaks references). "D" is a dict subclass which doesn't define __getitem__, "E" is a dict subclass which does (simply calling dict.__getitem__ and returning the result) $ python-dict/python lookup-speed.py Timing 4000000 name lookups type builtin global local overhead dict 2.81268 2.04795 1.26585 0.18117 D 5.08308 2.85853 1.23625 0.17944 E 81.62213 64.60481 1.26524 0.17973 This is a ~3% slowdown on the test with the heaviest use of builtins and globals that I can think of offhand. Slowdown: builtin global dict 3% 3% D 80% 40% E *28 *31 I'm about done pretending to cheerlead for this patch, though if I fix the reference problem I'll post the revised diffs. Jeff
"JE" == Jeff Epler <jepler@unpythonic.net> writes:
JE> I'm about done pretending to cheerlead for this patch, though if JE> I fix the reference problem I'll post the revised diffs. I'm not enthusiastic about this patch, but I think it's really helpful that you've done this work. A concrete patch, albeit incomplete, and some real performance measurements improves the quality of discussion on this issue. Did you post the lookup-speed.py script in an earlier message? I don't think I know what exactly you are measuring. Jeremy
BTW, the (final) patch, test script, and commentary should be uploaded to the SF patch manager. --Guido van Rossum (home page: http://www.python.org/~guido/)
1. You've got to show your test code. 2. Have you tried pybench? --Guido van Rossum (home page: http://www.python.org/~guido/)
Here's the lookup-speed script. num = 1000000 code = """ x = None r = range(%d) def builtin_lookup(): for i in r: len; len; len; len len; len; len; len len; len; len; len len; len; len; len def global_lookup(): for i in r: x; x; x; x x; x; x; x x; x; x; x x; x; x; x def local_lookup(): x = None for i in r: x; x; x; x x; x; x; x x; x; x; x x; x; x; x def overhead(): for i in r: pass """ % num class D(dict): pass class E(dict): __getitem__ = lambda s, k: dict.__getitem__(s, k) print "Timing %d name lookups" % (num*4) print "%8s %8s %8s %8s %8s" % ("type", "builtin", "global", "local", "overhead") import time for dc in dict, D, E: d = dc() exec code in d print "%8s" % dc.__name__, for f in ("builtin_lookup()", "global_lookup()", "local_lookup()", "overhead()"): t0 = time.time() exec f in d t1 = time.time() print "%8.5f" % (t1-t0), print
Jeff Epler wrote:
If all you want is a way to define a default value for a failing dictionary lookup, wouldn't it be better to simply add this feature to the standard dictionary implementation ? I don't think that many people will use this feature, yet everybody would pay for the performance hit. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
I don't think that many people will use this feature, yet everybody would pay for the performance hit.
I wonder how many reasonable real-life scenarios actually exist for this feature? My personal use would be for COM, to implement case-insensitive namespaces. In some cases, case-insensitivity is actually part of the implementation specification, and I can imagine other scenarios where case insensitive namespaces are "reasonable" even in the face of Python's excellent decision not to go this route. I wonder is this is actually the root of many requests to see this feature implemented? If not, it would be interesting to see what the competing desires really are. Mark.
Excellent observation. Perhaps the people who want this can get together and write a PEP that explains the motivation and why the alternatives don't work? (E.g. when you need case-insensitive variable names, perhaps it's just as easy to preprocess the source code?) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I can think of two 'real world' use cases for this patch that I remember. 1) When I was consulting for a finite element modeling company that was using Python to manipulate _very_ large objects through a database. I don't recall the details, but basically they wanted to be able to control database connection lifetimes in a way that wasn't possible at the time (Python 1.4 or so, I think =). These days I think that proxies would probably be leveraged to provide the capability, but namespace lookup would still be useful -- one objection they had to using proxies is that they didn't want their users to think in terms of objects with attributes -- they wanted their users to say: a = b + c and have the system figure out at runtime whether to evaluate b, a DB connection needed to be established, or not. None of the alternatives I thought of at the time "worked" for various reasons. Sorry I'm so vague, it's been a long time. (I don't even remember how the problem was resolved). 2) The other case where I would have used this is in building a high-power interactive shell for scientific users doing modeling (think Matlab on steroids), where I wanted the interpreter to be able to resolve names at runtime from pickled states -- that way the memory load was dependent on actual usage, but the "impression" to the user would have been that the system "never forgot". In both of these cases, Guido would be right in saying that that's not Python, but that's somewhat irrelevant -- the end-users would have had a "better" user experience with the feature on, and other aspects of the system would have made it quite clear that this wasn't vanilla Python. I like the feature, not necessarily for inclusion in core Python but as a way of experimenting with alternate universes in a research mode: "What would a PHP-like 'isset()' feel like?" "Could we do something akin to Tcl's TclObj dual representations?" Name resolution has always felt like the least "hookable" of Python's mechanisms (as opposed to attribute lookup, exception handling, etc.), and I wonder what good things could come of making it more open to experimentation. --david PS: Thinking of PHP, the fact that I can look up variables defined in sessions or cookies or whatnot really really easily is one of the reasons I'm liking PHP more than Python for web programming these days. I don't know enough of the technical details of the web to know automatically where what data is stored and in what format -- and PHP doesn't make me. Maybe web folks would use the dict-subclass feature to emulate this aspect of PHP.
I think (2) could be resolved using "ghosts" -- this is what Zope's persistency mechanism uses. The names must be present, but their content is loaded on demand. I expect that (1) probably could also be solved this way.
That's a good question.
But why not make it an attribute lookup? That's how Zope (again :-) deals with this. Remember acquisition? :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
That assumes that you can "preload" all of the names at initialization, correct? IOW, that you know the contents of the namespace in advance. Not a huge limitation, but not the same as lazy ghost-creation on lookup.
Yah, I remember. I'm not saying that one can't come up with a pythonic yet low-impact way of presenting web constructs to users. I guess I'm just expressing frustration with the fact that in PHP I don't really need to understand the web, whereas in Python I feel I have to. Seems wrong. =) --da
Correct. I don't see this as a huge limitation -- all "autoload" features that I'm familiar with (Emacs, Tcl; how about Perl?) require you to know at least the auto-loaded names in advance. (Also in my defence you only mentioned the cost of unpickling everything as an argument. :-)
I guess this depends on which Python application you use. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 2:11 PM -0500 10/29/02, Guido van Rossum wrote:
Perl doesn't require it. If a method call is made but that method name doesn't appear in the object's inheritance hierarchy, perl walks up the tree looking for a method named AUTOLOAD and if it finds one calls that. The AUTOLOAD method then has the opportunity to do something--satisfy the call, dispatch to a parent class, or pitch a fit. It's valid for an AUTOLOAD method to satisfy the request without instantiating a method for it. (In which case the next invocation of that method will end up back in AUTOLOAD) Works for subroutines too, though the hierarchy isn't supposed to be walked for those. (Older versions of perl would do it, but that's been deprecated) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk
Um, as long we're doing *method* lookup, Python's __getattr__ can do all that.
OK, so if there's a global AUTOLOAD and no global x, looking for x will call AUTOLOAD to satisfy x, right? Python doesn't do this, and this is what the "hookable namespace" feature is aiming at. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 2:26 PM -0500 10/29/02, Guido van Rossum wrote:
Sorry--it's a twisty conversation of little passages. I got somewhat turned around.
If there's an AUTOLOAD in a package, no X in that package, and you call X, then that package's AUTOLOAD will get called instead, yes. So if you had: print(Foo::Bar()); to print the output of the subroutine Bar in the package Foo, and there was no subroutine Bar in the package Foo but there *is* a subroutine AUTOLOAD in the package Foo, then Foo's AUTOLOAD will get called instead. Appropriate variables are filled in so that AUTOLOAD method can tell what sub it was called for. Only works for subs, not for global variables. (That's for perl 6, I expect) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk
If at some point it's decided that this patch is a good idea I think we can get rid of the performance hit in the normal case by duplicating eval_code2() with some preprocessor magic. eval_code2() would then call out to eval_code2_non_dict() if the globals weren't in a dict. Hmm, I wonder whether there are other situations where this idea would work too, i.e. situations where eval_code2() could decide at startup that it could take a fast path... -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -
I don't see the extra DECREF here (and below) needed to compensate for the fact that PyObject_GetItem() returns the object with incremented refcount but PyDict_GetItem() doesn't. What are you going to do with all the *other* places where PyDict_GetItem() is used? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tue, Oct 29, 2002 at 07:09:22AM -0500, Guido van Rossum wrote:
oops. I believe this can be fixed, if only by a DECREF in all paths and an INCREF in the fast path. However, this will worsen the "fast case" performance figures at least a tiny bit. [guido again]
What are you going to do with all the *other* places where PyDict_GetItem() is used?
Which places do you have in mind? The common feature request seems to be for globals() only, and I can't immediately think of a way of accessing globals() that my patch doesn't (attempt to) cover. Unfortunately, I'm not particularly excited about this capability myself. What I think this does show, however, is that using a sequence like (not quite C code) inline PyObject *PyObject_GetItem_FastForDicts(o, k) { if(PyDict_CheckExact(o)) { ret = PyDict_GetItem(o, k); Py_INCREF(ret); return ret; } return PyObject_GetItem(o, k); } will impose in-the-noise penalties for 'dict' instances, modest penalties for 'dict' subclass instances without __getitem__, and provide the full power for general objects. On Tue, Oct 29, 2002 at 08:57:01AM +0100, M.-A. Lemburg wrote:
It doesn't seem to be quite this simple, because you almost certainly want to have items in the builtin namespace appear even if you want to give a default on what would eventually be a NameError for a LOAD_GLOBAL/LOAD_NAME. You end up with something like class D(dict): def __getitem__(self, item): try: return dict.__getitem__(self, item) except KeyError: try: return getattr(__builtins__, item) except AttributeError: return "XX" which is more complicated that a "simple" defaulting dictionary. Unfortunately, this is another time when I am not a strong proponent of a feature, but merely wrote the patch to see if the "common wisdom" on which the rejection was based is valid. (common wisdom's been right in earlier cases, but I'm tempted to call the speed penalties for this change entirely tolerable if the purposes of people who do want this patch are good ones) On Tue, Oct 29, 2002 at 12:11:12AM -0500, Raymond Hettinger wrote [in a private message]:
It took me awhile to figure this out... LOAD_NAME first tries to load from the locals dict, where I did not change the way that name resolution functions. eval() uses the same dict for locals and globals if the locals (third arg) are not directly specified. The following works: class SpreadSheet(dict): def __getitem__(self, key): return eval(dict.__getitem__(self,key), self, {}) Jeff
I'm already unhappy about the performance hit of your patch. We're trying very hard to make global and built-in access as fast as possible. This patch adds extra local variable references to the "fast path". More INCREFS and DECREFS will slow it down even more. Not just because there's more executed code, but also because there's more code, period, which may reduce the I-cache effectiveness.
There are lots of other places where PyDict_*Item is used for an object that could be a dict subclass. Unless you've tracked all these down I think it wouldn't be wise to apply this patch. We either say (as we say now) "you can't override __getitem__ in a dict passed into the Python interpreter and expect it to work" or we say "you can override __getitem__ and it will work". It would be really bad to say "you can override __getitem__ and it will usually work but not in these twelve situations, because we didn't bother to change legacy code."
Unfortunately, I'm not particularly excited about this capability myself.
Me neither, to say the least. I still feel hostile, both because of the performance hit, and because I worry that it will be used to create obfuscated code that *looks* like Python but isn't Python.
I'd like to see more benchmarking of this (as well as more motivation).
Another deficiency of the patch.
--Guido van Rossum (home page: http://www.python.org/~guido/)
Still working from the version which is missing DECREFs, I get these timings from a program intended to more directly measure the speed of local, global, and builtin name lookups: Vanilla Python CVS. In effect these are three runs of the same thing, so you get some idea of the timing variations on my system. $ python-cvs/python lookup-speed.py Timing 4000000 name lookups type builtin global local overhead dict 2.74015 1.99215 1.21323 0.17610 D 2.69901 1.99129 1.24662 0.17832 E 2.73839 2.03171 1.25233 0.17843 CVS with my original patch (leaks references). "D" is a dict subclass which doesn't define __getitem__, "E" is a dict subclass which does (simply calling dict.__getitem__ and returning the result) $ python-dict/python lookup-speed.py Timing 4000000 name lookups type builtin global local overhead dict 2.81268 2.04795 1.26585 0.18117 D 5.08308 2.85853 1.23625 0.17944 E 81.62213 64.60481 1.26524 0.17973 This is a ~3% slowdown on the test with the heaviest use of builtins and globals that I can think of offhand. Slowdown: builtin global dict 3% 3% D 80% 40% E *28 *31 I'm about done pretending to cheerlead for this patch, though if I fix the reference problem I'll post the revised diffs. Jeff
"JE" == Jeff Epler <jepler@unpythonic.net> writes:
JE> I'm about done pretending to cheerlead for this patch, though if JE> I fix the reference problem I'll post the revised diffs. I'm not enthusiastic about this patch, but I think it's really helpful that you've done this work. A concrete patch, albeit incomplete, and some real performance measurements improves the quality of discussion on this issue. Did you post the lookup-speed.py script in an earlier message? I don't think I know what exactly you are measuring. Jeremy
BTW, the (final) patch, test script, and commentary should be uploaded to the SF patch manager. --Guido van Rossum (home page: http://www.python.org/~guido/)
1. You've got to show your test code. 2. Have you tried pybench? --Guido van Rossum (home page: http://www.python.org/~guido/)
Here's the lookup-speed script. num = 1000000 code = """ x = None r = range(%d) def builtin_lookup(): for i in r: len; len; len; len len; len; len; len len; len; len; len len; len; len; len def global_lookup(): for i in r: x; x; x; x x; x; x; x x; x; x; x x; x; x; x def local_lookup(): x = None for i in r: x; x; x; x x; x; x; x x; x; x; x x; x; x; x def overhead(): for i in r: pass """ % num class D(dict): pass class E(dict): __getitem__ = lambda s, k: dict.__getitem__(s, k) print "Timing %d name lookups" % (num*4) print "%8s %8s %8s %8s %8s" % ("type", "builtin", "global", "local", "overhead") import time for dc in dict, D, E: d = dc() exec code in d print "%8s" % dc.__name__, for f in ("builtin_lookup()", "global_lookup()", "local_lookup()", "overhead()"): t0 = time.time() exec f in d t1 = time.time() print "%8.5f" % (t1-t0), print
participants (8)
-
Dan Sugalski -
David Ascher -
Guido van Rossum -
Jack Jansen -
Jeff Epler -
Jeremy Hylton -
M.-A. Lemburg -
Mark Hammond