Multiple dicts for string interpolation?

Every once in awhile I want to perform string interpolation using more than one dictionary. One way is to build a dictionary that's a union of multiple dictionaries: dict = {} dict.update(d1) dict.update(d2) ... s = format % dict Another way is the MultiDict approach that Digital Creations (used to?) use in their DocumentTemplate module (I can't remember the exact usage any more): dict = MultiDict() dict.append(d1) dict.append(d2) ... s = format % dict A MultiDict object maintains a list of the dicts it's been fed and searches them in order when __getitem__ is called. I'd like to propose a third alternative. How about if the string interpolation function accepted a tuple of dictionaries directly: s = format % (d1, d2) It would only be used when named interpolation was expected. I don't think there would be any conflict with current % operator semantics. Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/ 847-971-7098

Gut feeling: it's dangerous to fill up every possible dark corner with specific semantics that are occasionally useful, because if you go too far you lose useful redundancy, and you end up with Perl. Not sure whether this particular suggestion is "going too far." I think it depends on to what extent this is a common, useful idiom. Do you have evidence of that? Examples? --Guido van Rossum (home page: http://www.python.org/~guido/)

>> I'd like to propose a third alternative. How about if the string >> interpolation function accepted a tuple of dictionaries directly: >> >> s = format % (d1, d2) Guido> Gut feeling: it's dangerous to fill up every possible dark corner Guido> with specific semantics that are occasionally useful, because if Guido> you go too far you lose useful redundancy, and you end up with Guido> Perl. Yeah, I am kind of taking advantage of the fact that the format operator doesn't happen to use tuples of dicts already, though this seems like a natural extension of the current semantics. Currently, you can have Style of Simple form Complex form Interpolation ------- ----------- ------------ sprintf string tuple of strings named dict tuple of dicts It does complicate the decision making process in the string format routine a bit. Guido> I think it depends on to what extent this is a common, useful Guido> idiom. Do you have evidence of that? Examples? Well, the first place I ran into it was in DocumentTemplates a few years ago. They used an idiom heavily which may have now been replaced by acquisition where you'd effectively push and pop value dicts onto a stack as you entered and exited nested blocks of DTML code. Their solution was a special dict-like object. The example that made the light click for me this evening was having an object whose class dict stores constants and whose instant dict stores varying values. It seemed cleaner to me to do obj = Class() ... s = format % (Class.__dict__, obj.__dict__) than go through the intermediate step of building a separate dict which would just get discarded as soon as this bit of string building was complete. (I will perform this once for each of several thousand instances.) It's not a big deal. If it seems too obscure the other obvious solutions are not gruesome. Skip

Oh crap... Of course, the table should have been Style of Simple form Complex form Interpolation ------- ----------- ------------ sprintf string tuple of strings named dict (empty fourth cell...) Skip

On Tue, 25 Jan 2000, Skip Montanaro wrote:
Implementation of acquisition basically uses a MultiDict underneath. Consider acquisition as a cumulative context composed from the containers of the target of a web request. (Actually, a distinction is made between the object containment hierarchy of the target and the successive components of the path along which the target is reached by the request, with the containment contexts taking precedence - but that's probably not important here:-) Add in incidental things like the server environment (from which you can get HTTP_REFERER and cookies and so forth). Each of the components can be a dictionary or a MultiDict (or a sequence of pairs, i think), and they're ultimately composed in a MultiDict. I think another place in zope where multidicts play prominently is in the security mechanism, where any object can have local roles, and the ultimate role of a user within a context is composed from the union across the containment hierarchy. There probably are lots of other places where multidicts are used. Suffice to say that there's a lot of composing of contexts that goes on in Zope in general, acquistion being a prime example but not the only one, and multidicts play heavily in many. I would be surprised if this need to combine contexts is peculiar to web server, or general server applications.
I suppose we'd be pretty happy to have something like MultiDict as part of python... Ken klm@digicool.com (Who's the only one left in fredericksburg to speak up, at the moment:-)

Ken Manheimer wrote:
No it doesn't. Acquisition achieves combination of multiple namespaces in much the same way that inheritence does (through delagation).
Acquisition plays a role in security, but MultiDicts-like things are not used.
Note that Zope actually uses two separate flavors. The one used most in Zope (in DocumentTemplate) has very specific hooks to work with Zope's security machinery. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

On Tue, 25 Jan 2000, Skip Montanaro wrote:
That's really interesting. I wrote a bunch of Python wrappers for a UI toolkit a little while ago, and there were two main pieces of machinery i built to make the toolkit pleasant to use: 1. a reasonably nice C++ extension class kit (this is where i tried overloading operator new for the first time, so it would allocate memory that could be freed by PyMem_DEL -- i don't know if CXX uses the same approach) 2. a second layer of Python wrapper classes for the extension classes that implements extra methods in Python, and maintains a hierarchy of default keyword argument values along the inheritance hierarchy of widgets The second of these things involved implementing exactly the kind of dictionary stack that you mentioned. The programming idiom for building widgets goes something like: defaultstack = {} # maps class object to list of defaults dictionaries class Label(Component): defaults = _dict(text="Label", align=LEFT) class Button(Label): defaults = _dict(text="Button", align=CENTER, shadow=2) ... w = Window(...) Button.push(fg="white", bg="red", font="-*-courier-bold-r-normal--14-*-*-*-*-*-*-*") a = Button(w, text="one") b = Button(w, text="two") c = Button(w, text="three") Button.pop() This way you can install some options for a while and make a bunch of widgets with your defaults, then pop the options and go back to the previous state. The layers of dictionaries that get composed in every widget's __init__ function are (in order of priority): 1. any non-None keyword arguments to __init__ 2. any non-None values in class.defaults 3. any non-None values in class.__bases__[0].defaults 4. any non-None values in class.__bases__[0].__bases__[0].defaults etc. When a new set of defaults is push()ed, the class's current defaults are saved on the defaultstack for the class, and restored when pop() gets called. I don't *know* if this is really the best way to do things, but it has seemed to work out pretty well in practice. It makes it more convenient to deal with all the options you have to throw around especially when doing UI stuff. Anyway, i noticed the same sort of thing today while wondering about using keyword arguments for HTML tag attributes. (Perhaps HTMLgen does this sort of thing already -- sorry i haven't looked at it.) Anyway, the following sort of helper might be useful in general: class Default: def __init__(self, cl, **defaults): self.cl = cl self.defaults = defaults def __call__(self, *args, **kw): for key, value in self.defaults: if not kw.has_key(key): kw[key] = value return apply(self.cl, args, kw) Then you could do the same thing as above with: MyButton = Default(Button, fg="white", bg="red", font="-*-courier-bold-r-normal--14-*-*-*-*-*-*-*") a = MyButton(w, text="one") b = MyButton(w, text="two") c = MyButton(w, text="three") This is probably a cleaner way to do things. I haven't tried it, but it might be a nice thing to have in Tkinter. -- ?!ng

[Skip, wants to interpolate multiple dicts via "%", suggests passing a tuple of dicts: format % (d1, d2, ...)] [Guido]
You yourself raised one last century <wink>: simply wanting to interpolate from both locals() and globals(). At the time, the idea of a new dict-like mapping object (capturing Python's lookup rules) appealed to you. I still like that, and note that the apparent need becomes more acute if "deep nesting" is ever added. I wasn't aware of the MultiDict approach Skip mentioned, but thought it looked spot on for the general case! Skip, is the long-windedness of dict = MultiDict() dict.append(d1) dict.append(d2) ... s = format % dict the part you didn't like about that? If so, how about changing the constructor to def __init__(self, *dicts): ... instead so you could use it as a one-liner format % MultiDict(d1, d2, ...) ? That's exactly the same as the tuple idea, except there's a nice descriptive word in the middle of it <wink>.

Tim> Skip, is the long-windedness of Tim> dict = MultiDict() Tim> dict.append(d1) Tim> dict.append(d2) Tim> ... Tim> s = format % dict Tim> the part you didn't like about that? If so, how about changing the Tim> constructor to Tim> def __init__(self, *dicts): Tim> ... Tim> instead so you could use it as a one-liner Tim> format % MultiDict(d1, d2, ...) Tim> ? That's exactly the same as the tuple idea, except there's a nice Tim> descriptive word in the middle of it <wink>. The long-windedness was part of it. The performance hit of composing dictionaries thousands of times to perform a single format operation was also a consideration. Okay, side excursion into the Zope source tree... What I was calling MultiDict is actually MultiMapping (written in C, BTW). As a side effect of my Zope install here, I even already have it in sys.path (go figure!). And it turns out to work just as Tim surmised: >>> d1 = {"a": 1} >>> d2 = {"b": 2} >>> d = MultiMapping.MultiMapping(d1, d2) >>> d["b"] 2 >>> d["a"] 1 Dang! Turns out Jim Fulton has a time machine also. I guess the next question is to extend Ken's comment about getting it into the Python core. Would that be something possible for 1.6? I used a Python version of MultiMapping in an ancient version of DocumentTemplate. I'm sure the C version has been around for at least two or three years and would appear pretty darn stable, since it seems to be at the core of a lot of Zope's coolness. Skip

Skip Montanaro wrote:
(snip)
Yes. I'm sure you'd have to de-ExtensionClass-ify it to get it into the core. :)
since it seems to be at the core of a lot of Zope's coolness.
Actually, it isn't, as there is a separate similar thing (TemplateDict) used in DocumentTemplate. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Tim Peters wrote:
(snip)
I wasn't aware of the MultiDict approach Skip mentioned,
See the MultiMapping module in ExtensionClass. You can get the latest flavor of this in the latest Zope release.
Note the rather important *stack* sematics of MultiMappings. We often push and pop namespaces on and off of MultiMappings in use.
This is exactly what the current MultiMapping "class" does. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Skip Montanaro wrote:
Actually, push (and pop). The namspaces are managed as a stack.
Yes. In the current semantics, you output the two dictionaries. Try: '%s %s' % ({'hello':'skip'},{}) Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Gut feeling: it's dangerous to fill up every possible dark corner with specific semantics that are occasionally useful, because if you go too far you lose useful redundancy, and you end up with Perl. Not sure whether this particular suggestion is "going too far." I think it depends on to what extent this is a common, useful idiom. Do you have evidence of that? Examples? --Guido van Rossum (home page: http://www.python.org/~guido/)

>> I'd like to propose a third alternative. How about if the string >> interpolation function accepted a tuple of dictionaries directly: >> >> s = format % (d1, d2) Guido> Gut feeling: it's dangerous to fill up every possible dark corner Guido> with specific semantics that are occasionally useful, because if Guido> you go too far you lose useful redundancy, and you end up with Guido> Perl. Yeah, I am kind of taking advantage of the fact that the format operator doesn't happen to use tuples of dicts already, though this seems like a natural extension of the current semantics. Currently, you can have Style of Simple form Complex form Interpolation ------- ----------- ------------ sprintf string tuple of strings named dict tuple of dicts It does complicate the decision making process in the string format routine a bit. Guido> I think it depends on to what extent this is a common, useful Guido> idiom. Do you have evidence of that? Examples? Well, the first place I ran into it was in DocumentTemplates a few years ago. They used an idiom heavily which may have now been replaced by acquisition where you'd effectively push and pop value dicts onto a stack as you entered and exited nested blocks of DTML code. Their solution was a special dict-like object. The example that made the light click for me this evening was having an object whose class dict stores constants and whose instant dict stores varying values. It seemed cleaner to me to do obj = Class() ... s = format % (Class.__dict__, obj.__dict__) than go through the intermediate step of building a separate dict which would just get discarded as soon as this bit of string building was complete. (I will perform this once for each of several thousand instances.) It's not a big deal. If it seems too obscure the other obvious solutions are not gruesome. Skip

Oh crap... Of course, the table should have been Style of Simple form Complex form Interpolation ------- ----------- ------------ sprintf string tuple of strings named dict (empty fourth cell...) Skip

On Tue, 25 Jan 2000, Skip Montanaro wrote:
Implementation of acquisition basically uses a MultiDict underneath. Consider acquisition as a cumulative context composed from the containers of the target of a web request. (Actually, a distinction is made between the object containment hierarchy of the target and the successive components of the path along which the target is reached by the request, with the containment contexts taking precedence - but that's probably not important here:-) Add in incidental things like the server environment (from which you can get HTTP_REFERER and cookies and so forth). Each of the components can be a dictionary or a MultiDict (or a sequence of pairs, i think), and they're ultimately composed in a MultiDict. I think another place in zope where multidicts play prominently is in the security mechanism, where any object can have local roles, and the ultimate role of a user within a context is composed from the union across the containment hierarchy. There probably are lots of other places where multidicts are used. Suffice to say that there's a lot of composing of contexts that goes on in Zope in general, acquistion being a prime example but not the only one, and multidicts play heavily in many. I would be surprised if this need to combine contexts is peculiar to web server, or general server applications.
I suppose we'd be pretty happy to have something like MultiDict as part of python... Ken klm@digicool.com (Who's the only one left in fredericksburg to speak up, at the moment:-)

Ken Manheimer wrote:
No it doesn't. Acquisition achieves combination of multiple namespaces in much the same way that inheritence does (through delagation).
Acquisition plays a role in security, but MultiDicts-like things are not used.
Note that Zope actually uses two separate flavors. The one used most in Zope (in DocumentTemplate) has very specific hooks to work with Zope's security machinery. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

On Tue, 25 Jan 2000, Skip Montanaro wrote:
That's really interesting. I wrote a bunch of Python wrappers for a UI toolkit a little while ago, and there were two main pieces of machinery i built to make the toolkit pleasant to use: 1. a reasonably nice C++ extension class kit (this is where i tried overloading operator new for the first time, so it would allocate memory that could be freed by PyMem_DEL -- i don't know if CXX uses the same approach) 2. a second layer of Python wrapper classes for the extension classes that implements extra methods in Python, and maintains a hierarchy of default keyword argument values along the inheritance hierarchy of widgets The second of these things involved implementing exactly the kind of dictionary stack that you mentioned. The programming idiom for building widgets goes something like: defaultstack = {} # maps class object to list of defaults dictionaries class Label(Component): defaults = _dict(text="Label", align=LEFT) class Button(Label): defaults = _dict(text="Button", align=CENTER, shadow=2) ... w = Window(...) Button.push(fg="white", bg="red", font="-*-courier-bold-r-normal--14-*-*-*-*-*-*-*") a = Button(w, text="one") b = Button(w, text="two") c = Button(w, text="three") Button.pop() This way you can install some options for a while and make a bunch of widgets with your defaults, then pop the options and go back to the previous state. The layers of dictionaries that get composed in every widget's __init__ function are (in order of priority): 1. any non-None keyword arguments to __init__ 2. any non-None values in class.defaults 3. any non-None values in class.__bases__[0].defaults 4. any non-None values in class.__bases__[0].__bases__[0].defaults etc. When a new set of defaults is push()ed, the class's current defaults are saved on the defaultstack for the class, and restored when pop() gets called. I don't *know* if this is really the best way to do things, but it has seemed to work out pretty well in practice. It makes it more convenient to deal with all the options you have to throw around especially when doing UI stuff. Anyway, i noticed the same sort of thing today while wondering about using keyword arguments for HTML tag attributes. (Perhaps HTMLgen does this sort of thing already -- sorry i haven't looked at it.) Anyway, the following sort of helper might be useful in general: class Default: def __init__(self, cl, **defaults): self.cl = cl self.defaults = defaults def __call__(self, *args, **kw): for key, value in self.defaults: if not kw.has_key(key): kw[key] = value return apply(self.cl, args, kw) Then you could do the same thing as above with: MyButton = Default(Button, fg="white", bg="red", font="-*-courier-bold-r-normal--14-*-*-*-*-*-*-*") a = MyButton(w, text="one") b = MyButton(w, text="two") c = MyButton(w, text="three") This is probably a cleaner way to do things. I haven't tried it, but it might be a nice thing to have in Tkinter. -- ?!ng

[Skip, wants to interpolate multiple dicts via "%", suggests passing a tuple of dicts: format % (d1, d2, ...)] [Guido]
You yourself raised one last century <wink>: simply wanting to interpolate from both locals() and globals(). At the time, the idea of a new dict-like mapping object (capturing Python's lookup rules) appealed to you. I still like that, and note that the apparent need becomes more acute if "deep nesting" is ever added. I wasn't aware of the MultiDict approach Skip mentioned, but thought it looked spot on for the general case! Skip, is the long-windedness of dict = MultiDict() dict.append(d1) dict.append(d2) ... s = format % dict the part you didn't like about that? If so, how about changing the constructor to def __init__(self, *dicts): ... instead so you could use it as a one-liner format % MultiDict(d1, d2, ...) ? That's exactly the same as the tuple idea, except there's a nice descriptive word in the middle of it <wink>.

Tim> Skip, is the long-windedness of Tim> dict = MultiDict() Tim> dict.append(d1) Tim> dict.append(d2) Tim> ... Tim> s = format % dict Tim> the part you didn't like about that? If so, how about changing the Tim> constructor to Tim> def __init__(self, *dicts): Tim> ... Tim> instead so you could use it as a one-liner Tim> format % MultiDict(d1, d2, ...) Tim> ? That's exactly the same as the tuple idea, except there's a nice Tim> descriptive word in the middle of it <wink>. The long-windedness was part of it. The performance hit of composing dictionaries thousands of times to perform a single format operation was also a consideration. Okay, side excursion into the Zope source tree... What I was calling MultiDict is actually MultiMapping (written in C, BTW). As a side effect of my Zope install here, I even already have it in sys.path (go figure!). And it turns out to work just as Tim surmised: >>> d1 = {"a": 1} >>> d2 = {"b": 2} >>> d = MultiMapping.MultiMapping(d1, d2) >>> d["b"] 2 >>> d["a"] 1 Dang! Turns out Jim Fulton has a time machine also. I guess the next question is to extend Ken's comment about getting it into the Python core. Would that be something possible for 1.6? I used a Python version of MultiMapping in an ancient version of DocumentTemplate. I'm sure the C version has been around for at least two or three years and would appear pretty darn stable, since it seems to be at the core of a lot of Zope's coolness. Skip

Skip Montanaro wrote:
(snip)
Yes. I'm sure you'd have to de-ExtensionClass-ify it to get it into the core. :)
since it seems to be at the core of a lot of Zope's coolness.
Actually, it isn't, as there is a separate similar thing (TemplateDict) used in DocumentTemplate. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Tim Peters wrote:
(snip)
I wasn't aware of the MultiDict approach Skip mentioned,
See the MultiMapping module in ExtensionClass. You can get the latest flavor of this in the latest Zope release.
Note the rather important *stack* sematics of MultiMappings. We often push and pop namespaces on and off of MultiMappings in use.
This is exactly what the current MultiMapping "class" does. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Skip Montanaro wrote:
Actually, push (and pop). The namspaces are managed as a stack.
Yes. In the current semantics, you output the two dictionaries. Try: '%s %s' % ({'hello':'skip'},{}) Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
participants (6)
-
Guido van Rossum
-
Jim Fulton
-
Ka-Ping Yee
-
Ken Manheimer
-
Skip Montanaro
-
Tim Peters