Config file template motivation for PEP 463 or an update to format string spec
One way that I like to create templates for config files is to write the config file in a way that can be used with `str.format` if I read in the entire config file as a string before re-saving it or sending it somewhere.* The config file contains a half dozen or dozen variables. Another several dozen variables are pretty much permanent. This approach works for me in the sense that batch submits over several combinations of variables is just a few lines of code, and when I revisit a job after quite some time, the curly braces in the config file template remind me what settings (might) require my attention. The problem lies in what to do about defaults. Including a formatted variable in the template file requires that the variable be included in my call to `format` otherwise I'll get a `KeyError`. For my part, I can work around this by including a companion '.py' file that includes a `dict` of default values. What I pine for is the ability to discard this extra '.py' file, and its WYSINWYG heresy by enabling a way to enable defaults in the processing of formatted strings. One such way would be to modify the format string spec. I am not sure how this might be done, so I cannot make a smart proposal on that one. Another way, although it might not be the most popular, is to allow for inline exception handling of the sort proposed in PEP 463. To me, the following does not look so bad: Config file: ``` ... BAUD: {baud except 9600} ... ``` Template updating code snippets: ``` with open('config.yaml', 'r') as f: txt = f.read() options = {'baud': 19200} new_txt1 = txt.format(options) # BAUD: 19200 is now in new_txt1 options = {} new_txt2 = txt.format(options) # BAUD: 9600 is now in new_txt2 ``` Curious to know anyone's thoughts on the subject, particularly anyone else who is in the habit of hastily writing config files. Apologies if bringing up a rejected PEP on my first thread causes any irritation, but I swear I would not have done it if string formatting weren't so useful. I thank you for my cookie and now I would like some milk. * - I'm aware that there can be security risks in this approach, but I am working in a walled off system (not public) where it is unlikely that I or anyone else can do much damage even if we wanted to.
On Fri, 1 Apr 2022 at 23:51, <brian.patrick.mccall@gmail.com> wrote:
One way that I like to create templates for config files is to write the config file in a way that can be used with `str.format` if I read in the entire config file as a string before re-saving it or sending it somewhere.* The config file contains a half dozen or dozen variables. Another several dozen variables are pretty much permanent. This approach works for me in the sense that batch submits over several combinations of variables is just a few lines of code, and when I revisit a job after quite some time, the curly braces in the config file template remind me what settings (might) require my attention. The problem lies in what to do about defaults. Including a formatted variable in the template file requires that the variable be included in my call to `format` otherwise I'll get a `KeyError`. For my part, I can work around this by including a companion '.py' file that includes a `dict` of default values. What I pine for is the ability to discard this extra '.py' file, and its WYS INWYG heresy by enabling a way to enable defaults in the processing of formatted strings. One such way would be to modify the format string spec. I am not sure how this might be done, so I cannot make a smart proposal on that one. Another way, although it might not be the most popular, is to allow for inline exception handling of the sort proposed in PEP 463.
To me, the following does not look so bad: Config file: ``` ... BAUD: {baud except 9600} ... ```
Template updating code snippets: ``` with open('config.yaml', 'r') as f: txt = f.read() options = {'baud': 19200} new_txt1 = txt.format(options) # BAUD: 19200 is now in new_txt1 options = {} new_txt2 = txt.format(options) # BAUD: 9600 is now in new_txt2 ```
Curious to know anyone's thoughts on the subject, particularly anyone else who is in the habit of hastily writing config files. Apologies if bringing up a rejected PEP on my first thread causes any irritation, but I swear I would not have done it if string formatting weren't so useful. I thank you for my cookie and now I would like some milk.
Nothing wrong with bringing up rejected PEPs, either to reference when discussing another feature (like this), or to specifically revisit the original proposal (although in that case you'd have to respond to the reasons for rejection). But in this case, I think exception handling is kinda overkill for defaults. What you really want is a way to say "put this thing, and use this default". It might be easier to abuse the format string for this. I'll leave you to polish this (for starters, the options should really be contained inside Options, not referenced as a global), but here's a bit of an idea:
class Defaultable(str): ... def __format__(self, fmt): ... return self ... class Default: ... def __format__(self, fmt): ... return fmt ... class Options: ... def __getitem__(self, key): ... if key in options: ... return Defaultable(options[key]) ... return Default() ... options = {'baud': 19200} "BAUD: {baud:9600}".format_map(Options()) 'BAUD: 19200' options = {} "BAUD: {baud:9600}".format_map(Options()) 'BAUD: 9600'
The basic idea is: Regardless of the "format string", return the string unchanged, but if we don't have a string to return, use the format string instead. If you need format strings as well, the easiest way would probably be to do something like "{baud:9600:10s}" and handle the subdivision in your own code. This is all a bit of an abuse of notation, though, so do whatever makes the most sense for you. ChrisA
Oh, I didn't even know about `format_map`! Or `__format__` for that matter. This makes everything much easier. For my purposes, I don't even need to use `__format__` since I am not using format strings. In a show of appreciation, here is a URL to a GIF of James Corden bowing: https://media.giphy.com/media/l2R0eYcNq9rJUsVAA/giphy.gif ``` class Options(UserDict): def __getitem__(self, key): if key in self: return super().__getitem__(key) key, *default = key.split('?') if key in self: return super().__getitem__(key) return ''.join(default) options = {'baud': 19200} result = 'BAUD: {baud?9600}'.format_map(Options(options)) print(result) options = {} result = 'BAUD: {baud?9600}'.format_map(Options(options)) print(result) ```
My first thought was that this is a job for string.Template, which has always struck me as subclassing-friendly. Took about a half hour of wrangling the regex to get here: from string import Template, _sentinel_dict class TemplateWithDefaults(Template): braceidpattern = r'([_a-z][_a-z0-9]*))((:(?P<default>([^}]*)))?' def substitute(self, mapping=_sentinel_dict, /, **kws): if mapping is _sentinel_dict: mapping = kws elif kws: mapping = _ChainMap(kws, mapping) def convert(mo): named = mo.group('named') or mo.group('braced') if named is not None: try: return str(mapping[named]) except KeyError: default = mo.group('default') if not default: raise return default if mo.group('escaped') is not None: return self.delimiter if mo.group('invalid') is not None: self._invalid(mo) raise ValueError('Unrecognized named group in pattern', self.pattern) return self.pattern.sub(convert, self.template) template = TemplateWithDefaults('Where ${prep:was} my default? ${here}.') options = {'prep':'is', 'here':'Nowhere'} print(template.substitute(options)) print(template.substitute(here='Here')) print(template.substitute()) # Should fail with key error on 'here' On Fri, Apr 1, 2022 at 8:13 AM Brian McCall <brian.patrick.mccall@gmail.com> wrote:
Oh, I didn't even know about `format_map`! Or `__format__` for that matter. This makes everything much easier. For my purposes, I don't even need to use `__format__` since I am not using format strings.
In a show of appreciation, here is a URL to a GIF of James Corden bowing: https://media.giphy.com/media/l2R0eYcNq9rJUsVAA/giphy.gif
``` class Options(UserDict): def __getitem__(self, key): if key in self: return super().__getitem__(key) key, *default = key.split('?') if key in self: return super().__getitem__(key) return ''.join(default)
options = {'baud': 19200} result = 'BAUD: {baud?9600}'.format_map(Options(options)) print(result)
options = {} result = 'BAUD: {baud?9600}'.format_map(Options(options)) print(result)
``` _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HPV3AJ... Code of Conduct: http://python.org/psf/codeofconduct/
Have you considered using a templating library like Jinja2 or Mako? That's my first thought whenever I want something that's too complicated for `str.format()`. In addition to handling default values, these libraries would also give you the ability to encode some simple logic in your files, which may be useful. -Kale On 4/1/22 02:26, brian.patrick.mccall@gmail.com wrote:
One way that I like to create templates for config files is to write the config file in a way that can be used with `str.format` if I read in the entire config file as a string before re-saving it or sending it somewhere.* The config file contains a half dozen or dozen variables. Another several dozen variables are pretty much permanent. This approach works for me in the sense that batch submits over several combinations of variables is just a few lines of code, and when I revisit a job after quite some time, the curly braces in the config file template remind me what settings (might) require my attention. The problem lies in what to do about defaults. Including a formatted variable in the template file requires that the variable be included in my call to `format` otherwise I'll get a `KeyError`. For my part, I can work around this by including a companion '.py' file that includes a `dict` of default values. What I pine for is the ability to discard this extra '.py' file, and its WYS INWYG heresy by enabling a way to enable defaults in the processing of formatted strings. One such way would be to modify the format string spec. I am not sure how this might be done, so I cannot make a smart proposal on that one. Another way, although it might not be the most popular, is to allow for inline exception handling of the sort proposed in PEP 463.
To me, the following does not look so bad: Config file: ``` ... BAUD: {baud except 9600} ... ```
Template updating code snippets: ``` with open('config.yaml', 'r') as f: txt = f.read() options = {'baud': 19200} new_txt1 = txt.format(options) # BAUD: 19200 is now in new_txt1 options = {} new_txt2 = txt.format(options) # BAUD: 9600 is now in new_txt2 ```
Curious to know anyone's thoughts on the subject, particularly anyone else who is in the habit of hastily writing config files. Apologies if bringing up a rejected PEP on my first thread causes any irritation, but I swear I would not have done it if string formatting weren't so useful. I thank you for my cookie and now I would like some milk.
* - I'm aware that there can be security risks in this approach, but I am working in a walled off system (not public) where it is unlikely that I or anyone else can do much damage even if we wanted to. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/L2VQKT... Code of Conduct: http://python.org/psf/codeofconduct/
participants (5)
-
Brian McCall -
brian.patrick.mccall＠gmail.com
-
Chris Angelico -
Eric Fahlgren -
Kale Kundert