The "right" way to use config files
Terry Reedy
tjreedy at udel.edu
Sat Aug 9 18:30:17 EDT 2014
On 8/9/2014 2:14 PM, Fabien wrote:
> On 09.08.2014 19:29, Terry Reedy wrote:
>> If possible, functions should *return* their results, or yield their
>> results in chunks (as generators). Let the driver function decide where
>> to put results. Aside from separating concerns, this makes testing much
>> easier.
>
> I see. But then this is also true for parameters, right? And yet we
> return to my original question ;-)
>
>
> Let's say my configfile looks like this:
>
> -----------------
> ### app/config.cfg
> # General params
> output_dir = '..'
> input_file = '..'
>
> # Func 1 params
> [func1]
> enable = True
> threshold = 0.1
> maxite = 1
> -----------------
>
> And I have a myconfig module which looks like:
>
> -----------------
> ### app/myconfig.py
>
> import ConfigObj
>
> parser = obj() # parser will be instanciated by initialize
Try parser = object() to actually run, but the line is not needed.
Instead put "parser: instantiated by initialize" in the docstring.
>
> def initialize(cfgfile=None):
> global parser
> parser = ConfigObj(cfgfile, file_error=True)
> -----------------
>
> My main program could look like this:
>
> -----------------
> ### app/mainprogram_1.py
>
> import myconfig
>
> def func1():
> # the params are in the cfg
> threshold = myconfig.parser['func1'].as_float('threshold')
> maxite = myconfig.parser['func1'].as_long('maxite')
>
> # dummy operations
> score = 100.
> ite = 1
> while (score > threshold) and (ite < maxite):
> score /= 10
> ite += 1
>
> # dummy return
> return score
>
> def main():
> myconfig.initialize(sys.argv[1])
>
> if myconfig.parser['func1'].as_bool('enable'):
> results = func1()
>
> if __name__ == '__main__':
> main()
> -----------------
The advantage of TDD is that it forces one to make code testable as you
do. Old code may not be designed to be so easily testable, as I have
learned trying to add tests to idlelib. For the above, I would consider
def func1_algo(threshhold, maxite): # possible separte file
score = 100.
ite = 1
while (score > threshold) and (ite < maxite):
score /= 10
ite += 1
return score
def func1(): # interface wrapper
threshold = myconfig.parser['func1'].as_float('threshold')
maxite = myconfig.parser['func1'].as_long('maxite')
return func1_algo(threshhold, maxite)
This is a slight bit of extra work, but now you can separately test (and
modify) the algorithm and the interfacing. Testing the algorithm is
easy, which encourages testing multiple i/o pairs.
for in, out in iopairs:
assert func1_algo(in) == out # or self.assertEqual, or ...
(or close enough for float outputs)
As for the interfacing: you can write and read multiple versions of
config.cfg (relatively slow), use something like unittest.mock to mock
the myconfig module, or write something fairly simple (py3 code).
class Entry(dict):
def as_bool(self, name):
s = self[name]
return True if s == 'True' else False if s == 'False' else None
def as_int(self, name):
return int(self[name])
as_long = as_int
def as_float(self, name):
return float(self[name])
class Config(object):
def initialize(self, argv):
pass
myconfig = Config() # a module is like a singleton class
myconfig.initialize('a') # test that does not raise
# In use for testing, uncomment the following two lines
# import mainprogram_1.py as mp1
# mp1.myconfig = myconfig
f1_cfg = Entry({
'enable': 'True',
'threshold': '0.1',
'maxite': '1',
})
myconfig.parser = {'func1': f1_cfg}
print(myconfig.parser['func1'].as_float('threshold') == 0.1)
print(myconfig.parser['func1'].as_long('maxite') == 1)
print(myconfig.parser['func1'].as_bool('enable') == True)
f1_cfg['maxite'] = 5
print(myconfig.parser['func1'].as_int('maxite') == 5)
# prints True 4 times
Notice that you inject the mock myconfig into the tested module just
one. After that, you can change anything within parser or replace parser
with a new dict.
> Or like this:
>
> -----------------
> ### app/mainprogram_2.py
>
> import myconfig
>
> def func1(threshold=None, maxite=None):
These should not have defaults; avoid extra work!
> # dummy operations
> score = 100.
> ite = 1
> while (score > threshold) and (ite < maxite):
> score /= 10
> ite += 1
>
> # dummy return
> return score
>
> def main():
> myconfig.initialize(sys.argv[1])
>
> if myconfig.parser['func1'].as_bool('enable'):
> # the params are in the cfg
> threshold = myconfig.parser['func1'].as_float('threshold')
> maxite = myconfig.parser['func1'].as_long('maxite')
> results = func1(threshold=threshold, maxite=maxite)
>
> if __name__ == '__main__':
> main()
> -----------------
>
> In this case, program2 is easier to test/understand, but if the
> parameters become numerous it could be a pain...
This is equivalent to what i wrote except for putting the wrapper inline
in main(). Testing is the same for either.
--
Terry Jan Reedy
More information about the Python-list
mailing list