[Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version

[Chris Angelico]

On Thu, Apr 16, 2015 at 1:53 AM, Simon Kennedy <sffjunkie at gmail.com> wrote:
> On Wednesday, 15 April 2015 12:44:59 UTC+1, Chris Angelico wrote:
>> There is a fundamental difficulty with the unification of the
>> disparate, in that you have to start coping with all sorts of
>> differences and try to make sense of them *in code*. When, for
>> instance, you want to unify config handling, what you're saying is
>> either that every config handler has to devolve to the lowest common
>> denominator, or else that your unified config handler has to become
>> the least common multiple.
>
>
> The stdlib is full of modules which don't attempt everything but instead
> attempt to do a useful subset and everyone seems to be happy they're
> included.
>
> Why not configuration file handling?
>
> I assume we're writing Python code and that eventually all config data needs
> to end up in memory that Python can access.
>
> Pulling numbers out of my posterior, lets say 80% of projects only need
> nothing more than a text file with a set of sections with name/value pairs
> under them. That can be parsed into a dict.

A dict of dicts, presumably. This is already handled:

https://docs.python.org/3/library/configparser.html

Note that it has a couple of "See also" references to other perfectly
viable config file formats.

> And 5% want to use Python code. That when it gets exec'ed a dict comes out
> the other end

Why a dict?

> And 5% want to use a database. We can produce an adapter (say sqlite) that
> spits out a dict or they can.

Why a dict?

> And 5% want to use an etcd store.
> That leaves 5% that want to do something funky. They're on their own
>
> There aren't that many configuration storage systems in the world.
>
> Then you define the order which sources (command line / env var / global
> config etc.) are used and merge them into a complete configuration
>
> Why not 'bless' some code which simplifies the 95%
>
> There's a distribution on PyPi called 'layeredconfig' which does much of
> what I think could be done. A few Python projects I've encountered seem to
> follow a similar pattern.
>
> I'm not sure that I see the demons you appear to see in config file
> handling. Perhaps you could give a real world example

There aren't demons - there are just different requirements. Some
programs use YAML configs, for which you need to hunt down a
third-party package. Some use JSON, some use configparser, some use
Python code directly. Not all of them render down to a dict.

>> The latter may seem tempting ("you can use
>> this module to do ANYTHING!"), but you quickly fall foul of the
>> Inner-Platform Effect.
>>
>> http://thedailywtf.com/articles/The_Inner-Platform_Effect
>> https://en.wikipedia.org/wiki/Inner-platform_effect
>>
>
> That's not a term I've encountered before. But perhaps it's part of my
> problem but I looked at the Wikipedia page and I'm not sure it makes a case
> that the effect is bad.
>
> For example the first is about Firefox's FTP. I use Sublime Text and some of
> the packages you can install do similar things but I'm not sure I would say
> they were wrong to wrap git for example.

I don't know about *wrapping* git; the inner-platform effect would be
if the Sublime package actually *reimplements* git, which is what
happened with Firefox's FTP module.

> The only downside seems to be "they tend to be slower and if poorly coded,
> less reliable as well.[citation needed]". But you could say that about any
> code.

Look at the article on thedailywtf.com rather than the Wikipedia page,
then. Have a think about what it's doing, then decide for yourself if
it's worthwhile.

Steven cited four config file handlers that already ship with Python.
Add in YAML as well, because it's worth adding, and then come up with
a way for a Python program to be able to use any of them. How is your
uberconfigparser going to handle this? Probably something like this:

import config
with open("app_name.json") as f:
    my_config = config.parser(format=config.JSON, source=f)

The only effective way to handle everything would be to have a
mode-switch that lets you pick a format. And if you're doing that, you
may as well just write it like this:

import json
with open("app_name.json") as f:
    my_config = json.load(f)

The umbrella module provides nothing that can't be done better with actual code.

ChrisA