Dear all,

Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.

Following Britton's comment, I tried to keep the migration as simple as possible. It can be achieved by  typing  `yt config migrate` once. Note that toml is case-sensitive, while ConfigParser is not. To accommodate for this, the migration script will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10).

Now, here are the novelties:

The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.

Cheers,
Corentin

On 24/11/2020 10:01, Britton Smith wrote:
Hi there,

I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.

Britton

On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org> wrote:
Hi,
I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini.
Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?

I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).

Cheers,
Clément


On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com> wrote:

Hi Corentin,

This is a great idea.

I was originally opposed to toml.  But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks.  I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.

On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr> wrote:

Dear yt devs,

TLDR: should we change the configuration format from .ini to another one (e.g. toml)?


Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.

Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).

I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:

  1. Keep the .ini file format and build on top of that.
    Pros:
    • no need to write a script to convert the old configuration files to a new format
    • easy to write for the user
    Cons:
    • may become increasingly cumbersome to maintain if the available configuration keys are extended
    • no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
  2. Switch to toml
    Pros:
    • the language support semantically boolean/str/numbers (and parse them accordingly)
    • language is standardized & similar to .INI
    • easy to write (but see below)
    • possible to have nested configs, e.g. to set defaults for a range of different fields:

      [yt.fields.gas]
      # Use linear scale by default for gas fields …
      use_log = false

      [yt.fields.gas.pressure]
      # … except for pressure fields
      use_log = true

      That would be parsed natively as
      {'yt': {
          'fields': {
         
          'gas': {
                  'use_log': False,
                 
      'pressure': {
                      'use_log': True

                  }
              }
          }
      }
    • the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number).
    Cons:
    • users will have to migrate their old config (if any), though we could provide a migration script
    • yet-another dependency (but we could vendor the parser, there are small implementations)
    • may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)
    • for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)
    • there are some gotchas, for example boolean true is written "true" not "True"
  3. Switch to yaml or json?

What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).

Regards,
Corentin

--
Dr. Corentin Cadiou
Post Doctoral Research Assistant
Cosmoparticle Initiative Hub, desk 27
University College London (UCL)
Gower St, Bloomsbury, London WC1E 6BT

mobile: +33.6.43.18.66.83
_______________________________________________
yt-dev mailing list -- yt-dev@python.org
To unsubscribe send an email to yt-dev-leave@python.org
https://mail.python.org/mailman3/lists/yt-dev.python.org/
Member address: matthewturk@gmail.com

_______________________________________________
yt-dev mailing list -- yt-dev@python.org
To unsubscribe send an email to yt-dev-leave@python.org
https://mail.python.org/mailman3/lists/yt-dev.python.org/
Member address: brittonsmith@gmail.com

_______________________________________________
yt-dev mailing list -- yt-dev@python.org
To unsubscribe send an email to yt-dev-leave@python.org
https://mail.python.org/mailman3/lists/yt-dev.python.org/
Member address: corentin.cadiou@iap.fr
-- 
Dr. Corentin Cadiou
Post Doctoral Research Assistant
Cosmoparticle Initiative Hub, desk 27
University College London (UCL)
Gower St, Bloomsbury, London WC1E 6BT

mobile: +33.6.43.18.66.83