Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?

I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).

Cheers, Clément

On 23 Nov 2020, at 16:05, Matthew Turk matthewturk@gmail.com wrote:

Hi Corentin,

This is a great idea.

I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.

On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU corentin.cadiou@iap.fr wrote:

...

Dear yt devs,

TLDR: should we change the configuration format from .ini to another one (e.g. toml)?

---

Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.

Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).

I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:

• Keep the .ini file format and build on top of that.

Pros:

• no need to write a script to convert the old configuration files to a new format
• easy to write for the user

Cons:

• may become increasingly cumbersome to maintain if the available configuration keys are extended
• no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
• Switch to toml

Pros:

• the language support semantically boolean/str/numbers (and parse them accordingly)
• language is standardized & similar to .INI
• easy to write (but see below)
• possible to have nested configs, e.g. to set defaults for a range of different fields:

[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false

[yt.fields.gas.pressure] # … except for pressure fields use_log = true

That would be parsed natively as {'yt': { 'fields': {'gas': {'use_log': False,'pressure':{ 'use_log': True }}} }

• the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number).

Cons:

• users will have to migrate their old config (if any), though we could provide a migration script

• yet-another dependency (but we could vendor the parser, there are small implementations)

• may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)

• for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)

• there are some gotchas, for example boolean true is written "true" not "True"

• Switch to yaml or json?

What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).

Regards, Corentin

-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT

mobile: +33.6.43.18.66.83

---

yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com

Dear all,

Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.

Following Britton's comment, I tried to keep the migration as simple as possible. It can be achieved by  typing  `yt config migrate` once. Note that toml is case-sensitive, while ConfigParser is not. To accommodate for this, the migration script will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10).

Now, here are the novelties:

* the format is now toml. * the configuration is loaded from ~/.config/yt/yt.toml (instead of ~/config/yt/ytrc), and if a local yt.toml file is found in the current folder or any of its parent, it is also loaded (and overrides previous preferences). You can use `yt config print-path --local` to ask yt where it found the local configuration file. * the local configuration file can be configured from the CLI in the same way as the global one, provided you add the `--local` flag, e.g. `yt config set --local yt loglevel 1` will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. * the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example

> import yt > yt.config.ytcfg["foo", "bar", "baz"] = 10 > yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime. > yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml.

This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.

* I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration

[fields] cmap = "arbre"

[fields.gas] cmap = "viridis"

[fields.gas.temperature] cmap = "plasma"

then you can use `ytcfg.get_most_specific` to find the most specific option corresponding to your request. For example

> yt.config.ytcfg.get_most_specific("fields", "gas", "temperature", "cmap")       # full match with fields.gas.temperature:cmap 'plasma' > yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap")           # partial match with fields.gas:cmap 'viridis' > yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap")  # partial match with fields:cmap 'arbre'

This enable us to set options for example for all fields, for all fields of type `gas` and for specific fields individually.

The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.

Cheers, Corentin

On 24/11/2020 10:01, Britton Smith wrote:

Hi there,

I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.

Britton

On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org mailto:yt-dev@python.org> wrote:

Hi,
I support changing the format to toml, which feels infinitely less
controversial than yaml or json since it’s already so close to ini.
Correct me if I’m wrong, but that would allow for the config to be
accessed internally as a dictionary instead of through a
ConfigParser instance, right ?

I also support Matt’s suggestion for migration strategy, provided
we don’t expect silent parsing errors to occur (e.g. True being
parsed as a string instead of a boolean. This example is not
relevant but you get the idea).

Cheers,
Clément

...

On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com
<mailto:matthewturk@gmail.com>> wrote:

Hi Corentin,

This is a great idea.

I was originally opposed to toml.  But, I think it's now a
transitive dependency for yt anyway, so we should be safe using
it, and I think that it's reasonably common enough that it will
be usable for folks.  I would wager than migration is not going
to be too terribly difficult, as we can probably parse a config,
see if it matches the defaults, and if not, just bail and say
that it should be converted.

On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU
<corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:

    Dear yt devs,

    TLDR: should we change the configuration format from .ini to
    another one (e.g. toml)?

    ------------------------------------------------------------------------

    Last meeting we discussed what steps were required for the
    4.0 release and one of the point was whether we should change
    how we handle configurations files (e.g. to get
    https://github.com/yt-project/yt/pull/1931
    <https://github.com/yt-project/yt/pull/1931> eventually in).
    Given that changing the format of yt's configuration is a big
    breaking change, this would have its place in a yt-3 to 4
    transition plan.

    Amon the possibilities of a new or improved configuration
    system, users may be able to set some preferences per-field
    (e.g. have a configuration that implements the user's wish
    that "velocity_divergence should be displayed using a linear
    scale, with the seismic colormap"), or to pick their
    favourite unit system (e.g. cgs for the astro community, mks
    for the others).

    I know this has already been discussed earlier (both in PR
    comments and /via/ email), but let's take a moment to think
    about it with a fresh perspective. I can see three possible
    options:

     1. Keep the .ini file format and build on top of that.
        _Pros:_
          * no need to write a script to convert the old
            configuration files to a new format
          * easy to write for the user
        _Cons:_
          * may become increasingly cumbersome to maintain if the
            available configuration keys are extended
          * no support for subgroups within one configuration
            group, so the configuration file may become cluttered
            or we'd need to handle this manually in the codebase
     2. Switch to toml
        _Pros:_
          * the language support semantically boolean/str/numbers
            (and parse them accordingly)
          * language is standardized & similar to .INI
          * easy to write (but see below)
          * possible to have nested configs, e.g. to set defaults
            for a range of different fields:

            [yt.fields.gas]
            # Use linear scale by default for gas fields …
            use_log = false

            [yt.fields.gas.pressure]
            # … except for pressure fields
            use_log = true

            That would be parsed natively as
            {'yt': {
                'fields': {
            'gas': {
            'use_log': False,
            'pressure':{
                            'use_log': True
            }
            }
            }
            }
          * the syntax make it explicit what type each value is,
            so we could check the validity of the config when yt
            is loaded (for ex. [yt]\nloglevel=nan is invalid
            because loglevel should be a number).
        _Cons:_
          * users will have to migrate their old config (if any),
            though we could provide a migration script
          * yet-another dependency (but we could vendor the
            parser, there are small implementations)
          * may be harder to write (how does a user know the
            expected type of each key? Should [yt]\n loglevel by
            a string or a number?)
          * for developers: we'd need to change the way the
            config is supported in the codebase (currently we're
            relying on ConfigParser)
          * there are some gotchas, for example boolean true is
            written "true" not "True"
     3. Switch to yaml or json?

    What is your opinion? As a starting point, you could think
    about how you'd implement the example given above to
    deactivate the log scale for plots made with any field with
    type "gas", except pressure (is this something we want? If
    so, what's the best solution to achieve it in a meaningful
    way for the user?).

    Regards,
    Corentin

    --
    Dr. Corentin Cadiou
    Post Doctoral Research Assistant
    Cosmoparticle Initiative Hub, desk 27
    University College London (UCL)
    Gower St, Bloomsbury, London WC1E 6BT

    mobile: +33.6.43.18.66.83

    _______________________________________________
    yt-dev mailing list -- yt-dev@python.org
    <mailto:yt-dev@python.org>
    To unsubscribe send an email to yt-dev-leave@python.org
    <mailto:yt-dev-leave@python.org>
    https://mail.python.org/mailman3/lists/yt-dev.python.org/
    <https://mail.python.org/mailman3/lists/yt-dev.python.org/>
    Member address: matthewturk@gmail.com
    <mailto:matthewturk@gmail.com>

_______________________________________________
yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org>
To unsubscribe send an email to yt-dev-leave@python.org
<mailto:yt-dev-leave@python.org>
https://mail.python.org/mailman3/lists/yt-dev.python.org/
<https://mail.python.org/mailman3/lists/yt-dev.python.org/>
Member address: brittonsmith@gmail.com <mailto:brittonsmith@gmail.com>

---

yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr

Hi Clement,

On 26/11/2020 16:01, Clément Robert via yt-dev wrote:

Hi Corentin, thanks a lot for getting so much done in that direction so quickly. Regarding the `get_most_specific` feature, I’m wondering how flexible it is. For instance, can I specify a cmap for all “temperature” fields, independently from the ftype ? I imagine it would read something like

[fields.temperature] cmap = “plasma”

Currently, the `get_most_specific` feature finds the longest consecutive common path between the config and the requested config item, starting from the left only. In other words, if you have

[fields.temperature] cmap = "plasma"

it will return "plasma" if you call `ytcfg.get_most_specific("fields", "temperature", "cmap")` but will fail with `ytcfg.get_most_specific("fields", "gas", "temperature", "cmap")`. The reason is that otherwise we may end up with ambiguous results, i.e. what should `ytcfg.get_most_specific("fields", "gas", "temperature", "cmap")` return with the following config?

[fields.temperature] cmap = "plasma"

[fields.gas] cmap = "not_plasma"

We could of course fix the ambiguity by setting priority rules (ex. in this case, we could decide a field name is more specific than a field type, so it should have priority). What do others think about this?

Another point I’d like to raise, and it’s not mentioned in your email (but we discussed it in private), is the behaviour in case more than one config file is found (in cwd and the global one for instance). So currently, according to our discussion, both config files would be taken into account, the local one taking priority in case of overlapping. I can imagine that one would like to build nested configurations from nested directories in a project (although I _hope_ no-one actually thinks this a good idea !), but what I’m worried about is the reproducibility among participants in a shared project. I wrote this issue a few month back https://github.com/yt-project/yt/issues/2635 https://github.com/yt-project/yt/issues/2635 Basically, allowing local config files enable them to be shared along with a notebook or script in a repo, but if the global config file is still having an effect, then the results may vary from one machine to the other and I don’t think that’s a desirable trait in general. On the other hand, it would be nice that the possibility remains available, but I think it should only be used if and when this behaviour is explicitly asked for. In other words, this could be a parameter defined in the local-most file !

TLDR; to ease the process of others weighing in, we have three options (I'd actually be in favor of Clement's option):

1. do not support local config files, 2. Clement's proposition: load the local config file (if any), otherwise load the global config file, 3. Corentin's proposition: load both the local config file (if any) and the global config file + provide a way to dump the current config /via/ a CLI argument.

*More details:*

The way the PR is implemented /currently/ is that it loads the config from ~/.config/yt/yt.toml, then it walks up the folder tree until it finds a yt.toml file. At the first match (if any), it reads it, overrides whatever was defined earlier and stops walking the tree up (i.e. at most one local yt.toml file will ever be read, in the same spirit as git stopping at the first parent folder that contains a .git directory).

If I am correct, your proposition is the following: if any local config file is found, then do not read the global config (~/.config/yt/yt.toml). If we were to adopt this proposition, which file should `yt config set foo bar` configure?

Another option would be to read both (as is currently the case in my PR), but have the cli option `yt config list` dump the full config (either global + local, or defaults + global + local). Currently, `yt config list` shows only the global config and `yt config list --local` only the local one.

Clément

...

On 26 Nov 2020, at 16:03, Corentin CADIOU <corentin.cadiou@iap.fr mailto:corentin.cadiou@iap.fr> wrote:

Dear all,

Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.

Following Britton's comment, I tried to keep the migration as simple as possible. It can be achieved by  typing  `yt config migrate` once. Note that toml is case-sensitive, while ConfigParser is not. To accommodate for this, the migration script will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10).

Now, here are the novelties:

• the format is now toml.

• the configuration is loaded from ~/.config/yt/yt.toml (instead of ~/config/yt/ytrc), and if a local yt.toml file is found in the current folder or any of its parent, it is also loaded (and overrides previous preferences). You can use `yt config print-path --local` to ask yt where it found the local configuration file.

• the local configuration file can be configured from the CLI in the same way as the global one, provided you add the `--local` flag, e.g. `yt config set --local yt loglevel 1` will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory.

• the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example

  ...

  import yt yt.config.ytcfg["foo", "bar", "baz"] = 10 yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now"

  [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime.

  ...

  yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel"

  [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml.

  This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.

• I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration

  [fields] cmap = "arbre"

  [fields.gas] cmap = "viridis"

  [fields.gas.temperature] cmap = "plasma"

  then you can use `ytcfg.get_most_specific` to find the most specific option corresponding to your request. For example

  ...

  yt.config.ytcfg.get_most_specific("fields", "gas",

  "temperature", "cmap")       # full match with fields.gas.temperature:cmap 'plasma'

  ...

  yt.config.ytcfg.get_most_specific("fields", "gas", "density",

  "cmap")           # partial match with fields.gas:cmap 'viridis'

  ...

  yt.config.ytcfg.get_most_specific("fields", "io",

  "particle_velocity", "cmap")  # partial match with fields:cmap 'arbre'

  This enable us to set options for example for all fields, for all fields of type `gas` and for specific fields individually.

The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.

Cheers, Corentin

On 24/11/2020 10:01, Britton Smith wrote:

...

Hi there,

I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.

Britton

On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org mailto:yt-dev@python.org> wrote:

Hi,
I support changing the format to toml, which feels infinitely
less controversial than yaml or json since it’s already so close
to ini.
Correct me if I’m wrong, but that would allow for the config to
be accessed internally as a dictionary instead of through a
ConfigParser instance, right ?

I also support Matt’s suggestion for migration strategy,
provided we don’t expect silent parsing errors to occur (e.g.
True being parsed as a string instead of a boolean. This example
is not relevant but you get the idea).

Cheers,
Clément

...

On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com
<mailto:matthewturk@gmail.com>> wrote:

Hi Corentin,

This is a great idea.

I was originally opposed to toml.  But, I think it's now a
transitive dependency for yt anyway, so we should be safe using
it, and I think that it's reasonably common enough that it will
be usable for folks.  I would wager than migration is not going
to be too terribly difficult, as we can probably parse a
config, see if it matches the defaults, and if not, just bail
and say that it should be converted.

On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU
<corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:

    Dear yt devs,

    TLDR: should we change the configuration format from .ini
    to another one (e.g. toml)?

    ------------------------------------------------------------------------

    Last meeting we discussed what steps were required for the
    4.0 release and one of the point was whether we should
    change how we handle configurations files (e.g. to get
    https://github.com/yt-project/yt/pull/1931
    <https://github.com/yt-project/yt/pull/1931> eventually
    in). Given that changing the format of yt's configuration
    is a big breaking change, this would have its place in a
    yt-3 to 4 transition plan.

    Amon the possibilities of a new or improved configuration
    system, users may be able to set some preferences per-field
    (e.g. have a configuration that implements the user's wish
    that "velocity_divergence should be displayed using a
    linear scale, with the seismic colormap"), or to pick their
    favourite unit system (e.g. cgs for the astro community,
    mks for the others).

    I know this has already been discussed earlier (both in PR
    comments and /via/ email), but let's take a moment to think
    about it with a fresh perspective. I can see three possible
    options:

     1. Keep the .ini file format and build on top of that.
        _Pros:_
          * no need to write a script to convert the old
            configuration files to a new format
          * easy to write for the user
        _Cons:_
          * may become increasingly cumbersome to maintain if
            the available configuration keys are extended
          * no support for subgroups within one configuration
            group, so the configuration file may become
            cluttered or we'd need to handle this manually in
            the codebase
     2. Switch to toml
        _Pros:_
          * the language support semantically
            boolean/str/numbers (and parse them accordingly)
          * language is standardized & similar to .INI
          * easy to write (but see below)
          * possible to have nested configs, e.g. to set
            defaults for a range of different fields:

            [yt.fields.gas]
            # Use linear scale by default for gas fields …
            use_log = false

            [yt.fields.gas.pressure]
            # … except for pressure fields
            use_log = true

            That would be parsed natively as
            {'yt': {
                'fields': {
            'gas': {
            'use_log': False,
            'pressure':{
            'use_log': True
            }
            }
            }
            }
          * the syntax make it explicit what type each value
            is, so we could check the validity of the config
            when yt is loaded (for ex. [yt]\nloglevel=nan is
            invalid because loglevel should be a number).
        _Cons:_
          * users will have to migrate their old config (if
            any), though we could provide a migration script
          * yet-another dependency (but we could vendor the
            parser, there are small implementations)
          * may be harder to write (how does a user know the
            expected type of each key? Should [yt]\n loglevel
            by a string or a number?)
          * for developers: we'd need to change the way the
            config is supported in the codebase (currently
            we're relying on ConfigParser)
          * there are some gotchas, for example boolean true is
            written "true" not "True"
     3. Switch to yaml or json?

    What is your opinion? As a starting point, you could think
    about how you'd implement the example given above to
    deactivate the log scale for plots made with any field with
    type "gas", except pressure (is this something we want? If
    so, what's the best solution to achieve it in a meaningful
    way for the user?).

    Regards,
    Corentin

    --
    Dr. Corentin Cadiou
    Post Doctoral Research Assistant
    Cosmoparticle Initiative Hub, desk 27
    University College London (UCL)
    Gower St, Bloomsbury, London WC1E 6BT

    mobile: +33.6.43.18.66.83

    _______________________________________________
    yt-dev mailing list -- yt-dev@python.org
    <mailto:yt-dev@python.org>
    To unsubscribe send an email to yt-dev-leave@python.org
    <mailto:yt-dev-leave@python.org>
    https://mail.python.org/mailman3/lists/yt-dev.python.org/
    <https://mail.python.org/mailman3/lists/yt-dev.python.org/>
    Member address: matthewturk@gmail.com
    <mailto:matthewturk@gmail.com>

_______________________________________________
yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org>
To unsubscribe send an email to yt-dev-leave@python.org
<mailto:yt-dev-leave@python.org>
https://mail.python.org/mailman3/lists/yt-dev.python.org/
<https://mail.python.org/mailman3/lists/yt-dev.python.org/>
Member address: brittonsmith@gmail.com
<mailto:brittonsmith@gmail.com>

---

yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr

-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT

mobile: +33.6.43.18.66.83 <OpenPGP_signature.sig>

---

yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr